mirror of https://github.com/infiniteCable2/openpilot.git synced 2026-09-04 15:23:42 +08:00

Files

T

James Vecellio-Grant c47aadee19 sunnypilot modeld: Sync Tinygrad (#1081 )

* Tinygrad bump from sync-20250627

* bump tinygrad_repo

* Reformat metadata generator to match driving_models.json

* bump tinygrad

* Revert "bump tinygrad"

This reverts commit f479dfd5021266e22b34deca06cc3c4d531fb7e8.

* revert me after SP model compiled

* Model recompiled successfully, initiate "revert me after SP model compiled"

This reverts commit 95706eb688c9f5522903bee21892b8871ff8f957.

* The "FillMe" placeholder caused an extra 10 seconds of work

* bump to 22Jul2025

* Update build-all-tinygrad-models.yaml

* Update build-all-tinygrad-models.yaml

* Update build-all-tinygrad-models.yaml

* Update build-all-tinygrad-models.yaml

* Update build-all-tinygrad-models.yaml

* Update build-all-tinygrad-models.yaml

* Update build-all-tinygrad-models.yaml

* Allow more dynamic short names

This should hopefully be future-proof for now.. It's robust enough to return the correct word-digit format (see example on how it generates from given display name below):

'Last Horizon V2 (November 22, 2024)' -> LHV2
'Alabama (November 25, 2024)' -> ALABAMA
'PlayStation (December 03, 2024)' -> PLAYSTAT
'Postal Service (December 09, 2024)' -> PS
'Null Pointer (December 13, 2024)' -> NP
'North America (December 16, 2024)' -> NA
'National Public Radio (December 18, 2024)' -> NPR
'Filet o Fish (March 7, 2025)' -> FOF
'Tomb Raider 2 (April 18, 2025)' -> TR2
'Tomb Raider 3 (April 22, 2025)' -> TR3
'Tomb Raider 4 (April 25, 2025)' -> TR4
'Tomb Raider 5 (April 25, 2025)' -> TR5
'Tomb Raider 6 (April 30, 2025)' -> TR6
'Tomb Raider 7 (May 07, 2025)' -> TR7
'Down to Ride (Revision: May 10, 2025)' -> DTR
'SP Vikander Model (May 16, 2025)' -> SPVM
'VFF Driving (May 15, 2025)' -> VFFD
'Secret Good Openpilot (May 16, 2025)' -> SGO
'Vegetarian Filet o Fish (May 29, 2025)' -> VFOF
'Down To Ride (Revision: May 30, 2025)' -> DTR
'Vegetarian Filet o Fish v2 (June 05, 2025)' -> VFOFV2
'Kerrygold Driving (June 08, 2025)' -> KD
'Tomb Raider 10 (June 16, 2025)' -> TR10
'Organic Kerrygold (June 17, 2025)' -> OK
'Liquid Crystal Driving (June 21, 2025)' -> LCD
'Vegetarian Filet o Fish v3 (June 21, 2025)' -> VFOFV3
'Vibe Model [Custom Model]' -> VMCM
'Tomb Raider 13 (June 27, 2025)' -> TR13
'Aggressive TR (June 28, 2025)' -> ATR
'Tomb Raider 14 (June 30, 2025)' -> TR14
'Cookiemonster Tomb Raider (July 02, 2025)' -> CTR
'Down to Ride (Revision: July 07, 2025)' -> DTR
'Simple Plan Driving (July 07, 2025)' -> SPD
'Down to Ride (Revision: July 08, 2025)' -> DTR
'Tomb Raider 15 (July 09, 2025)' -> TR15
'Tomb Raider 15 rev-2 (July 11, 2025)' -> TR15R2
'Le Tomb Raider 14 (July 14, 2025)' -> LTR14
'Le Tomb Raider 14h (July 17, 2025)' -> LTR14H
'Tomb Raider 16 (July 18, 2025)' -> TR16
'Tomb Raider 16v2 (July 21, 2025)' -> TR16V2

* Update build-all-tinygrad-models.yaml

* Update build-all-tinygrad-models.yaml

* No need to sleep 3 seconds, just send it

* try dynamic

* cleanup

* Update build-single-tinygrad-model.yaml

* bc devtekve said. also, this is repetitive af

* Revert "bc devtekve said. also, this is repetitive af"

This reverts commit 3a0c1562de6eae1d2d213d2f53cfb2edb4eef40c.

* maybe we could use a script instead that both build all

That both build all and sunnypilot-build-model reference

* refactor: consolidate model building steps into a single workflow

* tweak

* tweakx2

* tweakx3

* tweakx4

* dunno dunno...

* output dir

* lots of changes

* Revert "lots of changes"

This reverts commit 4aadb0ee2929fdf9d902335161b22072bef4e068.

* fail if all fail

* no inputs needed

* make it easier for us

* note failure and exit 0

* Update build-all-tinygrad-models.yaml

* not needed unless we really want it

* Update build-single-tinygrad-model.yaml

* Merge branch 'sync-20250627-tinygrad' of github.com:sunnypilot/sunnypilot into sync-20250627-tinygrad

* retry for failed ?

* always run this step because sometimes one build fails

which causes the matrix to fail, but most builds still have uploaded artifacts.

* strip

* no escape

* Update build-all-tinygrad-models.yaml

* Test case from terminal run

(openpilot) james@Mac sunnypilot % jq -c '[.bundles[] | select(.runner=="tinygrad") | {ref, display_name: (.display_name | gsub(" \\([^)]*\\)"; "")), is_20hz}]' \
  /Users/james/Documents/GitHub/sunnypilot-docs/docs/driving_models_v6.json > matrix.json

mkdir -p output
touch "output/model-Tomb Raider 16v2 (July 21, 2025)-544"
touch "output/model-Space Lab Model (July 24, 2025)-547"
touch "output/model-Space Lab Model v1 (July 24, 2025)-548"

built=(); while IFS= read -r line; do built+=("$line"); done < <(
  ls output | sed -E 's/^model-//' | sed -E 's/-[0-9]+$//' | sed -E 's/ \([^)]*\)//' | awk '{gsub(/^ +| +$/, ""); print}'
)

jq -c --argjson built "$(printf '%s\n' "${built[@]}" | jq -R . | jq -s .)" \
  'map(select(.display_name as $n | ($built | index($n | gsub("^ +| +$"; "")) | not)))' \
  matrix.json > retry_matrix.json

cat retry_matrix.json

[]
(openpilot) james@Mac sunnypilot %

* always

* great success

* add suffix to retry artifact so it doesn't conflict

* retry to get_model too

* and there haha

* unnecessary hyphen

* compare built to missing. include retries

* adjust copy of artifacts.

* Update build-all-tinygrad-models.yaml

* Update model selector versioning and add documentation

* Update retry condition for failed models in build-all-tinygrad-models.yaml

* Update retry condition for failed models in build-all-tinygrad-models.yaml

* Update build-single-tinygrad-model.yaml

* false

* default none because why not

* red diff? i think?

* meh ... not needed i guess

* error error error

* Nayan is watching... always watching mike wazowski

* string all the way

* lots of retries just in case because im scared

* more robust

* ONLY ONE!!!!!!

* delete.... a lot

* fix artifacts

* fix artifacts

* make sure each is unique :)

* skip files like artifact duhhhh

* artifact name dir

* concurrency

* copy here

* Update build-single-tinygrad-model.yaml

* Update build-single-tinygrad-model.yaml

* bump

* bump tinygrad

* max parallel? if not, i have the other remedy ready in build-all

* revert me!

* I resynced tinygrad woo hoo

* setup shouldnt fail

* pull

* big ole diff

* condition

* Update build-all-tinygrad-models.yaml

* not always() never always() never!!!

* not failure instead of great success

* Update build-all-tinygrad-models.yaml

* yay that worked. lets invoke build-single one last time

* these arent used and are just taking up 250MB space

* really frog?

* bump back to 3

* self-hosted, tici

* rename to trigger tests

* split sync

---------

Co-authored-by: DevTekVE <devtekve@gmail.com>

2025-07-31 06:50:23 -07:00

3.3 KiB

Raw Blame History

Model Selector Version Compatibility

This document explains the version compatibility mechanism used by the Model Selector system, and the rationale behind certain version constraints and JSON file management strategies.

Overview

The Model Selector is responsible for selecting and validating model bundles based on their metadata and version constraints. Each model bundle is distributed via a JSON file and includes a minimumSelectorVersion field indicating the minimum selector version required to load it.

To ensure robust compatibility and prevent mismatches between model expectations and selector capabilities, the selector enforces two version boundaries:

REQUIRED_MIN_SELECTOR_VERSION: the oldest selector version we support.
CURRENT_SELECTOR_VERSION: the current version of the selector logic.

Version Compatibility Check

A model bundle is considered compatible if:

REQUIRED_MIN_SELECTOR_VERSION <= bundle["minimumSelectorVersion"] <= CURRENT_SELECTOR_VERSION

This ensures:

Old bundles are rejected if they rely on deprecated selector behavior.
Future bundles are ignored if they expect logic that the current selector doesn't yet implement.

Handling Breaking Changes

When a deep change in selector behavior requires all models to be recompiled (e.g., due to a major architectural update), we:

Create a new JSON file (e.g., from models_v4.json to models_v5.json).
Assign updated minimumSelectorVersion values in the new bundles.

This allows older selector versions to continue using the previous JSON file, while newer versions point to the new one, preventing cross-contamination.

Why `REQUIRED_MIN_SELECTOR_VERSION` Still Matters

Despite using new JSON files to isolate breaking changes, REQUIRED_MIN_SELECTOR_VERSION plays a critical role:

1. Cached Bundle Validation

Model bundles are cached locally (e.g., in-memory or on disk). A user might have previously loaded a now-invalid bundle from an older JSON file.

REQUIRED_MIN_SELECTOR_VERSION prevents the selector from reloading or trusting that stale cached bundle, even if the original JSON is gone.

2. Explicit Deprecation Boundary

By raising REQUIRED_MIN_SELECTOR_VERSION, we declare older bundles officially unsupported, even if they technically still exist in a legacy JSON file.

3. Avoiding Race Conditions

Some clients may have intermittent access to updated JSONs. The runtime check ensures version compatibility is enforced independently of external file state.

Summary

Component	Purpose
`minimumSelectorVersion`	Declares the minimum selector version required to load a model bundle
`REQUIRED_MIN_SELECTOR_VERSION`	Prevents loading bundles that are too old (e.g., from stale cache)
`CURRENT_SELECTOR_VERSION`	Prevents loading bundles that are too new or forward-incompatible
JSON file renaming	Isolates bundles by selector generation to handle full recompiles

This layered strategy ensures safe evolution of the model selection system while maintaining backward compatibility and runtime protection against stale or incompatible bundles.

3.3 KiB Raw Blame History