StarPilot/docs/MODEL_REBUILD.md

StarPilot Model Rebuild

This is the supported workflow for changing the vendored tinygrad revision and releasing a new model manifest generation. A manifest generation represents one tinygrad ABI. Model behavior versions (v8 through v16) are independent and must remain unchanged when only tinygrad changes.

The current generation is v25, pinned to tinygrad e837e367aac9e1a66e689f4f32ce20ca9367df13. The supported compiler is comma@192.168.3.110; never substitute another comma without explicit approval.

Release Contract

• Keep every existing StarPilot model ID stable across manifest generations.
• Store HF artifacts under models/v25/<model-id>/.
• Store GitHub fallback artifacts on the Models branch under v25/<model-id>/.
• Name every logical artifact <model-id>_driving_tinygrad.pkl.
• Publish native chunks as .chunkNNofNN plus .chunkmanifest.
• Serialize every v25 driving artifact out-of-band; this applies to normal QCOM models as well as external-GPU models.
• Include artifact_sha256 and artifact_chunk_count in the manifest.
• Set uses_external_gpu: true only for models compiled for Chestnut.
• Do not rename an artifact from another tinygrad revision. PKLs must either use the exact v25 pin or be rebuilt with it.
• Do not add models absent from the existing StarPilot catalog unless the release explicitly requests them.

The downloader checks Hugging Face first and GitHub second. There is no GitLab fallback. The HF manifest lives only at manifests/model_names_v25.json, old artifacts live under models/v24/, and current artifacts live under models/v25/. The v25 downloader never probes unversioned or v24 artifact paths; missing v25 artifacts fail safely instead of loading an incompatible pickle.

Tinygrad Bump

1. Record the exact tinygrad commit used by the compatible source catalog.
2. Replace tinygrad_repo/ from that commit, excluding nested Git metadata.
3. Write the full SHA to tinygrad_repo/TINYGRAD_COMMIT.
4. Review upstream modeld, compiler, parser, and camera-warp changes. Merge required ABI changes into StarPilot's existing multi-model runtime; never replace StarPilot modeld.py wholesale.
5. Increment MANIFEST_CANDIDATES to a new single version. Do not fall back to the prior manifest because its PKLs target a different tinygrad ABI.
6. Sync the exact tree to the compiler before building anything:

./dev sync
rsync -az --delete --exclude=.git --exclude=__pycache__ -e ssh \
  tinygrad_repo/ comma@192.168.3.110:/data/openpilot/tinygrad_repo/
rsync -az -e ssh selfdrive/modeld/ \
  comma@192.168.3.110:/data/openpilot/selfdrive/modeld/
rsync -az -e ssh scripts/model_compiler.py \
  comma@192.168.3.110:/data/openpilot/scripts/model_compiler.py
rsync -az -e ssh models comma@192.168.3.110:/data/openpilot/models

Confirm the device marker before compiling:

ssh comma@192.168.3.110 \
  'cat /data/openpilot/tinygrad_repo/TINYGRAD_COMMIT'

Reuse Compatible Artifacts

Reusing an artifact is preferred when its catalog records the exact same tinygrad SHA and exact same source-model commit. Display names and release dates are not sufficient proof. Copy compatible chunks server-side so the Mac never stores a second multi-gigabyte artifact, but rename every destination chunk to the stable StarPilot model ID.

Example:

hf buckets cp \
  'hf://datasets/<source>/<path>/<source-file>.chunk01of02' \
  'hf://buckets/StarPilot-Driving/StarPilot-Resources/models/v25/pop223/pop223_driving_tinygrad.pkl.chunk01of02'

Write 2 to pop223_driving_tinygrad.pkl.chunkmanifest, upload it last, and put the source artifact's full SHA-256 and chunk count into the v25 manifest. Upload the manifest only after every listed artifact directory is complete.

Compile Missing Models

Archived sources live under:

hf://buckets/StarPilot-Driving/StarPilot-Resources/onnx/<source-id>/

Stage one model at a time in /data/openpilot/uncompiledmodels; this avoids filling the comma and prevents ./models from selecting stale input files.

./models --<model-id> --version <behavior-version>
./models --<gpu-model-id> --version v16 --gpu

The default input is a single supercombo ONNX. For legacy sources use:

./models --<model-id> --input-format split --version <behavior-version>

Every non-local release build emits an OOB artifact as native chunks and removes the temporary full PKL. ./models --local-<id> intentionally keeps one OOB PKL for local use.

The resumable bulk helper is:

STAR_PILOT_MODEL_REMOTE=comma@192.168.3.110 \
python3 scripts/model_rebuild_pipeline.py compile \
  --workspace /Volumes/T5/StarPilot-Model-Rebuild \
  --source-map scripts/model_source_map_v25.json \
  --base-manifest ~/StarPilot-Resources/model_names_v25.json

Failures are recorded under results/; rerun the same command to resume.

Driver Monitoring And Default

Driver monitoring is built once per tinygrad generation:

./models --dm \
  --input-dir /data/openpilot/uncompiledmodels \
  --output-dir /tmp/dm_artifacts

Replace these four files together:

• dmonitoring_model_tinygrad.pkl
• dmonitoring_model_metadata.pkl
• dm_warp_1928x1208_tinygrad.pkl
• dm_warp_1344x760_tinygrad.pkl

Recompile RDF V4 with the same pin and replace the built-in selfdrive/modeld/models/driving_tinygrad.pkl native chunk set. Never commit a full built-in PKL over the repository limit.

Validation

Run repository tests first:

./dev sync
./.venv/bin/pytest -q -n0 \
  starpilot/assets/tests/test_model_pipeline.py \
  common/tests/test_file_chunker.py \
  scripts/tests/test_model_release.py

For representative v8, v11, v12, v15, v16, and GPU artifacts, validate both camera resolutions on real QCOM and require finite plan, lane-line, road-edge, lead, pose, and action outputs. Then start modeld and confirm stable modelV2 publication. Validate DM driverStateV2 at both resolutions.

Device Migration

When ModelManifestVersion changes, the model manager retains the selected model ID but deletes every non-local downloaded driving artifact from the old generation, including full PKLs, .pNN parts, native chunks, and chunk manifests. It then downloads that ID's v25 chunks. Local models and DM files are not deleted. If the selected v25 artifact cannot be downloaded and verified, the manager selects the built-in RDF V4 model.

Test this explicitly before release by starting with a v24 selected model and checking that no v24 driving artifact remains under /data/models after the v25 manifest is applied.