tiny my BUTT

2026-08-21 16:23:46 +08:00 · 2026-06-22 22:03:09 -05:00
parent bb36fe4287
commit d97100bd14
865 changed files with 190538 additions and 156895 deletions
@@ -0,0 +1,134 @@
+# StarPilot Unified Model Rebuild
+
+This workflow rebuilds StarPilot driving and driver-monitoring artifacts for the vendored tinygrad revision. Driving-model behavior versions remain manifest metadata; every runtime driving artifact uses the `tinygrad_single_v1` layout.
+
+## Safety
+
+- The supported build device is `comma@192.168.3.110`.
+- Never run these commands against `192.168.3.109`.
+- Do not compile normal and big-GPU artifacts together. This workflow builds normal QCOM artifacts only.
+- Keep source ONNX files and compiled PKLs on the T5 workspace, not the comma.
+
+## Workspace
+
+The default workspace is:
+
+```text
+/Volumes/T5/StarPilot-Model-Rebuild-2026-06-22/
+```
+
+Important directories:
+
+- `onnx/<model-id>/`: ID-prefixed source ONNX files.
+- `compiled/`: completed unified driving PKLs.
+- `driver-monitoring/`: DM ONNX, model PKL, metadata, and camera warps.
+- `ready-for-resources/`: flat repository-upload handoff.
+- `external-upload/`: artifacts over 100 MiB plus `handoff.json`.
+- `logs/`: one remote compilation log per model.
+- `results/`: source and artifact checksum records.
+- `manifests/`: generated `model_names_v22.json`.
+
+## Initialize And Extract
+
+```bash
+python3 scripts/model_rebuild_pipeline.py init
+python3 scripts/model_rebuild_pipeline.py extract \
+  --base-manifest /path/to/model_names_v21.json
+```
+
+Extraction streams Git blobs directly to disk. LFS pointers are resolved from the local object cache or fetched by object ID, then checked against the pointer SHA-256 and size. Binary ONNX data is never stored in a shell variable.
+
+To retry one source:
+
+```bash
+python3 scripts/model_rebuild_pipeline.py extract \
+  --model pop22 \
+  --base-manifest /path/to/model_names_v21.json
+```
+
+Source commits are defined in `scripts/model_source_map_v22.json`.
+
+## Compile
+
+Compile one model:
+
+```bash
+python3 scripts/model_rebuild_pipeline.py compile \
+  --model pop22 \
+  --base-manifest /path/to/model_names_v21.json
+```
+
+Compile or resume the full catalog:
+
+```bash
+python3 scripts/model_rebuild_pipeline.py compile \
+  --base-manifest /path/to/model_names_v21.json
+```
+
+Existing artifacts are skipped unless `--force` is passed. Each model is staged in its own remote input directory, compiled on `.110`, copied back to the T5, hashed, and copied into `ready-for-resources/`. Failures are written to `results/<id>_failure.json`; rerunning the same command resumes incomplete models.
+
+Validate one or all completed artifacts with synthetic camera inputs on QCOM:
+
+```bash
+python3 scripts/model_rebuild_pipeline.py validate \
+  --model pop22 \
+  --base-manifest /path/to/model_names_v21.json
+```
+
+The lower-level device compiler also supports direct use:
+
+```bash
+./models --model pop22 --input-format split --version v11
+./models --model deeprl3v2 --input-format supercombo --version v15
+```
+
+`--version` records behavioral semantics only. It does not change artifact layout.
+
+## Driver Monitoring
+
+Stage the current DM ONNX in `uncompiledmodels`, then run:
+
+```bash
+./models --dm \
+  --input-dir /data/openpilot/uncompiledmodels \
+  --output-dir /tmp/dm_artifacts
+```
+
+This builds:
+
+- `dmonitoring_model_tinygrad.pkl`
+- `dmonitoring_model_metadata.pkl`
+- `dm_warp_1928x1208_tinygrad.pkl`
+- `dm_warp_1344x760_tinygrad.pkl`
+
+All four files must be updated together.
+
+## Manifest
+
+Generate v22 after compilation:
+
+```bash
+python3 scripts/model_rebuild_pipeline.py manifest \
+  --base-manifest /path/to/model_names_v21.json
+```
+
+The generator preserves existing IDs and behavioral metadata, adds `deeprl3v2`, and writes:
+
+- `artifact_format`
+- `artifact_size`
+- `artifact_sha256`
+- optional `artifact_url`
+
+Files above 100 MiB are listed in `external-upload/handoff.json`. Upload those files to Dropbox, use a direct-download URL, add it as `artifact_url`, and regenerate or edit the final manifest without changing its size or SHA-256 fields.
+
+## Runtime Verification
+
+Compilation validates JIT capture/replay, pickle round-trip, finite outputs, metadata slices, and both camera warps. Before release:
+
+1. Select representative v8, v11, v12, v15, and supercombo models.
+2. Confirm `modeld` stays running.
+3. Confirm finite `modelV2` path, lane-line, lead, pose, and action data.
+4. Confirm `driverStateV2` on both supported camera resolutions.
+5. Test download, selection, deletion, randomization, migration, and fallback in QT, raylib/mici, and Galaxy.
+
+The built-in South Carolina artifact is `selfdrive/modeld/models/driving_tinygrad.pkl`. If migration cannot download the selected v22 artifact, StarPilot switches to that built-in model.