IQ.Pilot/tools/scripts/iqpilot_model_compile_guide.md

IQPilot Model Compile Guide

This guide documents the current, working IQPilot flow for compiling and publishing tinygrad supercombo model artifacts for comma 3X/4 class devices.

It is written to be usable by either a human operator or another agent without needing outside context.

Scope

This guide covers:

• compiling supercombo ONNX models into prebuilt QCOM tinygrad .pkl artifacts
• matching IQPilot's current pinned compile path
• validating artifacts on-device before publishing
• updating the IQModels repo and selector manifests
• common failure modes seen in current IQPilot

This guide is specifically for the current IQPilot model stack:

• compiler script: iqpilot/selfdrive/iqmodeld/tools/compile_supercombo.py
• runtime: iqpilot/selfdrive/iqmodeld/daemon.py
• tinygrad compile flags: iqpilot/selfdrive/iqmodeld/SConscript
• on-device harness: iqpilot/selfdrive/iqmodeld/tools/force_onroad_iqmodeld_test.py

Current Known-Good QCOM Compile Flags

IQPilot should compile QCOM supercombo artifacts with the same important flags stock openpilot uses for the proven QCOM tinygrad path:

DEV=QCOM
WARP_DEV=QCOM
IMAGE=2
FLOAT16=1
NOLOCALS=1
JIT_BATCH_SIZE=0
OPENPILOT_HACKS=1

The critical correction was OPENPILOT_HACKS=1.

Without that, models can compile into artifacts that look valid but behave differently from stock openpilot builds, including:

• PoseNET/JIT shape mismatches
• freezes when going onroad
• iqmodeld death or stall after model startup
• artifacts that drive on stock but fail on IQPilot

Current Expected Device Context

These instructions assume:

• IQPilot lives at /data/openpilot
• the repo on your Mac is /Users/t/Developer/iqpilot/iqpilot
• the target device is reachable over SSH as iq@<ip>
• the device has a working venv at /usr/local/venv
• tinygrad is available under /data/openpilot/tinygrad_repo

If your paths differ, adjust the commands accordingly.

1. Confirm IQPilot Source Is Using the Correct Flags

Before compiling anything, verify iqpilot/selfdrive/iqmodeld/SConscript is returning the current QCOM flags for larch64.

The effective flag set for the QCOM path should include:

DEV=QCOM IMAGE=2 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1

If you edit the file locally, copy it to the device before recompiling:

scp /Users/t/Developer/iqpilot/iqpilot/iqpilot/selfdrive/iqmodeld/SConscript \
  iq@<ip>:/data/openpilot/iqpilot/selfdrive/iqmodeld/SConscript

2. Identify the Source ONNX and Target Artifact Name

For each model, determine:

• source ONNX file
• published artifact filename
• selector entry in IQModels

Examples from recent working flow:

• TobyRL
  • source: driving_supercombo.onnx
  • artifact: driving_supercombo_tobyrl.pkl
• NoPP
  • source: driving_supercombo.onnx
  • artifact: driving_supercombo_nopp.pkl
• DRLV3
  • source: driving_supercombo.onnx
  • artifact: driving_supercombo_drl3.pkl

3. Copy the Source ONNX to the Device

Copy the chosen ONNX to a neutral working path on-device:

scp /path/to/driving_supercombo.onnx \
  iq@<ip>:/data/media/0/driving_supercombo_<name>_source.onnx

Example:

scp '/Volumes/New New Vault/IQModels/models/recompiled16/model-No PP (June 26, 2026)-165/driving_supercombo.onnx' \
  iq@192.168.173.10:/data/media/0/driving_supercombo_nopp_source.onnx

4. Prepare Writable Cache Directories on the Device

Do this once per session before compiling.

This avoids tinygrad falling back to unwritable root-owned cache paths and prevents failures caused by:

• /root/.cache write problems
• temporary directory permission failures
• unstable shell environments

ssh iq@<ip> "sudo mkdir -p /data/root-home /data/root-cache /data/root-uv-cache /data/root-tmp"

5. Compile the Model On-Device

Use the exact compile environment below.

This is the current known-good command:

ssh iq@<ip> "sudo env \
HOME=/data/root-home \
XDG_CACHE_HOME=/data/root-cache \
UV_CACHE_DIR=/data/root-uv-cache \
TMPDIR=/data/root-tmp \
PYTHONPATH=/data/openpilot:/data/openpilot/tinygrad_repo \
PATH=/usr/local/venv/bin:/usr/sbin:/usr/bin:/sbin:/bin \
VIRTUAL_ENV=/usr/local/venv \
DEV=QCOM \
WARP_DEV=QCOM \
IMAGE=2 \
FLOAT16=1 \
NOLOCALS=1 \
JIT_BATCH_SIZE=0 \
OPENPILOT_HACKS=1 \
/usr/local/venv/bin/python \
/data/openpilot/iqpilot/selfdrive/iqmodeld/tools/compile_supercombo.py \
  --model-size 512x256 \
  --camera-resolutions 1344x760 1928x1208 \
  --onnx /data/media/0/driving_supercombo_<name>_source.onnx \
  --output /data/media/0/models/driving_supercombo_<name>.recompiled.pkl \
  --frame-skip 4 \
  --expected-device QCOM"

Example: NoPP

ssh iq@192.168.173.10 "sudo env \
HOME=/data/root-home \
XDG_CACHE_HOME=/data/root-cache \
UV_CACHE_DIR=/data/root-uv-cache \
TMPDIR=/data/root-tmp \
PYTHONPATH=/data/openpilot:/data/openpilot/tinygrad_repo \
PATH=/usr/local/venv/bin:/usr/sbin:/usr/bin:/sbin:/bin \
VIRTUAL_ENV=/usr/local/venv \
DEV=QCOM \
WARP_DEV=QCOM \
IMAGE=2 \
FLOAT16=1 \
NOLOCALS=1 \
JIT_BATCH_SIZE=0 \
OPENPILOT_HACKS=1 \
/usr/local/venv/bin/python \
/data/openpilot/iqpilot/selfdrive/iqmodeld/tools/compile_supercombo.py \
  --model-size 512x256 \
  --camera-resolutions 1344x760 1928x1208 \
  --onnx /data/media/0/driving_supercombo_nopp_source.onnx \
  --output /data/media/0/models/driving_supercombo_nopp.recompiled.pkl \
  --frame-skip 4 \
  --expected-device QCOM"

6. What Success Looks Like

On success, the compile ends with output similar to:

Saved JITs to /data/media/0/models/driving_supercombo_<name>.recompiled.pkl (119.07 MB)

Current good rebuilt supercombo artifacts have been around 114M to 119 MB.

If you get a tiny file, do not publish it.

Always verify both size and hash:

ssh iq@<ip> "ls -lh /data/media/0/models/driving_supercombo_<name>.recompiled.pkl"
ssh iq@<ip> "sha256sum /data/media/0/models/driving_supercombo_<name>.recompiled.pkl"

7. Expected Warnings During Compile

These warnings have appeared during successful builds and are not by themselves proof of a bad artifact:

input desire_pulse has mismatch on dtype. Expected dtypes.half, received dtypes.float.
input traffic_convention has mismatch on dtype. Expected dtypes.half, received dtypes.float.
input action_t has mismatch on dtype. Expected dtypes.half, received dtypes.float.
input features_buffer has mismatch on dtype. Expected dtypes.half, received dtypes.float.

Treat them as informational unless the resulting model fails runtime validation.

8. Install the Rebuilt Artifact On the Device

Before publishing to IQModels, install the rebuilt artifact locally on the device being tested.

Back up the current file first:

ssh iq@<ip> "cp /data/media/0/models/driving_supercombo_<name>.pkl /data/media/0/models/driving_supercombo_<name>.pkl.bak.\$(date +%s)"

Then replace it:

ssh iq@<ip> "sudo cp /data/media/0/models/driving_supercombo_<name>.recompiled.pkl /data/media/0/models/driving_supercombo_<name>.pkl"

Verify the active file hash:

ssh iq@<ip> "sha256sum /data/media/0/models/driving_supercombo_<name>.pkl"

9. Validate the Model Before Publishing

Do not push a model just because it compiled.

Current IQPilot validation should include both of these:

9.1 Offroad/on-device iqmodeld harness validation

Use the current harness:

iqpilot/selfdrive/iqmodeld/tools/force_onroad_iqmodeld_test.py

This script creates a minimal fake onroad environment and starts iqmodeld.

Run it on-device from the normal IQPilot tree:

ssh iq@<ip> "cd /data/openpilot && /usr/local/venv/bin/python /data/openpilot/iqpilot/selfdrive/iqmodeld/tools/force_onroad_iqmodeld_test.py --speed 15.0"

What you want:

• iqmodeld stays alive
• no PoseNET/JIT mismatch crash
• no immediate deadlock
• no model startup freeze

9.2 Real launcher / real device validation

The harness is necessary, but not sufficient.

Also validate through the actual launcher/runtime:

• boot IQPilot normally
• select the model through the selector
• clear model cache if required by your workflow
• confirm the model loads
• confirm it survives transition to onroad
• confirm modelV2, livePose, iqPlan, and related services stay alive
• confirm the UI does not freeze

For any model intended for release, a real onroad validation pass is strongly recommended before publishing.

10. Runtime Signals to Watch

When a model is bad, the system often does not fail cleanly.

Useful checks:

ssh iq@<ip> "tmux capture-pane -pt 0"
ssh iq@<ip> "cat /data/community/crashes/error.log"
ssh iq@<ip> "ls -1t /data/community/crashes | head"

Common bad signals:

• tinygrad.engine.jit.JitError: args mismatch in JIT
• expected_input_info showing CPU on one side and QCOM on the other
• shape mismatch like arg=2 vs arg=5
• iqmodeld disappearing or zombifying
• modelV2, livePose, or iqPlan dying
• UI frozen onroad
• commIssue spam caused by model pipeline collapse

11. Common Failure Modes and Meaning

Failure: args mismatch in JIT

Typical causes:

• compiled artifact does not match current runtime expectations
• wrong frame_skip
• wrong device capture path
• CPU-built artifact being used on QCOM runtime
• stale/old artifact still being loaded

Failure: artifact references CPU in crash output

Typical cause:

• artifact compiled on the wrong backend or with the wrong compile environment

Fix:

• rebuild on-device with DEV=QCOM, WARP_DEV=QCOM, and OPENPILOT_HACKS=1

Failure: tiny file after scp

Typical cause:

• interrupted copy
• partial transfer

Fix:

• verify local file size
• verify local hash
• do not publish until the copied file matches the device hash exactly

Failure: tinygrad cache or permission errors

Typical causes:

• /root/.cache unwritable
• shell launched outside the expected manager environment

Fix:

• use the writable cache dirs from this guide
• use the exact sudo env ... compile command above

Failure: model compiles but freezes onroad

Possible causes:

• wrong compile flags despite a successful build
• artifact differs from stock QCOM compile assumptions
• runtime deadlock only visible under real launch conditions

Fix:

• recompile with the current known-good flags
• run the harness
• then validate through normal IQPilot startup and onroad transition

12. Pull the Final Artifact Back to the Workstation

Once the device artifact is validated, pull it back locally:

scp -o StrictHostKeyChecking=no \
  iq@<ip>:/data/media/0/models/driving_supercombo_<name>.pkl \
  /tmp/driving_supercombo_<name>.recompiled.full

Verify locally:

ls -lh /tmp/driving_supercombo_<name>.recompiled.full
sha256sum /tmp/driving_supercombo_<name>.recompiled.full

The local hash must match the device hash exactly.

13. Publish to IQModels

Mounted example path:

/Volumes/New New Vault/IQModels

Replace the existing artifact in the model directory:

cp /tmp/driving_supercombo_<name>.recompiled.full \
  '/Volumes/New New Vault/IQModels/models/recompiled16/model-<Model Name>/driving_supercombo_<name>.pkl'

Then update both:

• docs/model_selector_b.json
• docs/model_fetcher_b.json

Update the sha256 values to the rebuilt artifact hash. If the file path and artifact filename stay the same, the URL does not need to change.

14. Verify Published Repo State

Before committing:

git -C '/Volumes/New New Vault/IQModels' status --short
git -C '/Volumes/New New Vault/IQModels' diff -- docs/model_selector_b.json docs/model_fetcher_b.json
sha256sum '/Volumes/New New Vault/IQModels/models/recompiled16/model-<Model Name>/driving_supercombo_<name>.pkl'

Make sure:

• repo artifact hash matches the rebuilt local hash
• selector SHA matches the repo artifact hash
• only intended model files and manifest files are changed

15. Commit and Push

Example:

git -C '/Volumes/New New Vault/IQModels' add -- \
  docs/model_selector_b.json \
  docs/model_fetcher_b.json \
  'models/recompiled16/model-<Model Name>/driving_supercombo_<name>.pkl'

git -C '/Volumes/New New Vault/IQModels' commit -m 'Recompile <Model Name> with corrected QCOM flags'
git -C '/Volumes/New New Vault/IQModels' push

16. Post-Publish Validation

After pushing:

• redownload the model through the selector on a device
• confirm the downloaded file SHA matches the published selector SHA
• confirm the device loads the rebuilt model
• confirm the model survives onroad entry

Useful command:

python3 - <<'PY'
from openpilot.common.params import Params
print(Params().get("ModelManager_ActiveBundle"))
PY

17. Recommended Release Checklist

Use this every time.

1. Confirm SConscript contains the current QCOM flags, especially OPENPILOT_HACKS=1.
2. Copy the source ONNX to the device.
3. Compile on-device with the exact environment from this guide.
4. Verify artifact size is full-sized, not truncated.
5. Verify artifact hash.
6. Install the rebuilt artifact on the test device.
7. Run the force_onroad_iqmodeld_test.py harness.
8. Validate under the real launcher path.
9. Confirm no crash logs, no PoseNET/JIT mismatch, no freeze.
10. Pull the artifact back locally and verify hash match.
11. Replace the published IQModels artifact.
12. Update selector and fetcher SHA values.
13. Verify repo hash, manifest hash, and git diff.
14. Commit and push.
15. Redownload through selector and validate one more time.

18. Notes for Future Agents

• Do not assume a successful compile means the model is good.
• Do not assume a selector SHA mismatch is the only cause of failure.
• Do not publish partially copied .pkl files.
• Do not skip the real launcher/onroad validation path.
• If a model works on stock openpilot but fails on IQPilot, compare the QCOM compile path first.
• When in doubt, rebuild on-device with the exact current flags in this guide and validate again.