models: persist model selection per catalog across chestnut state changes (#1960 )

models: fix current model not updating on chestnut status (#1959 )
* models: preserve user model selection across reboots and power cycles * no * again * idk * over
2026-09-11 00:23:43 +08:00 · 2026-08-25 01:12:12 -04:00 · 2026-08-25 00:41:31 -04:00 · 2026-08-24 23:31:52 -04:00 · 2026-08-24 22:52:29 -04:00 · 2026-08-24 22:42:45 -04:00
35 changed files with 1412 additions and 599 deletions
@@ -0,0 +1,501 @@
+name: Build default models
+
+on:
+  workflow_dispatch:
+    inputs:
+      target:
+        description: 'Model target to build'
+        required: true
+        type: choice
+        options:
+          - small
+          - big
+          - dm
+  workflow_call:
+    inputs:
+      target:
+        description: 'Model target to build (small, big, or dm)'
+        required: true
+        type: string
+
+concurrency:
+  group: build-default-models-${{ inputs.target }}
+  cancel-in-progress: false
+
+env:
+  HF_REPO: sunnypilot/sunnypilot_models_v1
+
+jobs:
+  resolve:
+    runs-on: ubuntu-24.04
+    outputs:
+      model_name: ${{ steps.resolve.outputs.model_name }}
+      onnx_ref: ${{ steps.resolve.outputs.onnx_ref }}
+      onnx_path: ${{ steps.resolve.outputs.onnx_path }}
+      hf_defaults_path: ${{ steps.resolve.outputs.hf_defaults_path }}
+      tinygrad_ref: ${{ steps.resolve.outputs.tinygrad_ref }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive
+
+      - id: resolve
+        run: |
+          export PYTHONPATH=${{ github.workspace }}
+
+          if [ "${{ inputs.target }}" = "big" ]; then
+            NAME=$(python3 -c "from openpilot.sunnypilot.models.model_name import DEFAULT_BIG_MODEL; print(DEFAULT_BIG_MODEL)")
+            ONNX_PATH="openpilot/selfdrive/modeld/models/big_driving_supercombo.onnx"
+            HF_DEFAULTS_PATH="models/defaults/big"
+          elif [ "${{ inputs.target }}" = "dm" ]; then
+            ONNX_PATH="openpilot/selfdrive/modeld/models/dmonitoring_model.onnx"
+            HF_DEFAULTS_PATH="models/defaults/dm"
+            NAME="dmonitoring_model ($(git log -1 --format=%cd --date=format:'%B %d, %Y' -- "$ONNX_PATH"))"
+          else
+            NAME=$(python3 -c "from openpilot.sunnypilot.models.model_name import DEFAULT_MODEL; print(DEFAULT_MODEL)")
+            ONNX_PATH="openpilot/selfdrive/modeld/models/driving_supercombo.onnx"
+            HF_DEFAULTS_PATH="models/defaults/small"
+          fi
+
+          ONNX_REF=$(git log -1 --format='%H' -- "$ONNX_PATH")
+          TINYGRAD_REF=$(python3 openpilot/sunnypilot/models/tinygrad_ref.py)
+          if [ -z "$TINYGRAD_REF" ]; then
+            echo "::error::Failed to resolve tinygrad ref"
+            exit 1
+          fi
+
+          echo "model_name=${NAME}" >> $GITHUB_OUTPUT
+          echo "onnx_ref=${ONNX_REF}" >> $GITHUB_OUTPUT
+          echo "onnx_path=${ONNX_PATH}" >> $GITHUB_OUTPUT
+          echo "hf_defaults_path=${HF_DEFAULTS_PATH}" >> $GITHUB_OUTPUT
+          echo "tinygrad_ref=${TINYGRAD_REF}" >> $GITHUB_OUTPUT
+
+  build_small_model:
+    needs: resolve
+    if: ${{ inputs.target == 'small' }}
+    runs-on: [self-hosted, tici]
+    env:
+      SMALL_ONNX: openpilot/selfdrive/modeld/models/driving_supercombo.onnx
+      SMALL_PKL: openpilot/selfdrive/modeld/models/driving_tinygrad.pkl
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive
+
+      - name: Pull ONNX via LFS
+        run: git lfs pull -I "${{ env.SMALL_ONNX }}"
+
+      - name: Set environment variables
+        run: |
+          source /etc/profile
+          export UV_PROJECT_ENVIRONMENT=${HOME}/venv
+          export UV_PYTHON_PREFERENCE=managed
+          export UV_PYTHON_INSTALL_DIR=${HOME}/uv/python
+          export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
+          uv sync --frozen
+          printenv >> $GITHUB_ENV
+
+      - name: Disable powersave
+        run: |
+          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+          PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
+
+      - name: Compile small model with stock compiler
+        run: |
+          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+          export PYTHONPATH="${PYTHONPATH}:${{ github.workspace }}/tinygrad_repo:${{ github.workspace }}"
+
+          MODEL_SIZE=$(python3 -c "from openpilot.common.transformations.model import MEDMODEL_INPUT_SIZE as s; print(f'{s[0]}x{s[1]}')")
+          CAMERA_RES=$(python3 -c "from openpilot.common.transformations.camera import _ar_ox_fisheye as a, _os_fisheye as o; print(f'{a.width}x{a.height} {o.width}x{o.height}')")
+          FRAME_SKIP=$(python3 -c "from openpilot.selfdrive.modeld.constants import ModelConstants as MC; print(MC.MODEL_RUN_FREQ // MC.MODEL_CONTEXT_FREQ)")
+
+          TG_FLAGS="DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1"
+
+          env ${TG_FLAGS} python3 \
+            ${{ github.workspace }}/openpilot/selfdrive/modeld/compile_modeld.py \
+            --onnx ${{ github.workspace }}/${{ env.SMALL_ONNX }} \
+            --model-size $MODEL_SIZE \
+            --camera-resolutions $CAMERA_RES \
+            --frame-skip $FRAME_SKIP \
+            --output ${{ github.workspace }}/${{ env.SMALL_PKL }}
+
+      - name: Chunk small pkl
+        run: |
+          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+          export PYTHONPATH=${{ github.workspace }}
+          python3 -c "
+          from openpilot.common.file_chunker import chunk_file, get_chunk_targets
+          import os
+          pkl = '${{ github.workspace }}/${{ env.SMALL_PKL }}'
+          size = os.path.getsize(pkl)
+          targets = get_chunk_targets(pkl, size)
+          chunk_file(pkl, targets)
+          print(f'Chunked into {len(targets)} files')
+          "
+
+      - name: Prepare output
+        env:
+          MODEL_NAME: ${{ needs.resolve.outputs.model_name }}
+        run: |
+          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+          export PYTHONPATH=${{ github.workspace }}
+          MODELS_DIR="${{ github.workspace }}/openpilot/selfdrive/modeld/models"
+          OUTPUT_DIR="${{ github.workspace }}/small_output"
+          PKL_BASE="driving_tinygrad.pkl"
+          mkdir -p "$OUTPUT_DIR"
+
+          cp "$MODELS_DIR/${PKL_BASE}".chunk* "$OUTPUT_DIR/"
+          cp "$MODELS_DIR/${PKL_BASE}.chunkmanifest" "$OUTPUT_DIR/"
+
+          python3 "${{ github.workspace }}/release/ci/model_generator.py" \
+            --model-dir "$MODELS_DIR" \
+            --output-dir "$OUTPUT_DIR" \
+            --custom-name "$MODEL_NAME" \
+            --upstream-branch "${{ needs.resolve.outputs.onnx_ref }}"
+
+          echo "model-${MODEL_NAME}-${{ github.run_number }}" > "$OUTPUT_DIR/artifact_name.txt"
+
+      - name: Upload small model artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: model-${{ needs.resolve.outputs.model_name }}-${{ github.run_number }}
+          path: ${{ github.workspace }}/small_output/
+
+      - name: Upload artifact name file
+        uses: actions/upload-artifact@v4
+        with:
+          name: artifact-name-${{ needs.resolve.outputs.model_name }}
+          path: ${{ github.workspace }}/small_output/artifact_name.txt
+
+      - name: Re-enable powersave
+        if: always()
+        run: |
+          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+          PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
+
+  build_big_model:
+    needs: resolve
+    if: ${{ inputs.target == 'big' }}
+    runs-on: [self-hosted, usbgpu]
+    env:
+      BIG_ONNX: openpilot/selfdrive/modeld/models/big_driving_supercombo.onnx
+      BIG_PKL: openpilot/selfdrive/modeld/models/big_driving_tinygrad.pkl
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive
+
+      - name: Pull big ONNX via LFS
+        run: git lfs pull -I "${{ env.BIG_ONNX }}"
+
+      - name: Set environment variables
+        run: |
+          source /etc/profile
+          export UV_PROJECT_ENVIRONMENT=${HOME}/venv
+          export UV_PYTHON_PREFERENCE=managed
+          export UV_PYTHON_INSTALL_DIR=${HOME}/uv/python
+          export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
+          uv sync --frozen
+          printenv >> $GITHUB_ENV
+
+      - name: Disable powersave
+        run: |
+          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+          PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
+
+      - name: Wait for chestnut PCIe link
+        run: |
+          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+          export PYTHONPATH="${PYTHONPATH}:${{ github.workspace }}/tinygrad_repo:${{ github.workspace }}"
+          python3 -c "
+          import time
+          from openpilot.system.hardware.chestnut.flash import link_up
+          for i in range(10):
+              if link_up():
+                  print(f'PCIe link up after {i+1} attempt(s)')
+                  break
+              time.sleep(1)
+          else:
+              raise RuntimeError('Chestnut PCIe link not ready after 10 attempts')
+          "
+
+      - name: Compile big model with stock compiler
+        run: |
+          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+          export PYTHONPATH="${PYTHONPATH}:${{ github.workspace }}/tinygrad_repo:${{ github.workspace }}"
+
+          MODEL_SIZE=$(python3 -c "from openpilot.common.transformations.model import MEDMODEL_INPUT_SIZE as s; print(f'{s[0]}x{s[1]}')")
+          CAMERA_RES=$(python3 -c "from openpilot.common.transformations.camera import _ar_ox_fisheye as a, _os_fisheye as o; print(f'{a.width}x{a.height} {o.width}x{o.height}')")
+          FRAME_SKIP=$(python3 -c "from openpilot.selfdrive.modeld.constants import ModelConstants as MC; print(MC.MODEL_RUN_FREQ // MC.MODEL_CONTEXT_FREQ)")
+
+          TG_FLAGS="DEBUG=2 DEV=USB+AMD:LLVM WARP_DEV=QCOM FLOAT16=1 JIT_BATCH_SIZE=0 GMMU=0 TC_OPT=2"
+
+          env ${TG_FLAGS} python3 \
+            ${{ github.workspace }}/openpilot/selfdrive/modeld/compile_modeld.py \
+            --onnx ${{ github.workspace }}/${{ env.BIG_ONNX }} \
+            --model-size $MODEL_SIZE \
+            --camera-resolutions $CAMERA_RES \
+            --frame-skip $FRAME_SKIP \
+            --output ${{ github.workspace }}/${{ env.BIG_PKL }}
+
+      - name: Chunk big pkl
+        run: |
+          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+          export PYTHONPATH=${{ github.workspace }}
+          python3 -c "
+          from openpilot.common.file_chunker import chunk_file, get_chunk_targets
+          import os
+          pkl = '${{ github.workspace }}/${{ env.BIG_PKL }}'
+          size = os.path.getsize(pkl)
+          targets = get_chunk_targets(pkl, size)
+          chunk_file(pkl, targets)
+          print(f'Chunked into {len(targets)} files')
+          "
+
+      - name: Prepare output
+        env:
+          MODEL_NAME: ${{ needs.resolve.outputs.model_name }}
+        run: |
+          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+          export PYTHONPATH=${{ github.workspace }}
+          MODELS_DIR="${{ github.workspace }}/openpilot/selfdrive/modeld/models"
+          OUTPUT_DIR="${{ github.workspace }}/big_output"
+          PKL_BASE="big_driving_tinygrad.pkl"
+          mkdir -p "$OUTPUT_DIR"
+
+          cp "$MODELS_DIR/${PKL_BASE}".chunk* "$OUTPUT_DIR/"
+          cp "$MODELS_DIR/${PKL_BASE}.chunkmanifest" "$OUTPUT_DIR/"
+
+          python3 "${{ github.workspace }}/release/ci/model_generator.py" \
+            --model-dir "$MODELS_DIR" \
+            --output-dir "$OUTPUT_DIR" \
+            --custom-name "$MODEL_NAME" \
+            --upstream-branch "${{ needs.resolve.outputs.onnx_ref }}"
+
+          echo "model-${MODEL_NAME}-${{ github.run_number }}" > "$OUTPUT_DIR/artifact_name.txt"
+
+      - name: Upload big model artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: model-${{ needs.resolve.outputs.model_name }}-${{ github.run_number }}
+          path: ${{ github.workspace }}/big_output/
+
+      - name: Upload artifact name file
+        uses: actions/upload-artifact@v4
+        with:
+          name: artifact-name-${{ needs.resolve.outputs.model_name }}
+          path: ${{ github.workspace }}/big_output/artifact_name.txt
+
+      - name: Re-enable powersave
+        if: always()
+        run: |
+          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+          PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
+
+  upload_defaults:
+    needs: [ resolve, build_small_model, build_big_model, build_dm_model ]
+    if: |
+      ${{
+        !cancelled() &&
+        (inputs.target == 'big' && needs.build_big_model.result == 'success' ||
+         inputs.target == 'small' && needs.build_small_model.result == 'success' ||
+         inputs.target == 'dm' && needs.build_dm_model.result == 'success')
+      }}
+    runs-on: ubuntu-24.04
+    permissions:
+      id-token: write
+      contents: write
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Pull ONNX via LFS
+        run: git lfs pull -I "${{ needs.resolve.outputs.onnx_path }}"
+
+      - name: Install huggingface_hub
+        run: pip install --upgrade "huggingface_hub>=0.22.0"
+
+      - name: Download artifact name
+        if: ${{ inputs.target == 'small' || inputs.target == 'big' }}
+        uses: actions/download-artifact@v4
+        with:
+          name: artifact-name-${{ needs.resolve.outputs.model_name }}
+          path: artifact_name
+
+      - name: Read artifact name
+        if: ${{ inputs.target == 'small' || inputs.target == 'big' }}
+        id: artifact
+        run: |
+          ARTIFACT_NAME=$(cat artifact_name/artifact_name.txt)
+          echo "artifact_name=$ARTIFACT_NAME" >> $GITHUB_OUTPUT
+
+      - name: Download model artifact
+        if: ${{ inputs.target == 'small' || inputs.target == 'big' }}
+        uses: actions/download-artifact@v4
+        with:
+          name: ${{ steps.artifact.outputs.artifact_name }}
+          path: output
+
+      - name: Upload model to HF
+        if: ${{ inputs.target == 'small' || inputs.target == 'big' }}
+        env:
+          HF_OIDC_RESOURCE: datasets/${{ env.HF_REPO }}
+          ARTIFACT_NAME: ${{ steps.artifact.outputs.artifact_name }}
+        run: |
+          rm -f output/artifact_name.txt
+          export PYTHONPATH=$(pwd)
+          python3 release/ci/upload_default_model.py \
+            --hf-repo "${{ env.HF_REPO }}" \
+            --hf-defaults-path "${{ needs.resolve.outputs.hf_defaults_path }}" \
+            --artifact-name "$ARTIFACT_NAME" \
+            --model-dir output \
+            --onnx-path "${{ needs.resolve.outputs.onnx_path }}" \
+            --onnx-ref "${{ needs.resolve.outputs.onnx_ref }}" \
+            --model-name "${{ needs.resolve.outputs.model_name }}" \
+            --tinygrad-ref "${{ needs.resolve.outputs.tinygrad_ref }}" \
+            --run-number "${{ github.run_number }}"
+
+      - name: Download DM artifact
+        if: ${{ inputs.target == 'dm' }}
+        uses: actions/download-artifact@v4
+        with:
+          name: dm-model-${{ github.run_number }}
+          path: dm_output
+
+      - name: Generate DM metadata and upload to HF
+        if: ${{ inputs.target == 'dm' }}
+        env:
+          HF_OIDC_RESOURCE: datasets/${{ env.HF_REPO }}
+        run: |
+          export PYTHONPATH=$(pwd)
+          python3 -c "
+          import json, hashlib
+          from pathlib import Path
+          from datetime import datetime, UTC
+
+          dm_dir = Path('dm_output')
+          manifest = list(dm_dir.glob('*.chunkmanifest'))
+          assert manifest, 'No chunkmanifest found'
+          pkl_name = manifest[0].name.removesuffix('.chunkmanifest')
+          num_chunks = int(manifest[0].read_text().strip())
+
+          chunks = []
+          for i in range(num_chunks):
+              chunk = dm_dir / f'{pkl_name}.chunk{i+1:02d}of{num_chunks:02d}'
+              chunks.append({
+                  'file_name': chunk.name,
+                  'sha256': hashlib.sha256(chunk.read_bytes()).hexdigest()
+              })
+
+          digest = hashlib.sha256()
+          for c in chunks:
+              with open(dm_dir / c['file_name'], 'rb') as f:
+                  while block := f.read(1024*1024):
+                      digest.update(block)
+
+          metadata = {
+              'bundles': [{
+                  'short_name': 'DMMODEL',
+                  'display_name': '${{ needs.resolve.outputs.model_name }}',
+                  'ref': '${{ needs.resolve.outputs.onnx_ref }}',
+                  'runner': 'tinygrad',
+                  'build_time': datetime.now(UTC).strftime('%Y-%m-%dT%H:%M:%SZ'),
+                  'models': [{
+                      'type': 'chunked',
+                      'artifact': {
+                          'file_name': pkl_name,
+                          'download_uri': {'url': '', 'sha256': digest.hexdigest()},
+                          'chunks': chunks
+                      }
+                  }]
+              }]
+          }
+          with open(dm_dir / 'metadata.json', 'w') as f:
+              json.dump(metadata, f, indent=2)
+          print('Generated DM metadata.json')
+          "
+
+          python3 release/ci/upload_default_model.py \
+            --hf-repo "${{ env.HF_REPO }}" \
+            --hf-defaults-path "${{ needs.resolve.outputs.hf_defaults_path }}" \
+            --artifact-name "dm-model-${{ github.run_number }}" \
+            --model-dir dm_output \
+            --onnx-path "${{ needs.resolve.outputs.onnx_path }}" \
+            --onnx-ref "${{ needs.resolve.outputs.onnx_ref }}" \
+            --model-name "${{ needs.resolve.outputs.model_name }}" \
+            --tinygrad-ref "${{ needs.resolve.outputs.tinygrad_ref }}" \
+            --run-number "${{ github.run_number }}"
+
+  build_dm_model:
+    needs: resolve
+    if: ${{ inputs.target == 'dm' }}
+    runs-on: [self-hosted, tici]
+    env:
+      DM_ONNX: openpilot/selfdrive/modeld/models/dmonitoring_model.onnx
+      DM_PKL: openpilot/selfdrive/modeld/models/dmonitoring_model_tinygrad.pkl
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive
+
+      - name: Pull DM ONNX via LFS
+        run: git lfs pull -I "${{ env.DM_ONNX }}"
+
+      - name: Set environment variables
+        run: |
+          source /etc/profile
+          export UV_PROJECT_ENVIRONMENT=${HOME}/venv
+          export UV_PYTHON_PREFERENCE=managed
+          export UV_PYTHON_INSTALL_DIR=${HOME}/uv/python
+          export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
+          uv sync --frozen
+          printenv >> $GITHUB_ENV
+
+      - name: Disable powersave
+        run: |
+          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+          PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
+
+      - name: Compile DM model
+        run: |
+          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+          export PYTHONPATH="${PYTHONPATH}:${{ github.workspace }}/tinygrad_repo:${{ github.workspace }}"
+
+          TG_FLAGS="DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1"
+
+          taskset -c 7 env ${TG_FLAGS} python3 \
+            ${{ github.workspace }}/tinygrad_repo/examples/openpilot/compile3.py \
+            ${{ github.workspace }}/${{ env.DM_ONNX }} \
+            ${{ github.workspace }}/${{ env.DM_PKL }}
+
+      - name: Chunk DM pkl
+        run: |
+          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+          export PYTHONPATH=${{ github.workspace }}
+          python3 -c "
+          from openpilot.common.file_chunker import chunk_file, get_chunk_targets
+          import os
+          pkl = '${{ github.workspace }}/${{ env.DM_PKL }}'
+          size = os.path.getsize(pkl)
+          targets = get_chunk_targets(pkl, size)
+          chunk_file(pkl, targets)
+          print(f'Chunked {pkl} into {len(targets)} chunks')
+          "
+
+      - name: Prepare DM output
+        run: |
+          mkdir -p dm_output
+          cp ${{ github.workspace }}/${{ env.DM_PKL }}.chunk* dm_output/
+          cp ${{ github.workspace }}/${{ env.DM_PKL }}.chunkmanifest dm_output/
+
+      - name: Upload DM artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: dm-model-${{ github.run_number }}
+          path: dm_output/
+
+      - name: Re-enable powersave
+        if: always()
+        run: |
+          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+          PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
+
@@ -0,0 +1,66 @@
+name: Download HF model chunks
+description: Resolve and download model chunks from HuggingFace in parallel
+
+inputs:
+  hf_repo:
+    description: HuggingFace dataset repo
+    required: true
+  models:
+    description: 'JSON array of {hf_path, onnx_hash, canonical} objects'
+    required: true
+  dest_dir:
+    description: Destination directory for downloaded chunks
+    required: true
+
+runs:
+  using: composite
+  steps:
+    - name: Download model chunks
+      shell: bash
+      env:
+        HF_REPO: ${{ inputs.hf_repo }}
+        MODELS_JSON: ${{ inputs.models }}
+        DEST_DIR: ${{ inputs.dest_dir }}
+      run: |
+        set -eo pipefail
+        DOWNLOAD_LIST=$(mktemp)
+
+        resolve_chunks() {
+          local HF_PATH="$1" ONNX_HASH="$2" CANONICAL="$3" DEST_DIR="$4"
+          local JSON_URL="https://huggingface.co/datasets/${HF_REPO}/resolve/main/${HF_PATH}/default_models.json"
+          local DEFAULTS BUNDLE ARTIFACT BASE_URL NUM_CHUNKS
+          DEFAULTS=$(curl -fsSL "$JSON_URL")
+          BUNDLE=$(echo "$DEFAULTS" | jq --arg hash "$ONNX_HASH" '.bundles[] | select(.onnx_sha256 == $hash)')
+          ARTIFACT=$(echo "$BUNDLE" | jq -r '.models[0].artifact')
+          BASE_URL=$(echo "$ARTIFACT" | jq -r '.download_uri.url' | sed 's|/[^/]*$||')
+          NUM_CHUNKS=$(echo "$ARTIFACT" | jq -r '.chunks | length')
+
+          mkdir -p "$DEST_DIR"
+          while IFS= read -r CHUNK_NAME; do
+            CHUNK_IDX=$(echo "$CHUNK_NAME" | grep -oP 'chunk\K[0-9]+of[0-9]+' || true)
+            if [ -z "$CHUNK_IDX" ]; then
+              echo "::error::Failed to parse chunk index from: $CHUNK_NAME"
+              return 1
+            fi
+            ENCODED_URL=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${BASE_URL}/${CHUNK_NAME}', safe=':/'))")
+            printf '%s\t%s\n' "$ENCODED_URL" "${DEST_DIR}/${CANONICAL}.chunk${CHUNK_IDX}" >> "$DOWNLOAD_LIST"
+          done < <(echo "$ARTIFACT" | jq -r '.chunks[].file_name')
+          echo "$NUM_CHUNKS" > "${DEST_DIR}/${CANONICAL}.chunkmanifest"
+        }
+
+        echo "$MODELS_JSON" | jq -c '.[]' | while IFS= read -r model; do
+          HF_PATH=$(echo "$model" | jq -r '.hf_path')
+          ONNX_HASH=$(echo "$model" | jq -r '.onnx_hash')
+          CANONICAL=$(echo "$model" | jq -r '.canonical')
+          resolve_chunks "$HF_PATH" "$ONNX_HASH" "$CANONICAL" "$DEST_DIR"
+        done
+
+        TOTAL=$(wc -l < "$DOWNLOAD_LIST")
+        echo "Downloading $TOTAL chunks with 8 parallel connections..."
+        xargs -P8 -d'\n' -I{} bash -c '
+          URL="${1%%	*}"
+          DEST="${1#*	}"
+          echo "Downloading $(basename "$DEST")"
+          curl -fsSL --retry 3 --retry-delay 5 -o "$DEST" "$URL"
+        ' _ {} < "$DOWNLOAD_LIST"
+        rm -f "$DOWNLOAD_LIST"
@@ -103,20 +103,25 @@ jobs:
      - run: |
          cd ${{ github.workspace }}/openpilot/openpilot
          if [ "${{ inputs.target_hardware }}" != "usbgpu" ]; then
-            git lfs pull -X "selfdrive/modeld/models/big_*.onnx" -X "selfdrive/modeld/models/dmonitoring_*.onnx"
+            git lfs pull -X "**/selfdrive/modeld/models/big_*.onnx,**/selfdrive/modeld/models/dmonitoring_*.onnx"
            rm -f selfdrive/modeld/models/big_*.onnx selfdrive/modeld/models/dmonitoring_*.onnx
          else
-            git lfs pull -I "selfdrive/modeld/models/big_*.onnx"
+            git lfs pull -I "**/selfdrive/modeld/models/big_*.onnx" -X ""
            find selfdrive/modeld/models -name "*.onnx" ! -name "big_*.onnx" -delete
          fi
+          if grep -lIF "version https://git-lfs.github.com/spec/v1" selfdrive/modeld/models/*.onnx; then
+            echo "::error::the ONNX files above are still LFS pointers, not real models"
+            exit 1
+          fi
      - name: 'Upload Artifact'
        uses: actions/upload-artifact@v4
        with:
          name: models-${{ env.REF }}${{ inputs.artifact_suffix  }}
          path: ${{ github.workspace }}/openpilot/openpilot/selfdrive/modeld/models/*.onnx
+          if-no-files-found: error

  build_model:
-    runs-on: [self-hosted, tici]
+    runs-on: [self-hosted, usbgpu]
    needs: get_model
    env:
      MODEL_NAME: ${{ inputs.custom_name || inputs.upstream_branch }} (${{ needs.get_model.outputs.model_date }})
@@ -127,7 +132,6 @@ jobs:
          fetch-depth: 1
          submodules: recursive

-      - run: git lfs pull

      - name: Set environment variables
        id: set-env
@@ -160,7 +164,7 @@ jobs:
          fi
          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
          PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
-          rm -rf ${{ env.MODELS_DIR }}/*.onnx
+          rm -rf ${{ env.MODELS_DIR }}/*.onnx*

      - name: Download model artifacts
        uses: actions/download-artifact@v4
@@ -180,34 +184,48 @@ jobs:
          MODEL_SIZE=$(python3 -c "from openpilot.common.transformations.model import MEDMODEL_INPUT_SIZE as s; print(f'{s[0]}x{s[1]}')")
          CAMERA_RES=$(python3 -c "from openpilot.common.transformations.camera import _ar_ox_fisheye as a, _os_fisheye as o; print(f'{a.width}x{a.height} {o.width}x{o.height}')")

+          TG_FLAGS_QCOM="DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1"
          if [ "${{ inputs.target_hardware }}" == "usbgpu" ]; then
            echo "USBGPU build"
            export USBGPU=1
-            TG_FLAGS="DEBUG=2 DEV=USB+AMD:LLVM WARP_DEV=QCOM FLOAT16=1 JIT_BATCH_SIZE=0 GMMU=0 TC_OPT=2"
+            TG_FLAGS="DEBUG=1 DEV=USB+AMD:LLVM WARP_DEV=QCOM FLOAT16=1 JIT_BATCH_SIZE=0 GMMU=0 TC_OPT=2"
            OUTPUT_PKL="${{ env.MODELS_DIR }}/big_driving_tinygrad.pkl"
          else
            echo "QCOM build"
-            TG_FLAGS="DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1"
+            TG_FLAGS="$TG_FLAGS_QCOM"
            OUTPUT_PKL="${{ env.MODELS_DIR }}/driving_tinygrad.pkl"
          fi

          # Generate metadata for all ONNX files
          find "${{ env.MODELS_DIR }}" -maxdepth 1 -name '*.onnx' | while IFS= read -r onnx_file; do
            echo "Generating metadata: $onnx_file"
-            env ${TG_FLAGS} python3 "${{ env.MODELS_DIR }}/../get_model_metadata.py" "$onnx_file" || true
+            env ${TG_FLAGS_QCOM} python3 "${{ env.MODELS_DIR }}/../get_model_metadata.py" "$onnx_file" || true
          done

          # Detect model type and build compile args
-          VISION_ONNX="${{ env.MODELS_DIR }}/driving_vision.onnx"
-          POLICY_ONNX="${{ env.MODELS_DIR }}/driving_policy.onnx"
-          OFF_POLICY_ONNX="${{ env.MODELS_DIR }}/driving_off_policy.onnx"
-          ON_POLICY_ONNX="${{ env.MODELS_DIR }}/driving_on_policy.onnx"
+          VISION_ONNX=""
+          for f in "${{ env.MODELS_DIR }}/driving_vision.onnx" "${{ env.MODELS_DIR }}/big_driving_vision.onnx"; do
+            [ -f "$f" ] && VISION_ONNX="$f" && break
+          done
+
+          POLICY_ONNX=""
+          for f in "${{ env.MODELS_DIR }}/driving_policy.onnx" "${{ env.MODELS_DIR }}/big_driving_policy.onnx"; do
+            [ -f "$f" ] && POLICY_ONNX="$f" && break
+          done
+
+          OFF_POLICY_ONNX=""
+          for f in "${{ env.MODELS_DIR }}/driving_off_policy.onnx" "${{ env.MODELS_DIR }}/big_driving_off_policy.onnx"; do
+            [ -f "$f" ] && OFF_POLICY_ONNX="$f" && break
+          done
+
+          ON_POLICY_ONNX=""
+          for f in "${{ env.MODELS_DIR }}/driving_on_policy.onnx" "${{ env.MODELS_DIR }}/big_driving_on_policy.onnx"; do
+            [ -f "$f" ] && ON_POLICY_ONNX="$f" && break
+          done
+
          SUPERCOMBO_ONNX=""
-          for f in "${{ env.MODELS_DIR }}/supercombo.onnx" "${{ env.MODELS_DIR }}/driving_supercombo.onnx"; do
-            if [ -f "$f" ]; then
-              SUPERCOMBO_ONNX="$f"
-              break
-            fi
+          for f in "${{ env.MODELS_DIR }}/supercombo.onnx" "${{ env.MODELS_DIR }}/driving_supercombo.onnx" "${{ env.MODELS_DIR }}/big_supercombo.onnx" "${{ env.MODELS_DIR }}/big_driving_supercombo.onnx"; do
+            [ -f "$f" ] && SUPERCOMBO_ONNX="$f" && break
          done

          MODEL_TYPE="" ONNX_ARGS="" OUTPUT_NAME=""
@@ -36,8 +36,11 @@ jobs:
      publish_concurrency_group: ${{ steps.strategy.outputs.publish_concurrency_group }}
      is_stable_branch: ${{ steps.strategy.outputs.is_stable_branch }}
      build: ${{ steps.strategy.outputs.build }}
+      include_big_model: ${{ steps.strategy.outputs.include_big_model }}
    steps:
      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
      - name: Extract deploy strategy
        id: strategy
        run: |
@@ -78,6 +81,9 @@ jobs:
            stable_version=$(cat openpilot/sunnypilot/common/version.h | grep SUNNYPILOT_VERSION | sed -e 's/[^0-9|.]//g');
            echo "version=$([ "$is_stable_branch" = "true" ] && echo "$stable_version" || echo "$BUILD")" >> $GITHUB_OUTPUT
            echo "extra_version_identifier=${environment}" >> $GITHUB_OUTPUT
+
+            include_big_model="$(echo "$CONFIG" | jq -r '.include_big_model // false')";
+            echo "include_big_model=$include_big_model" >> $GITHUB_OUTPUT
          fi
          echo "build=$BUILD" >> $GITHUB_OUTPUT
          cat $GITHUB_OUTPUT
@@ -92,6 +98,8 @@ jobs:
      }}
    steps:
      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
      - name: Wait for Tests
        uses: ./.github/workflows/wait-for-action # Path to where you place the action
        with:
@@ -115,6 +123,7 @@ jobs:
    steps:
      - uses: actions/checkout@v4
        with:
+          fetch-depth: 1
          submodules: recursive
          ref: ${{ env.SOURCE_BRANCH }}
          repository: ${{ github.event.pull_request.head.repo.fork && github.event.pull_request.head.repo.full_name || github.repository }}
@@ -161,7 +170,7 @@ jobs:
          scons -j1 cache_dir="$SCONS_CACHE" --minimal \
            openpilot/selfdrive/locationd openpilot/sunnypilot/selfdrive/locationd
          echo "Building rest of sunnypilot"
-          /usr/bin/time -v scons -j$(nproc) cache_dir="$SCONS_CACHE" --minimal
+          SKIP_TINYGRAD_COMPILE=1 /usr/bin/time -v scons -j$(nproc) cache_dir="$SCONS_CACHE" --minimal
          touch ${BUILD_DIR}/prebuilt
          if [[ "${{ runner.debug }}" == "1" ]]; then
            ls -la ${BUILD_DIR}
@@ -203,22 +212,212 @@ jobs:
          source ${UV_PROJECT_ENVIRONMENT}/bin/activate
          PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable

+  prepare_chestnut:
+    needs: [ prepare_strategy ]
+    runs-on: ubuntu-24.04
+    if: ${{ needs.prepare_strategy.outputs.include_big_model == 'true' }}
+    outputs:
+      onnx_sha256: ${{ steps.resolve.outputs.onnx_sha256 }}
+    env:
+      GH_REPO: ${{ github.repository }}
+      HF_REPO: sunnypilot/sunnypilot_models_v1
+      HF_DEFAULTS_PATH: models/defaults/big
+    steps:
+      - name: Resolve ONNX hash and tinygrad ref via API
+        id: resolve
+        run: |
+          REF="${{ github.head_ref || github.ref_name }}"
+
+          ONNX_HASH=$(gh api "repos/${GH_REPO}/contents/openpilot/selfdrive/modeld/models/big_driving_supercombo.onnx?ref=${REF}" --jq '.content' | base64 -d | grep '^oid sha256:' | cut -d: -f2)
+          echo "ONNX hash: $ONNX_HASH"
+          echo "onnx_sha256=$ONNX_HASH" >> $GITHUB_OUTPUT
+
+          TINYGRAD_REF=$(gh api "repos/${GH_REPO}/contents/tinygrad_repo?ref=${REF}" --jq '.sha')
+          echo "tinygrad ref: $TINYGRAD_REF"
+
+          JSON_URL="https://huggingface.co/datasets/${HF_REPO}/resolve/main/${HF_DEFAULTS_PATH}/default_models.json"
+
+          check_defaults() {
+            DEFAULTS=$(curl -fsSL "$JSON_URL" 2>/dev/null) || return 1
+            TINYGRAD_MATCH=$(echo "$DEFAULTS" | jq -r --arg ref "$TINYGRAD_REF" '.tinygrad_ref == $ref' 2>/dev/null)
+            [ "$TINYGRAD_MATCH" = "true" ] || return 1
+            BUNDLE=$(echo "$DEFAULTS" | jq --arg hash "$ONNX_HASH" '.bundles[] | select(.onnx_sha256 == $hash)' 2>/dev/null)
+            [ -n "$BUNDLE" ] && [ "$BUNDLE" != "null" ]
+          }
+
+          if check_defaults; then
+            echo "HF defaults match repo ONNX hash and tinygrad ref"
+            exit 0
+          fi
+
+          echo "No matching model on HF — dispatching build"
+          gh workflow run build-default-models.yaml --ref "$REF" -f target=big
+
+          echo "Polling HF for big model availability..."
+          for i in $(seq 1 90); do
+            sleep 30
+            if check_defaults; then
+              echo "Big model available on HF after $((i * 30))s"
+              exit 0
+            fi
+            echo "Poll $i/90: not yet available"
+          done
+
+          echo "::error::Big model not available on HF after 45 minutes"
+          exit 1
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Cancel run on failure
+        if: failure()
+        run: gh run cancel ${{ github.run_id }}
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+  prepare_small_model:
+    needs: [ prepare_strategy ]
+    runs-on: ubuntu-24.04
+    outputs:
+      driving_onnx_sha256: ${{ steps.resolve.outputs.driving_onnx_sha256 }}
+    env:
+      GH_REPO: ${{ github.repository }}
+      HF_REPO: sunnypilot/sunnypilot_models_v1
+      HF_DEFAULTS_PATH: models/defaults/small
+    steps:
+      - name: Resolve ONNX hash and tinygrad ref via API
+        id: resolve
+        run: |
+          REF="${{ github.head_ref || github.ref_name }}"
+
+          DRIVING_HASH=$(gh api "repos/${GH_REPO}/contents/openpilot/selfdrive/modeld/models/driving_supercombo.onnx?ref=${REF}" --jq '.content' | base64 -d | grep '^oid sha256:' | cut -d: -f2)
+          echo "Driving ONNX hash: $DRIVING_HASH"
+          echo "driving_onnx_sha256=$DRIVING_HASH" >> $GITHUB_OUTPUT
+
+          TINYGRAD_REF=$(gh api "repos/${GH_REPO}/contents/tinygrad_repo?ref=${REF}" --jq '.sha')
+          echo "tinygrad ref: $TINYGRAD_REF"
+
+          JSON_URL="https://huggingface.co/datasets/${HF_REPO}/resolve/main/${HF_DEFAULTS_PATH}/default_models.json"
+
+          check_defaults() {
+            DEFAULTS=$(curl -fsSL "$JSON_URL" 2>/dev/null) || return 1
+            TINYGRAD_MATCH=$(echo "$DEFAULTS" | jq -r --arg ref "$TINYGRAD_REF" '.tinygrad_ref == $ref' 2>/dev/null)
+            [ "$TINYGRAD_MATCH" = "true" ] || return 1
+            DRIVING=$(echo "$DEFAULTS" | jq --arg hash "$DRIVING_HASH" '.bundles[] | select(.onnx_sha256 == $hash)' 2>/dev/null)
+            [ -n "$DRIVING" ] && [ "$DRIVING" != "null" ] || return 1
+          }
+
+          if check_defaults; then
+            echo "HF defaults match repo ONNX hash and tinygrad ref"
+            exit 0
+          fi
+
+          echo "No matching model on HF — dispatching build"
+          gh workflow run build-default-models.yaml --ref "$REF" -f target=small
+
+          echo "Polling HF for model availability..."
+          for i in $(seq 1 60); do
+            sleep 30
+            if check_defaults; then
+              echo "Model available on HF after $((i * 30))s"
+              exit 0
+            fi
+            echo "Poll $i/60: not yet available"
+          done
+
+          echo "::error::Small driving model not available on HF after 30 minutes"
+          exit 1
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Cancel run on failure
+        if: failure()
+        run: gh run cancel ${{ github.run_id }}
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+  prepare_dm_model:
+    needs: [ prepare_strategy ]
+    runs-on: ubuntu-24.04
+    outputs:
+      dm_onnx_sha256: ${{ steps.resolve.outputs.dm_onnx_sha256 }}
+    env:
+      GH_REPO: ${{ github.repository }}
+      HF_REPO: sunnypilot/sunnypilot_models_v1
+      HF_DEFAULTS_PATH: models/defaults/dm
+    steps:
+      - name: Resolve ONNX hash and tinygrad ref via API
+        id: resolve
+        run: |
+          REF="${{ github.head_ref || github.ref_name }}"
+
+          DM_HASH=$(gh api "repos/${GH_REPO}/contents/openpilot/selfdrive/modeld/models/dmonitoring_model.onnx?ref=${REF}" --jq '.content' | base64 -d | grep '^oid sha256:' | cut -d: -f2)
+          echo "DM ONNX hash: $DM_HASH"
+          echo "dm_onnx_sha256=$DM_HASH" >> $GITHUB_OUTPUT
+
+          TINYGRAD_REF=$(gh api "repos/${GH_REPO}/contents/tinygrad_repo?ref=${REF}" --jq '.sha')
+          echo "tinygrad ref: $TINYGRAD_REF"
+
+          JSON_URL="https://huggingface.co/datasets/${HF_REPO}/resolve/main/${HF_DEFAULTS_PATH}/default_models.json"
+
+          check_defaults() {
+            DEFAULTS=$(curl -fsSL "$JSON_URL" 2>/dev/null) || return 1
+            TINYGRAD_MATCH=$(echo "$DEFAULTS" | jq -r --arg ref "$TINYGRAD_REF" '.tinygrad_ref == $ref' 2>/dev/null)
+            [ "$TINYGRAD_MATCH" = "true" ] || return 1
+            DM=$(echo "$DEFAULTS" | jq --arg hash "$DM_HASH" '.bundles[] | select(.onnx_sha256 == $hash)' 2>/dev/null)
+            [ -n "$DM" ] && [ "$DM" != "null" ] || return 1
+          }
+
+          if check_defaults; then
+            echo "HF defaults match DM ONNX hash and tinygrad ref"
+            exit 0
+          fi
+
+          echo "No matching DM model on HF — dispatching build"
+          gh workflow run build-default-models.yaml --ref "$REF" -f target=dm
+
+          echo "Polling HF for DM model availability..."
+          for i in $(seq 1 60); do
+            sleep 30
+            if check_defaults; then
+              echo "DM model available on HF after $((i * 30))s"
+              exit 0
+            fi
+            echo "Poll $i/60: not yet available"
+          done
+
+          echo "::error::DM model not available on HF after 30 minutes"
+          exit 1
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Cancel run on failure
+        if: failure()
+        run: gh run cancel ${{ github.run_id }}
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  publish:
    concurrency:
-      # We do a bit of a hack here to avoid canceling the publishing job if a new commit comes in while we're publishing by adding the sha to the group name.
-      # This means that if multiple commits come in while we're publishing, they will be queued up and publish one after the other.
-      # Otherwise, if a job is waiting to be published due to environment wait time, it would be canceled by a new commit and restart the wait time.
      group: ${{ needs.prepare_strategy.outputs.publish_concurrency_group }}
      cancel-in-progress: ${{ needs.prepare_strategy.outputs.cancel_publish_in_progress == 'true' }}
-    if: ${{ (always() && !cancelled() && !failure()) && needs.build.result == 'success' && needs.prepare_strategy.result == 'success' && (!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt')) }}
-    needs: [ build, prepare_strategy ]
+    if: ${{
+      always() && !cancelled() &&
+      needs.build.result == 'success' &&
+      needs.prepare_strategy.result == 'success' &&
+      needs.prepare_small_model.result == 'success' &&
+      needs.prepare_dm_model.result == 'success' &&
+      (!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt')) &&
+      (needs.prepare_strategy.outputs.include_big_model != 'true' || needs.prepare_chestnut.result == 'success')
+      }}
+    needs: [ build, prepare_strategy, prepare_chestnut, prepare_small_model, prepare_dm_model ]
    runs-on: ubuntu-24.04
    environment: ${{ needs.prepare_strategy.outputs.environment }}
    steps:
      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 1

-      - name: Download build artifacts
+      - name: Download prebuilt artifact
        uses: actions/download-artifact@v4
        with:
          name: prebuilt
@@ -228,6 +427,17 @@ jobs:
          mkdir -p ${{ env.OUTPUT_DIR }}
          tar xzf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }}

+      - name: Download model chunks from HF
+        uses: ./.github/workflows/download-hf-model-chunks
+        with:
+          hf_repo: sunnypilot/sunnypilot_models_v1
+          dest_dir: ${{ env.OUTPUT_DIR }}/openpilot/selfdrive/modeld/models
+          models: |
+            [
+              {"hf_path": "models/defaults/small", "onnx_hash": "${{ needs.prepare_small_model.outputs.driving_onnx_sha256 }}", "canonical": "driving_tinygrad.pkl"},
+              {"hf_path": "models/defaults/dm", "onnx_hash": "${{ needs.prepare_dm_model.outputs.dm_onnx_sha256 }}", "canonical": "dmonitoring_model_tinygrad.pkl"}
+            ]
+
      - name: Configure Git
        run: |
          git config --global user.email "github-actions[bot]@users.noreply.github.com"
@@ -255,11 +465,77 @@ jobs:
            git tag -f -a ${TAG} -m "${{ needs.prepare_strategy.outputs.environment }} @ ${{ needs.prepare_strategy.outputs.version }} of build ${{ needs.prepare_strategy.outputs.build }}."
            git push -f origin ${TAG}

+  publish_chestnut:
+    concurrency:
+      group: ${{ needs.prepare_strategy.outputs.publish_concurrency_group }}-chestnut
+      cancel-in-progress: ${{ needs.prepare_strategy.outputs.cancel_publish_in_progress == 'true' }}
+    if: ${{
+      always() && !cancelled() &&
+      needs.build.result == 'success' &&
+      needs.prepare_strategy.result == 'success' &&
+      needs.prepare_small_model.result == 'success' &&
+      needs.prepare_dm_model.result == 'success' &&
+      needs.prepare_chestnut.result == 'success' &&
+      (!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
+      }}
+    needs: [ build, prepare_strategy, prepare_chestnut, prepare_small_model, prepare_dm_model ]
+    runs-on: ubuntu-24.04
+    environment: ${{ needs.prepare_strategy.outputs.environment }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+
+      - name: Download prebuilt artifact
+        uses: actions/download-artifact@v4
+        with:
+          name: prebuilt
+
+      - name: Untar prebuilt
+        run: |
+          mkdir -p ${{ env.OUTPUT_DIR }}
+          tar xzf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }}
+
+      - name: Download model chunks from HF
+        uses: ./.github/workflows/download-hf-model-chunks
+        with:
+          hf_repo: sunnypilot/sunnypilot_models_v1
+          dest_dir: ${{ env.OUTPUT_DIR }}/openpilot/selfdrive/modeld/models
+          models: |
+            [
+              {"hf_path": "models/defaults/small", "onnx_hash": "${{ needs.prepare_small_model.outputs.driving_onnx_sha256 }}", "canonical": "driving_tinygrad.pkl"},
+              {"hf_path": "models/defaults/dm", "onnx_hash": "${{ needs.prepare_dm_model.outputs.dm_onnx_sha256 }}", "canonical": "dmonitoring_model_tinygrad.pkl"},
+              {"hf_path": "models/defaults/big", "onnx_hash": "${{ needs.prepare_chestnut.outputs.onnx_sha256 }}", "canonical": "big_driving_tinygrad.pkl"}
+            ]
+
+      - name: Configure Git
+        run: |
+          git config --global user.email "github-actions[bot]@users.noreply.github.com"
+          git config --global user.name "github-actions[bot]"
+
+      - name: Publish chestnut branch
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          CHESTNUT_BRANCH="${{ needs.prepare_strategy.outputs.new_branch }}-chestnut"
+
+          ${{ env.CI_DIR }}/publish.sh \
+            "${{ github.workspace }}" \
+            "${{ env.OUTPUT_DIR }}" \
+            "$CHESTNUT_BRANCH" \
+            "${{ needs.prepare_strategy.outputs.version }}" \
+            "https://x-access-token:${{github.token}}@github.com/sunnypilot/sunnypilot.git" \
+            "${{ needs.prepare_strategy.outputs.extra_version_identifier }}"
+
  notify:
    needs:
    - prepare_strategy
    - build
    - publish
+    - publish_chestnut
+    - prepare_chestnut
+    - prepare_small_model
+    - prepare_dm_model
    runs-on: ubuntu-24.04
    if: ${{ (always() && !cancelled() && !failure())
      && needs.publish.result == 'success'
@@ -267,6 +543,8 @@ jobs:
      && (fromJSON(vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES_V2)[github.head_ref || github.ref_name] != null) }}
    steps:
      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 1

      - name: Prepare notification message
        id: message
@@ -279,6 +557,7 @@ jobs:
          export commit_short_sha="${commit_short_sha:0:7}"
          export extra_version_identifier="${{ needs.prepare_strategy.outputs.extra_version_identifier || github.run_number }}"
          export PUBLIC_REPO_URL="${{ env.PUBLIC_REPO_URL }}"
+          export chestnut_branch="${{ needs.prepare_chestnut.result == 'success' && format('{0}-chestnut', needs.prepare_strategy.outputs.new_branch) || '' }}"

          MESSAGE=$(cat << 'EOF' | envsubst
          ${{ vars.DISCOURSE_GENERAL_UPDATE_NOTICE }}
@@ -24,7 +24,9 @@ function agnos_init {
    if $AGNOS_PY --verify $MANIFEST; then
      sudo reboot
    fi
-    $DIR/openpilot/common/hardware/comma/updater $AGNOS_PY $MANIFEST
+    while true; do
+      $DIR/openpilot/common/hardware/comma/updater $AGNOS_PY $MANIFEST
+    done
  fi
 }

@@ -59,7 +59,7 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
    {"IsDriverViewEnabled", {CLEAR_ON_MANAGER_START, BOOL}},
    {"IsEngaged", {PERSISTENT, BOOL}},
    {"IsLdwEnabled", {PERSISTENT | BACKUP, BOOL}},
-    {"IsLiveStreaming", {CLEAR_ON_MANAGER_START, BOOL}},
+    {"IsLiveStreaming", {CLEAR_ON_MANAGER_START | CLEAR_ON_IGNITION_ON, BOOL}},
    {"IsMetric", {PERSISTENT | BACKUP, BOOL}},
    {"IsOffroad", {CLEAR_ON_MANAGER_START, BOOL}},
    {"IsRhdDetected", {PERSISTENT, BOOL}},
@@ -196,6 +196,8 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
    // Model Manager params
    {"ModelManager_ActiveBundle", {PERSISTENT, JSON}},
    {"ModelManager_ActiveJson", {CLEAR_ON_MANAGER_START, STRING}},
+    {"ModelManager_PrevBundle", {PERSISTENT, JSON}},
+    {"ModelManager_PrevBundle_USBGPU", {PERSISTENT, JSON}},
    {"ModelManager_ClearCache", {CLEAR_ON_MANAGER_START, BOOL}},
    {"ModelManager_DownloadIndex", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, INT}},
    {"ModelManager_Favs", {PERSISTENT | BACKUP, STRING}},
@@ -16,6 +16,15 @@ MASTER_SP_BRANCHES = ['master']
 RELEASE_BRANCHES = ['release-tizi-staging', 'release-mici-staging', 'release-tizi', 'release-mici', 'nightly']
 TESTED_BRANCHES = RELEASE_BRANCHES + ['devel-staging', 'nightly-dev'] + RELEASE_SP_BRANCHES + TESTED_SP_BRANCHES

+CHESTNUT_BRANCHES = {
+  "staging": "staging-chestnut",
+  "dev": "dev-chestnut",
+  "release-mici": "release-chestnut",
+  "release-tizi": "release-chestnut",
+  "release-mici-staging": "release-chestnut-staging",
+  "release-tizi-staging": "release-chestnut-staging",
+}
+
 SP_BRANCH_MIGRATIONS = {
  ("tici", "staging-c3-new"): "staging-tici",
  ("tici", "dev-c3-new"): "staging-tici",
@@ -73,44 +73,45 @@ compile_modeld_script = [
 model_w, model_h = MEDMODEL_INPUT_SIZE
 frame_skip = ModelConstants.MODEL_RUN_FREQ // ModelConstants.MODEL_CONTEXT_FREQ

-for usbgpu in [False, True] if USBGPU else [False]:
-  target_pkl_path = File(modeld_pkl_path(usbgpu)).abspath
-  # BIG_INTO_SMALL=1 builds the default target from the big model, e.g. to test it without a USB GPU
-  file_prefix, cmd_flags = ('big_', usbgpu_tg_flags) if usbgpu else ('big_' if os.getenv('BIG_INTO_SMALL') else '', tg_flags)
-  driving_onnx_deps = get_existing_chunks(File(f"models/{file_prefix}driving_supercombo.onnx").abspath)
-  camera_res_args = ' '.join(f'{cw}x{ch}' for cw, ch in CAMERA_CONFIGS)
-  # CPU 7 is isolated with isolcpus on AGNOS, so explicitly pin the compiler to it.
-  taskset = 'taskset -c 7 ' if arch == 'comma_arm64' else ''
-  cmd = (f'{cmd_flags} {mac_brew_string} {taskset}python3 {modeld_dir}/compile_modeld.py '
-        f'--model-size {model_w}x{model_h} '
-        f'--camera-resolutions {camera_res_args} '
-        f'--onnx {File(f"models/{file_prefix}driving_supercombo.onnx").abspath} '
-        f'--output {target_pkl_path} --frame-skip {frame_skip}')
-  onnx_sizes_sum = sum(os.path.getsize(f) for f in driving_onnx_deps)
-  chunk_targets = get_chunk_targets(target_pkl_path, estimate_pickle_max_size(onnx_sizes_sum))
-  def do_compile(target, source, env, command=cmd, pkl=target_pkl_path, chunks=chunk_targets):
-    from openpilot.system.hardware.chestnut.flash import link_up
-    # chestnut can enumerate before its PCIe link is up due to varying 12V power behavior across cars
-    for _ in range(10):
-      if link_up():
-        break
-      time.sleep(1)
-    else:
-      print("Chestnut not ready, skipping big model build")
-      return
-    if ret := env.Execute(command):
-      return ret
-    chunk_file(pkl, chunks)
-  def do_chunk(target, source, env, pkl=target_pkl_path, chunks=chunk_targets):
-    chunk_file(pkl, chunks)
-  actions = Action(do_compile, "  [USBGPU] $TARGET") if usbgpu else [cmd, Action(do_chunk, "  [CHUNK] $TARGET")]
-  node = lenv.Command(
-    chunk_targets,
-    tinygrad_files + compile_modeld_script + driving_onnx_deps + [Value(chunk_targets), chunker_file],
-    actions,
-  )
-  if usbgpu:
-    lenv.SideEffect(usbgpu_lock, node)
+if not os.getenv('SKIP_TINYGRAD_COMPILE'):
+  for usbgpu in [False, True] if USBGPU else [False]:
+    target_pkl_path = File(modeld_pkl_path(usbgpu)).abspath
+    # BIG_INTO_SMALL=1 builds the default target from the big model, e.g. to test it without a USB GPU
+    file_prefix, cmd_flags = ('big_', usbgpu_tg_flags) if usbgpu else ('big_' if os.getenv('BIG_INTO_SMALL') else '', tg_flags)
+    driving_onnx_deps = get_existing_chunks(File(f"models/{file_prefix}driving_supercombo.onnx").abspath)
+    camera_res_args = ' '.join(f'{cw}x{ch}' for cw, ch in CAMERA_CONFIGS)
+    # CPU 7 is isolated with isolcpus on AGNOS, so explicitly pin the compiler to it.
+    taskset = 'taskset -c 7 ' if arch == 'comma_arm64' else ''
+    cmd = (f'{cmd_flags} {mac_brew_string} {taskset}python3 {modeld_dir}/compile_modeld.py '
+          f'--model-size {model_w}x{model_h} '
+          f'--camera-resolutions {camera_res_args} '
+          f'--onnx {File(f"models/{file_prefix}driving_supercombo.onnx").abspath} '
+          f'--output {target_pkl_path} --frame-skip {frame_skip}')
+    onnx_sizes_sum = sum(os.path.getsize(f) for f in driving_onnx_deps)
+    chunk_targets = get_chunk_targets(target_pkl_path, estimate_pickle_max_size(onnx_sizes_sum))
+    def do_compile(target, source, env, command=cmd, pkl=target_pkl_path, chunks=chunk_targets):
+      from openpilot.system.hardware.chestnut.flash import link_up
+      # chestnut can enumerate before its PCIe link is up due to varying 12V power behavior across cars
+      for _ in range(10):
+        if link_up():
+          break
+        time.sleep(1)
+      else:
+        print("Chestnut not ready, skipping big model build")
+        return
+      if ret := env.Execute(command):
+        return ret
+      chunk_file(pkl, chunks)
+    def do_chunk(target, source, env, pkl=target_pkl_path, chunks=chunk_targets):
+      chunk_file(pkl, chunks)
+    actions = Action(do_compile, "  [USBGPU] $TARGET") if usbgpu else [cmd, Action(do_chunk, "  [CHUNK] $TARGET")]
+    node = lenv.Command(
+      chunk_targets,
+      tinygrad_files + compile_modeld_script + driving_onnx_deps + [Value(chunk_targets), chunker_file],
+      actions,
+    )
+    if usbgpu:
+      lenv.SideEffect(usbgpu_lock, node)

 # get model metadata
 fn = File(f"models/dmonitoring_model").abspath
@@ -142,4 +143,5 @@ def tg_compile(flags, model_name):
     Action(do_chunk, "  [CHUNK] $TARGET")],
  )

-tg_compile(tg_flags, 'dmonitoring_model')
+if not os.getenv('SKIP_TINYGRAD_COMPILE'):
+  tg_compile(tg_flags, 'dmonitoring_model')
@@ -18,7 +18,7 @@
    "_comment": "Set extra field to the failed reason."
  },
  "Offroad_ChestnutBranch": {
-    "text": "Chestnut detected! Switch to the release-chestnut branch to use chestnut-class models.",
+    "text": "Chestnut detected! Switch to the %1 branch to use chestnut-class models.",
    "severity": 0
  },
  "Offroad_UnregisteredHardware": {
@@ -149,11 +149,15 @@ class BigButton(Widget):
  def set_touch_valid_callback(self, touch_callback: Callable[[], bool]) -> None:
    super().set_touch_valid_callback(lambda: touch_callback() and self._grow_animation_until is None)

-  def _width_hint(self) -> int:
-    # A value moves the title to the top, where it shares space with the icon.
+  def _title_width_hint(self) -> int:
+    # A value moves the title to the top, where it shares space with the icon
    icon_size = self._txt_icon.width if self._txt_icon and self.value else 0
    return int(self._rect.width - self.LABEL_HORIZONTAL_PADDING * 2 - icon_size)

+  def _subtitle_width_hint(self) -> int:
+    # Bottom aligned, so it sits below the icon
+    return int(self._rect.width - self.LABEL_HORIZONTAL_PADDING * 2)
+
  def _get_label_font_size(self):
    if len(self.text) <= 18:
      return 48
@@ -228,14 +232,14 @@ class BigButton(Widget):

    label_color = LABEL_COLOR if self.enabled else rl.Color(255, 255, 255, int(255 * 0.35))
    self._label.set_color(label_color)
-    label_rect = rl.Rectangle(label_x, btn_y + self.LABEL_VERTICAL_PADDING, self._width_hint(),
+    label_rect = rl.Rectangle(label_x, btn_y + self.LABEL_VERTICAL_PADDING, self._title_width_hint(),
                              self._rect.height - self.LABEL_VERTICAL_PADDING * 2)
    self._label.render(label_rect)

    if self.value:
-      label_y = btn_y + self.LABEL_VERTICAL_PADDING + self._label.get_content_height(self._width_hint())
+      label_y = label_rect.y + self._label.get_content_height(int(label_rect.width))
      sub_label_height = btn_y + self._rect.height - self.LABEL_VERTICAL_PADDING - label_y
-      sub_label_rect = rl.Rectangle(label_x, label_y, self._width_hint(), sub_label_height)
+      sub_label_rect = rl.Rectangle(label_x, label_y, self._subtitle_width_hint(), sub_label_height)
      self._sub_label.render(sub_label_rect)

    # ICON -------------------------------------------------------------------
@@ -312,9 +316,6 @@ class BigMultiToggle(BigToggle):

    self.set_value(self._options[0])

-  def _width_hint(self) -> int:
-    return int(self._rect.width - self.LABEL_HORIZONTAL_PADDING * 2 - self._txt_enabled_toggle.width)
-
  def _handle_mouse_release(self, mouse_pos: MousePos):
    super()._handle_mouse_release(mouse_pos)
    cur_idx = self._options.index(self.value)
@@ -363,9 +364,6 @@ class GreyBigButton(BigButton):
  def LABEL_VERTICAL_PADDING(self):
    return BigButton.LABEL_VERTICAL_PADDING if self._label.text else 18

-  def _width_hint(self) -> int:
-    return int(self._rect.width - self.LABEL_HORIZONTAL_PADDING * 2)
-
  def _get_label_font_size(self):
    return 36

@@ -192,7 +192,7 @@ class ModelRenderer(Widget, ChevronMetrics, ModelRendererSP):

    max_idx = self._get_path_length_idx(path_x_array, max_distance)
    self._path.projected_points = self._map_line_to_polygon(
-      self._path.raw_points, 0.9, self._path_offset_z, max_idx, max_distance, allow_invert=False
+      self._path.raw_points, self._get_path_half_width(), self._path_offset_z, max_idx, max_distance, allow_invert=False
    )

    self._update_experimental_gradient()
@@ -292,7 +292,7 @@ class ModelRenderer(Widget, ChevronMetrics, ModelRendererSP):
    allow_throttle = sm['longitudinalPlan'].allowThrottle or not self._longitudinal_control
    self._blend_filter.update(int(allow_throttle))

-    if ui_state.rainbow_path:
+    if ui_state.rainbow_path and self._lateral_active:
      self.rainbow_path.draw_rainbow_path(self._rect, self._path)
      return

@@ -10,7 +10,7 @@ import time
 import pyray as rl

 from openpilot.cereal import custom
-from openpilot.sunnypilot.models.default_model import DEFAULT_MODEL
+from openpilot.sunnypilot.models.default_model import get_default_model
 from openpilot.common.constants import CV
 from openpilot.selfdrive.ui.ui_state import device, ui_state
 from openpilot.system.ui.lib.multilang import tr
@@ -115,8 +115,12 @@ class ModelsLayout(Widget):
  def calculate_cache_size():
    cache_size = 0.0
    if os.path.exists(CUSTOM_MODEL_PATH):
-      cache_size = sum(os.path.getsize(os.path.join(CUSTOM_MODEL_PATH, file)) for file in os.listdir(CUSTOM_MODEL_PATH)) / (1024**2)
-    return cache_size
+      for file in os.listdir(CUSTOM_MODEL_PATH):
+        try:
+          cache_size += os.path.getsize(os.path.join(CUSTOM_MODEL_PATH, file))
+        except OSError:
+          continue
+    return cache_size / (1024**2)

  def _clear_cache(self):
    def _callback(response):
@@ -178,27 +182,14 @@ class ModelsLayout(Widget):
    # circled_slash is authored grey; tinting it again only darkens it
    return {"name": name, "text_color": rl.GRAY, "icon": "icons/circled_slash.png", "icon_color": rl.WHITE}

-  @staticmethod
-  def _show_reset_params_dialog():
-    def _callback(response):
-      if response == DialogResult.CONFIRM:
-        ui_state.params.remove("CalibrationParams")
-        ui_state.params.remove("LiveTorqueParameters")
-    msg = tr("Model download has started in the background. We suggest resetting calibration. Would you like to do that now?")
-    dialog = ConfirmDialog(msg, tr("Reset Calibration"), callback=_callback)
-    gui_app.push_widget(dialog)
-
  def _on_model_selected(self, result):
    if result != DialogResult.CONFIRM:
      return
    selected_ref = self.model_dialog.selection_ref
    if selected_ref == "Default":
      ui_state.params.remove("ModelManager_ActiveBundle")
-      self._show_reset_params_dialog()
    elif selected_bundle := next((bundle for bundle in self.model_manager.availableBundles if bundle.ref == selected_ref), None):
      ui_state.params.put("ModelManager_DownloadIndex", selected_bundle.index)
-      if self.model_manager.activeBundle and selected_bundle.generation != self.model_manager.activeBundle.generation:
-        self._show_reset_params_dialog()
    self.model_dialog = None

  @staticmethod
@@ -211,7 +202,8 @@ class ModelsLayout(Widget):
    for bundle in bundles:
      folders.setdefault(next((ov_ride.value for ov_ride in bundle.overrides if ov_ride.key == "folder"), ""), []).append(bundle)

-    folders_list = [TreeFolder("", [TreeNode("Default", {'display_name': f"{DEFAULT_MODEL} (Default)", 'short_name': "Default"})])]
+    folders_list = [TreeFolder("", [TreeNode("Default", {'display_name': f"{get_default_model()} (Default)",
+                                                         'short_name': "Default"})])]
    for folder, folder_bundles in sorted(folders.items(), key=lambda x: max((bundle.index for bundle in x[1]), default=-1), reverse=True):
      folder_bundles.sort(key=lambda bundle: bundle.index, reverse=True)
      name = folder + (f" - (Updated: {m.group(1)})" if folder_bundles and (m := re.search(r'\(([^)]*)\)[^(]*$', folder_bundles[0].displayName)) else "")
@@ -249,7 +241,8 @@ class ModelsLayout(Widget):
    self._update_lagd_description(live_delay)
    self.model_manager = ui_state.sm["modelManagerSP"]
    self._handle_bundle_download_progress()
-    active_name = self.model_manager.activeBundle.displayName if self.model_manager and self.model_manager.activeBundle.ref else f"{DEFAULT_MODEL} (Default)"
+    default_label = f"{get_default_model()} (Default)"
+    active_name = self.model_manager.activeBundle.displayName if self.model_manager and self.model_manager.activeBundle.ref else default_label
    self.current_model_item.action_item.set_value(active_name)

    if not ui_state.is_offroad():
@@ -7,7 +7,7 @@ See the LICENSE.md file in the root directory for more details.
 import pyray as rl

 from openpilot.cereal import custom
-from openpilot.sunnypilot.models.default_model import DEFAULT_MODEL
+from openpilot.sunnypilot.models.default_model import get_default_model
 from openpilot.selfdrive.ui.mici.widgets.button import BigButton
 from openpilot.selfdrive.ui.sunnypilot.layouts.settings.models import ModelsLayout
 from openpilot.selfdrive.ui.ui_state import ui_state, device
@@ -27,7 +27,7 @@ class CurrentModelInfo(Widget):
    subheader_color = rl.Color(255, 255, 255, int(255 * 0.9 * 0.65))
    max_width = int(self._rect.width - 20)
    self.current_model_header = UnifiedLabel(tr("active model"), 48, max_width=max_width, text_color=header_color, font_weight=FontWeight.DISPLAY)
-    default_text = f"{DEFAULT_MODEL} (Default)".lower()
+    default_text = f"{get_default_model()} (Default)".lower()
    self.current_model_text = UnifiedLabel(default_text, 32, max_width=max_width, text_color=subheader_color, font_weight=FontWeight.ROMAN, scroll=True)

    self.info_header = UnifiedLabel("cache size", 48, max_width=max_width, text_color=header_color, font_weight=FontWeight.DISPLAY)
@@ -95,7 +95,7 @@ class ModelsLayoutMici(NavScroller):

    folders = self._get_grouped_bundles(favorites)
    folder_buttons = []
-    default_btn = BigButton(f"{DEFAULT_MODEL} (Default)".lower())
+    default_btn = BigButton(f"{get_default_model()} (Default)".lower())
    default_btn.set_click_callback(self._select_default)
    folder_buttons.append(default_btn)

@@ -162,7 +162,8 @@ class ModelsLayoutMici(NavScroller):
    self._was_downloading = is_downloading

    self.current_model_info.current_model_header.set_text(tr("active model"))
-    model_text = manager.activeBundle.displayName.lower() if manager.activeBundle.ref else f"{DEFAULT_MODEL} (Default)".lower()
+    default_model_text = f"{get_default_model()} (Default)".lower()
+    model_text = manager.activeBundle.displayName.lower() if manager.activeBundle.ref else default_model_text
    self.current_model_info.current_model_text.set_text(model_text)
    self.current_model_info.info_header.set_text(tr("cache size"))
    self.current_model_info.info_text.set_text(f"{ModelsLayout.calculate_cache_size():.2f} MB")
@@ -191,4 +192,3 @@ class ModelsLayoutMici(NavScroller):
      self.current_model_info.info_header.set_text(tr("progress") + self._download_progress)
      self.current_model_info.info_header._shimmer = True
      self.current_model_info.info_text.set_text(f"{progress/count:.2f}%")
-
@@ -4,11 +4,29 @@ Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
 This file is part of sunnypilot and is licensed under the MIT License.
 See the LICENSE.md file in the root directory for more details.
 """
+from openpilot.common.filter_simple import FirstOrderFilter
+from openpilot.selfdrive.ui.ui_state import ui_state, UIStatus
 from openpilot.selfdrive.ui.sunnypilot.onroad.chevron_metrics import ChevronMetrics
 from openpilot.selfdrive.ui.sunnypilot.onroad.rainbow_path import RainbowPath
+from openpilot.selfdrive.ui.sunnypilot.ui_state import MADSState
+from openpilot.system.ui.lib.application import gui_app


 class ModelRendererSP:
  def __init__(self):
    self.rainbow_path = RainbowPath()
    self.chevron_metrics = ChevronMetrics()
+    self._width_filter = FirstOrderFilter(0.9, 0.1, 1 / gui_app.target_fps)
+
+  @property
+  def _lateral_active(self) -> bool:
+    sm = ui_state.sm
+    if sm.valid["selfdriveStateSP"]:
+      mads = sm["selfdriveStateSP"].mads
+      if mads.available:
+        return mads.enabled and mads.state != MADSState.paused
+    return ui_state.status in (UIStatus.ENGAGED, UIStatus.LAT_ONLY)
+
+  def _get_path_half_width(self) -> float:
+    target = 0.9 if self._lateral_active else 0.40
+    return self._width_filter.update(target)
@@ -7,6 +7,7 @@ See the LICENSE.md file in the root directory for more details.
 """

 import argparse
+import math
 import os
 import tempfile
 import time
@@ -66,14 +67,15 @@ def get_policy_npy_shapes(input_shapes: dict, is_supercombo: bool = False) -> tu
  if desire_key:
    shapes['desire'] = (input_shapes[desire_key][2],)

-  if is_supercombo and 'features_buffer' in input_shapes:
-    fb = input_shapes['features_buffer']
-    shapes['prev_feat'] = (fb[0], fb[2])
-
  for key, shape in input_shapes.items():
    if key not in (desire_key, 'features_buffer') and 'img' not in key:
      shapes[key] = tuple(shape)

+  if is_supercombo and 'features_buffer' in input_shapes:
+    fb = input_shapes['features_buffer']
+    feat_dim = math.prod(fb[2:])
+    shapes['prev_feat'] = (fb[0], feat_dim)
+
  sizes = [int(np.prod(size)) for size in shapes.values()]
  return shapes, sizes

@@ -117,8 +119,9 @@ def generate_queues_and_npy(input_shapes: dict, frame_skip: int, device: str = D
  }

  if features_buffer:
+    feat_dim = math.prod(features_buffer[2:])
    feat_q_len = frame_skip * features_buffer[1] if is_supercombo else frame_skip * (features_buffer[1] - 1) + 1
-    queues['feat_q'] = Tensor(np.zeros((feat_q_len, features_buffer[0], features_buffer[2]),
+    queues['feat_q'] = Tensor(np.zeros((feat_q_len, features_buffer[0], feat_dim),
                       dtype=np.float32), device=device).contiguous().realize()

  queues.update({key: Tensor(value, device='NPY').realize() for key, value in npy_arrays.items() if key in ('tfm', 'big_tfm')})
@@ -183,14 +186,14 @@ def make_run_policy(vision_runner, policy_runners: list, features_slice: slice,
    warped_dev = warped.to(Device.DEFAULT)
    Tensor.realize(packed_npy_inputs_dev, warped_dev)

-    img = shift_and_sample(img_q, warped_dev[0:1], sample_skip_fn).realize()
-    big_img = shift_and_sample(big_img_q, warped_dev[1:2], sample_skip_fn).realize()
+    img = shift_and_sample(img_q, warped_dev[0:1], sample_skip_fn)
+    big_img = shift_and_sample(big_img_q, warped_dev[1:2], sample_skip_fn)

    unpacked_tensors = [tensor.reshape(shape) for tensor, shape in zip(packed_npy_inputs_dev.split(npy_sizes), npy_shapes.values(), strict=True)]
    unpacked_dict = dict(zip(npy_shapes.keys(), unpacked_tensors, strict=True))

    desire_dev = unpacked_dict['desire']
-    desire_buf = shift_and_sample(desire_q, desire_dev.reshape(1, 1, -1), sample_desire_fn).realize()
+    desire_buf = shift_and_sample(desire_q, desire_dev.reshape(1, 1, -1), sample_desire_fn)

    inputs = {desire_key: desire_buf}
    for key, tensor_val in unpacked_dict.items():
@@ -199,7 +202,7 @@ def make_run_policy(vision_runner, policy_runners: list, features_slice: slice,

    if 'prev_feat' in unpacked_dict:
      prev_feat_dev = unpacked_dict['prev_feat']
-      inputs['features_buffer'] = shift_and_sample(feat_q, prev_feat_dev.reshape(1, 1, -1), sample_skip_fn).realize()
+      inputs['features_buffer'] = shift_and_sample(feat_q, prev_feat_dev.reshape(1, 1, -1), sample_skip_fn).reshape(input_shapes['features_buffer'])

    if vision_runner:
      vision_out_cast = next(iter(vision_runner({road_key: img, wide_key: big_img}).values())).cast('float32').realize()
@@ -211,7 +214,7 @@ def make_run_policy(vision_runner, policy_runners: list, features_slice: slice,

    inputs.update({road_key: img, wide_key: big_img})
    if 'features_buffer' not in inputs:
-      inputs['features_buffer'] = sample_skip_fn(feat_q)
+      inputs['features_buffer'] = sample_skip_fn(feat_q).reshape(input_shapes['features_buffer'])

    policy_out = next(iter(policy_runners[0](inputs).values())).cast('float32').realize()
    if 'features_buffer' not in inputs and features_slice is not None:
@@ -272,18 +275,17 @@ def _parse_size(size_str: str) -> tuple[int, int]:
  return int(width), int(height)


-def read_file_chunked_to_shm(path):
+def read_file_chunked_to_disk(path):
  if not path:
    return None
  import atexit
  import shutil
  from openpilot.common.file_chunker import open_file_chunked
-  from openpilot.common.hardware.hw import Paths
-  shm_path = os.path.join(Paths.shm_path(), os.path.basename(path))
-  atexit.register(lambda: os.path.exists(shm_path) and os.remove(shm_path))
-  with open(shm_path, 'wb') as dst, open_file_chunked(path) as src:
-    shutil.copyfileobj(src, dst)
-  return shm_path
+  tmp_path = f'{path}.unchunked'
+  with open(tmp_path, 'wb') as f, open_file_chunked(path) as src:
+    shutil.copyfileobj(src, f)
+  atexit.register(lambda: os.path.exists(tmp_path) and os.remove(tmp_path))
+  return tmp_path


 def _load_policy_runners(args: argparse.Namespace) -> tuple[list, list]:
@@ -327,11 +329,11 @@ if __name__ == "__main__":
  model_w, model_h = args.model_size
  output_data = {}

-  args.vision_onnx = read_file_chunked_to_shm(args.vision_onnx)
-  args.policy_onnx = read_file_chunked_to_shm(args.policy_onnx)
-  args.off_policy_onnx = read_file_chunked_to_shm(args.off_policy_onnx)
-  args.on_policy_onnx = read_file_chunked_to_shm(args.on_policy_onnx)
-  args.supercombo_onnx = read_file_chunked_to_shm(args.supercombo_onnx)
+  args.vision_onnx = read_file_chunked_to_disk(args.vision_onnx)
+  args.policy_onnx = read_file_chunked_to_disk(args.policy_onnx)
+  args.off_policy_onnx = read_file_chunked_to_disk(args.off_policy_onnx)
+  args.on_policy_onnx = read_file_chunked_to_disk(args.on_policy_onnx)
+  args.supercombo_onnx = read_file_chunked_to_disk(args.supercombo_onnx)

  vision_runner = OnnxRunner(args.vision_onnx) if args.vision_onnx else None

@@ -5,10 +5,15 @@ This file is part of sunnypilot and is licensed under the MIT License.
 See the LICENSE.md file in the root directory for more details.
 """

+import os
+import tempfile
+from pathlib import Path
+
 import numpy as np
 from openpilot.common.parameterized import parameterized

-from openpilot.sunnypilot.modeld_v2.compile_modeld import derive_frame_skip, _detect_desire_key
+from openpilot.common.file_chunker import chunk_file, get_chunk_targets
+from openpilot.sunnypilot.modeld_v2.compile_modeld import derive_frame_skip, _detect_desire_key, read_file_chunked_to_disk
 from openpilot.common.test import OpenpilotTestCase


@@ -160,3 +165,115 @@ class TestOutputSlicePreservation(OpenpilotTestCase):
    policy_slices = {'plan': slice(0, 495), 'meta': slice(495, 550)}
    assert set(vision_slices.keys()) & set(policy_slices.keys()) == set(), \
      "vision and policy slices should not overlap in keys"
+
+
+class TestReadFileChunkedToDisk(OpenpilotTestCase):
+  def test_none_passthrough(self):
+    assert read_file_chunked_to_disk(None) is None
+
+  def test_unchunked_source_staged_on_disk(self):
+    with tempfile.TemporaryDirectory() as d:
+      src = Path(d) / "driving_supercombo.onnx"
+      payload = os.urandom(1024)
+      src.write_bytes(payload)
+
+      out = Path(read_file_chunked_to_disk(str(src)))
+
+      assert out.parent == Path(d)
+      assert out.name == "driving_supercombo.onnx.unchunked"
+      assert out.read_bytes() == payload
+
+  def test_chunked_source_reassembled_on_disk(self):
+    with tempfile.TemporaryDirectory() as d:
+      src = Path(d) / "driving_supercombo.onnx"
+      payload = os.urandom(4096)
+      src.write_bytes(payload)
+      chunk_file(str(src), get_chunk_targets(str(src), len(payload)))
+      assert not src.exists()
+
+      out = Path(read_file_chunked_to_disk(str(src)))
+
+      assert out.parent == Path(d)
+      assert out.read_bytes() == payload
+
+
+class Test4DFeaturesBuffer(OpenpilotTestCase):
+  def test_get_policy_npy_shapes_4d(self):
+    from openpilot.sunnypilot.modeld_v2.compile_modeld import get_policy_npy_shapes
+    input_shapes = {
+      'desire_pulse': (1, 25, 8),
+      'features_buffer': (1, 24, 32, 512),  # compare 4d to 3d for regression
+      'traffic_convention': (1, 2),
+      'action_t': (1, 2)
+    }
+    shapes, sizes = get_policy_npy_shapes(input_shapes, is_supercombo=True)
+    assert shapes['prev_feat'] == (1, 16384)
+    assert sizes == [8, 2, 2, 16384]
+
+  def test_get_policy_npy_shapes_3d(self):
+    from openpilot.sunnypilot.modeld_v2.compile_modeld import get_policy_npy_shapes
+    input_shapes = {
+      'desire_pulse': (1, 25, 8),
+      'features_buffer': (1, 24, 512),
+      'traffic_convention': (1, 2),
+      'action_t': (1, 2)
+    }
+    shapes, sizes = get_policy_npy_shapes(input_shapes, is_supercombo=True)
+    assert shapes['prev_feat'] == (1, 512)
+    assert sizes == [8, 2, 2, 512]
+
+
+class TestStockCompileModeldEquivalence(OpenpilotTestCase):
+  def test_get_policy_npy_shapes_matches_stock(self):
+    from openpilot.selfdrive.modeld.compile_modeld import get_policy_npy_shapes as stock_get_policy_npy_shapes
+    from openpilot.sunnypilot.modeld_v2.compile_modeld import get_policy_npy_shapes as sunny_get_policy_npy_shapes
+
+    stock_input_shapes = {
+      'desire_pulse': (1, 25, 8),
+      'features_buffer': (1, 24, 512), # see below comment
+      'traffic_convention': (1, 2),
+      'action_t': (1, 2),
+    }
+
+    stock_shapes, stock_sizes = stock_get_policy_npy_shapes(stock_input_shapes)
+    sunny_shapes, sunny_sizes = sunny_get_policy_npy_shapes(stock_input_shapes, is_supercombo=True)
+
+    assert sunny_shapes == stock_shapes
+    assert sunny_sizes == stock_sizes
+    assert sunny_shapes['prev_feat'] == (1, 512)
+
+  def test_make_input_queues_full_stock_equivalence(self):
+    from openpilot.selfdrive.modeld.compile_modeld import make_input_queues as stock_make_input_queues
+    from openpilot.sunnypilot.modeld_v2.compile_modeld import make_supercombo_input_queues as sunny_make_supercombo_input_queues
+    input_shapes = {
+      'img': (1, 12, 128, 256),
+      'desire_pulse': (1, 25, 8),
+      'features_buffer': (1, 24, 512),  # when https://github.com/commaai/openpilot/pull/38681 merges, update to 1,24,32,512
+      'traffic_convention': (1, 2),
+      'action_t': (1, 2),
+    }
+    frame_skip = 4
+
+    stock_queues, stock_npy = stock_make_input_queues(input_shapes, frame_skip, device='NPY')
+    sunny_queues, sunny_npy = sunny_make_supercombo_input_queues(input_shapes, frame_skip, device='NPY')
+    assert set(sunny_queues.keys()) == set(stock_queues.keys())
+    for key in stock_queues:
+      assert sunny_queues[key].shape == stock_queues[key].shape, \
+        f"Queue shape mismatch for {key}: sunny {sunny_queues[key].shape} != stock {stock_queues[key].shape}"
+    assert set(sunny_npy.keys()) == set(stock_npy.keys())
+    for key in stock_npy:
+      assert sunny_npy[key].shape == stock_npy[key].shape, \
+        f"Numpy array shape mismatch for {key}: sunny {sunny_npy[key].shape} != stock {stock_npy[key].shape}"
+
+  def test_make_warp_queues_stock_equivalence(self):
+    from openpilot.selfdrive.modeld.compile_modeld import make_warp_input_queues as stock_make_warp_queues
+    from openpilot.sunnypilot.modeld_v2.compile_modeld import make_warp_queues as sunny_make_warp_queues
+    stock_vision_shapes = {'img': (1, 12, 128, 256)}  # for now?
+    stock_queues, stock_npy = stock_make_warp_queues(stock_vision_shapes, frame_skip=4, device='NPY')
+    sunny_queues, sunny_npy = sunny_make_warp_queues(device='NPY')
+
+    assert set(sunny_npy.keys()) == set(stock_npy.keys()) == {'tfm', 'big_tfm'}
+    for key in sunny_npy:
+      assert sunny_npy[key].shape == stock_npy[key].shape == (3, 3)
+
+
@@ -3,8 +3,17 @@ import os
 import hashlib

 from openpilot.common.basedir import BASEDIR
+from openpilot.selfdrive.ui.ui_state import ui_state
 from openpilot.sunnypilot import get_file_hash
-from openpilot.sunnypilot.models.model_name import DEFAULT_MODEL
+from openpilot.sunnypilot.models.model_name import DEFAULT_MODEL, DEFAULT_BIG_MODEL
+
+
+def get_default_model() -> str:
+  show_big_model = (ui_state.usbgpu
+                    and (ui_state.usbgpu_active or ui_state.usbgpu_loading or ui_state.is_offroad()))
+
+  return DEFAULT_BIG_MODEL if show_big_model else DEFAULT_MODEL
+

 DEFAULT_MODEL_NAME_PATH = os.path.join(BASEDIR, "openpilot", "sunnypilot", "models", "model_name.py")
 MODEL_HASH_PATH = os.path.join(BASEDIR, "openpilot", "sunnypilot", "models", "tests", "model_hash")
@@ -13,7 +22,6 @@ SUPERCOMBO_ONNX_PATH = os.path.join(BASEDIR, "openpilot", "selfdrive", "modeld",

 def update_model_hash():
  supercombo_hash = get_file_hash(SUPERCOMBO_ONNX_PATH)
-
  combined_hash = hashlib.sha256(supercombo_hash.encode()).hexdigest()

  with open(MODEL_HASH_PATH, "w") as f:
@@ -22,40 +30,28 @@ def update_model_hash():
  print(f"Generated and updated new combined model hash to {MODEL_HASH_PATH}")


-def get_current_default_model_name():
-  print("[GET DEFAULT MODEL NAME]")
-  name = DEFAULT_MODEL
-  print(f'Current default model name: "{name}"')
-
-  return name
-
-
-def update_default_model_name(name: str):
-  print("[CHANGE DEFAULT MODEL NAME]")
+def update_default_model_names(default_model_name: str, default_big_model_name: str):
+  print("[CHANGE DEFAULT MODEL NAMES]")
  with open(DEFAULT_MODEL_NAME_PATH, "w") as f:
-    f.write(f'DEFAULT_MODEL = "{name}"\n')
-  print(f'New default model name: "{name}"')
+    f.write(f'DEFAULT_MODEL = "{default_model_name}"\n')
+    f.write(f'DEFAULT_BIG_MODEL = "{default_big_model_name}"\n')
+
+  print(f'New default small model name: "{default_model_name}"')
+  print(f'New default big model name: "{default_big_model_name}"')
  print("[DONE]")


 if __name__ == "__main__":
-  parser = argparse.ArgumentParser(description="Update default model name and hash")
-  parser.add_argument("--new_name", type=str, help="New default model name")
+  parser = argparse.ArgumentParser(description="Update default model names and hash")
+  parser.add_argument("--new_small_model_name", type=str, help="New default small model name")
+  parser.add_argument("--new_big_model_name", type=str, help="New default big model name")
  args = parser.parse_args()

-  if not args.new_name:
-    print("Warning: No new default model name provided. Use --new_name to specify")
-    print("Default model name and hash will not be updated! (aborted)")
-    exit(0)
+  if args.new_small_model_name is None and args.new_big_model_name is None:
+    new_name = input(f'Enter new default small model name (current: "{DEFAULT_MODEL}", leave empty to keep): ').strip()
+    new_big_model_name = input(f'Enter new default big model name (current: "{DEFAULT_BIG_MODEL}", leave empty to keep): ').strip()
+  else:
+    new_name, new_big_model_name = args.new_small_model_name, args.new_big_model_name

-  current_name = get_current_default_model_name()
-  new_name = args.new_name
-  if current_name == new_name:
-    print(f'Proposed default model name: "{new_name}"')
-    confirm = input("Proposed default model name is the same as the current default model name. Confirm? (y/n): ").upper().strip()
-    if confirm != "Y":
-      print("Default model name and hash will not be updated! (aborted)")
-      exit(0)
-
-  update_default_model_name(new_name)
+  update_default_model_names(new_name or DEFAULT_MODEL, new_big_model_name or DEFAULT_BIG_MODEL)
  update_model_hash()
@@ -13,8 +13,6 @@ from openpilot.common.params import Params
 from openpilot.common.swaglog import cloudlog
 from openpilot.common.hardware.hw import Paths
 from openpilot.sunnypilot.models.helpers import is_bundle_version_compatible
-from openpilot.selfdrive.modeld.helpers import usbgpu_present
-
 from openpilot.cereal import custom


@@ -140,8 +138,8 @@ class ModelCache:

 class ModelFetcher:
  """Handles fetching and caching of model data from remote source"""
-  MODEL_URL = "https://raw.githubusercontent.com/sunnypilot/sunnypilot-models/refs/heads/gh-pages/docs/driving_models_v20.json"
-  MODEL_URL_USBGPU = "https://raw.githubusercontent.com/sunnypilot/sunnypilot-models/refs/heads/gh-pages/docs/driving_models_usbgpu_v20.json"
+  MODEL_URL = "https://raw.githubusercontent.com/sunnypilot/sunnypilot-models/refs/heads/gh-pages/docs/driving_models_v21.json"
+  MODEL_URL_USBGPU = "https://raw.githubusercontent.com/sunnypilot/sunnypilot-models/refs/heads/gh-pages/docs/driving_models_usbgpu_v22.json"

  def __init__(self, params: Params):
    self.params = params
@@ -149,11 +147,10 @@ class ModelFetcher:
    self._is_usbgpu: bool | None = None
    self.model_cache = ModelCache(params)
    self.model_url = self.MODEL_URL
-    self._update_model_source()

-  def _update_model_source(self) -> None:
-    """Updates what json to use based on usbgpu availability"""
-    is_usbgpu = usbgpu_present()
+  def _update_model_source(self, chestnut_present: bool) -> None:
+    """Updates what json to use based on chestnut hardware presence via deviceState"""
+    is_usbgpu = chestnut_present
    if is_usbgpu != self._is_usbgpu:
      self._is_usbgpu = is_usbgpu
      self.model_cache = ModelCache(self.params, suffix="_USBGPU" if is_usbgpu else "")
@@ -191,9 +188,9 @@ class ModelFetcher:

    return None

-  def get_available_bundles(self) -> list[custom.ModelManagerSP.ModelBundle]:
+  def get_available_bundles(self, chestnut_present: bool = False) -> list[custom.ModelManagerSP.ModelBundle]:
    """Gets the list of available models, with smart cache handling"""
-    self._update_model_source()
+    self._update_model_source(chestnut_present)
    cached_data, is_expired = self.model_cache.get()

    if cached_data and not is_expired:
@@ -210,10 +207,12 @@ class ModelFetcher:
    cloudlog.warning("Failed to fetch fresh data. Using expired cache as fallback")
    return self.model_parser.parse_models(cached_data)

+
 if __name__ == "__main__":
+  from openpilot.selfdrive.modeld.helpers import usbgpu_present
  params = Params()
  model_fetcher = ModelFetcher(params)
-  bundles = model_fetcher.get_available_bundles()
+  bundles = model_fetcher.get_available_bundles(chestnut_present=usbgpu_present())
  for bundle in bundles:
    for model in bundle.models:
      model_overrides = {override.key: override.value for override in bundle.overrides}
@@ -18,12 +18,11 @@ from openpilot.sunnypilot.models.constants import Meta, MetaSimPose, MetaTombRai
 from openpilot.common.hardware.hw import Paths

 # SET ME TO THE EXACT JSON VERSION WE SET IN SUNNYPILOT_MODELS REPO
-REQUIRED_JSON_VERSION = 17
+REQUIRED_JSON_VERSION = 18

 CUSTOM_MODEL_PATH = Paths.model_root()
 METADATA_PATH = Path(__file__).parent / '../models/supercombo_metadata.pkl'
 ModelManager = custom.ModelManagerSP
-_LAST_VALIDATED_RAW = None


 def _compute_hash(file_path: str) -> str | None:
@@ -86,11 +85,11 @@ def _bundle_needs_reset(active_bundle: custom.ModelManagerSP.ModelBundle, availa
  if available_bundles is not None:
    matching_bundle = None
    for bundle in available_bundles:
-      if getattr(active_bundle, 'ref', None) and getattr(bundle, 'ref', None):
+      if active_bundle.ref and bundle.ref:
        if active_bundle.ref == bundle.ref:
          matching_bundle = bundle
          break
-      elif getattr(active_bundle, 'internalName', None) == getattr(bundle, 'internalName', None):
+      elif active_bundle.internalName == bundle.internalName:
        matching_bundle = bundle
        break

@@ -98,36 +97,42 @@ def _bundle_needs_reset(active_bundle: custom.ModelManagerSP.ModelBundle, availa
      return True
    if active_bundle.minimumSelectorVersion != matching_bundle.minimumSelectorVersion:
      return True
-
-    active_runner = getattr(active_bundle, 'runner', None)
-    matching_runner = getattr(matching_bundle, 'runner', None)
-    if active_runner is not None and matching_runner is not None:
-      if getattr(active_runner, 'raw', active_runner) != getattr(matching_runner, 'raw', matching_runner):
-        return True
+    if active_bundle.runner.raw != matching_bundle.runner.raw:
+      return True
    if set(_bundle_artifacts(active_bundle)) != set(_bundle_artifacts(matching_bundle)):
      return True

-  return not _bundle_is_valid_locally(active_bundle)
+  # missing files trigger re-download, not selection reset
+  return False


-def validate_active_bundle(params: Params, available_bundles: list[custom.ModelManagerSP.ModelBundle] | None = None) -> None:
-  global _LAST_VALIDATED_RAW
+def _prev_bundle_key(is_usbgpu: bool) -> str:
+  return "ModelManager_PrevBundle_USBGPU" if is_usbgpu else "ModelManager_PrevBundle"

+
+def validate_active_bundle(params: Params, available_bundles: list[custom.ModelManagerSP.ModelBundle] | None = None,
+                           is_usbgpu: bool = False) -> None:
  raw_bundle = params.get("ModelManager_ActiveBundle")
  if not raw_bundle:
-    return
-
-  if raw_bundle == _LAST_VALIDATED_RAW:
+    prev = params.get(_prev_bundle_key(is_usbgpu))
+    if prev and (prev_bundle := get_active_bundle(params, raw_bundle_dict=prev)) is not None:
+      if not _bundle_needs_reset(prev_bundle, available_bundles):
+        params.put("ModelManager_ActiveBundle", prev, block=True)
    return

  active_bundle = get_active_bundle(params, raw_bundle_dict=raw_bundle)
  if active_bundle is None or _bundle_needs_reset(active_bundle, available_bundles):
    cloudlog.warning("Active model bundle invalid; resetting to default")
+    params.put(_prev_bundle_key(not is_usbgpu), raw_bundle, block=True)
+
+    prev = params.get(_prev_bundle_key(is_usbgpu))
+    if prev and (prev_bundle := get_active_bundle(params, raw_bundle_dict=prev)) is not None:
+      if not _bundle_needs_reset(prev_bundle, available_bundles):
+        params.put("ModelManager_ActiveBundle", prev, block=True)
+        return
+
    params.remove("ModelManager_ActiveBundle")
    params.put("ModelRunnerTypeCache", int(custom.ModelManagerSP.Runner.stock), block=True)
-    _LAST_VALIDATED_RAW = None
-  else:
-    _LAST_VALIDATED_RAW = raw_bundle


 def get_active_bundle(params: Params | None = None, raw_bundle_dict: dict | bytes | None = None) -> "custom.ModelManagerSP.ModelBundle | None":
@@ -30,6 +30,7 @@ class ModelManagerSP:
    self.params = Params()
    self.model_fetcher = ModelFetcher(self.params)
    self.pm = messaging.PubMaster(["modelManagerSP"])
+    self.sm = messaging.SubMaster(["deviceState"])
    self.available_models: list[custom.ModelManagerSP.ModelBundle] = []
    self.selected_bundle: custom.ModelManagerSP.ModelBundle = None
    self.active_bundle: custom.ModelManagerSP.ModelBundle = get_active_bundle(self.params)
@@ -143,13 +144,17 @@ class ModelManagerSP:
      is_cached = False
      if len(artifact.chunks) > 0:
        from openpilot.common.file_chunker import get_chunk_name
+        num_chunks = len(artifact.chunks)
        chunks_valid = True
        for i, chunk in enumerate(artifact.chunks):
-          chunk_path = get_chunk_name(full_path, i, len(artifact.chunks))
+          chunk_path = get_chunk_name(full_path, i, num_chunks)
          if not await verify_file(chunk_path, chunk.sha256):
            chunks_valid = False
            break
-        if chunks_valid and len(artifact.chunks) > 0:
+          artifact.downloadProgress.progress = ((i + 1) / num_chunks) * 100
+          self._sync_artifact_progress(artifact)
+          self._report_status()
+        if chunks_valid and num_chunks > 0:
          is_cached = True
      else:
        if await verify_file(full_path, expected_hash):
@@ -216,6 +221,9 @@ class ModelManagerSP:
    """Downloads all models in a bundle"""
    self.selected_bundle = model_bundle
    self.selected_bundle.status = custom.ModelManagerSP.DownloadStatus.downloading
+    for model in self.selected_bundle.models:
+      model.artifact.downloadProgress.status = custom.ModelManagerSP.DownloadStatus.downloading
+    self._report_status()
    os.makedirs(destination_path, exist_ok=True)

    try:
@@ -249,18 +257,27 @@ class ModelManagerSP:
    """Main entry point for downloading a model bundle"""
    asyncio.run(self._download_bundle(model_bundle, destination_path))

+  BOOT_SETTLE_TICKS = 10  # seconds at 1 Hz before validating active bundle
+
  def main_thread(self) -> None:
    """Main thread for model management"""
    rk = Ratekeeper(1, print_delay_threshold=None)
+    boot_ticks = 0

    while True:
      try:
-        self.available_models = self.model_fetcher.get_available_bundles()
-        validate_active_bundle(self.params, self.available_models)
+        self.sm.update(0)
+        chestnut_present = self.sm['deviceState'].chestnutPresent
+        self.available_models = self.model_fetcher.get_available_bundles(chestnut_present)
+        if boot_ticks >= self.BOOT_SETTLE_TICKS:
+          validate_active_bundle(self.params, self.available_models, is_usbgpu=chestnut_present)
+        boot_ticks = min(boot_ticks + 1, self.BOOT_SETTLE_TICKS)
        self.active_bundle = get_active_bundle(self.params)

        if (index_to_download := self.params.get("ModelManager_DownloadIndex")) is not None:
-          if model_to_download := next((model for model in self.available_models if model.index == index_to_download), None):
+          if self.active_bundle and self.active_bundle.index == index_to_download:
+            self.params.remove("ModelManager_DownloadIndex")
+          elif model_to_download := next((model for model in self.available_models if model.index == index_to_download), None):
            try:
              self.download(model_to_download, Paths.model_root())
            except Exception as e:
@@ -1 +1,2 @@
 DEFAULT_MODEL = "CD210"
+DEFAULT_BIG_MODEL = "Lebowski"
@@ -20,4 +20,4 @@ class TestDefaultModel(OpenpilotTestCase):
    with open(MODEL_HASH_PATH) as f:
      current_hash = f.read().strip()

-    assert combined_hash == current_hash, "Run sunnypilot/models/default_model.py to update the default model name and hash"
+    assert combined_hash == current_hash, "Run openpilot/sunnypilot/models/default_model.py to update the default model name and hash"
@@ -83,9 +83,14 @@ class TestLocationdProc(OpenpilotTestCase):
      self.pm.send(msg.which(), msg)
      if msg.which() == "cameraOdometry":
        self.pm.wait_for_readers_to_update(msg.which(), timeout=1, dt=0.005)
-    time.sleep(1)  # wait for async params write
+    for _ in range(50):
+      val = self.params.get('LastGPSPositionLLK')
+      if val is not None:
+        break
+      time.sleep(0.1)

-    lastGPS = json.loads(self.params.get('LastGPSPositionLLK'))
+    self.assertIsNotNone(val, "LastGPSPositionLLK not written within 5s")
+    lastGPS = json.loads(val)
    self.assertAlmostEqual(lastGPS['latitude'], self.lat, delta=0.001)
    self.assertAlmostEqual(lastGPS['longitude'], self.lon, delta=0.001)
    self.assertAlmostEqual(lastGPS['altitude'], self.alt, delta=0.001)
@@ -28,7 +28,7 @@ from websocket import (ABNF, WebSocket, WebSocketException, WebSocketTimeoutExce
                       create_connection, WebSocketConnectionClosedException)

 import openpilot.cereal.messaging as messaging
-from openpilot.sunnypilot.models.default_model import DEFAULT_MODEL
+from openpilot.sunnypilot.models.model_name import DEFAULT_MODEL, DEFAULT_BIG_MODEL
 from openpilot.sunnypilot.selfdrive.car.sync_sunnylink_params import update_car_list_param
 from openpilot.sunnypilot.sunnylink.api import SunnylinkApi
 from openpilot.sunnypilot.sunnylink.utils import sunnylink_need_register, sunnylink_ready, get_param_as_byte, save_param_from_base64_encoded_string
@@ -182,6 +182,8 @@ def getParamsMetadata() -> str:
    schema["capabilities"] = generate_capabilities()
    schema["capability_labels"] = CAPABILITY_LABELS
    schema["default_model"] = DEFAULT_MODEL
+    schema["default_big_model"] = DEFAULT_BIG_MODEL
+    schema["usbgpu_active"] = params.get_bool("UsbGpuActive")
    raw = json.dumps(schema, separators=(",", ":")).encode("utf-8")
    return base64.b64encode(gzip.compress(raw)).decode("utf-8")
  except Exception:
@@ -828,20 +828,22 @@ def startStream(sdp: str, enabled: bool) -> dict:
  bridge_services_in = []

  # stale car params case taken care of by webrtcd being shut off on ignition
-  cp_bytes = Params().get("CarParamsPersistent")
+  cp_bytes = params.get("CarParamsPersistent")
  if cp_bytes is not None:
    with car.CarParams.from_bytes(cp_bytes) as CP:
      if CP.notCar:
        bridge_services_in.append("testJoystick")
-  else:
-    raise Exception("failed to get CarParamsPersistent")

  if params.get_bool("IsOffroad"):
    # manager owns camerad/stream_encoderd/webrtcd; flip the param and let it bring them up.
    # webrtcd clears IsLiveStreaming when the session ends
    params.put_bool("IsLiveStreaming", True)
    # wait for webrtcd end points to wake up
-    wait_for_webrtcd()
+    try:
+      wait_for_webrtcd()
+    except TimeoutError:
+      cloudlog.event("athena.startStream.webrtcd_offroad_start_timeout", error=True)
+      raise

  return post_stream_request(StreamRequestBody(sdp, ["wideRoad"], enabled, bridge_services_in, ["carState", "deviceState"]))

@@ -27,7 +27,7 @@ from openpilot.common.swaglog import cloudlog
 from openpilot.sunnypilot.system.statsd import statlog
 from openpilot.system.hardware.power_monitoring import PowerMonitoring
 from openpilot.system.hardware.fan_controller import FanController
-from openpilot.common.version import terms_version, training_version, get_build_metadata, terms_version_sp
+from openpilot.common.version import terms_version, training_version, get_build_metadata, terms_version_sp, CHESTNUT_BRANCHES


 ThermalStatus = log.DeviceState.ThermalStatus
@@ -301,7 +301,11 @@ def hardware_thread(end_event, hw_queue) -> None:

    set_usb_state(msg.deviceState, last_hw_state.usb_state)
    chestnut.update(started_ts is None, last_hw_state.usb_state)
-    set_offroad_alert_if_changed("Offroad_ChestnutBranch", msg.deviceState.chestnutPresent and not big_model_available)
+    current_channel = get_build_metadata().channel
+    chestnut_target = CHESTNUT_BRANCHES.get(current_channel)
+    chestnut_needs_switch = msg.deviceState.chestnutPresent and not big_model_available and chestnut_target is not None
+    set_offroad_alert_if_changed("Offroad_ChestnutBranch", chestnut_needs_switch,
+                                 extra_text=chestnut_target if chestnut_needs_switch else None)

    # this subset is only used for offroad
    temp_sources = [
@@ -1,40 +0,0 @@
-#!/usr/bin/env bash
-
-# Define the service name
-SERVICE_NAME="actions.runner.sunnypilot.$(uname -n)"
-
-# Function to control the service
-control_service() {
-    local action=$1  # Store the function argument in a local variable
-    sudo systemctl $action ${SERVICE_NAME}
-}
-
-service_exists_and_is_loaded() {
-    sudo systemctl status ${SERVICE_NAME} &>/dev/null
-    if [[ $? -ne 4 ]]; then
-        return 0  # Service is known to systemd (i.e., loaded)
-    else
-        return 1  # Service is unknown to systemd (i.e., not loaded)
-    fi
-}
-
-# Check for required argument
-if [[ -z $1 ]] || { [[ $1 != "start" ]] && [[ $1 != "stop" ]]; }; then
-    echo "Usage: $0 {start|stop}"
-    exit 1
-fi
-
-# Store the script argument in a descriptive variable
-ACTION=$1
-
-# Trap EXIT signal (Ctrl+C) and stop the service
-trap 'control_service stop ; exit' SIGINT SIGKILL EXIT
-
-# Enter the main loop
-while true; do
-    # Check if the service is actually present on the system
-    if service_exists_and_is_loaded; then
-        control_service $ACTION  # Call the function with the specified action
-    fi
-    sleep 1  # Pause before the next iteration
-done
@@ -68,10 +68,6 @@ def only_offroad(started: bool, params: Params, CP: car.CarParams) -> bool:
 def livestream(started: bool, params: Params, CP: car.CarParams) -> bool:
  return params.get_bool("IsLiveStreaming")

-def use_github_runner(started, params, CP: car.CarParams) -> bool:
-  return not PC and params.get_bool("EnableGithubRunner") and (
-    not params.get_bool("NetworkMetered") and not params.get_bool("GithubRunnerSufficientVoltage"))
-
 def use_copyparty(started, params, CP: car.CarParams) -> bool:
  return bool(params.get_bool("EnableCopyparty"))

@@ -110,15 +106,12 @@ def or_(*fns):
 def and_(*fns):
  return lambda *args: operator.and_(*(fn(*args) for fn in fns))

-def not_(*fns):
-  return lambda *args: operator.not_(*(fn(*args) for fn in fns))
-
 procs = [
  DaemonProcess("manage_athenad", "openpilot.system.athena.manage_athenad", "AthenadPid"),

  NativeProcess("loggerd", "openpilot/system/loggerd", ["./loggerd"], logging),
  NativeProcess("encoderd", "openpilot/system/loggerd", ["./encoderd"], only_onroad),
-  NativeProcess("stream_encoderd", "openpilot/system/loggerd", ["./encoderd", "--stream"], or_(and_(livestream, not_(iscar)), notcar)),
+  NativeProcess("stream_encoderd", "openpilot/system/loggerd", ["./encoderd", "--stream"], or_(livestream, notcar)),
  PythonProcess("logmessaged", "openpilot.system.logmessaged", always_run),

  NativeProcess("camerad", "openpilot/system/camerad", ["./camerad"], or_(driverview, livestream), enabled=not WEBCAM),
@@ -163,7 +156,7 @@ procs = [

  # debug procs
  NativeProcess("bridge", "openpilot/cereal/messaging", ["./bridge"], notcar),
-  PythonProcess("webrtcd", "openpilot.system.webrtc.webrtcd", or_(and_(livestream, not_(iscar)), notcar)),
+  PythonProcess("webrtcd", "openpilot.system.webrtc.webrtcd", or_(livestream, notcar)),
  PythonProcess("joystick", "openpilot.tools.joystick.joystick_control", and_(joystick, iscar)),

  # sunnylink <3
@@ -189,10 +182,6 @@ procs += [
  NativeProcess("locationd_llk", "openpilot/sunnypilot/selfdrive/locationd", ["./locationd"], only_onroad),
 ]

-if os.path.exists("./github_runner.sh"):
-  procs += [NativeProcess("github_runner_start", "openpilot/system/manager",
-                          ["./github_runner.sh", "start"], and_(only_offroad, use_github_runner), sigkill=False)]
-
 if os.path.exists("../../sunnypilot/sunnylink/uploader.py"):
  procs += [PythonProcess("sunnylink_uploader", "openpilot.sunnypilot.sunnylink.uploader", use_sunnylink_uploader_shim)]

@@ -23,9 +23,9 @@ def post_stream_request(body: StreamRequestBody) -> dict:
    ret["time"] = (t_end - t_start) * 1000
    return ret
  except requests.ConnectTimeout as e:
-    raise Exception("webrtc took too long to respond.") from e
+    raise Exception("device took too long to respond.") from e
  except requests.ConnectionError as e:
-    raise Exception("webrtc server on device is not running.") from e
+    raise Exception("turn car ignition off to use livestreaming.") from e


 def wait_for_webrtcd(max_retries: float = 10) -> None:
@@ -37,4 +37,4 @@ def wait_for_webrtcd(max_retries: float = 10) -> None:
    except requests.ConnectionError:
      attempts += 1
      time.sleep(0.5)
-  raise TimeoutError("webrtcd did not initialize in time.")
+  raise TimeoutError("livestreaming service did not initialize in time.")
@@ -21,10 +21,16 @@ from typing import Any
 from openpilot.system.webrtc.helpers import StreamRequestBody
 from openpilot.system.webrtc.schema import generate_field
 from openpilot.common.params import Params
+from openpilot.common.swaglog import cloudlog
 from openpilot.cereal import messaging, log

 SESSION_TIMEOUT_SECONDS = 300

+
+# ice candidate parser for logging
+def _ice_candidates(sdp: str) -> list[str]:
+  return [line.removeprefix("a=") for line in sdp.splitlines() if line.startswith("a=candidate:")]
+
 # socket trick: route lookup for 8.8.8.8 (nothing is sent or actually connected to)
 # return the source interfaces IP which is the default interface of the device
 def _default_route_ip() -> str | None:
@@ -253,7 +259,7 @@ class StreamSession:
    self._cleanup_lock = asyncio.Lock()
    self._cleanup_done = False
    self.logger = logging.getLogger("webrtcd")
-    self.logger.info(
+    cloudlog.warning(
      "New stream session (%s), video cameras %s, video enabled %s, incoming services %s, outgoing services %s",
      self.identifier, [t.id for t in self.video_tracks], body.enabled, body.bridge_services_in, body.bridge_services_out,
    )
@@ -329,9 +335,12 @@ class StreamSession:
  async def run(self):
    try:
      self.params.put("LivestreamRequestKeyframe", True)
+
+      # avoid datachannel race by adding messange_handler immediately
+      self.stream.set_message_handler(self.message_handler)
+
      await asyncio.wait_for(self.stream.wait_for_connection(), timeout=15)
      if self.stream.has_messaging_channel():
-        self.stream.set_message_handler(self.message_handler)
        if self.incoming_bridge is not None:
          await self.shared_pub_master.add_services_if_needed(self.incoming_bridge_services)
        if self.outgoing_bridge is not None:
@@ -341,14 +350,18 @@ class StreamSession:
      if self.bitrate_controller is not None:
        self.bitrate_controller.start()

-      self.logger.info("Stream session (%s) connected", self.identifier)
+      with cloudlog.ctx(session_id=self.identifier):
+        cloudlog.warning("webrtcd.session.connected")
      if self.is_body:
        await self.run_body_session()
      else:
        await self.run_normal_session()
-      self.logger.info("Stream session (%s) ended", self.identifier)
+      with cloudlog.ctx(session_id=self.identifier):
+        cloudlog.warning("webrtcd.session.ended")
    except Exception:
      self.logger.exception("Stream session failure")
+      with cloudlog.ctx(session_id=self.identifier):
+        cloudlog.exception("webrtcd.session.exception")
    finally:
      await self.post_run_cleanup()

@@ -422,15 +435,25 @@ async def handle_get_stream(state: ServerState, raw_body: bytes, content_type: s
    stream_dict[session.identifier] = session
    try:
      answer = await asyncio.wait_for(session.get_answer(), timeout=30)
+      cloudlog.event(
+        "webrtcd.session.ice_candidates",
+        session_id=session.identifier,
+        offer_candidates=_ice_candidates(body.sdp),
+        answer_candidates=_ice_candidates(answer.sdp),
+      )
    except TimeoutError:
      await session.stop()
      stream_dict.pop(session.identifier, None)
      logging.getLogger("webrtcd").exception("Timed out creating stream answer")
+      with cloudlog.ctx(session_id=session.identifier):
+        cloudlog.warning("webrtcd.session.answer_timeout")
      raise
    except Exception:
      await session.stop()
      stream_dict.pop(session.identifier, None)
      logging.getLogger("webrtcd").exception("Failed to create stream answer")
+      with cloudlog.ctx(session_id=session.identifier):
+        cloudlog.exception("webrtcd.session.answer_exception")
      raise
    session.start()

@@ -1,260 +0,0 @@
-#!/usr/bin/env bash
-set -e
-
-# Default values
-DEFAULT_REPO_URL="https://github.com/sunnypilot"
-START_AT_BOOT=false
-RESTORE_MODE=false
-RUNNER_VERSION="2.325.0"
-
-# Parse command line arguments
-while [[ $# -gt 0 ]]; do
-    case $1 in
-        --start-at-boot)
-            START_AT_BOOT=true
-            shift
-            ;;
-        --token)
-            GITHUB_TOKEN="$2"
-            shift 2
-            ;;
-        --repo)
-            REPO_URL="$2"
-            shift 2
-            ;;
-        --restore)
-            RESTORE_MODE=true
-            shift
-            ;;
-        *)
-            if [ -z "$GITHUB_TOKEN" ]; then
-                GITHUB_TOKEN="$1"
-            elif [ -z "$REPO_URL" ]; then
-                REPO_URL="$1"
-            fi
-            shift
-            ;;
-    esac
-done
-
-# Determine BASE_DIR based on mount point
-if mountpoint -q /data/media; then
-    BASE_DIR="/data/media/0/github"
-else
-    BASE_DIR="/data/github"
-fi
-
-# Constants
-RUNNER_USER="github-runner"
-USER_GROUPS="comma,gpu,gpio,sudo"
-RUNNER_DIR="${BASE_DIR}/runner"
-BUILDS_DIR="${BASE_DIR}/builds"
-LOGS_DIR="${BASE_DIR}/logs"
-CACHE_DIR="${BASE_DIR}/cache"
-OPENPILOT_DIR="${BASE_DIR}/openpilot"
-
-# Basic utility functions (no dependencies)
-remount_rw() {
-    sudo mount -o remount,rw /
-}
-
-remount_ro() {
-    sync || true  # Try to sync but continue even if it fails
-    sudo mount -o remount,ro /  # Always try to remount as read-only
-}
-
-# Always ensure we try to remount as read-only on exit
-trap remount_ro EXIT
-
-setup_runner_user() {
-    sudo useradd --comment 'GitHub Runner' --create-home --home-dir ${BASE_DIR} ${RUNNER_USER} --shell /bin/bash -G ${USER_GROUPS} || sudo usermod -aG ${USER_GROUPS} ${RUNNER_USER}
-}
-
-create_sudoers_entry() {
-    sudo grep -qxF "${RUNNER_USER} ALL=(ALL) NOPASSWD: ALL" /etc/sudoers || echo "${RUNNER_USER} ALL=(ALL) NOPASSWD: ALL" | sudo tee -a /etc/sudoers
-}
-
-set_directory_permissions() {
-    sudo chown -R ${RUNNER_USER}:comma "$BASE_DIR"
-    sudo chmod -R g+rwx "$BASE_DIR"
-    sudo find "$BASE_DIR" -type d -exec chmod g+s {} +
-}
-
-setup_directories() {
-    echo "Creating necessary directories..."
-    sudo mkdir -p "$RUNNER_DIR" "$BUILDS_DIR" "$LOGS_DIR" "$CACHE_DIR" "$OPENPILOT_DIR"
-    mkdir -p "/data/openpilot"
-    sudo chown -R comma:comma "/data/openpilot"
-    sync
-}
-
-wipe_bash_logout() {
-  export BASE_DIR
-  sudo -u ${RUNNER_USER} bash -c "touch ${BASE_DIR}/.bash_logout"
-  sudo -u ${RUNNER_USER} bash -c "truncate -s 0 '${BASE_DIR}/.bash_logout'"
-}
-
-# System configuration functions (depends on basic utility functions)
-setup_system_configs() {
-    echo "Setting up system configurations..."
-    remount_rw
-    setup_runner_user
-    create_sudoers_entry
-    remount_ro
-    set_directory_permissions
-    wipe_bash_logout
-}
-
-# Runner setup functions
-install_runner() {
-    echo "Downloading and setting up runner..."
-    cd "$RUNNER_DIR"
-    curl -o actions-runner-linux-arm64-${RUNNER_VERSION}.tar.gz -L https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-arm64-${RUNNER_VERSION}.tar.gz
-    sudo -u ${RUNNER_USER} tar -xzf ./actions-runner-linux-arm64-${RUNNER_VERSION}.tar.gz
-    sudo rm ./actions-runner-linux-arm64-${RUNNER_VERSION}.tar.gz
-    sudo chmod +x ./config.sh
-}
-
-configure_runner() {
-    remount_rw
-    echo "Configuring runner..."
-    cd "$RUNNER_DIR"
-    sudo -u ${RUNNER_USER} ./config.sh --url "$REPO_URL" --token "$GITHUB_TOKEN" --name $(hostname) --runnergroup "tici-tizi" --labels "tici" --work "$BUILDS_DIR" --unattended
-    remount_ro
-}
-
-create_service_template() {
-    echo "Creating service template..."
-    cat <<EOL > "$RUNNER_DIR/bin/actions.runner.service.template"
-[Unit]
-Description={{Description}}
-After=network-online.target nss-lookup.target time-sync.target
-Wants=network-online.target nss-lookup.target time-sync.target
-StartLimitInterval=5
-StartLimitBurst=10
-
-[Service]
-Type=simple
-User=root
-ExecStart=/usr/bin/unshare -m -- /bin/bash -c 'mount --bind ${OPENPILOT_DIR} /data/openpilot && setpriv --reuid={{User}} --regid={{User}} --init-groups env HOME=${BASE_DIR} USER={{User}} LOGNAME={{User}} MAIL=/var/mail/{{User}} {{RunnerRoot}}/runsvc.sh'
-WorkingDirectory={{RunnerRoot}}
-KillMode=process
-KillSignal=SIGTERM
-TimeoutStopSec=5min
-Restart=always
-RestartSec=120
-
-[Install]
-WantedBy=multi-user.target
-EOL
-}
-
-install_service() {
-    local service_name
-    if [ -f "${RUNNER_DIR}/.service" ]; then
-        service_name=$(cat "${RUNNER_DIR}/.service")
-    else
-        service_name="actions.runner.sunnypilot.$(uname -n)"
-    fi
-  
-    create_service_template
-    remount_rw
-    local service_path="/etc/systemd/system/${service_name}"
-    echo "Installing systemd service..."
-    if [ -f "${service_path}" ]; then
-        echo "Service ${service_path} found in systemd, we will delete it"
-        sudo rm -f "${service_path}"
-    fi
-    
-    cd "$RUNNER_DIR"
-    sudo ./svc.sh install $RUNNER_USER
-
-    if [ "$START_AT_BOOT" = false ]; then
-        sudo systemctl disable "${service_name}"
-    fi
-    remount_ro
-}
-
-check_restore_prerequisites() {
-    local can_restore=false
-    local service_name=""
-
-    # Check if base runner directory exists
-    if [ ! -d "${RUNNER_DIR}" ]; then
-        echo "ERROR: Runner directory ${RUNNER_DIR} does not exist"
-        echo "This directory is required for restore operations"
-        exit 1
-    fi
-
-    # First check if we have the required files for restoration
-    if [ -f "${RUNNER_DIR}/.credentials" ] && [ -f "${RUNNER_DIR}/.service" ]; then
-        can_restore=true
-        service_name=$(cat "${RUNNER_DIR}/.service")
-        echo "Found required runner configuration files"
-    else
-        echo "Missing required runner configuration files"
-        echo "Required: .credentials and .service files in ${RUNNER_DIR}"
-        exit 1
-    fi
-
-    if ! id "${RUNNER_USER}" &>/dev/null; then
-        echo "User ${RUNNER_USER} does not exist"
-    fi
-
-    # Only proceed if we can restore AND need to restore
-    if [ "$can_restore" = true ]; then
-        echo "Restoration is possible"
-        return 0
-    else
-        echo "No restoration possible"
-        exit 0
-    fi
-}
-
-perform_restore() {
-    echo "Starting runner restoration..."
-    setup_directories
-    setup_system_configs
-    install_service
-    echo "Runner restoration completed successfully"
-}
-
-perform_install() {
-    echo "Starting fresh installation..."
-    setup_directories
-    setup_system_configs
-    install_runner
-    set_directory_permissions
-    configure_runner
-    install_service
-    echo "Installation completed successfully"
-}
-
-main() {
-    if [ "$RESTORE_MODE" = true ]; then
-        echo "Running in restore mode - will only restore system configurations..."
-        check_restore_prerequisites
-        perform_restore
-    else
-        # Check required arguments for normal installation
-        if [ -z "$GITHUB_TOKEN" ]; then
-            echo "Usage: $0 [--start-at-boot] [--token <github_token>] [--repo <repository_url>] [--restore]"
-            echo "Required argument (except for --restore): github_token"
-            echo "Optional arguments:"
-            echo "  --start-at-boot    Enable auto-start at boot (default: false)"
-            echo "  --repo            Repository URL (default: ${DEFAULT_REPO_URL})"
-            echo "  --restore         Restore existing runner configuration"
-            exit 1
-        fi
-
-        # Set repository URL if not provided
-        REPO_URL="${REPO_URL:-$DEFAULT_REPO_URL}"
-        perform_install
-    fi
-
-    echo "Starting runner service..."
-    cd "$RUNNER_DIR"
-    sudo ./svc.sh start
-}
-
-main
@@ -53,24 +53,28 @@ def create_pkl_name(full_name: str) -> str:
  return pkl


-def _read_pkl_bytes(pkl_path: Path) -> bytes:
+def _hash_pkl(pkl_path: Path) -> str:
  manifest = Path(f"{pkl_path}.chunkmanifest")
  if manifest.exists():
    num_chunks = int(manifest.read_text().strip())
-    parts = []
-    for i in range(num_chunks):
-      chunk = Path(f"{pkl_path}.chunk{i + 1:02d}of{num_chunks:02d}")
-      parts.append(chunk.read_bytes())
-    return b''.join(parts)
-  return pkl_path.read_bytes()
+    paths = [Path(f"{pkl_path}.chunk{i + 1:02d}of{num_chunks:02d}") for i in range(num_chunks)]
+  else:
+    paths = [pkl_path]
+
+  digest = hashlib.sha256()
+  for path in paths:
+    with path.open('rb') as f:
+      while block := f.read(1024 * 1024):
+        digest.update(block)
+  return digest.hexdigest()


 def _find_driving_pkl(output_path: Path) -> Path | None:
-  for pattern in ('driving_tinygrad.pkl', 'driving_*_tinygrad.pkl'):
+  for pattern in ('*driving_tinygrad.pkl', '*driving_*_tinygrad.pkl'):
    matches = sorted(output_path.glob(pattern))
    if matches:
      return matches[0]
-  for pattern in ('driving_tinygrad.pkl.chunkmanifest', 'driving_*_tinygrad.pkl.chunkmanifest'):
+  for pattern in ('*driving_tinygrad.pkl.chunkmanifest', '*driving_*_tinygrad.pkl.chunkmanifest'):
    matches = sorted(output_path.glob(pattern))
    if matches:
      return Path(str(matches[0]).removesuffix('.chunkmanifest'))
@@ -86,8 +90,20 @@ def _rename_pkl_with_chunks(old_pkl: Path, new_pkl: Path) -> Path:
  return old_pkl.rename(new_pkl)


+def _hash_onnx_files(model_dir: Path) -> str | None:
+  onnx_files = sorted(model_dir.glob("*.onnx"))
+  if not onnx_files:
+    return None
+  digest = hashlib.sha256()
+  for f in onnx_files:
+    with f.open('rb') as fh:
+      while block := fh.read(1024 * 1024):
+        digest.update(block)
+  return digest.hexdigest()
+
+
 def generate_chunked_model(driving_pkl: Path) -> dict:
-  tinygrad_hash = hashlib.sha256(_read_pkl_bytes(driving_pkl)).hexdigest()
+  tinygrad_hash = _hash_pkl(driving_pkl)

  chunks_config = []
  manifest_file = Path(f"{driving_pkl}.chunkmanifest")
@@ -119,7 +135,8 @@ def generate_chunked_model(driving_pkl: Path) -> dict:
  }


-def create_metadata_json(models: list, output_dir: Path, custom_name=None, short_name=None, is_20hz=False, upstream_branch="unknown") -> None:
+def create_metadata_json(models: list, output_dir: Path, custom_name=None, short_name=None, is_20hz=False, upstream_branch="unknown",
+                         onnx_sha256=None, is_big=False) -> None:
  bundle_json = {
    "short_name": short_name,
    "display_name": custom_name or upstream_branch,
@@ -132,9 +149,13 @@ def create_metadata_json(models: list, output_dir: Path, custom_name=None, short
    "generation": "-1",
    "build_time": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ"),
    "overrides": {},
+    "is_big": is_big,
    "models": models,
  }

+  if onnx_sha256:
+    bundle_json["onnx_sha256"] = onnx_sha256
+
  # Write metadata to output_dir
  metadata_json = {
    "bundles": [bundle_json]
@@ -166,6 +187,8 @@ if __name__ == "__main__":
    print(f"No driving_tinygrad.pkl found in {_output_dir}", file=sys.stderr)
    sys.exit(1)

+  is_big = _driving_pkl.name.startswith('big_')
+
  if _pkl:
    new_pkl = _output_dir / f"driving_{_pkl}_tinygrad.pkl"
    if not new_pkl.exists():
@@ -174,4 +197,6 @@ if __name__ == "__main__":
      _driving_pkl = new_pkl

  _model_metadata = generate_chunked_model(_driving_pkl)
-  create_metadata_json([_model_metadata], _output_dir, args.custom_name, _short_name, args.is_20hz, args.upstream_branch)
+  _onnx_sha256 = _hash_onnx_files(Path(args.model_dir))
+  create_metadata_json([_model_metadata], _output_dir, args.custom_name, _short_name, args.is_20hz, args.upstream_branch,
+                       onnx_sha256=_onnx_sha256, is_big=is_big)
@@ -47,7 +47,7 @@ git rm -rf $OUTPUT_DIR/.git || true # Doing cleanup, but it might fail if the .g
 git remote remove origin || true # ensure cleanup
 git remote add origin $GIT_ORIGIN
 #git push origin -d $DEV_BRANCH || true # Ensuring we delete the remote branch if it exists as we are wiping it out
-git fetch origin $DEV_BRANCH || (git checkout -b $DEV_BRANCH && git commit --allow-empty -m "sunnypilot v$VERSION release" && git push -u origin $DEV_BRANCH)
+git fetch --depth 1 origin $DEV_BRANCH || (git checkout -b $DEV_BRANCH && git commit --allow-empty -m "sunnypilot v$VERSION release" && git push -u origin $DEV_BRANCH)

 echo "[-] committing version $VERSION T=$SECONDS"
 git add -f .
@@ -1,66 +0,0 @@
-#!/usr/bin/env bash
-# Determine BASE_DIR based on mount point
-if mountpoint -q /data/media; then
-    GITHUB_BASE_DIR="/data/media/0/github"
-else
-    GITHUB_BASE_DIR="/data/github"
-fi
-
-# Define directories and user
-BIN_DIR="$GITHUB_BASE_DIR/bin"
-BUILDS_DIR="$GITHUB_BASE_DIR/builds"
-OPENPILOT_DIR="$GITHUB_BASE_DIR/openpilot"
-LOGS_DIR="$GITHUB_BASE_DIR/logs"
-CACHE_DIR="$GITHUB_BASE_DIR/cache"
-RUNNER_USERNAME="github-runner"
-# Define the systemd service name
-SERVICE_NAME="github-runner"
-USER_GROUPS="comma,gpu,gpio,sudo"
-
-# Function to stop and disable the systemd service
-stop_and_uninstall_service() {
-    cd $GITHUB_BASE_DIR/runner
-    sudo ./svc.sh stop
-    sudo ./svc.sh uninstall
-}
-
-# Function to remove the systemd service file
-remove_runner() {
-    cd $GITHUB_BASE_DIR/runner
-    sudo rm .runner
-    sudo su -c './config.sh remove' github-runner
-}
-
-# Function to delete the Github Runner directories
-delete_directories() {
-    sudo rm -rf "$BIN_DIR/github-runner"
-    sudo rm -rf "$GITHUB_BASE_DIR" "$BIN_DIR" "$BUILDS_DIR" "$LOGS_DIR" "$CACHE_DIR" "$OPENPILOT_DIR"
-}
-
-# Function to remove the Github Runner user
-delete_user() {
-    for group in ${USER_GROUPS//,/ }
-    do
-       sudo gpasswd -d ${RUNNER_USERNAME} ${group}
-    done
-    sudo userdel -r ${RUNNER_USERNAME}
-}
-
-# Function to remove sudoers entry
-remove_sudoers_entry() {
-    sudo sed -i.bak "/${RUNNER_USERNAME} ALL=(ALL) NOPASSWD: ALL/d" /etc/sudoers
-}
-
-# Make filesystem writable
-sudo mount -o remount rw /
-
-# Ensure filesystem is remounted as read-only on script exit
-trap "sudo mount -o remount ro /" EXIT
-
-# Call functions
-stop_and_uninstall_service
-remove_runner
-delete_directories
-delete_user
-remove_sudoers_entry
-# End of uninstall script
@@ -0,0 +1,104 @@
+#!/usr/bin/env python3
+"""
+Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
+
+This file is part of sunnypilot and is licensed under the MIT License.
+See the LICENSE.md file in the root directory for more details.
+"""
+
+import argparse
+import hashlib
+import json
+import tempfile
+
+from huggingface_hub import HfApi, hf_hub_download
+
+
+def hash_file(path: str) -> str:
+  digest = hashlib.sha256()
+  with open(path, 'rb') as f:
+    while block := f.read(1024 * 1024):
+      digest.update(block)
+  return digest.hexdigest()
+
+
+def main():
+  parser = argparse.ArgumentParser()
+  parser.add_argument("--hf-repo", required=True)
+  parser.add_argument("--hf-defaults-path", required=True)
+  parser.add_argument("--artifact-name", required=True)
+  parser.add_argument("--model-dir", required=True)
+  parser.add_argument("--onnx-path", required=True)
+  parser.add_argument("--onnx-ref", required=True)
+  parser.add_argument("--model-name", required=True)
+  parser.add_argument("--tinygrad-ref", required=True)
+  parser.add_argument("--run-number", required=True)
+  args = parser.parse_args()
+
+  api = HfApi()
+  onnx_sha256 = hash_file(args.onnx_path)
+  short_ref = args.onnx_ref[:8]
+  folder_name = f"model-{args.model_name}-{short_ref}-{args.run_number}"
+
+  print(f"ONNX hash: {onnx_sha256}")
+  print(f"ONNX ref: {args.onnx_ref} (short: {short_ref})")
+  print(f"Folder: {folder_name}")
+
+  metadata_path = f"{args.model_dir}/metadata.json"
+  with open(metadata_path) as f:
+    metadata = json.load(f)
+
+  bundle = metadata['bundles'][0]
+  bundle['display_name'] = args.model_name
+  bundle['onnx_sha256'] = onnx_sha256
+  bundle['onnx_ref'] = args.onnx_ref
+
+  artifact = bundle['models'][0]['artifact']
+  hf_base = f"https://huggingface.co/datasets/{args.hf_repo}/resolve/main/{args.hf_defaults_path}/{folder_name}"
+  artifact['download_uri']['url'] = f"{hf_base}/{artifact['file_name']}"
+  for chunk in artifact.get('chunks', []):
+    chunk['url'] = f"{hf_base}/{chunk['file_name']}"
+
+  print(f"Uploading model to {args.hf_defaults_path}/{folder_name}/")
+  api.upload_folder(
+    folder_path=args.model_dir,
+    path_in_repo=f"{args.hf_defaults_path}/{folder_name}",
+    repo_id=args.hf_repo,
+    repo_type="dataset",
+  )
+
+  json_filename = f"{args.hf_defaults_path}/default_models.json"
+  try:
+    local_path = hf_hub_download(repo_id=args.hf_repo, repo_type='dataset', filename=json_filename)
+    with open(local_path) as f:
+      defaults_json = json.load(f)
+  except Exception:
+    defaults_json = {"tinygrad_ref": args.tinygrad_ref, "bundles": []}
+
+  defaults_json['tinygrad_ref'] = args.tinygrad_ref
+
+  existing_idx = next((i for i, b in enumerate(defaults_json['bundles'])
+                       if b.get('onnx_sha256') == onnx_sha256), None)
+  if existing_idx is not None:
+    defaults_json['bundles'][existing_idx] = bundle
+  else:
+    defaults_json['bundles'].append(bundle)
+
+  print(json.dumps(defaults_json, indent=2))
+
+  with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f:
+    json.dump(defaults_json, f, indent=2)
+    tmp_path = f.name
+
+  api.upload_file(
+    path_or_fileobj=tmp_path,
+    path_in_repo=json_filename,
+    repo_id=args.hf_repo,
+    repo_type="dataset",
+  )
+
+  print(f"Updated {json_filename}")
+
+
+if __name__ == "__main__":
+  main()
Author	SHA1	Message	Date
Jason Wen	25c25047b8	models: persist model selection per catalog across chestnut state changes (#1960 )	2026-08-25 01:12:12 -04:00
Jason Wen	cefe5737b9	models: fix current model not updating on chestnut status (#1959 ) * models: preserve user model selection across reboots and power cycles * no * again * idk * over	2026-08-25 00:41:31 -04:00
Jason Wen	760c19d3f9	ui/models: handle missing files during cache size calculation (#1958 )	2026-08-24 23:31:52 -04:00
James Vecellio-Grant	45814e3313	modeld_v2: spatial features (#1934 ) * modeld_v2: spatial features * Update fetcher.py * dont reshape non 4 dim arrays * realize for non compiled * Update compile_modeld.py * god dammit it was realize() * it was fucking frozen tinygrad. just need to recompile * bump * ci: add is_big flag to metadata.json to support backward compat * Update model_generator.py * Update sunnypilot-build-model.yaml * Update helpers.py * Revert "Update helpers.py" This reverts commit `3a955ca11a`. * Reapply "Update helpers.py" This reverts commit `ca9c6e1933`. * models: use less strict chestnut detection state --------- Co-authored-by: Jason Wen <haibin.wen3@gmail.com>	2026-08-24 22:52:29 -04:00
Jason Wen	2ba91d2be5	ci: add tinygrad ref check to prepare_chestnut and even faster prebuilt stages (#1957 ) * ci: faster prebuilt stages * tg check chestnut * zoomer!	2026-08-24 22:42:45 -04:00
Jason Wen	19f83b274f	ci: identical environment for publish_chestnut prebuilt	2026-08-24 22:06:20 -04:00
Jason Wen	d14d0b1dd0	ci: parallelize models chunk downloads and split branch publishing (#1955 ) * ci: parallelize model chunk downloads and better publish * ci: download all model chunks in parallel with xargs -P8 * split split * ew * must require	2026-08-24 21:48:53 -04:00
Jason Wen	6cc5f3aad8	ci: fix DM model build, separate HF defaults paths, nuke build races (#1956 ) * ci: fix DM model build, separate HF defaults paths, nuke build races * more split! * name * ci: download driving and DM model chunks into chestnut prebuilt output	2026-08-24 20:02:48 -04:00
Jason Wen	8e16c9babb	ci: offload small model compilation (#1952 ) * ci: compile default big model with stock modeld * Revert "Revert big RL model (#38627)" This reverts commit `516ec1e682`. * ci: compile default small model with stock modeld compiler * ci: offload small model compilation * Reapply "Revert big RL model (#38627)" This reverts commit `d06cfabb62`. * Reapply "Revert big RL model (#38627)" This reverts commit `d06cfabb62`.	2026-08-24 16:24:19 -04:00
Jason Wen	2bcfed5c71	ci: compile default models with stock modeld (#1954 ) * ci: compile default big model with stock modeld * Revert "Revert big RL model (#38627)" This reverts commit `516ec1e682`. * ci: compile default small model with stock modeld compiler * Reapply "Revert big RL model (#38627)" This reverts commit `d06cfabb62`.	2026-08-24 15:37:46 -04:00
Jason Wen	66cf334067	ci: unify default model build into single workflow (#1951 ) * ci: unify default model build into single workflow * ci: consolidate upload jobs and add tinygrad ref validation	2026-08-24 12:35:37 -04:00
Jason Wen	94ed0608e6	models: use less strict chestnut detection state (#1948 )	2026-08-24 01:40:31 -04:00
Jason Wen	0fbca979df	models: show big model list when Chestnut present (#1943 )	2026-08-23 20:19:57 -04:00
Jason Wen	dcddb2a0bd	models: revert icon override from this branch scope	2026-08-23 19:53:46 -04:00
Jason Wen	699eaf7957	include them!	2026-08-23 19:16:15 -04:00
Jason Wen	c246e6318a	Merge branch 'master' into models-good-detect	2026-08-23 16:56:44 -04:00
Jason Wen	718db8c62e	Sync: `commaai/openpilot:master` → `sunnypilot/sunnypilot:master` (#1944 )	2026-08-23 16:56:00 -04:00
Jason Wen	c2214d4c32	Merge commit '084747c75d2cbd23af65ab7a9e770bbd7b98bac9' into sync-20260823 # Conflicts: # openpilot/common/params_keys.h	2026-08-23 15:36:54 -04:00
Jason Wen	0de7fbf33d	new	2026-08-23 15:06:53 -04:00
Jason Wen	211f990f6b	models: fix sunnylink default model display and false big model re-downloading (#1941 ) * big needs small * no download * actually * send it	2026-08-23 04:04:46 -04:00
Jason Wen	97468e4fa4	[TIZI/TICI] ui: remove calibration reset dialog on model change (#1942 )	2026-08-23 03:48:52 -04:00
Jason Wen	6c6fba9a14	ci: fix flaky LLK test (#1940 )	2026-08-23 02:57:02 -04:00
Jason Wen	34621cf816	ci: refactor big model chunk handling (#1939 )	2026-08-23 02:48:10 -04:00
Jason Wen	086530b7c6	[TIZI/TICI] ui: fix path width during gas and steering override (#1938 )	2026-08-22 21:47:38 -04:00
Jason Wen	4f46433e2b	alerts: add branch metadata to chestnut offroad warning (#1936 ) * alerts: add branch metadata to chestnut offroad warning * all branches	2026-08-22 10:20:06 -04:00
Jason Wen	5a8567e3e7	ci: chestnut prebuilt branches (#1935 ) * ci: chestnut prebuilt branches * fix * nope * big * try again * diff * malformed * auth * more	2026-08-22 03:41:48 -04:00
Jason Wen	07558166c8	ci: only check default model on dispatch	2026-08-22 00:32:35 -04:00
Jason Wen	ca9338812e	ci: prep for chestnut prebuilts	2026-08-22 00:16:10 -04:00
granolaFPV	4667241fe7	[TIZI/TICI] ui: dynamic path width color (#1926 ) * Fix UI path color and thickness based on lateral steering state (Issue #1441) * Fix UI path color and thickness based on lateral control engagement (Issue #1441) * Fix UI path width and color based on MADS lateral engagement (Issue #1441) * Fix UI path width and color based on MADS lateral engagement (Issue #1441) * move to ModelRendererSP * match torque bar * same behavior across the board * simplify --------- Co-authored-by: Brennan Browne <brennanbrowne@google.com> Co-authored-by: Jason Wen <haibin.wen3@gmail.com>	2026-08-21 20:51:43 -04:00
Shane Smiskol	084747c75d	Fix button label widths (#38680 ) * Revert "ui: fix text and icon overlap on button (#38628)" This reverts commit `d9c4120f89`. * simple * can do this * fix eliding * Revert "fix eliding" This reverts commit b271a350182ad87f9942d7363383ee8ec72d0e36. * clean up * clean up	2026-08-21 15:41:12 -07:00
Marceline Milligan	a49c260927	ui: show default big model name when eGPU present/active (#1930 ) * Name the big default model in the device UI Build on the default big-model metadata from #1929 and resolve the displayed model from cached capability and modeld runtime state. Keep model selection behavior unchanged. Assisted-by: GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove get_default_model_label * simplify --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: nayan <nayan8teen@gmail.com>	2026-08-21 13:26:52 -04:00
Jason Wen	5ad2bfdb75	ci: deprecate GitHub runners (#1933 )	2026-08-21 00:35:38 -04:00
Jason Wen	b742557d62	sunnylink: add model resolver (#1931 ) * models: add get_default_model resolver for sunnylink * models: move get_default_model to default_model.py	2026-08-20 21:56:57 -04:00
Nayan	5ecd05aedf	models: add big model to default model resolution (#1929 ) * device * sunnylink * lint * lfs? * Revert "lfs?" This reverts commit `bcdaec6b4c`. * update path * Scope the default big model down to the sunnylink schema * Drop the mock-only default model test * Move the default model resolver out to separate PR --------- Co-authored-by: Jason Wen <haibin.wen3@gmail.com>	2026-08-20 21:31:32 -04:00
Jason Wen	5ae100aa1d	models fetcher: bump big model to v21	2026-08-20 19:19:59 -04:00
Jason Wen	be76a88b80	ci override LFS fetch exclude for real ONNX file retrieval (#1928 ) ci: override lfs.fetchexclude so the model fetch pulls real ONNX files instead of pointers	2026-08-20 19:12:59 -04:00
James Vecellio-Grant	049d225d5a	ci: Dedicated Model Runner (#1922 ) * ci: Dedicated Model Runner * recurse * not needed * fix wrapper * whoops * bypass * modeld_v2: restore chestnut link check before big model build * modeld_v2: stage onnx to disk instead of shared memory * ci: clear unchunked onnx temps before model build * ci: stream the pkl hash instead of loading it into memory --------- Co-authored-by: Jason Wen <haibin.wen3@gmail.com>	2026-08-20 16:18:17 -04:00
stef	555f48c5d2	params: remove livestream param on ignition (#38679 ) * remove livestream param on ignition * simplify process config	2026-08-19 14:21:59 -07:00
stef	dcf9d25bf3	webrtcd: more descriptive errors (#38677 ) more descriptive errors	2026-08-19 14:02:03 -07:00
stef	a8d1a280c6	webrtcd/athenad: we don't have to fail on no car params (#38678 ) we don't have to fail on no car params	2026-08-19 13:49:19 -07:00
stef	5b36799eec	webrtc: fix message handler race (#38675 ) open message handler early	2026-08-18 21:56:33 -07:00
stef	20fdc3d824	webrtcd: cloud logging (#38674 ) * logging * remove test * get rid of redudant try except * fix logger context	2026-08-18 21:13:57 -07:00
Shane Smiskol	7bd6cad821	Re-open agnos updater UI if crash (#38672 ) loop if crash	2026-08-18 19:06:48 -07:00