ci: Refactor model building workflows (#1096)

* Tinygrad bump from sync-20250627 * bump tinygrad_repo * Reformat metadata generator to match driving_models.json * bump tinygrad * Revert "bump tinygrad" This reverts commit f479dfd5021266e22b34deca06cc3c4d531fb7e8. * revert me after SP model compiled * Model recompiled successfully, initiate "revert me after SP model compiled" This reverts commit 95706eb688c9f5522903bee21892b8871ff8f957. * The "FillMe" placeholder caused an extra 10 seconds of work * bump to 22Jul2025 * Update build-all-tinygrad-models.yaml * Update build-all-tinygrad-models.yaml * Update build-all-tinygrad-models.yaml * Update build-all-tinygrad-models.yaml * Update build-all-tinygrad-models.yaml * Update build-all-tinygrad-models.yaml * Update build-all-tinygrad-models.yaml * Allow more dynamic short names This should hopefully be future-proof for now.. It's robust enough to return the correct word-digit format (see example on how it generates from given display name below): 'Last Horizon V2 (November 22, 2024)' -> LHV2 'Alabama (November 25, 2024)' -> ALABAMA 'PlayStation (December 03, 2024)' -> PLAYSTAT 'Postal Service (December 09, 2024)' -> PS 'Null Pointer (December 13, 2024)' -> NP 'North America (December 16, 2024)' -> NA 'National Public Radio (December 18, 2024)' -> NPR 'Filet o Fish (March 7, 2025)' -> FOF 'Tomb Raider 2 (April 18, 2025)' -> TR2 'Tomb Raider 3 (April 22, 2025)' -> TR3 'Tomb Raider 4 (April 25, 2025)' -> TR4 'Tomb Raider 5 (April 25, 2025)' -> TR5 'Tomb Raider 6 (April 30, 2025)' -> TR6 'Tomb Raider 7 (May 07, 2025)' -> TR7 'Down to Ride (Revision: May 10, 2025)' -> DTR 'SP Vikander Model (May 16, 2025)' -> SPVM 'VFF Driving (May 15, 2025)' -> VFFD 'Secret Good Openpilot (May 16, 2025)' -> SGO 'Vegetarian Filet o Fish (May 29, 2025)' -> VFOF 'Down To Ride (Revision: May 30, 2025)' -> DTR 'Vegetarian Filet o Fish v2 (June 05, 2025)' -> VFOFV2 'Kerrygold Driving (June 08, 2025)' -> KD 'Tomb Raider 10 (June 16, 2025)' -> TR10 'Organic Kerrygold (June 17, 2025)' -> OK 'Liquid Crystal Driving (June 21, 2025)' -> LCD 'Vegetarian Filet o Fish v3 (June 21, 2025)' -> VFOFV3 'Vibe Model [Custom Model]' -> VMCM 'Tomb Raider 13 (June 27, 2025)' -> TR13 'Aggressive TR (June 28, 2025)' -> ATR 'Tomb Raider 14 (June 30, 2025)' -> TR14 'Cookiemonster Tomb Raider (July 02, 2025)' -> CTR 'Down to Ride (Revision: July 07, 2025)' -> DTR 'Simple Plan Driving (July 07, 2025)' -> SPD 'Down to Ride (Revision: July 08, 2025)' -> DTR 'Tomb Raider 15 (July 09, 2025)' -> TR15 'Tomb Raider 15 rev-2 (July 11, 2025)' -> TR15R2 'Le Tomb Raider 14 (July 14, 2025)' -> LTR14 'Le Tomb Raider 14h (July 17, 2025)' -> LTR14H 'Tomb Raider 16 (July 18, 2025)' -> TR16 'Tomb Raider 16v2 (July 21, 2025)' -> TR16V2 * Update build-all-tinygrad-models.yaml * Update build-all-tinygrad-models.yaml * No need to sleep 3 seconds, just send it * try dynamic * cleanup * Update build-single-tinygrad-model.yaml * bc devtekve said. also, this is repetitive af * Revert "bc devtekve said. also, this is repetitive af" This reverts commit 3a0c1562de6eae1d2d213d2f53cfb2edb4eef40c. * maybe we could use a script instead that both build all That both build all and sunnypilot-build-model reference * refactor: consolidate model building steps into a single workflow * tweak * tweakx2 * tweakx3 * tweakx4 * dunno dunno... * output dir * lots of changes * Revert "lots of changes" This reverts commit 4aadb0ee2929fdf9d902335161b22072bef4e068. * fail if all fail * no inputs needed * make it easier for us * note failure and exit 0 * Update build-all-tinygrad-models.yaml * not needed unless we really want it * Update build-single-tinygrad-model.yaml * Merge branch 'sync-20250627-tinygrad' of github.com:sunnypilot/sunnypilot into sync-20250627-tinygrad * retry for failed ? * always run this step because sometimes one build fails which causes the matrix to fail, but most builds still have uploaded artifacts. * strip * no escape * Update build-all-tinygrad-models.yaml * Test case from terminal run (openpilot) james@Mac sunnypilot % jq -c '[.bundles[] | select(.runner=="tinygrad") | {ref, display_name: (.display_name | gsub(" \$[^)]*\$"; "")), is_20hz}]' \ /Users/james/Documents/GitHub/sunnypilot-docs/docs/driving_models_v6.json > matrix.json mkdir -p output touch "output/model-Tomb Raider 16v2 (July 21, 2025)-544" touch "output/model-Space Lab Model (July 24, 2025)-547" touch "output/model-Space Lab Model v1 (July 24, 2025)-548" built=(); while IFS= read -r line; do built+=("$line"); done < <( ls output | sed -E 's/^model-//' | sed -E 's/-[0-9]+$//' | sed -E 's/ $[^)]*$//' | awk '{gsub(/^ +| +$/, ""); print}' ) jq -c --argjson built "$(printf '%s\n' "${built[@]}" | jq -R . | jq -s .)" \ 'map(select(.display_name as $n | ($built | index($n | gsub("^ +| +$"; "")) | not)))' \ matrix.json > retry_matrix.json cat retry_matrix.json [] (openpilot) james@Mac sunnypilot % * always * great success * add suffix to retry artifact so it doesn't conflict * retry to get_model too * and there haha * unnecessary hyphen * compare built to missing. include retries * adjust copy of artifacts. * Update build-all-tinygrad-models.yaml * Update model selector versioning and add documentation * Update retry condition for failed models in build-all-tinygrad-models.yaml * Update retry condition for failed models in build-all-tinygrad-models.yaml * Update build-single-tinygrad-model.yaml * false * default none because why not * red diff? i think? * meh ... not needed i guess * error error error * Nayan is watching... always watching mike wazowski * string all the way * lots of retries just in case because im scared * more robust * ONLY ONE!!!!!! * delete.... a lot * fix artifacts * fix artifacts * make sure each is unique :) * skip files like artifact duhhhh * artifact name dir * concurrency * copy here * Update build-single-tinygrad-model.yaml * Update build-single-tinygrad-model.yaml * bump * bump tinygrad * max parallel? if not, i have the other remedy ready in build-all * revert me! * I resynced tinygrad woo hoo * setup shouldnt fail * pull * big ole diff * condition * Update build-all-tinygrad-models.yaml * not always() never always() never!!! * not failure instead of great success * Update build-all-tinygrad-models.yaml * yay that worked. lets invoke build-single one last time * these arent used and are just taking up 250MB space * really frog? * bump back to 3 * self-hosted, tici * rename to trigger tests * 2 and done --------- Co-authored-by: DevTekVE <devtekve@gmail.com>
2026-10-01 03:43:46 +08:00 · 2025-07-31 09:15:06 -07:00
parent c47aadee19
commit 6d41ce2032
4 changed files with 389 additions and 235 deletions
@@ -1,13 +1,36 @@
 name: Build Single Tinygrad Model and Push

 on:
+  workflow_call:
+    inputs:
+      upstream_branch:
+        description: 'Upstream commit to build from'
+        required: true
+        type: string
+      custom_name:
+        description: 'Custom name for the model (no date, only name)'
+        required: false
+        type: string
+      recompiled_dir:
+        description: 'Existing recompiled directory number (e.g. 3 for recompiled3)'
+        required: true
+        type: string
+      json_version:
+        description: 'driving_models version number to update (e.g. 5 for driving_models_v5.json)'
+        required: true
+        type: string
+      artifact_suffix:
+        description: 'Suffix for artifact name'
+        required: false
+        type: string
+        default: ''
+      bypass_push:
+        description: 'Bypass pushing to GitLab for build-all'
+        required: false
+        default: true
+        type: boolean
  workflow_dispatch:
    inputs:
-      build_model_ref:
-        description: 'Branch to use for build-model workflow'
-        required: false
-        default: 'master-new'
-        type: string
      upstream_branch:
        description: 'Upstream commit to build from'
        required: true
@@ -26,10 +49,12 @@ on:
        type: number
      model_folder:
        description: 'Model folder'
-        required: true
        type: choice
+        default: 'None'
        options:
+          - None
          - Simple Plan Models
+          - Space Lab Models
          - TR Models
          - DTR Models
          - Custom Merge Models
@@ -47,13 +72,27 @@ on:
        description: 'Minimum selector version'
        required: false
        type: number
+env:
+  RECOMPILED_DIR: recompiled${{ inputs.recompiled_dir }}
+  JSON_FILE: docs/docs/driving_models_v${{ inputs.json_version }}.json

 jobs:
-  build-single:
+  build_model:
+    uses: ./.github/workflows/sunnypilot-build-model.yaml
+    with:
+      upstream_branch: ${{ inputs.upstream_branch }}
+      custom_name: ${{ inputs.custom_name || inputs.upstream_branch }}
+      is_20hz: true
+      artifact_suffix: ${{ inputs.artifact_suffix }}
+    secrets: inherit
+
+  publish_model:
+    if: ${{ !inputs.bypass_push && !cancelled() }}
+    concurrency:
+      group: gitlab-push-${{ inputs.recompiled_dir }}
+      cancel-in-progress: false
+    needs: build_model
    runs-on: ubuntu-latest
-    env:
-      RECOMPILED_DIR: recompiled${{ github.event.inputs.recompiled_dir }}
-      JSON_FILE: docs/docs/driving_models_v${{ github.event.inputs.json_version }}.json
    steps:
      - name: Set up SSH
        uses: webfactory/ssh-agent@v0.9.0
@@ -96,66 +135,49 @@ jobs:
            exit 1
          fi

-      - name: Install dependencies
+      - name: Download artifact name file
+        uses: actions/download-artifact@v4
+        with:
+          name: artifact-name-${{ inputs.custom_name || inputs.upstream_branch }}
+          path: artifact_name
+
+      - name: Read artifact name
+        id: read-artifact-name
        run: |
-          sudo apt-get update
-          sudo apt-get install -y jq gh
-
-      - name: Build model
-        env:
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        run: |
-          START_TIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
-          gh workflow run sunnypilot-build-model.yaml \
-            --repo sunnypilot/sunnypilot \
-            --ref "${{ github.event.inputs.build_model_ref }}" \
-            -f upstream_branch="${{ github.event.inputs.upstream_branch }}" \
-            -f custom_name="${{ github.event.inputs.custom_name }}"
-
-          for i in {1..24}; do
-            RUN_ID=$(gh run list --repo sunnypilot/sunnypilot --workflow=sunnypilot-build-model.yaml --branch="${{ github.event.inputs.build_model_ref }}" --created ">$START_TIME" --limit=1 --json databaseId --jq '.[0].databaseId')
-            if [ -n "$RUN_ID" ]; then
-              break
-            fi
-            sleep 5
-          done
-
-          if [ -z "$RUN_ID" ]; then
-            echo "Could not find the triggered workflow run."
-            exit 1
-          fi
-
-          echo "Watching run ID: $RUN_ID"
-          gh run watch "$RUN_ID" --repo sunnypilot/sunnypilot
-          CONCLUSION=$(gh run view "$RUN_ID" --repo sunnypilot/sunnypilot --json conclusion --jq '.conclusion')
-          echo "Run concluded with: $CONCLUSION"
-          if [[ "$CONCLUSION" != "success" ]]; then
-            echo "Workflow run failed with conclusion: $CONCLUSION"
-            exit 1
-          fi
+          ARTIFACT_NAME=$(cat artifact_name/artifact_name.txt)
+          echo "artifact_name=$ARTIFACT_NAME" >> $GITHUB_OUTPUT

      - name: Download and extract model artifact
-        env:
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        uses: actions/download-artifact@v4
+        with:
+          name: ${{ steps.read-artifact-name.outputs.artifact_name }}
+          path: output
+
+      - name: Remove unwanted files
        run: |
-          ARTIFACT_DIR="gitlab_docs/models/$RECOMPILED_DIR"
-          RUN_ID=$(gh run list --repo sunnypilot/sunnypilot --workflow=sunnypilot-build-model.yaml --branch="${{ github.event.inputs.build_model_ref }}" --limit=1 --json databaseId --jq '.[0].databaseId')
-          ARTIFACT_NAME=$(gh api repos/sunnypilot/sunnypilot/actions/runs/$RUN_ID/artifacts --jq '.artifacts[] | select(.name | startswith("model-")) | .name')
-          echo "Downloading artifact: $ARTIFACT_NAME from run: $RUN_ID"
-          mkdir -p "$ARTIFACT_DIR/$ARTIFACT_NAME"
-          echo "Created directory: $ARTIFACT_DIR/$ARTIFACT_NAME"
-          gh run download "$RUN_ID" --repo sunnypilot/sunnypilot -n "$ARTIFACT_NAME" --dir "$ARTIFACT_DIR/$ARTIFACT_NAME"
-          echo "Downloaded artifact zip(s) to: $ARTIFACT_DIR/$ARTIFACT_NAME"
-          ZIP_PATH=$(find "$ARTIFACT_DIR/$ARTIFACT_NAME" -type f -name '*.zip' | head -n1)
-          if [ -n "$ZIP_PATH" ]; then
-            echo "Unzipping $ZIP_PATH to $ARTIFACT_DIR/$ARTIFACT_NAME"
-            unzip -o "$ZIP_PATH" -d "$ARTIFACT_DIR/$ARTIFACT_NAME"
-            rm -f "$ZIP_PATH"
-            echo "Unzipped and removed $ZIP_PATH"
-          else
-            echo "No zip file found in $ARTIFACT_DIR/$ARTIFACT_NAME"
-          fi
-          echo "Done processing $ARTIFACT_NAME"
+          find output -type f -name 'dmonitoring_model_tinygrad.pkl' -delete
+          find output -type f -name 'dmonitoring_model.onnx' -delete
+
+      - name: Copy model artifact(s) to GitLab recompiled dir
+        env:
+          ARTIFACT_NAME: ${{ steps.read-artifact-name.outputs.artifact_name }}
+        run: |
+          ARTIFACT_DIR="gitlab_docs/models/${RECOMPILED_DIR}/${ARTIFACT_NAME}"
+          mkdir -p "$ARTIFACT_DIR"
+          for path in output/*; do
+            if [ "$(basename "$path")" = "artifact_name.txt" ]; then
+              continue
+            fi
+            name="$(basename "$path")"
+            if [ -d "$path" ]; then
+              mkdir -p "$ARTIFACT_DIR/$name"
+              cp -r "$path"/* "$ARTIFACT_DIR/$name/"
+              echo "Copied dir $name -> $ARTIFACT_DIR/$name"
+            else
+              cp "$path" "$ARTIFACT_DIR/"
+              echo "Copied file $name -> $ARTIFACT_DIR/"
+            fi
+          done

      - name: Push recompiled dir to GitLab
        env:
@@ -163,28 +185,36 @@ jobs:
        run: |
          cd gitlab_docs
          git checkout main
+          git pull origin main
          for d in models/"$RECOMPILED_DIR"/*/; do
            git sparse-checkout add "$d"
          done
          git add models/"$RECOMPILED_DIR"
          git config --global user.name "GitHub Action"
          git config --global user.email "action@github.com"
-          git commit -m "Update $RECOMPILED_DIR with new/updated model from build-single-tinygrad-model" || echo "No changes to commit"
+          git commit -m "Create/Update $RECOMPILED_DIR with new/updated model from build-single-tinygrad-model" || echo "No changes to commit"
          git push origin main

+      - run: |
+          cd docs
+          git pull origin gh-pages
+
      - name: Run json_parser.py to update JSON
        run: |
-          FOLDER="${{ github.event.inputs.model_folder }}"
+          FOLDER="${{ inputs.model_folder }}"
          if [ "$FOLDER" = "Other" ]; then
-            FOLDER="${{ github.event.inputs.custom_model_folder }}"
+            FOLDER="${{ inputs.custom_model_folder }}"
          fi
          ARGS=""
-          [ -n "$FOLDER" ] && ARGS="$ARGS --model-folder \"$FOLDER\""
-          [ -n "${{ github.event.inputs.generation }}" ] && ARGS="$ARGS --generation \"${{ github.event.inputs.generation }}\""
-          [ -n "${{ github.event.inputs.version }}" ] && ARGS="$ARGS --version \"${{ github.event.inputs.version }}\""
+          if [ "$FOLDER" != "None" ] && [ -n "$FOLDER" ]; then
+            ARGS="$ARGS --model-folder \"$FOLDER\""
+          fi
+          [ -n "${{ inputs.generation }}" ] && ARGS="$ARGS --generation \"${{ inputs.generation }}\""
+          [ -n "${{ inputs.version }}" ] && ARGS="$ARGS --version \"${{ inputs.version }}\""
          eval python3 docs/json_parser.py \
            --json-path "$JSON_FILE" \
            --recompiled-dir "gitlab_docs/models/$RECOMPILED_DIR" \
+            --sort-by-date \
            $ARGS

      - name: Push updated JSON to GitHub docs repo