Compare commits

..

5 Commits

Author SHA1 Message Date
Isaac Barham 48b241fb6c modeld: add road-edge curb guard to prolong turn-in until curb opens 2026-11-20 16:00:36 -05:00
Isaac Barham 11dc83736f modeld: add apex/exit 1s-path blend and crawl-speed time cap to turn planner 2026-11-20 16:00:36 -05:00
Isaac Barham 2412ae892b modeld: tighten turn planner speed taper and apex release for rolling turns 2026-11-20 16:00:36 -05:00
Isaac Barham 75011ef0f0 modeld: add rear-axle protected turn planner for tight turns 2026-11-20 16:00:35 -05:00
Amy Jeanes 9489b2f481 models: retry a failed transfer from its resume state, keep the failure on the row (#2014)
* models: retry a failed transfer from its resume state, keep the failure on the row

A whole-file transfer whose pieces exhaust their retries (dropped link, DNS
blip, overloaded server) now gets up to four attempts, with a cancellable
2/4/8 s backoff between them. Every retry resumes from the sidecar, so nothing
already on disk is fetched again; speed is published as 0 while stalled.
Permanent HTTP errors raise at once, as before.

A failed download used to vanish from the Models row on the next manager tick
because the selection was cleared in `finally`. It now stays on the row, with
the failed status, until the next request; a finished or cancelled download
still clears it. The mici panel's error state names the failed model instead
of animating dots that never resolve.

With failures now sticky, a user cancel no longer marks the artifact or bundle
failed: it would otherwise flash "download failed" while the request handler
clears the row.

Tests: a transient failure resumes from the recorded pieces on retry and
fetches nothing twice, a persistent 503 gives up after every attempt, a 404
is never retried, a cancel during the backoff ends it at once, a cancel
leaves the bundle not failed, and the row state after a failed, finished and
cancelled request.

Assisted-by: Claude Code (Claude Opus 4.8)
Assisted-by: Claude Code (Claude Opus 5.5)

* remove unneeded comments

Assisted-by: Claude Code (Claude Opus 5.5)
2026-11-18 10:06:45 -08:00
19 changed files with 677 additions and 554 deletions
@@ -11,11 +11,6 @@ on:
description: 'Custom name for the model (no date, only name)'
required: false
type: string
from_sunnypilot:
description: 'The ref is on sunnypilot/sunnypilot (custom models): skip trying commaai/openpilot first'
required: false
type: boolean
default: false
recompiled_dir:
description: 'Existing recompiled directory number (e.g. 1 for recompiled1)'
required: true
@@ -39,11 +34,6 @@ on:
required: false
type: string
default: 'qcom'
on_device:
description: 'Compile on the tici / chestnut runner instead of on a hosted runner with tinygrad''s qcom / amd mock'
required: false
type: boolean
default: false
hf_repo:
description: 'Hugging Face dataset repository (e.g. sunnypilot/sunnypilot_models_v1)'
required: false
@@ -72,11 +62,6 @@ on:
description: 'Custom name for the model (no date, only name)'
required: false
type: string
from_sunnypilot:
description: 'The ref is on sunnypilot/sunnypilot (custom models): skip trying commaai/openpilot first'
required: false
type: boolean
default: true
is_20hz:
description: 'Is this a 20Hz model'
required: false
@@ -122,11 +107,6 @@ on:
options:
- qcom
- chestnut
on_device:
description: 'Compile on the tici / chestnut runner instead of on a hosted runner with tinygrad''s qcom / amd mock'
required: false
type: boolean
default: false
hf_repo:
description: 'Hugging Face dataset repository'
required: false
@@ -147,11 +127,9 @@ jobs:
with:
upstream_branch: ${{ inputs.upstream_branch }}
custom_name: ${{ inputs.custom_name || inputs.upstream_branch }}
from_sunnypilot: ${{ inputs.from_sunnypilot || false }}
is_20hz: ${{ inputs.is_20hz }}
artifact_suffix: ${{ inputs.artifact_suffix }}
target_hardware: ${{ inputs.target_hardware }}
on_device: ${{ inputs.on_device || false }}
secrets: inherit
publish_model:
@@ -1,61 +0,0 @@
name: Chestnut mock build parity
description: >-
Convert one commaai precompiled chestnut model (LS2) on this runner under tinygrad's amd mock, as a hosted chestnut build does, and require the
exact bytes the chestnut runner built (the published artifact). Needs setup-amd-mock, numpy and zstandard; tinygrad_repo checked out.
runs:
using: composite
steps:
- name: Key this check by everything its result depends on
id: key
shell: bash
env:
SOURCES: ${{ hashFiles('tinygrad_repo/tinygrad/**/*.py', 'tinygrad_repo/test/mockgpu/**', 'openpilot/sunnypilot/modeld_v2/*.py', '.github/workflows/chestnut-build-parity/action.yml') }}
run: echo "key=chestnut-build-parity-$SOURCES-$(sha256sum "$LLVM_PATH" | cut -c1-16)-$(python3 -c 'import platform; print(platform.python_version())')" >> "$GITHUB_OUTPUT"
- name: Passed before with all of it identical?
id: passed
uses: actions/cache/restore@v4
with:
path: ~/.chestnut_build_parity.passed
key: ${{ steps.key.outputs.key }}
- name: Rebuild LS2 on the mock and require the chestnut runner's artifact byte for byte
if: steps.passed.outputs.cache-hit != 'true'
shell: bash
env:
REF: b92fc4828664f8767bce5e92f1630ed319cbe509 # commaai/openpilot: precompiled big_driving_tinygrad.pkl, pins tinygrad ec91b74b5b
PKL_OID: b4dead3ee091d66ab002a6b8e716c26f4f79e3c931e4be5f48e017ba92f6e960
PKL_SIZE: "767228907"
TG_SHA: ec91b74b5b6b0fc924b72d3f27689679ca2438b9
WANT: 680a8d629785f2b1d1a2b0e40d0dfbcd0f7291c21e20c51259f1d768a22ec217 # driving_light_speed_2_november_04_2026_tinygrad.pkl, recompiled46
run: |
set -euo pipefail
work="$RUNNER_TEMP/chestnut_parity" && rm -rf "$work" && mkdir -p "$work"
href=$(curl -fsSL --retry 3 -X POST -H "Accept: application/vnd.git-lfs+json" -H "Content-Type: application/vnd.git-lfs+json" \
https://huggingface.co/commaai/openpilot-lfs.git/info/lfs/objects/batch \
-d "{\"operation\":\"download\",\"transfer\":[\"basic\"],\"objects\":[{\"oid\":\"$PKL_OID\",\"size\":$PKL_SIZE}]}" | jq -r '.objects[0].actions.download.href')
curl -fsSL --retry 3 -o "$work/big_driving_tinygrad.pkl" "$href"
echo "$PKL_OID $work/big_driving_tinygrad.pkl" | sha256sum -c -
tg="$work/tinygrad" && git init -q "$tg" && git -C "$tg" fetch -q --depth 1 https://github.com/tinygrad/tinygrad "$TG_SHA" && git -C "$tg" checkout -q FETCH_HEAD
CHECK_OOB=0 GMMU=0 PYTHONPATH="$GITHUB_WORKSPACE/tinygrad_repo" python3 tinygrad_repo/test/mockgpu/usb_server.py \
env DEV=USB+AMD PYTHONPATH="$tg:$GITHUB_WORKSPACE" python3 openpilot/sunnypilot/modeld_v2/convert_upstream_pkl.py "$work/big_driving_tinygrad.pkl" "$work/LS2.pkl"
got=$(sha256sum "$work/LS2.pkl" | cut -d' ' -f1)
if [ "$got" != "$WANT" ]; then
echo "::error::the mock's LS2 build ($got) is not the chestnut runner's ($WANT): this runner's LLVM or the mock no longer builds what the chestnut does"
exit 1
fi
echo "LS2 rebuilt on the mock: byte-identical to the chestnut runner's build"
touch ~/.chestnut_build_parity.passed
- name: Remember the pass
if: steps.passed.outputs.cache-hit != 'true'
uses: actions/cache/save@v4
with:
path: ~/.chestnut_build_parity.passed
key: ${{ steps.key.outputs.key }}
- name: Already passed
if: steps.passed.outputs.cache-hit == 'true'
shell: bash
run: |
echo "LS2 rebuild skipped: it already matched the chestnut runner's build with this exact mock, converter, LLVM and python"
@@ -1,60 +0,0 @@
name: Set up tinygrad's amd mock
description: LLVM 20 and ROCm comgr, as tinygrad_repo/.github/actions/setup-tinygrad installs them for the mock gpu
inputs:
emu_cache:
description: >-
Keep tinygrad's cache db (the emulator's compiled kernels and gate results) between runs under this name, e.g. the model's file name.
Empty: no cache.
default: ''
runs:
using: composite
steps:
- name: Install LLVM 20 and ROCm comgr
shell: bash
run: |
. /etc/os-release
curl -fsSL https://apt.llvm.org/llvm-snapshot.gpg.key | sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc > /dev/null
echo "deb http://apt.llvm.org/$VERSION_CODENAME/ llvm-toolchain-$VERSION_CODENAME-20 main" | sudo tee /etc/apt/sources.list.d/llvm.list
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://repo.radeon.com/rocm/rocm.gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/7.1 $VERSION_CODENAME main" | sudo tee /etc/apt/sources.list.d/rocm.list
printf 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600\n' | sudo tee /etc/apt/preferences.d/rocm-pin-600
sudo apt-get -qq update && sudo apt-get -qq install -y --no-install-recommends libllvm20 clang-20 lld-20 comgr
printf '/opt/rocm/lib\n/opt/rocm/lib64\n' | sudo tee /etc/ld.so.conf.d/rocm.conf && sudo ldconfig
echo "/usr/lib/llvm-20/bin" >> "$GITHUB_PATH"
llvm=$(ls /usr/lib/llvm-20/lib/libLLVM*.so* /usr/lib/*-linux-gnu/libLLVM-20.so* 2>/dev/null | head -1 || true)
[ -n "$llvm" ] || { echo "::error::libLLVM 20 not found"; exit 1; }
echo "LLVM_PATH=$llvm" >> "$GITHUB_ENV"
- name: Cache the chestnut's firmware
uses: actions/cache@v4
with:
path: ~/.cache/tinygrad/downloads/fw
key: chestnut-fw-${{ hashFiles('tinygrad_repo/tinygrad/runtime/autogen/am/fw.py') }}
- name: Fetch the chestnut's firmware (gitlab times out now and then; the mock boots from it)
shell: bash
run: |
for i in 1 2 3 4 5; do
PYTHONPATH="$GITHUB_WORKSPACE/tinygrad_repo" python3 -c "
from tinygrad.helpers import fetch_fw
from tinygrad.runtime.autogen.am import fw
for n, h in fw.hashes.items():
if n.startswith(('psp_14_0_2_', 'smu_14_0_2.', 'sdma_7_0_0.', 'gc_12_0_0_')): fetch_fw('amdgpu', n, h)" && exit 0
sleep $((i * 10))
done
echo "::error::could not fetch the chestnut's firmware from gitlab"; exit 1
- name: runner cpu
if: inputs.emu_cache != ''
id: cpu
shell: bash
run: echo "cpu=$( (grep -m1 '^model name' /proc/cpuinfo; grep -m1 '^flags' /proc/cpuinfo) | sha256sum | cut -c1-16)" >> "$GITHUB_OUTPUT"
- name: Keep the emulator's compiled kernels between runs
if: inputs.emu_cache != ''
uses: actions/cache@v4
with:
path: ~/.cache/tinygrad/cache.db
key: emu-${{ steps.cpu.outputs.cpu }}-${{ hashFiles('tinygrad_repo/tinygrad/**/*.py', 'tinygrad_repo/test/mockgpu/**') }}-${{ inputs.emu_cache }}
restore-keys: emu-${{ steps.cpu.outputs.cpu }}-${{ hashFiles('tinygrad_repo/tinygrad/**/*.py', 'tinygrad_repo/test/mockgpu/**') }}-
+17 -133
View File
@@ -20,11 +20,6 @@ on:
description: 'Custom name for the model (no date, only name)'
required: false
type: string
from_sunnypilot:
description: 'The ref is on sunnypilot/sunnypilot (custom models): skip trying commaai/openpilot first'
required: false
type: boolean
default: false
is_20hz:
description: 'Is this a 20Hz model'
required: false
@@ -40,11 +35,6 @@ on:
required: false
type: string
default: 'qcom'
on_device:
description: 'Compile on the tici / chestnut runner instead of on a hosted runner with tinygrad''s qcom / amd mock'
required: false
type: boolean
default: false
workflow_dispatch:
inputs:
upstream_branch:
@@ -56,11 +46,6 @@ on:
description: 'Custom name for the model (no date, only name)'
required: false
type: string
from_sunnypilot:
description: 'The ref is on sunnypilot/sunnypilot (custom models): skip trying commaai/openpilot first'
required: false
type: boolean
default: true
is_20hz:
description: 'Is this a 20Hz model'
required: false
@@ -74,11 +59,6 @@ on:
- qcom
- chestnut
default: 'qcom'
on_device:
description: 'Compile on the tici / chestnut runner instead of on a hosted runner with tinygrad''s qcom / amd mock'
required: false
type: boolean
default: false
run-name: Build model [${{ inputs.custom_name || inputs.upstream_branch }}] from ref [${{ inputs.upstream_branch }}]
@@ -94,7 +74,6 @@ jobs:
# Note: To allow dynamic models from both openpilot and sunnypilot (merges/mashups), we try commaai as default,
# and fallback to sunnypilot if the ref checkout fails.
- name: Checkout commaai/openpilot
if: ${{ !inputs.from_sunnypilot }}
id: checkout_upstream
continue-on-error: true
uses: actions/checkout@v4
@@ -106,7 +85,7 @@ jobs:
path: openpilot
- name: Fallback to sunnypilot/sunnypilot
if: inputs.from_sunnypilot || steps.checkout_upstream.outcome == 'failure'
if: steps.checkout_upstream.outcome == 'failure'
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot
@@ -131,14 +110,8 @@ jobs:
for f in "$@"; do
grep -qs "^version https://git-lfs" "$f" || continue
OID=$(awk -F'sha256:' '/^oid/{print $2}' "$f"); SIZE=$(awk '/^size/{print $2}' "$f")
HREF="https://huggingface.co/buckets/commaai/openpilot-lfs/resolve/sha256/$OID" # lfs.py's bucket (commaai/openpilot#39206)
curl -fsSL --retry 3 -r 0-0 -o /dev/null "$HREF" || HREF=$(curl -fsSL --retry 3 -X POST -H "Accept: application/vnd.git-lfs+json" -H "Content-Type: application/vnd.git-lfs+json" "$LFS_URL/objects/batch" \
-d "{\"operation\":\"download\",\"transfer\":[\"basic\"],\"objects\":[{\"oid\":\"$OID\",\"size\":$SIZE}]}" | jq -r '.objects[0].actions.download.href // empty')
if [ -z "$HREF" ]; then # not in openpilot's LFS: comma also publishes models to openpilot_driving_models
HF_PATH=$(curl -fsSL --retry 3 "https://huggingface.co/api/models/commaai/openpilot_driving_models/tree/main?recursive=true" | jq -r --arg oid "$OID" 'map(select(.lfs.oid == $oid))[0].path // empty')
[ -n "$HF_PATH" ] || { echo "::error::$f ($OID) is in neither openpilot's LFS nor commaai/openpilot_driving_models"; exit 1; }
HREF="https://huggingface.co/commaai/openpilot_driving_models/resolve/main/$HF_PATH"
fi
HREF=$(curl -fsSL --retry 3 -X POST -H "Accept: application/vnd.git-lfs+json" -H "Content-Type: application/vnd.git-lfs+json" "$LFS_URL/objects/batch" \
-d "{\"operation\":\"download\",\"transfer\":[\"basic\"],\"objects\":[{\"oid\":\"$OID\",\"size\":$SIZE}]}" | jq -r '.objects[0].actions.download.href')
aria2c -x 16 -s 16 -k 16M --max-tries=5 --retry-wait=5 --file-allocation=none --summary-interval=30 --allow-overwrite=true \
-d "$(dirname "$f")" -o "$(basename "$f")" "$HREF"
echo "$OID $f" | sha256sum -c -
@@ -146,9 +119,8 @@ jobs:
}
UPSTREAM_PKL=selfdrive/modeld/models/big_driving_tinygrad.pkl
if [ "${{ inputs.target_hardware }}" != "chestnut" ]; then
git lfs pull -X "**/selfdrive/modeld/models/big_*.onnx,**/selfdrive/modeld/models/dmonitoring_*.onnx,**/selfdrive/modeld/models/big_*.pkl,**/selfdrive/modeld/models/dmonitoring_*.pkl" || true
git lfs pull -X "**/selfdrive/modeld/models/big_*.onnx,**/selfdrive/modeld/models/dmonitoring_*.onnx,**/selfdrive/modeld/models/big_*.pkl,**/selfdrive/modeld/models/dmonitoring_*.pkl"
rm -f selfdrive/modeld/models/big_*.onnx selfdrive/modeld/models/dmonitoring_*.onnx selfdrive/modeld/models/big_*.pkl selfdrive/modeld/models/dmonitoring_*.pkl
fetch_lfs selfdrive/modeld/models/*.onnx
elif compgen -G "selfdrive/modeld/models/big_*.onnx" > /dev/null; then
fetch_lfs selfdrive/modeld/models/big_*.onnx selfdrive/modeld/models/big_*.pkl
find selfdrive/modeld/models -type f \( -name "*.onnx" -o -name "*.pkl" \) ! -name "big_*.onnx" ! -name "big_*.pkl" -delete
@@ -181,89 +153,18 @@ jobs:
if-no-files-found: error
build_model:
runs-on: ${{ inputs.on_device && fromJSON(inputs.target_hardware == 'chestnut' && '["self-hosted","chestnut"]' || '["self-hosted","tici"]') || inputs.target_hardware == 'chestnut' && 'ubuntu-24.04' || 'ubuntu-24.04-arm' }}
runs-on: [self-hosted, "${{ inputs.target_hardware == 'chestnut' && 'chestnut' || 'tici' }}"]
needs: get_model
env:
MODEL_NAME: ${{ inputs.custom_name || inputs.upstream_branch }} (${{ needs.get_model.outputs.model_date }})
REF: ${{ inputs.upstream_branch }}
MOCK_BUILD: ${{ !inputs.on_device }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
submodules: recursive
- name: Set up the hosted mock build
if: env.MOCK_BUILD == 'true'
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install LLVM 20 and ROCm comgr for the amd mock
if: env.MOCK_BUILD == 'true' && inputs.target_hardware == 'chestnut'
uses: ./.github/workflows/setup-amd-mock
with:
emu_cache: build
- name: Install the amd mock build dependencies
if: env.MOCK_BUILD == 'true' && inputs.target_hardware == 'chestnut'
run: |
python3 -m pip install -q numpy zstandard
- name: Install the qcom mock build dependencies
if: env.MOCK_BUILD == 'true' && inputs.target_hardware != 'chestnut'
id: qcom_deps
run: |
python3 -m pip install -q numpy zstandard tinymesa==25.2.7.2
if [ "$RUNNER_ARCH" = ARM64 ]; then # the qcomcl rootfs's compiler loads in this python, no qemu
sudo apt-get -qq update && sudo apt-get -qq install -y --no-install-recommends aria2
FS=$(PYTHONPATH="$TINYGRAD_PATH" python3 -c "from tinygrad.helpers import fetch; print(fetch('https://git.tinygrad.win/tinygrad/images/releases/download/v2/qcomcl.tar.gz', extract=True))")
echo "LLVM_QCOM_PATH=$FS/usr/lib/libllvm-qcom.so" >> "$GITHUB_ENV"
echo "compiler=native-$(sha256sum "$FS/usr/lib/libllvm-qcom.so" | cut -c1-16)" >> "$GITHUB_OUTPUT"
else
sudo apt-get -qq update && sudo apt-get -qq install -y --no-install-recommends qemu-user-static aria2
echo "compiler=qemu-$(dpkg-query -W -f='${Version}' qemu-user-static)" >> "$GITHUB_OUTPUT"
fi
- name: Compiler check passed before with all of it identical?
if: env.MOCK_BUILD == 'true' && inputs.target_hardware != 'chestnut'
id: qcom_compiler
env:
KEY: qcom-compiler-parity-${{ steps.qcom_deps.outputs.compiler }}-${{ hashFiles('tinygrad_repo/test/device/qcom/compiler_parity.py', 'tinygrad_repo/test/mockgpu/qcom/a630_compiled.pkl.xz', 'tinygrad_repo/tinygrad/runtime/support/compiler_qcom.py') }}
run: |
MARKER="ci/qcom-compiler-parity/$(printf '%s' "$KEY" | tr -c 'A-Za-z0-9._-' '_').txt"
CODE=$(curl -s -o /dev/null -w '%{http_code}' -I -L "https://huggingface.co/datasets/sunnypilot/sunnypilot_models_v1/resolve/main/$MARKER")
echo "$MARKER: HTTP $CODE"
{ echo "key=$KEY"; echo "marker=$MARKER"; echo "passed=$([ "$CODE" = 200 ] && echo true || echo false)"; } >> "$GITHUB_OUTPUT"
- name: Check the off-device qcom compiler builds the a630's bytes
if: env.MOCK_BUILD == 'true' && inputs.target_hardware != 'chestnut' && steps.qcom_compiler.outputs.passed != 'true'
run: |
# every kernel an a630 has compiled must compile to the same bytes here before a model is built here
PYTHONPATH="$TINYGRAD_PATH" python3 "$TINYGRAD_PATH/test/device/qcom/compiler_parity.py" check "$TINYGRAD_PATH/test/mockgpu/qcom/a630_compiled.pkl.xz"
- name: Remember the compiler check passed
if: env.MOCK_BUILD == 'true' && inputs.target_hardware != 'chestnut' && steps.qcom_compiler.outputs.passed != 'true'
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
KEY: ${{ steps.qcom_compiler.outputs.key }}
MARKER: ${{ steps.qcom_compiler.outputs.marker }}
run: |
python3 -m pip install -q --upgrade "huggingface_hub>=0.34"
printf '%s\n%s\n' "$KEY" "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" > "$RUNNER_TEMP/marker.txt"
hf upload --repo-type dataset sunnypilot/sunnypilot_models_v1 "$RUNNER_TEMP/marker.txt" "$MARKER" \
--commit-message "qcom compiler parity passed: $KEY" || echo "::warning::could not record the pass, the next build checks again"
- name: Kernels compiled by earlier builds
if: env.MOCK_BUILD == 'true' && inputs.target_hardware != 'chestnut'
uses: actions/cache@v4
with:
path: ~/.cache/tinygrad/cache.db
key: qcom-kernels-${{ steps.qcom_compiler.outputs.key }}-${{ github.run_id }}
restore-keys: qcom-kernels-${{ steps.qcom_compiler.outputs.key }}-
- name: Set environment variables
if: env.MOCK_BUILD != 'true'
id: set-env
run: |
# Set up common environment
@@ -276,7 +177,6 @@ jobs:
printenv >> $GITHUB_ENV
- name: Setup build environment
if: env.MOCK_BUILD != 'true'
run: |
mkdir -p "${BUILD_DIR}/"
sudo find $BUILD_DIR/ -mindepth 1 -delete
@@ -298,28 +198,21 @@ jobs:
- name: Build Model
run: |
if [ "$MOCK_BUILD" = "true" ]; then
TASKSET=""
export CHECK_OOB=0 GMMU=0 # as test/mockgpu runs the amd mock
else
source /etc/profile
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
TASKSET="taskset -c 7"
fi
source /etc/profile
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH="${PYTHONPATH}:${{ env.TINYGRAD_PATH }}:${{ github.workspace }}"
COMPILE_MODELD="${{ github.workspace }}/openpilot/sunnypilot/modeld_v2/compile_modeld.py"
MODEL_SIZE=$(python3 -c "from openpilot.common.transformations.model import MEDMODEL_INPUT_SIZE as s; print(f'{s[0]}x{s[1]}')")
CAMERA_RES=$(python3 -c "from openpilot.common.transformations.camera import _ar_ox_fisheye as a, _os_fisheye as o; print(f'{a.width}x{a.height} {o.width}x{o.height}')")
MOCK=$([ "$MOCK_BUILD" = "true" ] && echo MOCK || true)
TG_FLAGS_QCOM="DEV=${MOCK:+MOCK+}QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 OPENPILOT_HACKS=1"
TG_FLAGS_QCOM="DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 OPENPILOT_HACKS=1"
if [ "${{ inputs.target_hardware }}" == "chestnut" ]; then
echo "CHESTNUT build"
export CHESTNUT=1
TG_FLAGS="DEBUG=1 DEV=${MOCK}USB+AMD:LLVM FRAME_DEV=CPU FLOAT16=1 GMMU=0 TC_OPT=2 TC_MIN_GLOBALS=32"
TG_FLAGS="DEBUG=1 DEV=USB+AMD:LLVM FRAME_DEV=CPU FLOAT16=1 GMMU=0 TC_OPT=2 TC_MIN_GLOBALS=32"
OUTPUT_PKL="${{ env.MODELS_DIR }}/big_driving_tinygrad.pkl"
else
echo "QCOM build"
@@ -334,17 +227,15 @@ jobs:
echo "Converting commaai's precompiled model with tinygrad $TG_SHA"
UPSTREAM_TG="${{ runner.temp }}/upstream_tinygrad"
rm -rf "$UPSTREAM_TG" && git init -q "$UPSTREAM_TG" && git -C "$UPSTREAM_TG" fetch -q --depth 1 "$TG_URL" "$TG_SHA" && git -C "$UPSTREAM_TG" checkout -q FETCH_HEAD
ON_MOCK=""
if [ "$MOCK_BUILD" = "true" ]; then ON_MOCK="env PYTHONPATH=$TINYGRAD_PATH python3 $TINYGRAD_PATH/test/mockgpu/usb_server.py"; fi
CONVERTED="${{ runner.temp }}/big_driving_tinygrad_plain.pkl"
$ON_MOCK env DEV=USB+AMD PYTHONPATH="$UPSTREAM_TG:${{ github.workspace }}" $TASKSET python3 openpilot/sunnypilot/modeld_v2/convert_upstream_pkl.py \
env DEV=USB+AMD PYTHONPATH="$UPSTREAM_TG:${{ github.workspace }}" taskset -c 7 python3 openpilot/sunnypilot/modeld_v2/convert_upstream_pkl.py \
"${{ env.MODELS_DIR }}/big_driving_tinygrad.pkl" "$CONVERTED"
mv "$CONVERTED" "$OUTPUT_PKL"
elif NEW_ARCH_ONNX=$(find "${{ env.MODELS_DIR }}" -maxdepth 1 -name "${{ inputs.target_hardware == 'chestnut' && 'big_' || '' }}*driving*.onnx" -print -quit) &&
[ -n "$NEW_ARCH_ONNX" ] && grep -qa new_img "$NEW_ARCH_ONNX"; then # the new architecture: one ONNX taking the warped frames as new_img
echo "Compiling new architecture ONNX: $NEW_ARCH_ONNX"
env ${TG_FLAGS} $TASKSET python3 openpilot/sunnypilot/modeld_v2/compile_modeld_v2.py \
"$NEW_ARCH_ONNX" "$OUTPUT_PKL" --device-input "*" --benchmark-runs $([ "$MOCK_BUILD" = "true" ] && echo 0 || echo 1)
env ${TG_FLAGS} taskset -c 7 python3 openpilot/sunnypilot/modeld_v2/compile_modeld_v2.py \
"$NEW_ARCH_ONNX" "$OUTPUT_PKL" --device-input "*" --benchmark-runs 1
else
echo "Detected onnx architecture. Routing to native compile_modeld.py."
# Detect model type and build compile args
@@ -393,7 +284,7 @@ jobs:
if [ -n "$MODEL_TYPE" ]; then
echo "Detected: $MODEL_TYPE -> $OUTPUT_PKL"
env ${TG_FLAGS} $TASKSET python3 "$COMPILE_MODELD" \
env ${TG_FLAGS} taskset -c 7 python3 "$COMPILE_MODELD" \
--model-type $MODEL_TYPE \
--model-size $MODEL_SIZE \
--camera-resolutions $CAMERA_RES \
@@ -402,13 +293,6 @@ jobs:
fi
fi
- name: Replay its check frame on the mock and require the result bit for bit
if: env.MOCK_BUILD == 'true'
env:
DEV: ${{ inputs.target_hardware == 'chestnut' && 'MOCKUSB+AMD' || 'MOCK+QCOM' }}
PYTHONPATH: ${{ env.TINYGRAD_PATH }}:${{ github.workspace }}
run: python3 -m openpilot.sunnypilot.modeld_v2.mock_parity check "${{ env.MODELS_DIR }}/${{ inputs.target_hardware == 'chestnut' && 'big_' || '' }}driving_tinygrad.pkl"
- name: Prepare Output
run: |
sudo rm -rf ${{ env.OUTPUT_DIR }}
@@ -420,7 +304,7 @@ jobs:
--include='/${{ inputs.target_hardware == 'chestnut' && 'big_' || '' }}driving_tinygrad.pkl' \
--exclude='*' \
--delete-excluded \
$([ "$MOCK_BUILD" = "true" ] || echo --chown=comma:comma) \
--chown=comma:comma \
${{ env.MODELS_DIR }}/ ${{ env.OUTPUT_DIR }}/
python3 "${{ github.workspace }}/release/ci/model_generator.py" \
@@ -449,7 +333,7 @@ jobs:
path: ${{ env.OUTPUT_DIR }}/artifact_name.txt
- name: Re-enable powersave
if: always() && env.MOCK_BUILD != 'true'
if: always()
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
+13 -88
View File
@@ -28,7 +28,18 @@ jobs:
env:
DEV: CPU
run: tools/op.sh test -v openpilot/sunnypilot/modeld_v2/tests/test_plain_artifact.py
- uses: ./.github/workflows/setup-amd-mock
- name: Install LLVM 20 and ROCm comgr for the amd mock, as tinygrad_repo/.github/actions/setup-tinygrad does
run: |
. /etc/os-release
curl -fsSL https://apt.llvm.org/llvm-snapshot.gpg.key | sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc > /dev/null
echo "deb http://apt.llvm.org/$VERSION_CODENAME/ llvm-toolchain-$VERSION_CODENAME-20 main" | sudo tee /etc/apt/sources.list.d/llvm.list
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://repo.radeon.com/rocm/rocm.gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/7.1 $VERSION_CODENAME main" | sudo tee /etc/apt/sources.list.d/rocm.list
printf 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600\n' | sudo tee /etc/apt/preferences.d/rocm-pin-600
sudo apt-get -qq update && sudo apt-get -qq install -y --no-install-recommends libllvm20 clang-20 lld-20 comgr
printf '/opt/rocm/lib\n/opt/rocm/lib64\n' | sudo tee /etc/ld.so.conf.d/rocm.conf && sudo ldconfig
echo "/usr/lib/llvm-20/bin" >> "$GITHUB_PATH"
- name: Convert a pickled USB+AMD jit and replay it under tinygrad's amd mock
env:
DEV: MOCKUSB+AMD
@@ -69,8 +80,6 @@ jobs:
runs-on: ubuntu-latest
outputs:
models: ${{ steps.set-matrix.outputs.models }}
chestnut: ${{ steps.set-matrix.outputs.chestnut }}
qcom: ${{ steps.set-matrix.outputs.qcom }}
steps:
- uses: actions/checkout@v4
- name: Fetch and Parse json
@@ -82,22 +91,16 @@ jobs:
with open("openpilot/sunnypilot/models/fetcher.py", "r") as f:
urls = re.findall(r"MODEL_URL(?:_CHESTNUT)?\s*=\s*[\"'"'"']([^\"'"'"']+)[\"'"'"']", f.read())
artifacts, chestnut, qcom = [], [], []
artifacts = []
for url in urls:
data = json.loads(urllib.request.urlopen(url).read())
for bundle in data.get("bundles", []):
for model in bundle.get("models", []):
if "artifact" in model:
artifacts.append(model["artifact"])
if "chestnut" in url:
chestnut.append(model["artifact"])
elif "tinygrad" in model["artifact"]["file_name"]:
qcom.append(model["artifact"])
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
f.write(f"models={json.dumps(artifacts)}\n")
f.write(f"chestnut={json.dumps(chestnut)}\n")
f.write(f"qcom={json.dumps(qcom)}\n")
'
test-model:
@@ -134,81 +137,3 @@ jobs:
export PYTHONPATH="$GITHUB_WORKSPACE/tinygrad_repo:$GITHUB_WORKSPACE:$PYTHONPATH"
python3 -m pip uninstall -y tinygrad || uv pip uninstall -y tinygrad --system || true
python3 -m pytest openpilot/sunnypilot/modeld_v2/tests/test_models.py
chestnut-mock-parity:
name: Mock parity ${{ matrix.artifact.file_name }}
needs: generate-matrix
runs-on: ubuntu-24.04
strategy:
fail-fast: false
matrix:
artifact: ${{ fromJson(needs.generate-matrix.outputs.chestnut) }}
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: ./.github/workflows/setup-amd-mock
with:
emu_cache: ${{ matrix.artifact.file_name }}
- run: python3 -m pip install -q numpy
- name: Download model
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
MODEL_URL: ${{ matrix.artifact.download_uri.url }}
MODEL_SHA: ${{ matrix.artifact.download_uri.sha256 }}
shell: bash
run: |
AUTH=()
if [ -n "$HF_TOKEN" ]; then AUTH=(-H "Authorization: Bearer $HF_TOKEN"); fi
curl --retry 3 --retry-all-errors -s -S -f -L "${AUTH[@]}" -o "$RUNNER_TEMP/model.pkl" "$MODEL_URL"
if [ -n "$MODEL_SHA" ]; then echo "$MODEL_SHA $RUNNER_TEMP/model.pkl" | sha256sum -c -; fi
- name: Replay its check frame on the amd mock and require the chestnut's result bit for bit
env:
PYTHONPATH: ${{ github.workspace }}/tinygrad_repo:${{ github.workspace }}
run: python3 -m openpilot.sunnypilot.modeld_v2.mock_parity check "$RUNNER_TEMP/model.pkl"
qcom-mock-parity:
name: QCOM mock parity ${{ matrix.artifact.file_name }}
needs: generate-matrix
runs-on: ubuntu-24.04
strategy:
fail-fast: false
matrix:
artifact: ${{ fromJson(needs.generate-matrix.outputs.qcom) }}
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: python3 -m pip install -q numpy tinymesa==25.2.7.2
- name: Download model
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
MODEL_URL: ${{ matrix.artifact.download_uri.url }}
MODEL_SHA: ${{ matrix.artifact.download_uri.sha256 }}
shell: bash
run: |
AUTH=()
if [ -n "$HF_TOKEN" ]; then AUTH=(-H "Authorization: Bearer $HF_TOKEN"); fi
curl --retry 3 --retry-all-errors -s -S -f -L "${AUTH[@]}" -o "$RUNNER_TEMP/model.pkl" "$MODEL_URL"
if [ -n "$MODEL_SHA" ]; then echo "$MODEL_SHA $RUNNER_TEMP/model.pkl" | sha256sum -c -; fi
- name: Replay its check frame on the qcom mock and require the comma device's result bit for bit
env:
DEV: MOCK+QCOM
PYTHONPATH: ${{ github.workspace }}/tinygrad_repo:${{ github.workspace }}
run: python3 -m openpilot.sunnypilot.modeld_v2.mock_parity check "$RUNNER_TEMP/model.pkl"
chestnut-mock-build-parity:
name: Chestnut build on the amd mock = chestnut runner build
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: ./.github/workflows/setup-amd-mock
with:
emu_cache: chestnut-build-parity
- run: python3 -m pip install -q numpy zstandard
- uses: ./.github/workflows/chestnut-build-parity
+10 -3
View File
@@ -3,7 +3,7 @@ from collections.abc import Callable
import base64
import ctypes
import gc
from functools import cached_property
from functools import cached_property, partial
import os
os.environ['GMMU'] = '0' # for chestnut fast loading, noop for qcom
os.environ.setdefault('AM_POWER_LIMIT', '100')
@@ -43,6 +43,7 @@ from openpilot.selfdrive.modeld.constants import ModelConstants, Plan
from openpilot.selfdrive.modeld.helpers import MODELS_DIR, chestnut_present, chestnut_compiled, modeld_pkl_path, load_oob, wait_for_chestnut
from openpilot.sunnypilot.livedelay.helpers import get_lat_delay
from openpilot.sunnypilot.modeld_v2.modeld_base import ModelStateBase
from openpilot.sunnypilot.modeld_v2.turn_planner import TurnPlanner
from openpilot.sunnypilot.selfdrive.controls.lib.relc import RoadEdgeLaneChangeController
SEND_RAW_PRED = os.getenv('SEND_RAW_PRED')
@@ -54,7 +55,8 @@ BIG_MODEL_TIMEOUT = 60
def get_action_from_model(model_output: dict[str, np.ndarray], prev_action: log.ModelDataV2.Action,
lat_action_t: float, long_action_t: float, v_ego: float) -> log.ModelDataV2.Action:
lat_action_t: float, long_action_t: float, v_ego: float,
turn_shaper: Callable[[float], float] | None = None) -> log.ModelDataV2.Action:
if 'action' not in model_output:
plan = model_output['plan'][0]
desired_accel = get_accel_from_plan(plan[:,Plan.VELOCITY][:,0],
@@ -69,6 +71,8 @@ def get_action_from_model(model_output: dict[str, np.ndarray], prev_action: log.
else:
desired_accel = model_output['action'][0,1]
desired_curvature = model_output['action'][0,0] / (max(1.0, v_ego))**2
if turn_shaper is not None:
desired_curvature = turn_shaper(desired_curvature)
stop = should_stop(v_ego, desired_accel)
desired_accel = smooth_value(desired_accel, prev_action.desiredAcceleration, LONG_SMOOTH_SECONDS)
if v_ego > MIN_LAT_CONTROL_SPEED:
@@ -340,6 +344,7 @@ def main(demo=False):
DH = DesireHelper()
RELC = RoadEdgeLaneChangeController()
turn_planner = TurnPlanner(CP.wheelbase)
while True:
# Keep receiving frames until we are at least 1 frame ahead of previous extra frame
@@ -448,7 +453,9 @@ def main(demo=False):
if not sm['carControl'].latActive:
prev_action.desiredCurvature = sm['controlsState'].curvature
action = get_action_from_model(model_output, prev_action, lat_action_t, long_action_t, v_ego)
action = get_action_from_model(model_output, prev_action, lat_action_t, long_action_t, v_ego,
partial(turn_planner.update, model_output, v_ego, sm['controlsState'].curvature,
sm['carControl'].latActive, is_rhd))
prev_action = action
fill_model_msg(modelv2_send, model_output, action,
publish_state, meta_main.frame_id, meta_extra.frame_id, frame_id,
@@ -267,8 +267,8 @@ class ModelsLayoutMici(NavScroller):
self.current_model_info.info_text.set_text(info_text)
if manager.selectedBundle and manager.selectedBundle.status == custom.ModelManagerSP.DownloadStatus.failed:
self.current_model_info.info_header.set_text(tr("error") + self._download_progress)
self.current_model_info.info_text.set_text(tr("download failed"))
self.current_model_info.info_header.set_text(tr("error"))
self.current_model_info.info_text.set_text(f"{manager.selectedBundle.internalName.lower()} | {tr('download failed')}")
elif manager.selectedBundle and manager.selectedBundle.status == custom.ModelManagerSP.DownloadStatus.downloading:
self.cancel_download_btn.set_visible(ui_state.params.get("ModelManager_DownloadRef") is not None)
@@ -371,7 +371,7 @@ def _compile_warp_resolution_worker(cam_w, cam_h, model_w, model_h, benchmark_ru
if __name__ == "__main__":
if ('USB' in os.getenv('DEV', '') or os.getenv('CHESTNUT')) and not os.getenv('DEV', '').startswith('MOCK'):
if 'USB' in os.getenv('DEV', '') or os.getenv('CHESTNUT'):
from openpilot.system.hardware.chestnut.flash import link_up
for _ in range(10):
if link_up():
@@ -52,8 +52,8 @@ if __name__ == "__main__":
benchmark(jit, cb=lambda t, i=i: print(f" [{i + 1}/{2 + args.benchmark_runs}] {t * 1e3:.2f} ms"), **check_inputs)
assert len(recorded) == 1, f"expected one captured kernel list, got {len(recorded)}"
frame, _ = capture_frame(jit, check_inputs, 'output_buffers', run=False)
frame, _ = capture_frame(jit, check_inputs, 'output_buffers')
part = serialize_jit(jit, recorded[0], frame)
PlainJit(part[0], memoryview(part[1]), verify=not os.environ.get('DEV', '').startswith('MOCK')) # mock_parity checks a mock build bit for bit
PlainJit(part[0], memoryview(part[1]))
write_bundle(args.output, {'run': part}, {'metadata': metadata, 'input_specs': input_specs, 'output_specs': output_specs})
print(f"Saved model to {args.output} ({os.path.getsize(args.output) / 1e6:.2f} MB, {len(part[0]['calls'])} kernels)")
@@ -10,7 +10,6 @@ import dataclasses
import inspect
import itertools
import mmap
import os
import re
import struct
import tempfile
@@ -195,8 +194,7 @@ def convert(jit, kwargs: dict, frame: dict, target) -> tuple[dict, memoryview]:
base, at, nbytes = _address(argument_words[slot])
args.append({"buffer": buffers.setdefault(base, len(buffers)), "offset": at, "nbytes": nbytes or base.max_numel() * base.dtype.itemsize, "dtype": u8})
outs, ins = kernel[6]
device_target = dataclasses.asdict(dataclasses.replace(target, interface=target.interface.removeprefix("MOCK")))
calls.append({"kind": "program", "name": re.sub(r"\x1b\[[0-9;]*m", "", kernel[1]), "lib": programs[image_alloc], "target": device_target,
calls.append({"kind": "program", "name": re.sub(r"\x1b\[[0-9;]*m", "", kernel[1]), "lib": programs[image_alloc], "target": dataclasses.asdict(target),
"signature": [[f"data{i}", i, u8, [a.get("nbytes", 1)]] for i, a in enumerate(args)], "global_size": grid, "local_size": local,
"globals": list(range(len(args))), "outs": list(outs), "ins": list(ins), "args": args})
needs_init |= {args[i]["buffer"] for i in ins if "buffer" in args[i] and args[i]["buffer"] not in written}
@@ -237,16 +235,14 @@ def main():
frame, _ = capture_frame(artifact["run"], kwargs, "output_buffers")
part = convert(artifact["run"], kwargs, frame, Device[Device.DEFAULT].renderer.target)
if not os.getenv("MOCKUSB_SOCKET"):
plain = PlainJit(*part)
replay = {path[0]: Tensor(array, device=device).realize() for path, array, device in frame["inputs"] if len(path) == 1}
replay["output_buffers"] = {path[1]: Tensor(array, device=device).realize() for path, array, device in frame["inputs"] if len(path) == 2}
flatten(plain(**replay))
for index, (got, expected) in enumerate(zip([t.numpy() for t in replay["output_buffers"].values()], frame["outputs"], strict=True)):
assert np.array_equal(got, expected), f"output {index} of the plain artifact differs from the pickle"
plain = PlainJit(*part) # replays the frame, within tolerance
replay = {path[0]: Tensor(array, device=device).realize() for path, array, device in frame["inputs"] if len(path) == 1}
replay["output_buffers"] = {path[1]: Tensor(array, device=device).realize() for path, array, device in frame["inputs"] if len(path) == 2}
flatten(plain(**replay))
for index, (got, expected) in enumerate(zip([t.numpy() for t in replay["output_buffers"].values()], frame["outputs"], strict=True)):
assert np.array_equal(got, expected), f"output {index} of the plain artifact differs from the pickle"
write_bundle(args.output, {"run": part}, {key: artifact[key] for key in ("metadata", "input_specs", "output_specs")})
checked = "left to mock_parity" if os.getenv("MOCKUSB_SOCKET") else "bit-exact with the pickle"
print(f"converted {len(part[0]['calls'])} kernels, {checked}: {args.output}")
print(f"converted {len(part[0]['calls'])} kernels, bit-exact with the pickle: {args.output}")
if __name__ == "__main__":
@@ -1,153 +0,0 @@
import json
import os
import shutil
import struct
import sys
import tempfile
import time
os.environ.setdefault("DEV", "MOCKUSB+AMD")
QCOM = "QCOM" in os.environ["DEV"]
ON_MOCK = os.environ["DEV"].startswith("MOCK")
ARCH, HARDWARE = ("a630", "comma 3x/4 (a630)") if QCOM else ("gfx1200", "chestnut")
os.environ.setdefault("CHECK_OOB", "0")
os.environ.setdefault("GMMU", "0")
if sys.platform == "darwin" and os.environ["DEV"].startswith("MOCKUSB"): # the mock's VRAM is a memfd, which macOS lacks: an unlinked file is the same
from tinygrad.runtime.autogen import libc
def _memfd_create(name, flags):
fd, path = tempfile.mkstemp()
os.unlink(path)
return fd
libc.memfd_create = _memfd_create
import numpy as np
from openpilot.sunnypilot.modeld_v2.helpers import ALIGNMENT, MAGIC, PlainJit, _device, read_parts
class ParityError(RuntimeError):
pass
def _emulator():
try:
if QCOM:
from test.mockgpu.qcom import emu
else:
from test.mockgpu.amd import emu
except ImportError as e:
raise ParityError(f"tinygrad's {'qcom' if QCOM else 'amd'} mock is not importable: put tinygrad_repo on PYTHONPATH") from e
if not hasattr(emu, "hw_unverified"):
raise ParityError(f"this tinygrad's emulator has no {ARCH} rounding model, its check frames would not match a {HARDWARE}")
return emu
def unverified(header: dict, blob: memoryview) -> dict[str, int]:
emu, out = _emulator(), {}
for call in header["calls"]:
if call["kind"] != "program":
continue
if (arch := call["target"].get("arch", "")).split(",")[0] != ARCH: # IMAGE builds append ",IMAGE_PITCH_ALIGNMENT=64"
out[f"arch={arch}"] = out.get(f"arch={arch}", 0) + 1
continue
for sig, n in emu.hw_unverified(bytes(blob[slice(*call["lib"])])).items():
out[sig] = out.get(sig, 0) + n
return out
def _array(blob: memoryview, stored: dict) -> np.ndarray:
return np.frombuffer(blob[slice(*stored["data"])], dtype=np.dtype(stored["dtype"])).reshape(stored["shape"]).copy()
def replay(header: dict, blob: memoryview, shared: dict) -> list[np.ndarray]:
from tinygrad import Tensor
check = header["check"]
jit = PlainJit(header, blob, None, verify=False, shared=shared)
kwargs: dict = {}
for path, stored, device in check["inputs"]:
tensor = Tensor(_array(blob, stored), device=_device(device, None)).realize()
kwargs[path[0]] = tensor if len(path) == 1 else {**kwargs.get(path[0], {}), path[1]: tensor}
result = jit(**kwargs)
outputs = kwargs[check["output_container"]].values() if check["output_container"] else result
return [t.numpy() for t in (outputs if isinstance(outputs, (list, tuple)) or check["output_container"] else [outputs])]
def _parts_with_offsets(path: str) -> tuple[dict, dict]:
parts, _ = read_parts(path)
with open(path, "rb") as f:
head = f.read(len(MAGIC) + 8)
start = len(MAGIC) + 8 + struct.unpack_from("<Q", head, len(MAGIC))[0]
f.seek(len(MAGIC) + 8)
entries = json.loads(f.read(start - len(MAGIC) - 8))["parts"]
base = start + -start % ALIGNMENT
offsets = {tuple(e["key"]) if isinstance(e["key"], list) else e["key"]: base + e["blob"][0] for e in entries}
return parts, offsets
def _gate(parts: dict) -> None:
missing: dict[str, int] = {}
for header, blob in parts.values():
if "check" not in header:
raise ParityError("artifact has no check frame")
for sig, n in unverified(header, blob).items():
missing[sig] = missing.get(sig, 0) + n
if missing:
raise ParityError(f"kernels use instructions the emulator is not verified against a {HARDWARE} for: " +
", ".join(f"{s} ({n}x)" for s, n in sorted(missing.items(), key=lambda kv: -kv[1])))
def check(path: str) -> None:
parts, _ = read_parts(path)
if ON_MOCK:
_gate(parts)
shared: dict = {}
for key, (header, blob) in parts.items():
got = replay(header, blob, shared)
for i, (g, stored) in enumerate(zip(got, header["check"]["outputs"], strict=True)):
want = _array(blob, stored)
if g.shape != want.shape or not np.array_equal(g.view(np.uint8), want.view(np.uint8)):
n = int(np.count_nonzero(g != want)) if g.shape == want.shape else -1
where = "mock" if ON_MOCK else "device"
raise ParityError(f"part {key} output {i}: the {where}'s check frame differs from the stored one in {n} of {want.size} values")
def record(path: str, out: str) -> None:
parts, offsets = _parts_with_offsets(path)
_gate(parts)
shared, writes = {}, []
for key, (header, blob) in parts.items():
for g, stored in zip(replay(header, blob, shared), header["check"]["outputs"], strict=True):
want = np.dtype(stored["dtype"])
if tuple(g.shape) != tuple(stored["shape"]) or g.dtype != want:
raise ParityError(f"part {key}: replayed output {g.dtype}{g.shape} does not fit the stored {want}{tuple(stored['shape'])}")
writes.append((offsets[key] + stored["data"][0], np.ascontiguousarray(g).tobytes()))
shutil.copyfile(path, out)
with open(out, "r+b") as f:
for offset, data in writes:
f.seek(offset)
f.write(data)
def main(argv: list[str]) -> int:
if len(argv) < 2 or argv[0] not in ("check", "record") or (argv[0] == "record" and len(argv) != 3):
print("usage: mock_parity check ARTIFACT... | record ARTIFACT OUT", file=sys.stderr)
return 2
try:
if argv[0] == "record":
st = time.perf_counter()
record(argv[1], argv[2])
print(f"{argv[2]}: check frame recorded on the mock in {time.perf_counter() - st:.0f}s")
return 0
for path in argv[1:]:
st = time.perf_counter()
check(path)
print(f"{os.path.basename(path)}: bit-identical to its {HARDWARE} check frame on the {'mock' if ON_MOCK else 'device'} ({time.perf_counter() - st:.0f}s)")
return 0
except ParityError as e:
print(f"PARITY FAIL: {e}", file=sys.stderr)
return 1
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+9 -2
View File
@@ -41,6 +41,7 @@ from openpilot.selfdrive.modeld.modeld import ChestnutGpuState
from openpilot.sunnypilot.modeld_v2.fill_model_msg import fill_model_msg, fill_pose_msg, PublishState, get_curvature_from_output
from openpilot.sunnypilot.modeld_v2.parse_model_outputs import Parser
from openpilot.sunnypilot.modeld_v2.blendful import Blendful
from openpilot.sunnypilot.modeld_v2.turn_planner import TurnPlanner
from openpilot.sunnypilot.modeld_v2.constants import ModelConstants, Plan
from openpilot.sunnypilot.modeld_v2.meta_helper import load_meta_constants
from openpilot.sunnypilot.modeld_v2.camera_offset_helper import CameraOffsetHelper
@@ -231,7 +232,8 @@ class ModelState(ModelStateBase):
def get_action_from_model(self, model_output: dict[str, np.ndarray], prev_action: log.ModelDataV2.Action,
lat_action_t: float, long_action_t: float, v_ego: float,
blendful: Callable[[float], float] | None = None) -> log.ModelDataV2.Action:
blendful: Callable[[float], float] | None = None,
turn_shaper: Callable[[float], float] | None = None) -> log.ModelDataV2.Action:
if 'action' not in model_output:
plan = model_output['plan'][0]
desired_accel = get_accel_from_plan(plan[:, Plan.VELOCITY][:, 0], plan[:, Plan.ACCELERATION][:, 0], self.constants.T_IDXS,
@@ -245,6 +247,8 @@ class ModelState(ModelStateBase):
desired_curvature = model_output['action'][0, 0] / (max(2.5, v_ego))**2
if blendful is not None:
desired_curvature += blendful(desired_curvature)
if turn_shaper is not None:
desired_curvature = turn_shaper(desired_curvature)
stop = v_ego < 0.3 and desired_accel < 0.1
desired_accel = smooth_value(desired_accel, prev_action.desiredAcceleration, self.LONG_SMOOTH_SECONDS)
@@ -373,6 +377,7 @@ def main(demo=False):
RELC = RoadEdgeLaneChangeController()
blendful = Blendful()
blendful_enabled = False
turn_planner = TurnPlanner(CP.wheelbase)
while True:
# Keep receiving frames until we are at least 1 frame ahead of previous extra frame
@@ -492,7 +497,9 @@ def main(demo=False):
prev_action.desiredCurvature = sm['controlsState'].curvature
blendful_active = desire == log.Desire.none and DH.lane_change_state == log.LaneChangeState.off
action = model.get_action_from_model(model_output, prev_action, lat_action_t, long_action_t, v_ego,
partial(blendful.update, model_output, v_ego, blendful_active, enabled=blendful_enabled))
partial(blendful.update, model_output, v_ego, blendful_active, enabled=blendful_enabled),
partial(turn_planner.update, model_output, v_ego, sm['controlsState'].curvature,
sm['carControl'].latActive, is_rhd))
prev_action = action
fill_model_msg(drivingdata_send, modelv2_send, model_output, action,
publish_state, meta_main.frame_id, meta_extra.frame_id, frame_id,
@@ -30,10 +30,10 @@ def record_kernel_linear():
yield recorded
def capture_frame(jit, kwargs: dict, output_container: str | None = None, run: bool = True):
def capture_frame(jit, kwargs: dict, output_container: str | None = None):
inputs = [([name] if key is None else [name, key], tensor.numpy().copy(), tensor.device)
for name, value in kwargs.items() for key, tensor in (value.items() if isinstance(value, dict) else [(None, value)])]
result = jit(**kwargs) if run else None # run=False: the last call already left these inputs' outputs in output_container
result = jit(**kwargs)
outputs = kwargs[output_container].values() if output_container else flatten(result)
return {"inputs": inputs, "outputs": [tensor.numpy().copy() for tensor in outputs], "output_container": output_container}, result
@@ -80,8 +80,7 @@ def serialize_jit(jit, kernel_linear, check: dict | None = None) -> tuple[dict,
if ast.op is Ops.PROGRAM:
elf, info = ast.to_elf(), ast.arg
global_size, local_size = info.launch_dims({})
device_target = dataclasses.asdict(dataclasses.replace(elf.target, interface=elf.target.interface.removeprefix("MOCK")))
calls.append({"kind": "program", "name": elf.name, "lib": blob.add(bytes(elf.lib)), "target": device_target,
calls.append({"kind": "program", "name": elf.name, "lib": blob.add(bytes(elf.lib)), "target": dataclasses.asdict(elf.target),
"signature": kernel_signature(ast, elf),
"global_size": [int(x) for x in global_size], "local_size": [int(x) for x in local_size],
"globals": list(info.globals), "outs": list(info.outs), "ins": list(info.ins), "args": arguments})
@@ -17,4 +17,4 @@ class TestRegistryModels(unittest.TestCase):
self.skipTest("MODEL_BASE_NAME env var not set, skipping integration test.")
bundle = load_model(os.path.join(os.environ.get("MODEL_DIR", "/tmp/models"), base_name), {"QCOM": "CPU", "AMD": "CPU"}, verify=False, cache=False)
jits = [value for value in bundle.values() if isinstance(value, PlainJit)]
assert "metadata" in bundle and jits and all(len(jit.captured._linear.src) > 0 for jit in jits)
assert "metadata" in bundle and jits and all(len(jit.captured.linear.src) > 0 for jit in jits)
@@ -0,0 +1,239 @@
"""
Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
import math
import numpy as np
from openpilot.common.realtime import DT_MDL
from openpilot.common.test import OpenpilotTestCase
from openpilot.sunnypilot.modeld_v2.constants import ModelConstants, Plan
from openpilot.sunnypilot.modeld_v2.turn_planner import TurnPlanner, compute_plan_peak_curv
def make_plan(curv: float, speed: float = 4.0) -> dict[str, np.ndarray]:
"""Build a synthetic model_output dict with a constant-curvature arc plan."""
t = np.array(ModelConstants.T_IDXS, dtype=np.float32)
s = speed * t
plan = np.zeros((1, len(t), ModelConstants.PLAN_WIDTH), dtype=np.float32)
if abs(curv) < 1e-6:
plan[0, :, Plan.POSITION][:, 0] = s
plan[0, :, Plan.POSITION][:, 1] = 0.0
plan[0, :, Plan.T_FROM_CURRENT_EULER][:, 2] = 0.0
else:
r = 1.0 / curv
theta = s * curv
plan[0, :, Plan.POSITION][:, 0] = r * np.sin(theta)
plan[0, :, Plan.POSITION][:, 1] = r * (1.0 - np.cos(theta))
plan[0, :, Plan.T_FROM_CURRENT_EULER][:, 2] = theta
return {'plan': plan}
class TestTurnPlanner(OpenpilotTestCase):
def test_plan_peak_curv_extraction(self):
assert abs(compute_plan_peak_curv(make_plan(0.0, 5.0))) < 1e-5
pk = compute_plan_peak_curv(make_plan(0.12, 4.0))
assert abs(pk - 0.12) < 0.01
pk_l = compute_plan_peak_curv(make_plan(-0.15, 4.0))
assert abs(pk_l + 0.15) < 0.01
assert compute_plan_peak_curv({}) == 0.0
def test_untouched_at_highway_speed(self):
tp = TurnPlanner(wheelbase=2.89)
for i in range(60):
k = 0.01 * math.sin(i * 0.1)
out = tp.update(make_plan(k, 25.0), v_ego=25.0, actual_curv=k, active=True, is_rhd=False, model_curv=k)
assert out == k
def test_untouched_on_gentle_low_speed_curve(self):
tp = TurnPlanner(wheelbase=2.89)
for i in range(80):
k = 0.025 * min(1.0, i / 20.0)
out = tp.update(make_plan(k, 6.0), v_ego=6.0, actual_curv=k, active=True, is_rhd=False, model_curv=k)
assert out == k
def test_sign_preservation_and_magnitude_bound_under_jitter(self):
"""Output must never oppose model_curv or exceed APEX_PATH_BOOST_CAP * |model_curv|."""
tp = TurnPlanner(wheelbase=2.89)
# Reproduces a5 seg31 Turn 1: brief negative jitter then sharp positive right turn
ks = [-0.015] * 15 + [0.016, 0.05, 0.11, 0.14, 0.17, 0.18, 0.14, 0.08, 0.02, -0.01]
plan_right = make_plan(0.15, 3.0)
for k in ks:
out = tp.update(plan_right, v_ego=3.0, actual_curv=k * 0.7, active=True, is_rhd=False, model_curv=k)
assert abs(out) <= abs(k) * 1.35 + 1e-9
if k > 0:
assert out >= 0.0
elif k < 0:
assert out <= 0.0
def test_zero_lag_on_turn_exit_unwinding(self):
tp = TurnPlanner(wheelbase=2.89)
plan_tight = make_plan(0.14, 4.0)
# Ramp up entry and hold through apex so buffer is at peak curvature
for i in range(25):
k = 0.14 * (i / 25.0)
tp.update(plan_tight, v_ego=4.0, actual_curv=k * 0.8, active=True, is_rhd=False, model_curv=k)
for _ in range(15):
tp.update(plan_tight, v_ego=4.0, actual_curv=0.14, active=True, is_rhd=False, model_curv=0.14)
# Now unwind monotonically: output must equal model_curv with zero lag
for i in range(1, 20):
k = max(0.0, 0.14 * (1.0 - i / 15.0))
out = tp.update(make_plan(k, 4.0), v_ego=4.0, actual_curv=k + 0.01, active=True, is_rhd=False, model_curv=k)
assert abs(out - k) < 1e-3
def test_standstill_launch_and_mid_turn_pause(self):
tp = TurnPlanner(wheelbase=2.89)
# Stopped straight at stop bar, then launches into a sharp right turn
for _ in range(10):
tp.update(make_plan(0.12, 0.0), v_ego=0.0, actual_curv=0.0, active=True, is_rhd=False, model_curv=0.002)
# First few steps of launch should hold back early curvature ramp
outs = []
for i in range(1, 20):
k = 0.006 * i
outs.append(tp.update(make_plan(0.12, 2.5), v_ego=2.5, actual_curv=0.004 * i, active=True, is_rhd=False, model_curv=k))
assert outs[10] < 0.006 * 11 - 0.01
# Pause mid-turn (actual_curv = 0.05) should preserve turn_psi and not dip below 0.05 on resume
tp.turn_psi = math.radians(30.0)
tp.update(make_plan(0.08, 0.0), v_ego=0.0, actual_curv=0.05, active=True, is_rhd=False, model_curv=0.05)
assert tp.turn_psi >= math.radians(29.0)
resumed = tp.update(make_plan(0.08, 2.0), v_ego=2.0, actual_curv=0.05, active=True, is_rhd=False, model_curv=0.06)
assert resumed >= 0.05 - 1e-6
def test_wheelbase_and_rhd_scaling(self):
tp_short = TurnPlanner(wheelbase=2.4)
tp_long = TurnPlanner(wheelbase=3.4)
plan_r = make_plan(0.14, 4.2)
for _ in range(25):
tp_short.update(plan_r, 4.2, 0.0, True, False, 0.02)
tp_long.update(plan_r, 4.2, 0.0, True, False, 0.02)
assert tp_long.shift_dist > tp_short.shift_dist * 1.3
# RHD gives larger shift on left turns (k < 0) than right turns (k > 0)
tp_rhd_l = TurnPlanner(wheelbase=2.89)
tp_rhd_r = TurnPlanner(wheelbase=2.89)
for _ in range(25):
tp_rhd_l.update(make_plan(-0.14, 3.5), 3.5, 0.0, True, True, -0.02)
tp_rhd_r.update(make_plan(0.14, 3.5), 3.5, 0.0, True, True, 0.02)
assert tp_rhd_l.shift_dist > tp_rhd_r.shift_dist * 1.2
def test_closed_loop_90deg_turn_clears_curb_without_oncoming_overshoot(self):
"""Closed-loop Tesla Model Y 90-deg right turn around R=2.5 m curb."""
L, W_half, x_cam, lane_w, curb_r = 2.89, 0.96, 1.50, 3.60, 2.5
y_curb = lane_w / 2.0
x_dest = curb_r + lane_w / 2.0
x_yellow = x_dest + lane_w / 2.0
pts = [(x, 0.0) for x in np.linspace(-20, x_dest - 2.0, 200)]
pts += [(x_dest - 2.0 + 2.0 * math.cos(th), 2.0 + 2.0 * math.sin(th)) for th in np.linspace(-math.pi / 2, 0, 50)]
pts += [(x_dest, y) for y in np.linspace(2.0, 25.0, 200)]
ref = np.array(pts)
def run(use_planner: bool) -> tuple[float, float]:
X, Y, psi, k_act = -15.0, 0.0, 0.0, 0.0
v = 3.5
tp = TurnPlanner(wheelbase=L, dt=DT_MDL) if use_planner else None
min_clr, max_ovr = 99.0, -99.0
for _ in range(400):
xc, yc = X + x_cam * math.cos(psi), Y + x_cam * math.sin(psi)
i_near = int(np.argmin(np.hypot(ref[:, 0] - xc, ref[:, 1] - yc)))
i_tgt = i_near
while i_tgt < len(ref) - 1 and np.hypot(ref[i_tgt, 0] - xc, ref[i_tgt, 1] - yc) < 6.5:
i_tgt += 1
dx, dy = ref[i_tgt, 0] - xc, ref[i_tgt, 1] - yc
lx = dx * math.cos(psi) + dy * math.sin(psi)
ly = -dx * math.sin(psi) + dy * math.cos(psi)
k_mod = float(np.clip(2.0 * ly / max(lx**2 + ly**2, 1.0), -0.19, 0.19))
plan = make_plan(0.14 if (max(0.0, x_dest - xc) < 15.0 and psi < math.radians(75)) else 0.0, v)
k_cmd = tp.update(plan, v, k_act, True, False, k_mod) if tp is not None else k_mod
k_act += (float(np.clip(k_cmd, -0.19, 0.19)) - k_act) * (DT_MDL / (0.22 + DT_MDL))
ds = v * DT_MDL
psi += k_act * ds
X += ds * math.cos(psi)
Y += ds * math.sin(psi)
x_ir, y_ir = X - W_half * math.sin(psi), Y + W_half * math.cos(psi)
x_of, y_of = X + L * math.cos(psi) + W_half * math.sin(psi), Y + L * math.sin(psi) - W_half * math.cos(psi)
if x_ir <= 0.0:
clr = y_curb - y_ir
elif y_ir >= y_curb + curb_r:
clr = x_ir - curb_r
else:
clr = math.hypot(x_ir, y_ir - (y_curb + curb_r)) - curb_r
min_clr = min(min_clr, clr)
if y_of > 2.0:
max_ovr = max(max_ovr, x_of - x_yellow)
if Y > 15.0:
break
return min_clr, max_ovr
base_clr, _ = run(False)
plan_clr, plan_ovr = run(True)
assert base_clr < -0.5 # baseline curbs the rear wheel badly
assert plan_clr > 0.20 # TurnPlanner clears the curb by > 20 cm
assert plan_ovr < 0.0 # stays inside the oncoming yellow line
def test_rolling_speed_taper(self):
"""Shift is strong at low speed, light at rolling speed, 0 at 19+ mph, and time-capped at crawl."""
tp_crawl = TurnPlanner(wheelbase=2.89)
tp_low = TurnPlanner(wheelbase=2.89)
tp_roll = TurnPlanner(wheelbase=2.89)
tp_fast = TurnPlanner(wheelbase=2.89)
for _ in range(30):
tp_crawl.update(make_plan(0.14, 1.8), v_ego=1.8, actual_curv=0.0, active=True, is_rhd=False, model_curv=0.02)
tp_low.update(make_plan(0.14, 3.5), v_ego=3.5, actual_curv=0.0, active=True, is_rhd=False, model_curv=0.02)
tp_roll.update(make_plan(0.14, 6.0), v_ego=6.0, actual_curv=0.0, active=True, is_rhd=False, model_curv=0.02)
tp_fast.update(make_plan(0.14, 9.0), v_ego=9.0, actual_curv=0.0, active=True, is_rhd=False, model_curv=0.02)
assert tp_crawl.shift_dist <= 1.8 * 0.48 + 0.05 # time-capped so crawl starts don't stall
assert tp_low.shift_dist > 1.3
assert 0.3 < tp_roll.shift_dist < 0.7
assert tp_fast.shift_dist == 0.0
def test_apex_1s_path_boost_and_exit_unwind(self):
"""Once past entry, blends with 1s path to boost under-steering apex and unwind hanging exit."""
tp = TurnPlanner(wheelbase=2.89)
# Drive past entry (psi > 30 deg)
for _ in range(35):
tp.update(make_plan(0.14, 4.0), v_ego=4.0, actual_curv=0.12, active=True, is_rhd=False, model_curv=0.10)
# Apex (psi ~ 48 deg): action under-steers at 0.10 while 1s path says 0.14 -> boosted cleanly above 0.10
apex_out = tp.update(make_plan(0.14, 4.0), v_ego=4.0, actual_curv=0.12, active=True, is_rhd=False, model_curv=0.10)
assert apex_out > 0.115
# Advance past 60 deg into late-turn exit
for _ in range(20):
tp.update(make_plan(0.14, 4.0), v_ego=4.0, actual_curv=0.12, active=True, is_rhd=False, model_curv=0.14)
# Exit hang: action still stuck at 0.20 while 1s path already unwound to 0.10 -> pulls down cleanly
exit_out = tp.update(make_plan(0.10, 4.0), v_ego=4.0, actual_curv=0.15, active=True, is_rhd=False, model_curv=0.20)
assert exit_out < 0.15
def test_road_edge_curb_guard_holds_until_opening(self):
"""Confident straight curb beside car prolongs turn-in until curb flares open or disappears."""
x = np.array(ModelConstants.X_IDXS, dtype=np.float32)
def with_curb(plan_dict, e0=2.4, flare_5m=0.1, std=0.4):
edges = np.zeros((1, 2, len(x), 2), dtype=np.float32)
edges[0, 0, :, 0] = -5.0
edges[0, 1, :, 0] = e0 + (flare_5m / 25.0) * (x ** 2)
stds = np.full((1, 2, len(x), 2), std, dtype=np.float32)
return {**plan_dict, 'road_edges': edges, 'road_edges_stds': stds}
tp = TurnPlanner(wheelbase=2.89)
# Rolling 15 mph (6.7 m/s) right turn with solid straight curb right beside car -> curb guard holds it down
mo_curb = with_curb(make_plan(0.03, 6.7), e0=2.4, flare_5m=0.1, std=0.4)
for _ in range(5):
out_guarded = tp.update(mo_curb, v_ego=6.7, actual_curv=0.0, active=True, is_rhd=False, model_curv=0.035)
assert out_guarded < 0.016
# Curb flares open (intersection/turn lane opens) -> releases immediately!
mo_open = with_curb(make_plan(0.06, 6.7), e0=2.4, flare_5m=1.2, std=1.1)
out_open = tp.update(mo_open, v_ego=6.7, actual_curv=0.01, active=True, is_rhd=False, model_curv=0.06)
assert out_open > 0.04
# Normal curved road (curb itself curves at k=0.03 => flare_5m = 0.5 * 0.03 * 25 = 0.375 m) -> untouched
tp2 = TurnPlanner(wheelbase=2.89)
mo_curve = with_curb(make_plan(0.03, 6.7), e0=2.4, flare_5m=0.375, std=0.4)
out_curve = tp2.update(mo_curve, v_ego=6.7, actual_curv=0.03, active=True, is_rhd=False, model_curv=0.03)
assert abs(out_curve - 0.03) < 1e-6
@@ -0,0 +1,272 @@
"""
Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
import math
from collections import deque
import numpy as np
from openpilot.common.realtime import DT_MDL
from openpilot.sunnypilot.modeld_v2.constants import ModelConstants, Plan
# Two-stage rear-axle protected turn planner:
# Stage 1 (Entry): wheelbase-scaled spatial delay on curvature buildup so the
# inside rear tire clears the corner curb instead of off-tracking over it.
# Capped in time (MAX_SHIFT_TIME) so crawl-speed starts never feel stalled.
# Stage 2 (Apex & Exit): once past initial corner entry, smoothly blends
# action curvature with the model's own 1-second path curvature so tight
# apexes don't under-steer/go wide and exits unwind cleanly.
SHIFT_WHEELBASE_RATIO = 0.58 # fraction of wheelbase to delay turn-in (~1.68 m for 2.89 m WB)
WHEELBASE_CLIP = (2.0, 4.0) # m
OPPOSITE_SIDE_WEIGHT = 0.70 # lighter shift for non-curb-side turns
MAX_SHIFT_TIME = 0.48 # s, caps entry delay at crawl speeds (2-5 mph)
TURN_SPEED_TAPER = (7.5, 10.5) # m/s (~17-23.5 mph): overall active window
SHIFT_SPEED_BP = (3.5, 6.0, 8.5) # m/s (~7.8, 13.4, 19.0 mph)
SHIFT_SPEED_V = (1.0, 0.35, 0.0) # full from stop, light on rolling turns, 0 above 19 mph
CURV_GATE = (0.04, 0.09) # 1/m (R=25 m -> R=11 m): ignores gentle curves
ENTRY_PHASE_RELEASE_DEG = (8.0, 22.0) # deg: release entry shift crisply once rotating
APEX_BLEND_PHASE_DEG = (10.0, 25.0) # deg: ramp in 1s-path blend through apex & exit
APEX_PATH_BLEND_MAX = 0.65 # max weight on 1s path curvature during apex/exit
APEX_PATH_BOOST_CAP = 1.35 # max ratio vs |model_curv| for safety
PATH_AVG_HORIZON_S = 1.0 # s, horizon for average path curvature
SHIFT_BUILD_TAU = 0.15 # s
SHIFT_RELEASE_TAU = 0.07 # s
APEX_BLEND_TAU = 0.12 # s
BUFFER_MAX_DIST = 4.5 # m
MIN_MOVE_SPEED = 0.3 # m/s
MID_TURN_CURV_THRESH = 0.02 # 1/m
PLAN_S_RANGE = (1.0, 18.0) # m
PLAN_MIN_SEG_DS = 0.8 # m
# Road-edge curb guard: prevents premature turn-in (e.g. immediately on blinker)
# while a confident, unbroken curb still runs straight beside the vehicle.
CURB_STD_GATE = (0.55, 0.95) # m: confident curb below 0.55, gone/opening above 0.95
CURB_CLOSE_GATE = (2.5, 3.1) # m: curb close to lane below 2.5, open turn bay above 3.1
CURB_OPENING_GATE = (0.35, 0.90) # m: (edge_5m - edge_0m) flare indicating intersection opening
CURB_FOLLOW_MARGIN_K = 0.005 # 1/m: small lane-keeping allowance above curb's own curvature
CURB_CHECK_X = 5.0 # m: lookahead along road edge for curb wall check
CURB_GUARD_PHASE_DEG = (10.0, 22.0) # deg: only guards turn entry before car has rotated in
T_IDXS = np.array(ModelConstants.T_IDXS, dtype=np.float32)
X_IDXS = np.array(ModelConstants.X_IDXS, dtype=np.float32)
def compute_plan_metrics(model_output: dict[str, np.ndarray]) -> tuple[float, float]:
"""Return (peak_upcoming_curv, avg_1s_path_curv) from model_output['plan']."""
plan_arr = model_output.get('plan')
if plan_arr is None or plan_arr.ndim < 3 or plan_arr.shape[1] < 3:
return 0.0, 0.0
plan = plan_arr[0]
pos = plan[:, Plan.POSITION]
yaws = plan[:, Plan.T_FROM_CURRENT_EULER][:, 2]
ds = np.hypot(np.diff(pos[:, 0]), np.diff(pos[:, 1]))
s = np.concatenate(([0.0], np.cumsum(ds)))
peak_k = 0.0
n = len(yaws)
for i in range(n - 2):
if s[i] < PLAN_S_RANGE[0]:
continue
if s[i] > PLAN_S_RANGE[1]:
break
seg_ds = s[i + 2] - s[i]
if seg_ds < PLAN_MIN_SEG_DS:
continue
k = float((yaws[i + 2] - yaws[i]) / seg_ds)
if abs(k) > abs(peak_k):
peak_k = k
peak_k = float(np.clip(peak_k, -0.25, 0.25))
if len(yaws) == len(T_IDXS):
s_1s = float(np.interp(PATH_AVG_HORIZON_S, T_IDXS, s))
yaw_1s = float(np.interp(PATH_AVG_HORIZON_S, T_IDXS, yaws))
path_k_1s = float(np.clip(yaw_1s / s_1s, -0.25, 0.25)) if s_1s >= PLAN_MIN_SEG_DS else 0.0
else:
path_k_1s = 0.0
return peak_k, path_k_1s
def compute_plan_peak_curv(model_output: dict[str, np.ndarray]) -> float:
return compute_plan_metrics(model_output)[0]
def compute_curb_guard(model_output: dict[str, np.ndarray], turn_sign: float) -> tuple[float, float]:
"""
Check if a confident, unbroken road edge (curb) still runs straight beside the car
on the turning side (turn_sign > 0 = right edge, turn_sign < 0 = left edge).
Returns (curb_weight in [0, 1], max_safe_curv_abs).
"""
edges = model_output.get('road_edges')
stds = model_output.get('road_edges_stds')
if edges is None or stds is None or edges.ndim < 4 or edges.shape[1] < 2:
return 0.0, 1.0
idx = 1 if turn_sign > 0.0 else 0
edge_y = edges[0, idx, :, 0]
if len(edge_y) != len(X_IDXS):
return 0.0, 1.0
edge_std = float(stds[0, idx, 0, 0])
std_w = float(np.interp(edge_std, CURB_STD_GATE, [1.0, 0.0]))
if std_w <= 0.0:
return 0.0, 1.0
e0 = float(edge_y[0]) * turn_sign
e5 = float(np.interp(CURB_CHECK_X, X_IDXS, edge_y)) * turn_sign
if e0 <= 0.0:
return 0.0, 1.0
close_w = float(np.interp(e0, CURB_CLOSE_GATE, [1.0, 0.0]))
flare = e5 - e0
unopened_w = float(np.interp(flare, CURB_OPENING_GATE, [1.0, 0.0]))
curb_w = std_w * close_w * unopened_w
if curb_w <= 0.0:
return 0.0, 1.0
# Allow following the curb's own curvature (so normal curved roads are untouched) plus small margin.
k_curb = max(0.0, 2.0 * flare / (CURB_CHECK_X * CURB_CHECK_X))
return curb_w, k_curb + CURB_FOLLOW_MARGIN_K
class TurnPlanner:
"""
Two-stage rear-axle protected turn planner for tight low-speed turns.
Stage 1 (Entry): delays curvature buildup by a wheelbase-proportional distance
(time-capped at MAX_SHIFT_TIME) so the rear axle clears the inside curb.
Stage 2 (Apex & Exit): blends action curvature with the model's 1-second path
curvature once past entry, preventing wide apexes and exit hang-ups.
"""
def __init__(self, wheelbase: float = 2.7, dt: float = DT_MDL):
self.wheelbase = float(np.clip(wheelbase, *WHEELBASE_CLIP))
self.dt = dt
self.s = 0.0
self.buf_s: deque[float] = deque()
self.buf_k: deque[float] = deque()
self.turn_sign = 0.0
self.turn_psi = 0.0
self.shift_dist = 0.0
self.apex_blend = 0.0
self.plan_peak_curv = 0.0
self.path_curv_1s = 0.0
def reset(self) -> None:
self.s = 0.0
self.buf_s.clear()
self.buf_k.clear()
self.turn_sign = 0.0
self.turn_psi = 0.0
self.shift_dist = 0.0
self.apex_blend = 0.0
self.plan_peak_curv = 0.0
self.path_curv_1s = 0.0
def update(self, model_output: dict[str, np.ndarray], v_ego: float, actual_curv: float,
active: bool, is_rhd: bool, model_curv: float, enabled: bool = True) -> float:
if not enabled or not active:
self.reset()
return model_curv
if v_ego < MIN_MOVE_SPEED:
# Stopped straight (e.g., at a stop bar): reset turn phase so launch gets full protection.
# Stopped mid-turn (e.g., yielding to a pedestrian): preserve turn_psi so resume doesn't re-delay.
if abs(actual_curv) < MID_TURN_CURV_THRESH:
self.turn_psi = 0.0
self.turn_sign = 0.0
self.shift_dist = 0.0
self.apex_blend = 0.0
self.buf_s.clear()
self.buf_k.clear()
self.buf_s.append(self.s)
self.buf_k.append(float(actual_curv))
return model_curv
ds = v_ego * self.dt
self.s += ds
self.buf_s.append(self.s)
self.buf_k.append(float(model_curv))
while len(self.buf_s) > 2 and (self.s - self.buf_s[0]) > BUFFER_MAX_DIST:
self.buf_s.popleft()
self.buf_k.popleft()
turn_speed_w = float(np.interp(v_ego, TURN_SPEED_TAPER, [1.0, 0.0]))
if turn_speed_w <= 0.0 and self.shift_dist < 1e-3 and self.apex_blend < 1e-3:
self.turn_psi = 0.0
self.turn_sign = 0.0
self.plan_peak_curv = 0.0
self.path_curv_1s = 0.0
return model_curv
plan_peak, path_k_1s = compute_plan_metrics(model_output)
self.plan_peak_curv = plan_peak
self.path_curv_1s = path_k_1s
tightness = max(abs(model_curv), abs(plan_peak))
curv_w = float(np.interp(tightness, CURV_GATE, [0.0, 1.0]))
ref_k = model_curv if abs(model_curv) > 0.015 else plan_peak
if abs(ref_k) > 0.035 and turn_speed_w > 0.05:
new_sign = 1.0 if ref_k > 0.0 else -1.0
if new_sign != self.turn_sign:
self.turn_sign = new_sign
self.turn_psi = 0.0
self.turn_psi = max(0.0, self.turn_psi + actual_curv * self.turn_sign * ds)
else:
self.turn_psi = max(0.0, self.turn_psi - 0.5 * self.dt)
if self.turn_psi < 0.02:
self.turn_sign = 0.0
psi_deg = math.degrees(self.turn_psi)
# Stage 1: Entry rear-axle spatial shift (time-capped so crawl-speed starts stay responsive)
shift_speed_w = float(np.interp(v_ego, SHIFT_SPEED_BP, SHIFT_SPEED_V))
entry_phase_w = float(np.interp(psi_deg, ENTRY_PHASE_RELEASE_DEG, [1.0, 0.0]))
is_curb_side = (ref_k < 0.0) if is_rhd else (ref_k > 0.0)
side_w = 1.0 if is_curb_side else OPPOSITE_SIDE_WEIGHT
raw_shift = SHIFT_WHEELBASE_RATIO * self.wheelbase * shift_speed_w * curv_w * entry_phase_w * side_w
target_shift = min(raw_shift, v_ego * MAX_SHIFT_TIME)
tau = SHIFT_RELEASE_TAU if target_shift < self.shift_dist else SHIFT_BUILD_TAU
self.shift_dist += (target_shift - self.shift_dist) * (self.dt / (tau + self.dt))
out_k = model_curv
if self.shift_dist >= 0.05 and len(self.buf_s) >= 2:
k_delayed = float(np.interp(self.s - self.shift_dist, self.buf_s, self.buf_k))
# Same sign as model_curv (or 0), never exceeds |model_curv|, zero lag on unwind.
if model_curv >= 0.0:
out_k = float(np.clip(k_delayed, 0.0, model_curv))
else:
out_k = float(np.clip(k_delayed, model_curv, 0.0))
# Stage 1b: Road-edge curb guard during turn entry (prolongs straight travel while
# a confident, unbroken curb is still beside the car before the turn lane/intersection opens).
curb_phase_w = float(np.interp(psi_deg, CURB_GUARD_PHASE_DEG, [1.0, 0.0]))
if turn_speed_w > 0.05 and curb_phase_w > 0.01 and abs(out_k) > CURB_FOLLOW_MARGIN_K:
curb_w, k_cap = compute_curb_guard(model_output, 1.0 if out_k > 0.0 else -1.0)
guard_w = curb_w * turn_speed_w * curb_phase_w
if guard_w > 0.01 and abs(out_k) > k_cap:
capped = math.copysign(k_cap, out_k)
out_k = (1.0 - guard_w) * out_k + guard_w * capped
# Stage 2: Apex & Exit 1s-path blend (prevents wide apexes and exit hang-ups)
apex_phase_w = float(np.interp(psi_deg, APEX_BLEND_PHASE_DEG, [0.0, 1.0]))
target_blend = APEX_PATH_BLEND_MAX * turn_speed_w * curv_w * apex_phase_w
self.apex_blend += (target_blend - self.apex_blend) * (self.dt / (APEX_BLEND_TAU + self.dt))
if self.apex_blend > 0.01 and out_k * path_k_1s > 0.0:
# Through the apex, boost when 1s path asks for tighter curvature than action (prevents wide turns).
# Only pull curvature down toward 1s path late in the turn (psi > 60 deg) to prevent exit hang-ups.
if abs(path_k_1s) > abs(out_k) or psi_deg > 60.0:
max_boosted = abs(model_curv) * APEX_PATH_BOOST_CAP
target_k = float(np.clip(path_k_1s, -max_boosted, max_boosted))
out_k = (1.0 - self.apex_blend) * out_k + self.apex_blend * target_k
return out_k
+31 -8
View File
@@ -29,8 +29,11 @@ DOWNLOAD_TIMEOUT = (30, 30)
# HuggingFace throttles each connection to ~1-2 MB/s but not the connection count; 12 saturates the device link
MAX_CONCURRENT_PIECES = 12
PIECE_SIZE = 8 * 1024 * 1024
PIECE_ATTEMPTS = 3
PIECE_ATTEMPTS = 3 # per piece, before the transfer as a whole is retried
PIECE_BACKOFF = 1.0 # seconds, growing linearly: waits 1 s then 2 s between a piece's three attempts
DOWNLOAD_ATTEMPTS = 4 # whole-transfer attempts, each resuming from the sidecar
RETRY_BACKOFF = 2.0 # seconds, doubling: waits 2, 4 then 8 s between the four transfer attempts
RETRY_POLL = 0.25 # how often a backoff checks for a cancel
TRANSIENT_HTTP_STATUS = frozenset({408, 429, 500, 502, 503, 504})
REPORT_INTERVAL = 0.5 # seconds between progress publications
# byte ranges count encoded bytes but iter_content yields decoded ones, so ask for an unencoded body
@@ -298,6 +301,24 @@ class ModelManagerSP:
artifact.downloadProgress.resuming = False
del self._download_start_times[artifact.fileName]
async def _download_with_retries(self, url: str, path: str, artifact) -> None:
for attempt in range(DOWNLOAD_ATTEMPTS):
try:
await self._download_file(url, path, artifact)
return
except Exception as e:
if not _is_transient(e) or attempt == DOWNLOAD_ATTEMPTS - 1:
raise
delay = RETRY_BACKOFF * 2 ** attempt
cloudlog.warning(f"Retrying {artifact.fileName} in {delay:g}s after {type(e).__name__}: {e}")
progress = artifact.downloadProgress
self._set_progress(artifact, progress.status, progress.progress, progress.eta, 0.0) # stalled: no speed
deadline = time.monotonic() + delay
while (remaining := deadline - time.monotonic()) > 0:
if self._download_interrupted():
raise DownloadCancelled("Download cancelled") from e
await asyncio.sleep(min(RETRY_POLL, remaining))
async def _process_artifact(self, artifact, destination_path: str) -> None:
if not artifact.downloadUri.uri:
return None
@@ -323,7 +344,7 @@ class ModelManagerSP:
return
self._set_progress(artifact, status.downloading, 0)
await self._download_file(url, full_path, artifact)
await self._download_with_retries(url, full_path, artifact)
self._set_progress(artifact, status.verifying, 99)
if not await verify_file(full_path, expected_hash):
_remove_file(full_path)
@@ -332,14 +353,11 @@ class ModelManagerSP:
self._set_progress(artifact, status.downloaded, 100)
except DownloadCancelled:
except DownloadCancelled: # not a failure: the request handler clears the row
self._download_start_times.pop(artifact.fileName, None)
artifact.downloadProgress.status = status.failed
artifact.downloadProgress.eta = 0
artifact.downloadProgress.speed = 0
self._sync_artifact_progress(artifact)
if self.selected_bundle:
self.selected_bundle.status = status.failed
self._report_status()
raise
@@ -401,6 +419,8 @@ class ModelManagerSP:
self.params.put(ACTIVE_BUNDLE_KEYS[source], model_bundle.to_dict(), block=True)
self.active_bundle = get_active_bundle(self.params, chestnut=self.chestnut_present)
except DownloadCancelled:
raise # not a failure
except Exception:
if self.selected_bundle is not None:
self.selected_bundle.status = custom.ModelManagerSP.DownloadStatus.failed
@@ -428,11 +448,14 @@ class ModelManagerSP:
self._download_ref = ref_to_download
try:
self.download(model_to_download, Paths.model_root(), source)
except DownloadCancelled:
self.selected_bundle = None
except Exception as e:
cloudlog.exception(e)
cloudlog.exception(e) # a failure stays on the row until the next request
else:
self.selected_bundle = None
finally:
self._release_download_ref()
self.selected_bundle = None
def main_thread(self) -> None:
"""Main thread for model management"""
@@ -57,6 +57,8 @@ class ServerState:
fail_ranges: dict[tuple[int, int], int] = dataclasses.field(default_factory=dict) # range -> HTTP status, every time
fail_times: dict[tuple[int, int], int] = dataclasses.field(default_factory=dict) # range -> how many more requests 503
short_once: set[tuple[int, int]] = dataclasses.field(default_factory=set) # ranges whose first answer is cut short
fail_after_others: bool = False # hold each failure until every other piece has been sent
sent: set[tuple[int, int]] = dataclasses.field(default_factory=set) # ranges sent in full
stall_ranges: set[tuple[int, int]] = dataclasses.field(default_factory=set)
stall_event: threading.Event | None = None
stall_released_by_event: bool | None = None
@@ -90,6 +92,10 @@ class DownloadHandler(http.server.BaseHTTPRequestHandler):
state.fail_times[rng] -= 1
status = 503
if status is not None:
others = {piece_range(i) for i in range(NUM_PIECES)} - {rng}
deadline = time.monotonic() + 5
while state.fail_after_others and not others <= state.sent and time.monotonic() < deadline:
time.sleep(0.01)
self.send_response(status)
self.end_headers()
return
@@ -117,6 +123,8 @@ class DownloadHandler(http.server.BaseHTTPRequestHandler):
self.wfile.write(payload[100:])
else:
self.wfile.write(payload)
if rng is not None:
state.sent.add(rng)
@contextlib.contextmanager
@@ -152,7 +160,7 @@ class ManagerDownloadTestBase(OpenpilotTestCase):
self.addCleanup(self._tmp.cleanup)
self.dest = self._tmp.name
for name, value in (('PIECE_SIZE', PIECE_SIZE), ('REPORT_INTERVAL', 0.05), ('PIECE_BACKOFF', 0.01), ('SPEED_FLOOR', 100)):
for name, value in (('PIECE_SIZE', PIECE_SIZE), ('REPORT_INTERVAL', 0.05), ('PIECE_BACKOFF', 0.01), ('SPEED_FLOOR', 100), ('RETRY_BACKOFF', 0.01)):
self.enterContext(mock.patch.object(manager_module, name, value))
self.reported: list[float] = []
@@ -234,6 +242,17 @@ class ManagerDownloadTestBase(OpenpilotTestCase):
def download_bundle(self, source: str):
asyncio.run(self.manager._download_bundle(self._bundle, self.dest, source))
def request_download(self) -> dict:
self.make_artifact()
self._bundle.ref = "ref"
self._bundle.minimumSelectorVersion = helpers.REQUIRED_JSON_VERSION
self.manager.source_models = {"qcom": [self._bundle], "chestnut": []}
self.manager.params, store = self._make_params_with_store()
store["ModelManager_DownloadRef"] = "ref"
with mock.patch.object(manager_module.Paths, 'model_root', return_value=self.dest):
self.manager._process_download_requests()
return store
def read_path(self) -> bytes:
with open(self.path, 'rb') as f:
return f.read()
@@ -250,6 +269,7 @@ class ManagerDownloadTestBase(OpenpilotTestCase):
params.get.side_effect = get
params.put.side_effect = put
params.remove.side_effect = lambda key: put(key, None)
return params, store
@@ -329,6 +349,36 @@ class TestManagerDownload(ManagerDownloadTestBase):
assert os.path.isfile(self.path)
assert self.resume_state()["done"] == [i for i in range(NUM_PIECES) if i != 1]
def test_failed_transfer_resumes_on_retry(self):
self.server.fail_times = {piece_range(1): manager_module.PIECE_ATTEMPTS} # the first transfer gives up on piece 1
self.server.fail_after_others = True # by then every other piece is saved
self.process_artifact()
assert self.read_path() == WHOLE_BODY
assert self.piece_requests().count(piece_range(1)) == manager_module.PIECE_ATTEMPTS + 1, "the retry fetched piece 1"
others = [i for i in range(NUM_PIECES) if i != 1]
assert all(self.piece_requests().count(piece_range(i)) == 1 for i in others), "the retry skipped every saved piece"
def test_lasting_transient_failure_gives_up(self):
self.server.fail_ranges = {piece_range(1): 503}
with self.assertRaises(requests.exceptions.HTTPError):
self.process_artifact()
assert self.piece_requests().count(piece_range(1)) == manager_module.PIECE_ATTEMPTS * manager_module.DOWNLOAD_ATTEMPTS
def test_cancel_ends_retry_backoff(self):
self.enterContext(mock.patch.object(manager_module, 'RETRY_BACKOFF', 30.0))
self.manager._download_ref = b"ref"
def fail_then_cancel(*args):
self.params.get.return_value = None # the user cancels while the manager waits to retry
raise requests.exceptions.ConnectionError("dropped")
transfer = self.enterContext(mock.patch.object(self.manager, '_download_file', side_effect=fail_then_cancel))
started = time.monotonic()
with self.assertRaises(DownloadCancelled):
self.process_artifact()
assert time.monotonic() - started < 5, "the wait must end at the cancel, not after 30 s"
assert transfer.call_count == 1, "no second transfer after a cancel"
assert self.artifact.downloadProgress.status != custom.ModelManagerSP.DownloadStatus.failed
def test_removing_download_ref_cancels(self):
stall_event = self.stall(3)
refs = iter([b"ref"])
@@ -465,6 +515,23 @@ class TestProcessArtifact(ManagerDownloadTestBase):
self.download_bundle("qcom")
assert ACTIVE_BUNDLE_KEYS["qcom"] not in store
assert os.path.isfile(self.path), "cancel must not delete a cached model"
assert self.manager.selected_bundle.status != custom.ModelManagerSP.DownloadStatus.failed, "a cancel is not a failure"
def test_failed_download_stays_on_the_row(self):
self.server.fail_ranges = {piece_range(1): 404}
store = self.request_download()
selected = self.manager.selected_bundle
assert selected is not None and selected.status == custom.ModelManagerSP.DownloadStatus.failed, "a failure stays on the row"
assert store["ModelManager_DownloadRef"] is None, "the failed request is released"
def test_finished_download_clears_the_row(self):
self.request_download()
assert self.manager.selected_bundle is None
def test_cancelled_download_clears_the_row(self):
self.pm.send.side_effect = lambda *args: self.manager.params.remove("ModelManager_DownloadRef") # cancel at the first report
self.request_download()
assert self.manager.selected_bundle is None
def test_download_ref_is_released_only_if_unchanged(self):
for current, removed in ((b"new-ref", False), (b"ref", True)):