Compare commits

...

375 Commits

Author SHA1 Message Date
Jason Wen 175835273f cap it so we don't cry 2026-09-01 00:35:09 -04:00
Jason Wen 5c24e128d8 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025-wee 2026-09-01 00:16:21 -04:00
James Vecellio-Grant 98ed8111f6 modeld_v2: big to small model fallback (#1974) 2026-09-01 00:15:23 -04:00
royjr de197ba6fa chestnut: alert when big model ready (#1947)
egpu: alert when big model ready

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2026-08-30 16:20:16 -04:00
Jason Wen 1dd5a7c91d Sync: commaai/openpilot:mastersunnypilot/sunnypilot:master (#1973) 2026-08-28 12:58:26 -04:00
nayan acb784d207 Merge commit '4a13639cfd122ccb9113a4d6ce225dcbd8e61914' into sync-20260827
# Conflicts:
#	openpilot/selfdrive/modeld/SConscript
#	openpilot/selfdrive/modeld/modeld.py
#	openpilot/selfdrive/ui/mici/layouts/home.py
#	openpilot/selfdrive/ui/ui_state.py
#	tinygrad_repo
2026-08-28 12:39:53 -04:00
Jason Wen f46d9a7bba Bump opendbc 2026-08-28 01:18:40 -04:00
Jason Wen cfb38312db Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2026-08-27 16:57:56 -04:00
Nayan 4075befc5e osm: support map deletion via sunnylink (#1971)
delete delete
2026-08-27 11:23:26 -04:00
Jason Wen 9f43d2477d [MICI] ui: move and restyle the sunnylink pill in settings (#1972) 2026-08-27 03:52:57 -04:00
Nayan 2d6cc4c065 models: Model Selector upgrades (#1953)
* uh, i did not commit anything all this time

* slideee to the left, cha cha

* lint lint

* ui: unify model source predicate and per-source bundle lookup in model_info

* [TIZI/TICI] ui: disable the other-model row onroad like the active row

* [TIZI/TICI] ui: drop docstring that restates the function name

* [TIZI/TICI] ui: keep Favorites as the first model folder in the picker

* ui: record why model names read the params slots and not modelManagerSP

* ui: show the default model's name on the picker Default entries

* models: bind a download to its ref so cancel and reselect work everywhere

* models: resume partial chunked downloads and verify silently

* models: publish a verifying status so cached checks read as verification, not a stuck download

* [TIZI/TICI] ui: move download status onto each model's own row

* [TIZI/TICI] ui: show the row status description while it has text

* [TIZI/TICI] ui: restore the Model Status bar row

* models: a cancel interrupts verification immediately and keeps on-disk chunks

* models: a selection made mid-download queues instead of cancelling the transfer

* [TIZI/TICI] ui: Model Status shows both slots idle and the queued pick while busy

* [TIZI/TICI] ui: label the Model Status slots small and big and scroll long names

* models: start a queued download in the same tick and label empty slots (Default)

* ui: scroll Model Status names at the corrected speed

* [TIZI/TICI] ui: Model Status shows the big model failing over to small

* [TIZI/TICI] ui: stable model rows and a runner-matched failover note on Model Status

* [TIZI/TICI] ui: model rows show full names and the failover note reopens with the page

* ui: name the actually driving model runner-matched and bring mici to state parity

* fix ugly

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
Co-authored-by: James Vecellio-Grant <159560811+Discountchubbs@users.noreply.github.com>
2026-08-27 02:03:53 -04:00
Daniel Koepping 4a13639cfd reduce chestnut states (#38705)
ui: unify chestnut status presentation
2026-08-26 19:12:49 -07:00
YassineYousfi fa75fdd852 chestnut stats: overlap with gpu work (#38704)
* chestnut stats: overlap with gpu work

* ci

---------

Co-authored-by: elkoled <elkoled@gmail.com>
2026-08-26 17:57:55 -07:00
Daniel Koepping 5cfdb2f4da rename usbgpu to chestnut (#38703)
chestnut: rename eGPU interfaces
2026-08-26 15:42:59 -07:00
Daniel Koepping 63548ce10d bump tinygrad (#38702) 2026-08-26 15:20:04 -07:00
Daniel Koepping 980fb79c1a update orange GPU icon (#38701)
mici: update failed eGPU icon
2026-08-26 12:21:14 -07:00
Harald Schäfer d40df6f829 modeld: fall back on invalid big model outputs (#38700) 2026-08-26 12:06:53 -07:00
Nayan da28afca91 models: dual-slot backend (qcom/usbgpu) with ref-based downloads (#1966)
* models: dual-slot backend (qcom/usbgpu) with ref-based downloads

* models: restore get_active_source and the usbgpu-to-qcom fallback

* models: fix per-slot validation and cap mismatched-source refetches

* ui/models: select models by ref and seed the usbgpu slot on migration

* models: drop defensive attribute guards on capnp bundles

* models: remove vestigial fetcher state and dead fallbacks

* models: resolve the active bundle from the active source slot only

* models: pass the usbgpu kwarg through the modeld test stubs

* models: resolve the displayed model from the active slot in ui_state

* models: correct the validation memo type hint

* models: drop docstrings that restate the function name

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2026-08-26 02:34:02 -04:00
Jason Wen 15f201caed ui: use full big model failure detection for sidebar and home eGPU icons (#1969) 2026-08-25 21:07:44 -04:00
Jason Wen 1d4558c067 [TIZI/TICI] sidebar: show eGPU icon when chestnut is present (#1968)
* [tizi/tici] sidebar: show eGPU icon when chestnut is present

* matchy match

* fix
2026-08-25 20:47:52 -04:00
Jason Wen 78a766eb61 ui: fix scrolling label speed at non-60fps refresh rates (#1967)
* ui: fix scrolling label speed at non-60fps refresh rates

* send it

* nope

* more
2026-08-25 20:38:02 -04:00
Jason Wen b742b96c44 [MICI] ui: four-state eGPU icon for non-default big models (#1945)
* ui: four-state eGPU icon for non-default big models

* oops

* try this out

* align
2026-08-25 12:04:39 -04:00
Jason Wen 6fd4278e02 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-08-25 02:03:09 -04:00
Jason Wen 25c25047b8 models: persist model selection per catalog across chestnut state changes (#1960) 2026-08-25 01:12:12 -04:00
Jason Wen cefe5737b9 models: fix current model not updating on chestnut status (#1959)
* models: preserve user model selection across reboots and power cycles

* no

* again

* idk

* over
2026-08-25 00:41:31 -04:00
Jason Wen 760c19d3f9 ui/models: handle missing files during cache size calculation (#1958) 2026-08-24 23:31:52 -04:00
James Vecellio-Grant 45814e3313 modeld_v2: spatial features (#1934)
* modeld_v2: spatial features

* Update fetcher.py

* dont reshape non 4 dim arrays

* realize for non compiled

* Update compile_modeld.py

* god dammit it was realize()

* it was fucking frozen tinygrad. just need to recompile

* bump

* ci: add is_big flag to metadata.json to support backward compat

* Update model_generator.py

* Update sunnypilot-build-model.yaml

* Update helpers.py

* Revert "Update helpers.py"

This reverts commit 3a955ca11a.

* Reapply "Update helpers.py"

This reverts commit ca9c6e1933.

* models: use less strict chestnut detection state

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2026-08-24 22:52:29 -04:00
Jason Wen 2ba91d2be5 ci: add tinygrad ref check to prepare_chestnut and even faster prebuilt stages (#1957)
* ci: faster prebuilt stages

* tg check chestnut

* zoomer!
2026-08-24 22:42:45 -04:00
Jason Wen 19f83b274f ci: identical environment for publish_chestnut prebuilt 2026-08-24 22:06:20 -04:00
Jason Wen d14d0b1dd0 ci: parallelize models chunk downloads and split branch publishing (#1955)
* ci: parallelize model chunk downloads and better publish

* ci: download all model chunks in parallel with xargs -P8

* split split

* ew

* must require
2026-08-24 21:48:53 -04:00
Jason Wen 6cc5f3aad8 ci: fix DM model build, separate HF defaults paths, nuke build races (#1956)
* ci: fix DM model build, separate HF defaults paths, nuke build races

* more split!

* name

* ci: download driving and DM model chunks into chestnut prebuilt output
2026-08-24 20:02:48 -04:00
Jason Wen 8e16c9babb ci: offload small model compilation (#1952)
* ci: compile default big model with stock modeld

* Revert "Revert big RL model (#38627)"

This reverts commit 516ec1e682.

* ci: compile default small model with stock modeld compiler

* ci: offload small model compilation

* Reapply "Revert big RL model (#38627)"

This reverts commit d06cfabb62.

* Reapply "Revert big RL model (#38627)"

This reverts commit d06cfabb62.
2026-08-24 16:24:19 -04:00
Jason Wen 2bcfed5c71 ci: compile default models with stock modeld (#1954)
* ci: compile default big model with stock modeld

* Revert "Revert big RL model (#38627)"

This reverts commit 516ec1e682.

* ci: compile default small model with stock modeld compiler

* Reapply "Revert big RL model (#38627)"

This reverts commit d06cfabb62.
2026-08-24 15:37:46 -04:00
Jason Wen 66cf334067 ci: unify default model build into single workflow (#1951)
* ci: unify default model build into single workflow

* ci: consolidate upload jobs and add tinygrad ref validation
2026-08-24 12:35:37 -04:00
Jason Wen 94ed0608e6 models: use less strict chestnut detection state (#1948) 2026-08-24 01:40:31 -04:00
Jason Wen 0fbca979df models: show big model list when Chestnut present (#1943) 2026-08-23 20:19:57 -04:00
Jason Wen dcddb2a0bd models: revert icon override from this branch scope 2026-08-23 19:53:46 -04:00
Jason Wen 699eaf7957 include them! 2026-08-23 19:16:15 -04:00
Jason Wen c246e6318a Merge branch 'master' into models-good-detect 2026-08-23 16:56:44 -04:00
Jason Wen 718db8c62e Sync: commaai/openpilot:mastersunnypilot/sunnypilot:master (#1944) 2026-08-23 16:56:00 -04:00
Jason Wen c2214d4c32 Merge commit '084747c75d2cbd23af65ab7a9e770bbd7b98bac9' into sync-20260823
# Conflicts:
#	openpilot/common/params_keys.h
2026-08-23 15:36:54 -04:00
Jason Wen 0de7fbf33d new 2026-08-23 15:06:53 -04:00
DevTekVE 451cc3445a Bump opendbc 2026-08-23 16:35:28 +02:00
Jason Wen 211f990f6b models: fix sunnylink default model display and false big model re-downloading (#1941)
* big needs small

* no download

* actually

* send it
2026-08-23 04:04:46 -04:00
Jason Wen 97468e4fa4 [TIZI/TICI] ui: remove calibration reset dialog on model change (#1942) 2026-08-23 03:48:52 -04:00
Jason Wen 6c6fba9a14 ci: fix flaky LLK test (#1940) 2026-08-23 02:57:02 -04:00
Jason Wen 34621cf816 ci: refactor big model chunk handling (#1939) 2026-08-23 02:48:10 -04:00
Jason Wen 086530b7c6 [TIZI/TICI] ui: fix path width during gas and steering override (#1938) 2026-08-22 21:47:38 -04:00
Jason Wen 4f46433e2b alerts: add branch metadata to chestnut offroad warning (#1936)
* alerts: add branch metadata to chestnut offroad warning

* all branches
2026-08-22 10:20:06 -04:00
Jason Wen 5a8567e3e7 ci: chestnut prebuilt branches (#1935)
* ci: chestnut prebuilt branches

* fix

* nope

* big

* try again

* diff

* malformed

* auth

* more
2026-08-22 03:41:48 -04:00
Jason Wen 07558166c8 ci: only check default model on dispatch 2026-08-22 00:32:35 -04:00
Jason Wen ca9338812e ci: prep for chestnut prebuilts 2026-08-22 00:16:10 -04:00
granolaFPV 4667241fe7 [TIZI/TICI] ui: dynamic path width color (#1926)
* Fix UI path color and thickness based on lateral steering state (Issue #1441)

* Fix UI path color and thickness based on lateral control engagement (Issue #1441)

* Fix UI path width and color based on MADS lateral engagement (Issue #1441)

* Fix UI path width and color based on MADS lateral engagement (Issue #1441)

* move to ModelRendererSP

* match torque bar

* same behavior across the board

* simplify

---------

Co-authored-by: Brennan Browne <brennanbrowne@google.com>
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2026-08-21 20:51:43 -04:00
Shane Smiskol 084747c75d Fix button label widths (#38680)
* Revert "ui: fix text and icon overlap on button (#38628)"

This reverts commit d9c4120f89.

* simple

* can do this

* fix eliding

* Revert "fix eliding"

This reverts commit b271a350182ad87f9942d7363383ee8ec72d0e36.

* clean up

* clean up
2026-08-21 15:41:12 -07:00
Marceline Milligan a49c260927 ui: show default big model name when eGPU present/active (#1930)
* Name the big default model in the device UI

Build on the default big-model metadata from #1929 and resolve the displayed model from cached capability and modeld runtime state. Keep model selection behavior unchanged.

Assisted-by: GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove get_default_model_label

* simplify

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: nayan <nayan8teen@gmail.com>
2026-08-21 13:26:52 -04:00
Jason Wen 5ad2bfdb75 ci: deprecate GitHub runners (#1933) 2026-08-21 00:35:38 -04:00
Jason Wen b742557d62 sunnylink: add model resolver (#1931)
* models: add get_default_model resolver for sunnylink

* models: move get_default_model to default_model.py
2026-08-20 21:56:57 -04:00
Nayan 5ecd05aedf models: add big model to default model resolution (#1929)
* device

* sunnylink

* lint

* lfs?

* Revert "lfs?"

This reverts commit bcdaec6b4c.

* update path

* Scope the default big model down to the sunnylink schema

* Drop the mock-only default model test

* Move the default model resolver out to separate PR

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2026-08-20 21:31:32 -04:00
Jason Wen 5ae100aa1d models fetcher: bump big model to v21 2026-08-20 19:19:59 -04:00
Jason Wen be76a88b80 ci override LFS fetch exclude for real ONNX file retrieval (#1928)
ci: override lfs.fetchexclude so the model fetch pulls real ONNX files instead of pointers
2026-08-20 19:12:59 -04:00
James Vecellio-Grant 049d225d5a ci: Dedicated Model Runner (#1922)
* ci: Dedicated Model Runner

* recurse

* not needed

* fix wrapper

* whoops

* bypass

* modeld_v2: restore chestnut link check before big model build

* modeld_v2: stage onnx to disk instead of shared memory

* ci: clear unchunked onnx temps before model build

* ci: stream the pkl hash instead of loading it into memory

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2026-08-20 16:18:17 -04:00
stef 555f48c5d2 params: remove livestream param on ignition (#38679)
* remove livestream param on ignition

* simplify process config
2026-08-19 14:21:59 -07:00
stef dcf9d25bf3 webrtcd: more descriptive errors (#38677)
more descriptive errors
2026-08-19 14:02:03 -07:00
stef a8d1a280c6 webrtcd/athenad: we don't have to fail on no car params (#38678)
we don't have to fail on no car params
2026-08-19 13:49:19 -07:00
stef 5b36799eec webrtc: fix message handler race (#38675)
open message handler early
2026-08-18 21:56:33 -07:00
stef 20fdc3d824 webrtcd: cloud logging (#38674)
* logging

* remove test

* get rid of redudant try except

* fix logger context
2026-08-18 21:13:57 -07:00
Shane Smiskol 7bd6cad821 Re-open agnos updater UI if crash (#38672)
loop if crash
2026-08-18 19:06:48 -07:00
Jason Wen 1daa2d9081 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-08-17 10:29:01 -04:00
Jason Wen 7a9099818f Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2026-08-14 17:57:37 -04:00
Jason Wen ef4fa48f33 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-08-14 17:18:06 -04:00
Jason Wen e426ced4c0 bump 2026-08-11 17:47:18 -04:00
Jason Wen 7681ba763b Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2026-08-11 17:47:12 -04:00
Jason Wen b04d666a68 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-08-10 04:27:58 -04:00
Jason Wen fb026b86a3 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2026-08-06 21:58:57 -04:00
Jason Wen cff4c5e589 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-08-06 21:58:50 -04:00
Jason Wen e0cca951f3 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-07-25 08:52:04 -04:00
Jason Wen 2439c048ed Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2026-07-20 15:12:17 -04:00
Jason Wen 7b54d7d7a0 bump 2026-07-20 09:51:59 -04:00
Jason Wen b2bdb5787f Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
#	openpilot/tools/plotjuggler/layouts/analyzing-panda-block-angle-hkg.xml
#	openpilot/tools/plotjuggler/layouts/analyzing-torque-angle-hkg.xml
#	openpilot/tools/plotjuggler/layouts/hkg_angle_control.xml
#	openpilot/tools/plotjuggler/layouts/safety-limits-angle-kkg.xml
#	pyproject.toml
#	scripts/lint/lint.sh
2026-07-20 09:50:52 -04:00
Jason Wen 282f517b78 upstream changes 2026-06-09 00:18:40 -04:00
Jason Wen 1779144ff9 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
#	scripts/lint/lint.sh
2026-06-08 23:53:43 -04:00
Jason Wen 54fb2750ba Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2026-05-09 11:47:25 -04:00
DevTekVE 14431ab77f Merge branch 'master' into hkg-angle-steering-2025 2026-05-05 10:37:34 +02:00
Jason Wen 9d45db41b3 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
#	system/version.py
2026-05-03 16:08:40 -04:00
Jason Wen b57c593d92 bump 2026-04-20 23:31:05 -04:00
nayan 0674c42866 good bot
fix state
2026-04-19 08:52:19 +02:00
nayan 1f7bcf246a Bringing sl change to validate 2026-04-18 09:25:12 +02:00
DevTekVE b0512dc523 Revert "safety: dynamically relax lateral jerk limits during accel conflicts"
This reverts commit b0644a37e3b5a7b941bd4347f4c7ef0f25f44fbc.
2026-04-12 11:25:31 +02:00
DevTekVE cd88fd3850 Bringing shane's improvements on the angle steering branch 2026-04-10 12:09:12 +02:00
DevTekVE ff78eaeba1 Merge branch 'master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-04-10 11:38:19 +02:00
DevTekVE 92c8aeeb92 lint 2026-04-10 11:34:29 +02:00
DevTekVE ecaa20e7be Add source map configuration to VSCode launch settings
- Enables better debugging by mapping sources to `${workspaceFolder}/opendbc/safety`.
2026-04-05 15:10:21 +02:00
DevTekVE 859c98c9d8 Refactor PlotJuggler layouts and optimize custom math equations
- Introduced new tabs for `Smoothing and Torque ceilings`.
- Updated custom math equations for cleaner logic and added new snippets for angle smoothing, ceiling brackets, and roll compensation.
2026-04-05 12:14:13 +02:00
DevTekVE 69cbe8c6ed Enhance torque reduction logic with speed and steering error adjustments
- Introduced speed-dependent and error-sensitive dynamic torque ceilings.
- Improved interpolation for smoother torque application.
2026-04-04 11:55:14 +02:00
DevTekVE 1aafe92fc0 Remove hyundai_canfd_ccnc.dbc and update dependencies
- Deleted `_hyundai_canfd_ccnc.dbc` and its import references across related files.
- Merged relevant signals and comments into `hyundai_canfd_og.dbc` for consolidated usage.
- Cleaned up obsolete imports in `hyundai_canfd.dbc`.
2026-04-04 08:46:09 +02:00
DevTekVE adb6b9fb12 Test changing priority for reading dbc files to in memory first 2026-04-04 08:29:57 +02:00
DevTekVE f67a9f4624 Revised steering angle smoothing matrix and logic cleanup
- Adjusted `SMOOTHING_ANGLE_VEGO_MATRIX` to refine torque smoothing at mid-range speeds.
- Removed unused deadzone logic for cleaner and more consistent angle smoothing.
2026-04-03 18:10:05 +02:00
DevTekVE 094e834ac4 cleanup: remove unused datafile references and expand ignored patterns
- Deleted `<previouslyLoaded_Datafiles>` sections from PlotJuggler layouts to streamline configuration files.
- Added `.ipynb` files to the `pyproject.toml` ignore list for cleaner tooling.
2026-04-03 17:19:18 +02:00
DevTekVE e4067060b9 Adding ioniq 9 and updating ioniq 5 pe n-line fingerprint 2026-04-03 17:06:15 +02:00
DevTekVE 86a14640b5 Tune speed-dependent steering smoothing to eliminate EPS whine
Reimplemented an Exponential Moving Average (EMA) filter on the requested
steering angle (`apply_angle`). The model's raw high-frequency micro-corrections
at low speeds cause acoustic resonance (whine) in the EPS motor. This filter
dynamically adjusts the smoothing factor (alpha) based on vehicle speed to
silence the EPS at a crawl while maintaining zero-latency precision on the highway.

Key Behaviors & Speed Matrix:
* Deadzone: Ignores angle changes ≤ 0.1° to preserve straight-line tracking.
* 0 km/h (0 mph) -> Alpha: 0.05 (Max smoothing to eliminate stationary vibration)
* 30.6 km/h (19 mph) -> Alpha: 0.10 (Heavy smoothing for stable residential turning)
* 39.6 km/h (25 mph) -> Alpha: 0.30 (Moderate smoothing)
* 49.7 km/h (31 mph) -> Alpha: 0.60 (Light smoothing for responsive city driving)
* 80.0 km/h (50 mph) -> Alpha: 1.00 (Zero smoothing / raw signal for high-speed precision)
2026-04-03 16:13:05 +02:00
DevTekVE 59fb84f0d5 Remove HKG angle control tuning components and dependencies
- Deleted HKG-specific angle tuning settings and related UI elements.
    - Removed tuning parameter handling and smoothing logic from carcontroller.
    - Simplifies codebase and eliminates unused parameters.
2026-04-03 15:33:57 +02:00
DevTekVE 78de201ce7 Bring it back to 360 for hda1s 2026-04-03 15:14:37 +02:00
DevTekVE 17e49f69cc refactor: update parameterized tests and extend Hyundai Ioniq 5 PE model years
- Replaced `parameterized.expand` with new `parameterized` syntax for cleaner test definitions.
- Added 2026 model year to Hyundai Ioniq 5 PE in `car_list.json`.
2026-04-03 14:39:45 +02:00
DevTekVE bec75ebdd8 remove: TorqueReductionGainController and its associated tests
- Fully deprecates the no longer used torque reduction logic.
- Cleans up outdated functionality for clarity and maintainability.
2026-04-03 14:14:07 +02:00
DevTekVE 473832efa7 Merge branch 'master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-04-03 14:00:54 +02:00
DevTekVE 2cceeb3552 fix: clip steering angle to respect angle limits when latActive is off
- Prevents potential out-of-bound steering angle values.
- Ensures compliance with defined steering angle constraints.
2026-04-03 01:06:37 +02:00
DevTekVE 923a47f713 Reverting all the new control changes until we properly test them on a few variants to ensure it is safe.
Revert "upstream pending tune"

This reverts commit b51d9af9a0.

Revert "show torque reduction gain"

This reverts commit ded0b506d6.

Revert "must gate"

This reverts commit 8b60649eed.

Revert "bump"

This reverts commit 221c219fca.

Revert "temp: comment out blind-spot monitoring signals due to DBC changes"

This reverts commit 790a762a05.
2026-04-01 09:29:53 +02:00
DevTekVE 790a762a05 temp: comment out blind-spot monitoring signals due to DBC changes
- Avoided crash caused by missing signals in updated DBC definitions.
- Added a TODO to revisit and validate blind-spot logic.
2026-04-01 08:55:26 +02:00
Jason Wen 221c219fca bump 2026-03-31 21:57:58 -04:00
Jason Wen 8b60649eed must gate 2026-03-31 21:42:06 -04:00
Jason Wen ded0b506d6 show torque reduction gain 2026-03-31 07:46:13 -04:00
Jason Wen b51d9af9a0 upstream pending tune 2026-03-31 06:56:52 -04:00
Jason Wen 4460ce8166 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-03-18 03:43:10 -04:00
Jason Wen f1aa0c7f78 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2026-03-07 01:49:09 -05:00
Jason Wen e7c8126fd9 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-03-05 02:01:04 -05:00
Jason Wen 125999c364 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-03-01 16:34:41 -05:00
Jason Wen 5b25ea7f99 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2026-02-28 15:47:57 -05:00
Jason Wen b7e2631286 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-02-27 21:56:34 -05:00
Jason Wen a838871189 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-02-19 01:51:17 -05:00
Jason Wen 1827331599 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-02-17 20:12:13 -05:00
Jason Wen 5c777bbe01 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2026-02-13 23:41:06 -05:00
Jason Wen fb97f993d1 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2026-02-13 17:32:10 -05:00
Jason Wen 94b67077e3 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-02-12 23:35:46 -05:00
Jason Wen 14f17699b9 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2026-02-11 00:35:20 -05:00
Jason Wen 80e27d5cbb Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2026-02-10 23:41:27 -05:00
Jason Wen 039dbcd877 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2026-02-09 01:46:00 -05:00
Jason Wen 8c134ae555 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-02-08 20:04:01 -05:00
Jason Wen 276c7a2b34 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-02-02 22:40:14 -05:00
Jason Wen 0e2dbcebfa Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-01-26 11:45:29 -05:00
Jason Wen bbb7760a95 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2026-01-19 01:43:21 -05:00
Jason Wen 23a27b2fbf Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2026-01-09 18:48:43 -05:00
Jason Wen 8b78107a40 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2025-12-31 00:49:56 -05:00
Jason Wen c27b6007de wrong bump? 2025-12-26 10:08:31 -05:00
Jason Wen 36c2dce247 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2025-12-26 10:08:23 -05:00
Jason Wen 8035039731 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2025-12-23 00:52:24 -05:00
Jason Wen 8aa6c9440f Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2025-12-20 17:01:25 -05:00
Jason Wen 34ef40fd81 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2025-12-18 00:19:20 -05:00
Jason Wen 4d044d7618 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2025-12-15 02:25:04 -05:00
Jason Wen 6247e3dc84 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2025-12-13 01:59:06 -05:00
Jason Wen b1131289b7 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2025-12-07 01:37:16 -05:00
Jason Wen 35dc7d661e Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2025-12-07 00:11:54 -05:00
DevTekVE c39b2dad94 Merge branch 'master' into hkg-angle-steering-2025 2025-11-29 10:12:31 +01:00
Jason Wen e6b769245c Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
#	selfdrive/ui/sunnypilot/SConscript
#	selfdrive/ui/sunnypilot/qt/offroad/settings/lateral_panel.cc
#	selfdrive/ui/sunnypilot/qt/offroad/settings/lateral_panel.h
#	selfdrive/ui/sunnypilot/qt/offroad/settings/longitudinal_panel.cc
2025-11-25 18:54:02 -05:00
DevTekVE 8b94f8b2f8 Merge branch 'master' into hkg-angle-steering-2025 2025-11-06 18:31:16 +01:00
Jason Wen dec014cd17 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	sunnypilot/selfdrive/car/interfaces.py
2025-11-04 17:45:05 -05:00
DevTekVE 7b40272866 Add sunnypilot-specific stats logging and handling
- Introduced `StatLogSP` for sunnypilot-specific metrics.
- Integrated stats collection and submission pathways for sunnylink.
- Extended parameters and handlers to support additional metrics.
- Added gzip compression and base64 encoding for oversized payload handling.
2025-11-04 21:20:16 +01:00
DevTekVE 506456e7f0 Merge branch 'master' into hkg-angle-steering-2025 2025-11-01 13:31:23 +01:00
DevTekVE 4ea4b9d177 Merge branch 'master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2025-10-31 06:59:39 +01:00
DevTekVE 0e2313dc31 Merge branch 'master' into hkg-angle-steering-2025 2025-10-18 11:14:45 +02:00
Jason Wen 79ea7db103 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2025-10-17 23:42:01 -04:00
Jason Wen f9ae9192fa Merge branch 'ui-icbm-universal' into hkg-angle-steering-2025 2025-10-17 21:55:00 -04:00
Jason Wen 7ec23006c6 check this 2025-10-17 21:54:44 -04:00
Jason Wen 2be9447a6c Merge branch 'ui-icbm-universal' into hkg-angle-steering-2025 2025-10-17 21:19:09 -04:00
Jason Wen cfd926778e always init true 2025-10-17 21:19:03 -04:00
Jason Wen a97a67e3d0 need 2025-10-17 21:17:31 -04:00
Jason Wen 518b6de08d Merge branch 'ui-icbm-universal' into hkg-angle-steering-2025 2025-10-17 21:15:59 -04:00
Jason Wen 6933e3bcdb fix cruise toggles 2025-10-17 21:15:43 -04:00
Jason Wen 05e0ca8bee some more 2025-10-17 20:46:36 -04:00
Jason Wen 410614fcf3 single location 2025-10-17 20:26:18 -04:00
Jason Wen e2bc0996ef Merge branch 'ui-icbm-universal' into hkg-angle-steering-2025 2025-10-17 12:21:47 -04:00
Jason Wen 1be0c20cf5 oops 2025-10-17 12:21:37 -04:00
Jason Wen 839143b9ed oops 2025-10-17 12:20:23 -04:00
Jason Wen bce86637ae Merge branch 'ui-icbm-universal' into hkg-angle-steering-2025 2025-10-17 12:15:23 -04:00
Jason Wen b833d3ee89 ui: update ICBM-related settings handling 2025-10-17 12:14:44 -04:00
Jason Wen e0441dfb4b Merge branch 'sla-event' into hkg-angle-steering-2025 2025-10-17 11:58:42 -04:00
Jason Wen 62ec40bba6 Speed Limit Assist: update active event handling 2025-10-17 11:58:19 -04:00
Jason Wen 15c6d38028 bump 2025-10-16 17:05:58 -04:00
Jason Wen 56eb9f555c Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	sunnypilot/selfdrive/controls/lib/e2e_alerts_helper.py
2025-10-16 01:12:10 -04:00
Jason Wen 2f9951df02 Merge branch 'e2e-alert-state-machine' into hkg-angle-steering-2025 2025-10-15 23:55:40 -04:00
Jason Wen 6030bf4da3 less 2025-10-15 23:52:03 -04:00
Jason Wen 074694d660 lead depart: only arm if we have a confirmed close lead for over a second after allowing alert 2025-10-15 23:48:13 -04:00
Jason Wen df35f48f3b magic 2025-10-15 22:56:08 -04:00
Jason Wen 4fb9704540 time based 2025-10-15 22:51:40 -04:00
Jason Wen 48cbe266fc 10 frames for both 2025-10-15 22:50:42 -04:00
Jason Wen 21aa7ff367 rename 2025-10-15 22:47:08 -04:00
Jason Wen 7caf05dd51 not used 2025-10-15 22:41:00 -04:00
Jason Wen 2d779f5db9 E2E Helper: universal state machine 2025-10-15 22:38:57 -04:00
Jason Wen 18208f1da0 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2025-10-15 17:41:56 -04:00
Jason Wen 5a3c6ddf57 gate lka angle steering out of alpha long 2025-10-14 23:25:00 -04:00
Jason Wen eaa8732ab0 Merge branch 'e2e-alerts-cooldown' into hkg-angle-steering-2025 2025-10-14 14:44:00 -04:00
Jason Wen 6f0284c84f try preventing startup false trigger 2025-10-14 14:43:47 -04:00
Jason Wen 999ea03f23 try preventing startup false trigger 2025-10-14 14:42:42 -04:00
Jason Wen 0975db3ff1 Merge branch 'e2e-alerts-cooldown' into hkg-angle-steering-2025 2025-10-14 14:31:59 -04:00
Jason Wen 8d70a8b80a only when long not engaged 2025-10-14 14:31:48 -04:00
Jason Wen da93f92887 rename 2025-10-14 14:26:57 -04:00
Jason Wen 9117f6c071 Merge branch 'e2e-alerts-cooldown' into hkg-angle-steering-2025 2025-10-14 14:23:46 -04:00
Jason Wen 3567ff9691 introduce recent moving check 2025-10-14 14:22:41 -04:00
Jason Wen f44ae2ced9 too complicated 2025-10-14 14:08:40 -04:00
Jason Wen 376e0ca615 only allow one trigger per standstill session 2025-10-14 14:07:16 -04:00
Jason Wen 32b7686468 Merge branch 'master' into e2e-alerts-cooldown 2025-10-14 11:40:58 -04:00
nayan b9e0f52ea9 E2E Alert Cooldown 2025-10-14 07:43:15 -04:00
Jason Wen a3163b680f Merge branch 'sla-chimes' into hkg-angle-steering-2025 2025-10-14 02:01:05 -04:00
Jason Wen 8a927d808f Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2025-10-14 02:00:36 -04:00
Jason Wen bc7d5e474d Speed Limit Assist: audible alerts for certain states 2025-10-14 01:45:13 -04:00
Jason Wen 36f192b5fe Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2025-10-13 03:10:47 -04:00
Jason Wen ffd5cd4ac2 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2025-10-11 02:29:47 -04:00
Jason Wen e4a00fcd6c Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025 2025-10-10 17:28:16 -04:00
Jason Wen 0bdcb41103 Merge remote-tracking branch 'sunnypilot/sunnypilot/master' into hkg-angle-steering-2025
# Conflicts:
#	common/params_keys.h
#	opendbc_repo
#	selfdrive/ui/sunnypilot/qt/offroad/settings/lateral_panel.cc
2025-10-10 17:02:14 -04:00
DevTekVE 78051085ca Merge branch 'master' into hkg-angle-steering-2025 2025-09-23 07:54:32 +02:00
DevTekVE 4910d5809a bump opendbc 2025-09-23 07:45:39 +02:00
DevTekVE 8dd862ff28 yikes, becoming picky huh? 2025-09-14 22:50:21 +02:00
DevTekVE 1b57497da9 Merge remote-tracking branch 'origin/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2025-09-14 22:48:00 +02:00
DevTekVE 8d8d1ffc7a bump opendbc again 2025-09-14 22:46:20 +02:00
DevTekVE c5919d5495 wrong dbc lol 2025-09-14 22:42:19 +02:00
DevTekVE da71951c95 This is no longer in use nor needed. Bai! 2025-09-14 12:56:28 +02:00
DevTekVE 25a152cd8b Merge remote-tracking branch 'origin/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2025-09-14 12:44:23 +02:00
DevTekVE 9077a1082a Sorry for C3 :( but moving you out to a last working branch before I sync it up 2025-09-14 06:52:24 +02:00
DevTekVE 7ae7000254 Rework override behavior and feeling 2025-09-14 06:38:00 +02:00
DevTekVE 7029455706 better juggle 2025-09-13 08:32:27 +02:00
DevTekVE 3a71a62215 Merge branch 'master' into hkg-angle-steering-2025 2025-09-12 10:14:52 +02:00
DevTekVE 00622e8c33 Merge remote-tracking branch 'origin/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2025-09-05 09:46:41 +02:00
DevTekVE 549da3ee92 Merge branch 'master' into hkg-angle-steering-2025 2025-09-04 20:20:37 +02:00
DevTekVE e038a65ef8 Merge branch 'master' into hkg-angle-steering-2025 2025-08-31 13:14:33 +02:00
DevTekVE 8d0513c657 dbc: update CHECKSUM format for multiple messages to improve data integrity 2025-08-30 19:56:21 +02:00
DevTekVE 2cea48f4cd Merge remote-tracking branch 'origin/master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2025-08-30 15:27:45 +02:00
DevTekVE 8855a9ab65 Improve surprise jerk by safety blocks 2025-08-26 14:47:26 +02:00
DevTekVE 32321c01cc A bit more helpful safety block investigation help 2025-08-26 10:09:11 +02:00
DevTekVE 52cd65fefb lint dont bother me 2025-08-25 08:24:24 +02:00
DevTekVE 1fcdeccd40 Merge branch 'master' into hkg-angle-steering-2025
# Conflicts:
#	common/params_keys.h
#	opendbc_repo
2025-08-25 08:01:51 +02:00
DevTekVE 3517c36978 Revert "Add HkgAngleDebug structure and enhance angle debugging in car controller" 2025-08-24 19:02:48 +02:00
DevTekVE fb30c3c1e8 cleanup and honour params 2025-08-23 15:42:47 +02:00
DevTekVE 1c25e568d5 Update steering pressed logic to include hands-on-wheel detection for improved safety 2025-08-23 15:11:07 +02:00
DevTekVE f172122b7c Update steering pressed logic to include hands-on-wheel detection for improved safety 2025-08-22 19:55:16 +02:00
Jason Wen 67d6cdc7cd Merge remote-tracking branch 'sunnypilot/sunnypilot/hkg-angle-steering-2025' into hkg-angle-steering-2025 2025-08-21 16:39:36 -04:00
DevTekVE 6724085cfd Refactor angle limit calculations and adjust average road roll for improved steering dynamics 2025-08-21 20:09:57 +02:00
DevTekVE 6774f34eee Refine non-linear mapping in torque reduction gain calculation for improved steering response 2025-08-21 00:21:46 +02:00
DevTekVE 6d0402896d Adjust STEER_THRESHOLD and refine non-linear mapping in torque reduction gain calculation for improved steering response 2025-08-20 23:16:04 +02:00
DevTekVE 344021a3d9 Adjust non-linear mapping in torque reduction gain calculation for improved response 2025-08-20 19:54:31 +02:00
DevTekVE a9b85ab27d Refactor HkgAngleDebug structure to include current and baseline limits for angle parameters 2025-08-20 19:54:10 +02:00
DevTekVE 17204a46e4 Add HkgAngleDebug structure and enhance angle debugging in car controller 2025-08-20 18:39:53 +02:00
DevTekVE 7c4d415462 Enhance torque reduction gain calculation with non-linear mapping and smoothing 2025-08-20 00:12:43 +02:00
DevTekVE b8985b6d72 Enhance torque reduction gain calculation with non-linear mapping and smoothing 2025-08-19 19:50:13 +02:00
DevTekVE c669473f88 Refine torque reduction parameters and update UI for angle error analysis 2025-08-19 19:35:34 +02:00
DevTekVE 8751435bf5 Improving tq redc gain and override behavior 2025-08-19 10:08:51 +02:00
DevTekVE 30ae210761 Merge branch 'master' into hkg-angle-steering-2025 2025-08-19 10:07:42 +02:00
Jason Wen 3eb693f58b Merge remote-tracking branch 'sunnypilot/sunnypilot/hkg-angle-steering-2025' into hkg-angle-steering-2025 2025-08-18 12:30:29 -04:00
DevTekVE 7b4a31c5ac bugfix 2025-08-17 16:07:45 +02:00
DevTekVE 1527c8cf88 bump opendbc 2025-08-17 15:31:31 +02:00
DevTekVE ab0a7ae666 no joystick on this branch, causing issues 2025-08-17 15:25:40 +02:00
DevTekVE 11ec2f1f21 Apply suggestions from code review 2025-08-17 15:10:17 +02:00
DevTekVE 2c6808d37e Merge branch 'master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2025-08-17 15:02:28 +02:00
Jason Wen 2e96382c49 notebook init 2025-08-17 00:34:01 -04:00
DevTekVE 0239e440ca Adjust replay 2025-08-15 10:32:07 +02:00
Jason Wen 76b972daff Hyundai angle steering: STEERING_ANGLE_2 available on all cars 2025-08-14 01:56:46 -04:00
Jason Wen bc3ef3e7dd Hyundai angle steering: hugging no more - use the true steering angle signal from MDPS 2025-08-13 23:08:32 -04:00
DevTekVE 9839291dd0 Merge remote-tracking branch 'origin/master' into hkg-angle-steering-2025 2025-08-13 20:13:42 +02:00
DevTekVE 17cba328d6 refactor: update torque tuning configuration for angle steering
- Adjusted torque tuning configuration to avoid reliance on torque controller for Hyundai angle steering.
- Simplified control logic by removing unnecessary checks for torque control type.
refactor: clean up code formatting and improve test structure for torque reduction gain
2025-08-13 20:03:51 +02:00
DevTekVE 86093765d8 Merge branch 'master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2025-08-13 07:55:46 +02:00
DevTekVE 05fa1c8ae8 Adjust default params and cleanup 2025-08-12 21:36:22 +02:00
DevTekVE e8a40d6b85 Merge branch 'master' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
#	system/manager/process_config.py
2025-08-10 14:24:10 +02:00
DevTekVE c265e0bb85 Merge remote-tracking branch 'origin/master' into hkg-angle-steering-2025
# Conflicts:
#	common/params_keys.h
#	opendbc_repo
#	system/manager/manager.py
2025-08-02 09:56:58 +02:00
DevTekVE c6b118788b Merge branch 'master-new' into hkg-angle-steering-2025 2025-07-31 18:25:08 +02:00
DevTekVE b004f6dbdc Merge branch 'master-new' into hkg-angle-steering-2025 2025-07-31 18:17:34 +02:00
DevTekVE cd4930b680 Bump opendbc 2025-07-26 10:56:49 +02:00
DevTekVE f861aca628 Refactor vehicle model initialization and adjust angle limits for baseline model 2025-07-26 08:12:06 +02:00
DevTekVE 1ff0d8e2ee please don't bother me anymore! 2025-07-26 08:09:51 +02:00
DevTekVE d76d70764e Update slip factor precision for Hyundai steering parameters
- Adjusted `slip_factor` in Hyundai CANFD safety modes for improved consistency and accuracy.
- Ensured proper representation of `slip_factor` output in test logs.
2025-07-25 20:17:19 +02:00
DevTekVE dc73ce0b71 save tools replay 2025-07-25 19:47:31 +02:00
DevTekVE 3f666748af bump opendbc 2025-07-25 14:16:02 +02:00
DevTekVE 8de8a8838c bumo opendbc 2025-07-25 14:08:53 +02:00
DevTekVE f563b7eb71 Refactor steering angle limit application for improved safety and model compliance 2025-07-25 12:12:42 +02:00
DevTekVE 3c18b83708 save temp 2025-07-25 09:56:30 +02:00
DevTekVE bd35f5904b Enhance steering angle rate limiting and safety enforcement logic
- Introduced explicit post-rate limiting using model-specific dynamics.
- Improved low-speed smoothing and precision of applied angles.
2025-07-25 08:12:41 +02:00
DevTekVE 474f2737f6 Refactor steering angle limit logic for conservative seleion
- Removed unused lateral accel/jerk logic for simplicity.
- Updated angle limit calculations to choose the smallest delta for safer control.
2025-07-24 21:48:24 +02:00
DevTekVE ea1ac4a212 Revert "Add configurable max lateral accel and jerk parameters for Hyundai vehicles"
This reverts commit b95f8c5929.

Revert "Add baseline safety model and improve steering angle limiting logic"

This reverts commit b53cbb2e18.

Revert "Disable lateral accel/jerk params and ensure float consistency in angle limits"

This reverts commit 165d7c7b36.
2025-07-24 10:02:48 +02:00
DevTekVE 165d7c7b36 Disable lateral accel/jerk params and ensure float consistency in angle limits
- Commented out unused lateral accel/jerk parameters for clarity.
- Ensured `np.clip` always returns a float for precision.
2025-07-24 09:41:52 +02:00
DevTekVE b53cbb2e18 Add baseline safety model and improve steering angle limiting logic
- Introduced a baseline safety model (`GENESIS_GV80_2025`) for comparison.
- Enhanced steer angle limit calculation using both baseline and current limits for improved safety and precision.
2025-07-24 09:38:03 +02:00
DevTekVE b95f8c5929 Add configurable max lateral accel and jerk parameters for Hyundai vehicles
- Introduced user-configurable options for max lateral acceleration and jerk.
- Enables fine-tuning of vehicle handling for smoother control.
2025-07-24 08:39:10 +02:00
DevTekVE e564bb0b85 bumo openbc 2025-07-23 18:46:52 +02:00
DevTekVE e2ec8a7b13 Refine lateral control limits and simplify safety model handling
- Reduced max lateral acceleration and jerk by 20% for smoother handling.
- Removed unused `get_safety_CP` function, simplifying `VehicleModel` initialization.
2025-07-22 08:07:51 +02:00
DevTekVE 75c6f0f10e Test with gv80 as baseline for limits 2025-07-20 22:14:18 +02:00
DevTekVE af38044b42 Merge branch 'master-new' into hkg-angle-steering-2025 2025-07-20 10:14:46 +02:00
DevTekVE 684fa846d8 Merge branch 'master-new' into hkg-angle-steering-2025
# Conflicts:
#	.codespellignore
#	opendbc_repo
#	system/manager/manager.py
2025-07-19 21:51:46 +02:00
DevTekVE 416e722855 Update baseline model to IONIQ 5 PE for improved angle safety tuning
- Replaced conservative GENESIS_GV80_2025 model with IONIQ 5 PE parameters.
- Adjusted steering parameters (ratio, slip factor, wheelbase) for better lateral control performance.
2025-07-19 21:46:17 +02:00
DevTekVE 9fd4613bbb Add 2025 Kia EV6 support with updated radar and camera fingerprints 2025-07-09 09:47:31 +02:00
DevTekVE a0362e3c5f "Refined UI labels and tooltips for HKG tuning options to improve clarity and user understanding." 2025-06-29 16:48:53 +02:00
DevTekVE e32ef1cdc0 Fix incorrect torque sign usage in torque reduction calculation
Ensure `actuators.torque` uses its absolute value in the `calculate_angle_torque_reduction_gain` method to prevent sign-related issues during Hyundai steering angle control.
2025-06-29 14:33:30 +02:00
DevTekVE 20673ec8a6 Adjust warning font size in angle tuning settings panel. 2025-06-29 13:35:09 +02:00
DevTekVE 53fbdf7329 Rename "IdleTorque" to "ActiveTorque" for clarity.
The parameter name "HkgTuningAngleIdleTorqueReductionGain" was updated to "HkgTuningAngleActiveTorqueReductionGain" across multiple files for better clarity and alignment with its functionality. This change ensures consistency in naming conventions and improves code readability.
2025-06-29 13:23:56 +02:00
DevTekVE 6f72b74fac Add idle torque reduction for Hyundai lateral control
Introduced `ANGLE_IDLE_TORQUE_REDUCTION_GAIN` to manage torque when the vehicle is stationary, ensuring smoother handling and better lane centering. Updated parsing, parameters, and UI settings to support this new idle torque parameter. Adjusted torque calculation logic and smoothing factor behavior for enhanced control flexibility.
2025-06-29 13:22:18 +02:00
DevTekVE e83705a32e Merge branch 'master-new' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2025-06-29 09:44:01 +02:00
DevTekVE 95d36b9ba2 Refactor and enhance HKG angle tuning logic.
Introduced a toggle for angle smoothing factor and renamed related parameters for clarity. Refactored backend settings to use new parameter names and expanded smoothing matrices for better tuning granularity. Updated UI elements to reflect these changes, emphasizing usability and consistency.
2025-06-28 21:25:44 +02:00
DevTekVE b9e74254bd Merge branch 'master-new' into hkg-angle-steering-2025 2025-06-27 10:46:22 +02:00
DevTekVE b4405b200d Merge remote-tracking branch 'origin/master-new' into hkg-angle-steering-2025 2025-06-25 09:27:21 +02:00
DevTekVE a8a3fdac54 Rename parameter in calculate_target_torque for clarity. 2025-06-23 22:53:45 +02:00
DevTekVE 4eddc622a7 Refactor torque calculations in Hyundai controller
Rename methods and variables for clarity in torque reduction and override calculations. Adjust logic to streamline handling of steering inputs and improve maintainability.
2025-06-23 22:51:40 +02:00
DevTekVE 4e42ada240 Refactor torque management in Hyundai controller for cleaner override and ramp logic
Extract torque ramping and override functionality into dedicated methods within `LkasTorqueManager` to improve maintainability and reduce redundancy. Simplify `update` logic by delegating state-specific operations to new methods.
2025-06-23 22:47:51 +02:00
DevTekVE 6266217655 Introduce LkasTorqueManager for LKAS torque handling in Hyundai controller
Encapsulate LKAS torque calculations, ramping, and override logic into the new `LkasTorqueManager` class to improve modularity and maintainability. Replace existing torque logic with calls to the manager.
2025-06-23 22:20:10 +02:00
DevTekVE ea09d32e98 Remove lateral acceleration logic from Hyundai steering controller 2025-06-23 21:55:17 +02:00
DevTekVE 15958c88d3 Refactor lateral acceleration scaling logic in Hyundai controller
Move scaling of `max_angle_delta` under high lateral acceleration to improve clarity and prevent redundant operations.
2025-06-23 20:22:03 +02:00
DevTekVE b80d7fb5ea Adding GV70 electrified 2026 2025-06-22 14:30:59 +02:00
DevTekVE 1c3d25c6ff Refine lateral acceleration handling in Hyundai steering logic
Enforce absolute check for `real_a_lat` against `MAX_LATERAL_ACCEL` to improve angle scaling under high lateral acceleration conditions.
2025-06-22 11:03:35 +02:00
rav4kumar c9f22b32c7 Revert "Incorporate lateral acceleration in Hyundai angle steering logic"
This reverts commit c0524985bb.
2025-06-21 11:40:58 -07:00
DevTekVE c0524985bb Incorporate lateral acceleration in Hyundai angle steering logic
Add handling for IMU lateral acceleration to refine steering angle limits in CAN FD configurations. Parse and utilize `IMU_LatAccelVal` signal for enhanced lateral control accuracy.
2025-06-21 17:34:13 +02:00
DevTekVE 83839c7ea7 Add angle steering support and refactor related logic for Hyundai CAN FD.
Introduced support for CAN FD angle steering, including updated parameters, signal parsing, and new tests. Refactored related steering logic for clarity, reducing unused code and enhancing maintainability.
2025-06-21 13:38:50 +02:00
DevTekVE c1a1d4b4c3 Update lint script to exclude .xml files in layouts directory
Added `layouts/.*\.xml` to `IGNORED_FILES` in `lint.sh` to prevent linting of layout XML files.
2025-06-20 10:50:45 +02:00
DevTekVE b5af7a905a Merge branch 'master-new' into hkg-angle-steering-2025 2025-06-18 20:13:45 +02:00
DevTekVE 96b1b2f55f Update steering request logic in Hyundai controller
Ensure steering request activation depends on lateral control being active. This adds clarity and aligns better with control logic requirements.
2025-06-17 18:51:54 +02:00
DevTekVE 9361ba5d70 Refactor Hyundai steering angle handling logic
Streamline steering angle calculations and fault avoidance logic by removing redundant comments and unused code. Simplified `round_angle` implementation for clarity and consistency.
2025-06-17 12:08:06 +02:00
DevTekVE 4e9014311e Refactor steeringPressed logic in Hyundai carstate.py.
Revised the determination of `steeringPressed` to account for both hands-on-wheel detection and torque overriding in CAN FD setups. Simplified fallback logic for non-CAN FD configurations for better code clarity and maintainability.
2025-06-12 00:31:07 +02:00
DevTekVE 232873fc70 Refine steering press detection logic.
Adjusted the sensitivity and threshold values for `HOD_Dir_Status` in steering press updates, improving accuracy in detecting steering input. This change aligns with updated parameter requirements for better responsiveness.
2025-06-12 00:14:05 +02:00
DevTekVE 0a61fca9c9 Fix steering press detection for Hyundai models.
Updated the condition to detect steering press by changing HOD_Dir_Status threshold from `> 2` to `>= 2`. This ensures the detection logic aligns correctly with expected behavior.
2025-06-12 00:06:52 +02:00
DevTekVE 480bdc34dc Add support for CANFD angle steering in Hyundai cars
Introduced handling for the `HOD_FD_01_100ms` message when the CANFD angle steering flag is enabled. This ensures proper message parsing and extends compatibility for specific Hyundai vehicle configurations.
2025-06-12 00:04:44 +02:00
DevTekVE 716b475a13 Update Hyundai controls for HOD status and steer limits
Adjusted the steering override frame window and incorporated new HOD_Dir_Status to improve hands-on detection. Added parsing for new signals in Hyundai CAN FD, enhancing steering override responsiveness and reliability.
2025-06-12 00:01:31 +02:00
DevTekVE b1ec5ec034 Adjust override angle cap in Hyundai car controller
Increased the minimum override angle cap from 0.01 to 0.1 and explicitly cast the maximum cap to a float. This change improves consistency and ensures proper handling of steering limits.
2025-06-11 23:20:19 +02:00
DevTekVE 470613c2b7 Adjust Hyundai steer override parameters for improved control.
Reduced the override frame window and updated the angle cap logic to use MAX_ANGLE_RATE. These changes aim to enhance steering responsiveness and safety by fine-tuning steer angle limits.
2025-06-11 23:07:49 +02:00
DevTekVE 336c5b4154 Remove smoothing_factor from Hyundai car controller logic
The `smoothing_factor` parameter and related logic have been removed to simplify the steering angle smoothing approach. All references and usage of this parameter have been eliminated, relying solely on speed-based dynamic interpolation. This change streamlines the code while maintaining functionality.
2025-06-11 23:04:32 +02:00
DevTekVE abdb9dc750 Adjust Hyundai steering override frame logic
Reduced `OVERRIDE_FRAME_WINDOW` and updated condition to properly respect override frame limits. This ensures smoother handling and more precise steering adjustments under certain driving scenarios.
2025-06-11 22:49:04 +02:00
DevTekVE ab98683973 Refactor steering override logic in Hyundai carcontroller
Replaced `recently_overridden` with `frames_since_override` for better granularity and added dynamic override angle limits using interpolation. These changes enhance steering control accuracy during user overrides and improve overall code readability.
2025-06-11 22:42:05 +02:00
DevTekVE 186c24dbe6 Refine Hyundai steering override handling logic
Adjusted logic for recently overridden steering to improve angle limits and torque smoothing. Removed unused or redundant code, optimizing the functionality and maintaining cleaner readability.
2025-06-11 21:49:18 +02:00
DevTekVE 9cdf6340a1 Refactor steering angle smoothing for clarity and reuse.
Extracted the steering angle smoothing logic into a standalone function `sp_smooth_angle` to enhance readability and reusability. Adjusted angle smoothing parameters and introduced a maximum vehicle speed threshold for applying smoothing. Minor updates improve maintainability and ensure consistent behavior across speed ranges.
2025-06-11 10:07:44 +02:00
DevTekVE 2855b1341c Adjust steering thresholds for Hyundai CAN FD vehicles
Updated `STEER_THRESHOLD` to 350 and `NO_LONGER_OVERRIDING_THRESHOLD` to 150 for better alignment with Hyundai CAN FD steering behavior. These changes ensure improved compatibility and more accurate steering response.
2025-06-10 09:53:08 +02:00
DevTekVE cf28f99976 Revert "Add twilsonco's LKAS torque calculator for improved lateral control"
This reverts commit b1770fb0e7aece0e160b1b083cb260edbbdc53dd.
2025-06-10 09:40:59 +02:00
DevTekVE a39d67dc47 Fix apply_angle_last reset logic in Hyundai carcontroller
Re-enables resetting `apply_angle_last` to `steering_angle` when steering is recently overridden. This ensures proper handling of steering angle limits during transitions.
2025-06-08 19:04:02 +02:00
DevTekVE 7e75257f12 Refine Hyundai steering control logic.
Simplified torque ramp-up logic by combining conditions and adjusted `STEER_THRESHOLD` for CANFD angle steering. These changes aim to enhance control precision and maintain consistency in overrides.
2025-06-08 19:02:49 +02:00
DevTekVE df38449553 Reduce override timeout for Hyundai carcontroller
Decrease the override timeout from 100 to 50 frames, ensuring quicker recognition of driver input override. This improves responsiveness and aligns with refined control behavior.
2025-06-08 18:22:44 +02:00
DevTekVE 1385ef3bc5 Fix steering control behavior during user override
Removed restrictive rate limiting during recent user overrides to improve steering response. Adjusted logic to ensure correct handling of steering angle when lateral control is inactive or overridden.
2025-06-08 18:01:47 +02:00
DevTekVE 7c23c11c51 Refine steering logic with override detection.
Adjust steering behavior to account for recent user overrides, improving safety and control. Introduced a "recently_overridden" check to limit angle rates and torque adjustments when user intervention is detected.
2025-06-08 17:52:16 +02:00
DevTekVE aeff2e12ec Refine steering logic with user override handling.
Added logic to use the current steering angle when the steering wheel is pressed, ensuring smoother transitions during user overrides. Updated function parameters and implementation to reflect this enhancement.
2025-06-08 17:38:34 +02:00
DevTekVE 7274899671 Refactor Hyundai override logic for steering thresholds
Removed redundant `recently_overridden` logic and introduced a more robust approach for tracking user steering overrides. Added `NO_LONGER_OVERRIDING_THRESHOLD` and updated conditions to improve steer override handling. Adjustments ensure smoother torque transitions and more accurate steering state detection.
2025-06-08 17:00:44 +02:00
DevTekVE 6c00fd608f pass tests? 2025-06-08 12:28:11 +02:00
DevTekVE df6a034c11 Bump opendbc 2025-06-08 12:26:09 +02:00
DevTekVE acb109c290 adding plotjuggler stuff 2025-06-08 10:02:44 +02:00
DevTekVE b227b00249 Update torque clamping to use parameterized min torque
Replaced hardcoded `angle_min_active_torque` with `ANGLE_MIN_TORQUE` from params for better configurability and consistency. This ensures the torque clamping logic aligns with defined parameters.
2025-06-07 19:43:01 +02:00
DevTekVE 3ec9d6c18a Merge branch 'master-new' into hkg-angle-steering-2025
# Conflicts:
#	common/params_keys.h
2025-06-07 15:04:16 +02:00
DevTekVE 86db8b95f0 Refactor torque calculation and deactivate live tuning.
Updated torque calculation logic with a new optional parameter for minimum active torque, streamlining control behavior. Deactivated and cleaned up references to HkgAngleLiveTuning, simplifying configuration and reducing runtime complexities. Updated relevant UI and parameter descriptions for clarity.
2025-06-07 11:55:57 +02:00
DevTekVE 4cfff8a35f Merge branch 'master-new' into hkg-angle-steering-2025 2025-06-06 23:08:37 +02:00
DevTekVE 962fedf48c Merge branch 'master-new' into hkg-angle-steering-2025
# Conflicts:
#	opendbc/car/tests/routes.py
2025-06-06 20:49:06 +02:00
DevTekVE 04494414d1 Merge branch 'master-new' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2025-06-05 09:18:53 +02:00
DevTekVE 0b83576e9b Adjust torque ramping logic and update steering thresholds
Increase the override window and refine torque ramp-up behavior to avoid conflicts during recent overrides. Updated steering driver allowance and threshold values for CANFD angle steering to improve compatibility and performance.
2025-06-02 09:49:31 +02:00
DevTekVE ce4ef0f817 Refine steering override logic in Hyundai car controller
Added logic to track recent steering overrides and adjust LKAS torque behavior accordingly. This ensures smoother transitions when the steering is overridden and reduces potential conflicts with driver input. Updated CANFD-specific steering thresholds for enhanced compatibility.
2025-06-02 09:12:03 +02:00
DevTekVE f0b15c1c56 Adding twil's torque calculation 2025-06-01 19:04:34 +02:00
DevTekVE f898e9fdfe Merge branch 'master-new' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2025-06-01 09:46:45 +02:00
DevTekVE 8ee7804b0e Bump opendbc (no tesla controls, no twil yet) 2025-05-29 16:31:42 +02:00
DevTekVE 923228194e bump opendbc to prior tesla changes until i can pass safety validations 2025-05-28 12:44:44 +02:00
DevTekVE 8837b2e3f6 Merge branch 'master-new' into hkg-angle-steering-2025
# Conflicts:
#	opendbc_repo
2025-05-28 12:36:02 +02:00
DevTekVE f48c9dc1c2 bump opendbc 2025-05-25 17:34:24 +02:00
DevTekVE 74aa07a8cd Ingore something i dont control thx 2025-05-25 17:31:37 +02:00
DevTekVE 5236e4860f Make lint happy, maybe 2025-05-25 17:31:37 +02:00
DevTekVE 3d174da1c3 adding some of my tests and validaitons 2025-05-25 17:31:25 +02:00
DevTekVE 8faa40f3a3 clean 2025-05-25 17:31:25 +02:00
DevTekVE 3e03275f28 Add PlotJuggler layout for analyzing torque and angle data
This new layout visualizes actuator data, CAN steering messages, and car state variables. It provides multiple time-series plots to aid in debugging and analysis. Plugin configurations are also included for extended functionality.
2025-05-25 17:31:22 +02:00
DevTekVE 7595cf8a25 Refine Hyundai angle and torque control logic.
Simplified control flag handling for angle steering, adjusted torque calculations for smoother ramp rates, and updated tuning parameters for the Hyundai Ioniq 5 PE. Minor adjustment to return value handling in lateral control functions.
2025-05-25 17:31:22 +02:00
DevTekVE 2675d43adb bump opendbc
Remove duplicate STEER_ANGLE_SATURATION_THRESHOLD import

Cleaned up an unnecessary duplicate import of STEER_ANGLE_SATURATION_THRESHOLD from latcontrol_angle_torque. This simplifies the module imports and prevents potential redundancy or confusion.

Refactor lateral control to combine torque and angle logic

Merged functionalities of LatControlTorque and LatControlAngle into a single LatControlAngleTorque class. Refactored code to utilize methods from both parent classes, reducing duplication and improving maintainability.

Add angle-torque hybrid lateral control for Hyundai CAN FD

Introduces `LatControlAngleTorque` to enable hybrid angle and torque-based steering for specific Hyundai models. Updates related logic in carcontroller, interface, and controlsd to accommodate this new lateral control method. Adjusts torque parameters for enhanced control in supported models.
2025-05-25 17:31:21 +02:00
DevTekVE d9f4ce82e6 clean 2025-05-25 17:31:21 +02:00
DevTekVE 648a1845d8 cleanup the mess 2025-05-25 17:31:21 +02:00
DevTekVE a87eff6d1c Add HKG Angle Live Tuning parameter and update related handling 2025-05-25 17:31:21 +02:00
DevTekVE dd6ad37e23 Absolutely zero clue on this, I did it with AI and it's for me to play. Don't take this notebook seriously please 2025-05-25 17:31:21 +02:00
DevTekVE c5e778b939 How annoying the linter on a comment lol 2025-05-25 17:31:21 +02:00
DevTekVE a9ab81a77a useless but should keep linter happy 2025-05-25 17:31:21 +02:00
DevTekVE 2d40e1d8e5 Refactor torque parameter handling in Hyundai carcontroller
Replaced direct access to `params` with instance variables for torque parameters to improve code clarity and maintainability. Updated smoothing factor description in angle tuning settings to include speed-related behavior. This enhances readability and prepares for further tuning adjustments.
2025-05-25 17:31:20 +02:00
DevTekVE 7c8f367a5d Fix data type for HkgTuningOverridingCycles value
Updated the value of HkgTuningOverridingCycles to a string for consistency with other parameters in the tuning configuration. This ensures proper handling and avoids potential issues with type mismatches.

Add overriding cycles parameter for torque adjustment

Introduced "HkgTuningOverridingCycles" for configurable user override torque ramp-down cycles. Updated relevant logic in torque control and UI settings to handle the new parameter. This improves flexibility in adjusting steering torque override behavior.
2025-05-25 17:31:20 +02:00
DevTekVE ff4cf558aa Add HKG angle tuning settings with min/max torque parameters
Introduce separate angle tuning controls for HKG vehicles, including smoothing factor, min torque, and max torque parameters. Refactor developer panel to integrate the new settings into a dedicated UI panel, enhancing modularity and customization capabilities.
2025-05-25 17:31:20 +02:00
DevTekVE 0e151e51bc Update HKG Angle Smoothing Factor description in Developer Panel
Enhanced the description to clarify its effect on steering behavior. Included details on how the smoothing factor impacts steering smoothness using EMA, aiding user understanding.
2025-05-25 17:31:20 +02:00
DevTekVE 60cc0031b0 Revert "Revert "Revert the EMA calculation on the curvature to test another approach""
This reverts commit 58fcda8c
2025-05-25 17:31:20 +02:00
DevTekVE d24cac0998 Refactor steering angle logic for smoother control adjustments
Refactored the calculation and application of the steering angle to improve code clarity and ensure smoother transitions. Removed unused parameter update logic in `latcontrol_angle.py` and enhanced handling of driver overrides in `carcontroller.py`.
2025-05-25 17:31:20 +02:00
DevTekVE 45d110830c Fix typo in parameter access method.
Replaced `self._params` with `self.params` to correctly access the parameter `HkgTuningAngleSmoothingFactor`. This ensures the smoothing factor is updated as intended during the control loop.
2025-05-25 17:31:20 +02:00
DevTekVE ea3a9ae911 Improve angle smoothing by integrating dynamic parameter tuning
Introduced a dynamic smoothing factor using the `HkgTuningAngleSmoothingFactor` parameter. This allows more granular control over curvature smoothing based on customizable user input, enhancing driving smoothness. Added necessary logic to process and apply this parameter efficiently.
2025-05-25 17:31:19 +02:00
DevTekVE 6bff8c0e7c Revert "Revert the EMA calculation on the curvature to test another approach"
This reverts commit bd471b3498.
2025-05-25 17:31:19 +02:00
DevTekVE bc6b8802b8 Add HKG angle smoothing factor for steering adjustments
Introduced a new parameter, `HkgTuningAngleSmoothingFactor`, to apply exponential moving average (EMA) smoothing to steering angle changes, reducing sudden adjustments. Added associated UI controls, parameter persistence, and integration into Hyundai carcontroller logic for improved steering stability.
2025-05-25 17:31:19 +02:00
DevTekVE 252ef572d3 Revert the EMA calculation on the curvature to test another approach 2025-05-25 17:31:19 +02:00
DevTekVE 414d397e3f Handle missing pygame import gracefully
Wrap the pygame import in a try-except block to catch ImportError. This prevents the script from crashing and provides a clear message prompting the user to install pygame if it's missing.

Remove "inputs" package and update "pygame" dependency

The "inputs" package has been removed from the lockfile and dependency list, while "pygame" is now included universally without the "dev" extra marker. This change simplifies dependencies and ensures consistency across environments.

Update dependencies: replace 'inputs' with 'pygame'

Replaced the 'inputs' library with 'pygame' for joystickd dependencies in `pyproject.toml`. Additionally, removed a redundant 'pygame' entry from the general dependencies.

Ugly, I know, but soundd is unhappy with joystick

Allowing lat with mads

Invert steering input for joystick control

The steering axis input is now multiplied by -1 to reverse its direction. This ensures correct handling of the left stick's horizontal input, aligning behavior with expected control dynamics.

Refactor joystick control to use pygame for broader support

Replaced the `inputs` library with `pygame` for joystick handling, providing improved compatibility with Xbox and PlayStation controllers. Added initialization, adaptive mappings, deadzone handling, and enhanced event processing for robust joystick operation. Updated README with dependencies and usage information for Xbox controllers.
2025-05-25 17:31:19 +02:00
DevTekVE 11b7b3789d Adjust speed thresholds in filter_speed_matrox.
Updated the `filter_speed_matrox` values to improve curvature filtering behavior at different speeds. This change ensures better handling and stability across a wider range of driving conditions.
2025-05-25 17:31:19 +02:00
DevTekVE 871ac53717 Optimize curvature filtering by adding speed-dependent logic.
Introduced speed-based dynamic alpha adjustment using interpolation for smoother curvature filtering. This improves steering angle calculations by adapting filter sensitivity to vehicle speed, enhancing control performance.
2025-05-25 17:31:19 +02:00
DevTekVE 64ea66b6e6 chsnge alpha to nicer value 2025-05-25 17:31:19 +02:00
DevTekVE 6d7c6759b3 Adjust curvature handling and filtering parameters
Updated curvature breakpoints and torque scaling for improved control in sharp turns. Increased filter alpha for faster curvature response while maintaining system stability.
2025-05-25 17:31:18 +02:00
DevTekVE 4cea013570 Adjust curvature handling and filtering parameters
Updated curvature breakpoints in Hyundai carcontroller to improve torque scaling for curved driving. Slightly refined the filter coefficient in lateral control for smoother curvature filtering and more accurate steering adjustments.
2025-05-25 17:31:18 +02:00
DevTekVE eb375c0587 Refactor curvature-based steering angle and torque logic.
Introduced dynamic torque scaling based on curvature for smoother and more adaptive steering control. Replaced raw curvature inputs with filtered curvature for enhanced stability and reduced noise in steering angle calculations. Removed unused speed scaling logic to simplify the lateral control flow.
2025-05-25 17:31:18 +02:00
DevTekVE 7fd8a5a4bd Reapply "Significant improvement on the jerkiness"
This reverts commit 85ce84e7b7.
2025-05-25 17:31:18 +02:00
DevTekVE b3c90216bb Revert "Significant improvement on the jerkiness"
This reverts commit ea1af879ba2905b076ccfe65993a9db701d689dd.

Revert "More improvement but still not quite"

This reverts commit ad95493c5c61b2ace7c459d2ebc151ddaa80040f.

Revert "Adjust low-speed scaling for lateral control angle"

This reverts commit 6f789ac1ebb66b0239b4028303573c2d7d386b39.

Revert "Refactor speed-based steering scaling logic."

This reverts commit 1d40735ab8db8d470ff3b287a6b42847beffff7d.
2025-05-25 17:31:18 +02:00
DevTekVE 10f345f956 Refactor speed-based steering scaling logic.
Updated the steering angle computation to use a clearer and more descriptive speed-scaling configuration. Replaced low-speed-specific logic with a generalized approach based on speed breakpoints and corresponding influence factors. This improves maintainability and ensures smoother steering adjustments at varying speeds.
2025-05-25 17:31:18 +02:00
DevTekVE 956d2c36d0 Adjust low-speed scaling for lateral control angle
Refined the low-speed scaling parameters by modifying speed breakpoints and factors. This improves handling at lower speeds for smoother and more predictable behavior.
2025-05-25 17:31:18 +02:00
DevTekVE 55e688b6f2 More improvement but still not quite 2025-05-25 17:31:17 +02:00
DevTekVE f017954027 Significant improvement on the jerkiness 2025-05-25 17:31:17 +02:00
DevTekVE 7e992d11b1 bump panda and opendbc 2025-05-25 17:31:15 +02:00
83 changed files with 5456 additions and 958 deletions
+1
View File
@@ -2,5 +2,6 @@ Wen
REGIST
PullRequest
cancelled
indeces
FOF
NoO
@@ -8,13 +8,13 @@ on:
required: true
type: string
target_hardware:
description: 'Hardware target to compile for (qcom or usbgpu)'
description: 'Hardware target to compile for (qcom or chestnut)'
required: true
type: choice
default: 'qcom'
options:
- qcom
- usbgpu
- chestnut
hf_repo:
description: 'Hugging Face dataset repository'
required: false
@@ -59,7 +59,7 @@ jobs:
id: get-json
run: |
cd docs/docs
PREFIX="driving_models_${{ inputs.target_hardware == 'usbgpu' && 'usbgpu_' || '' }}v"
PREFIX="driving_models_${{ inputs.target_hardware == 'chestnut' && 'chestnut_' || '' }}v"
latest=$(ls ${PREFIX}*.json | sed -E "s/${PREFIX}([0-9]+)\.json/\1/" | sort -n | tail -1)
next=$((latest+1))
json_file="${PREFIX}${next}.json"
+501
View File
@@ -0,0 +1,501 @@
name: Build default models
on:
workflow_dispatch:
inputs:
target:
description: 'Model target to build'
required: true
type: choice
options:
- small
- big
- dm
workflow_call:
inputs:
target:
description: 'Model target to build (small, big, or dm)'
required: true
type: string
concurrency:
group: build-default-models-${{ inputs.target }}
cancel-in-progress: false
env:
HF_REPO: sunnypilot/sunnypilot_models_v1
jobs:
resolve:
runs-on: ubuntu-24.04
outputs:
model_name: ${{ steps.resolve.outputs.model_name }}
onnx_ref: ${{ steps.resolve.outputs.onnx_ref }}
onnx_path: ${{ steps.resolve.outputs.onnx_path }}
hf_defaults_path: ${{ steps.resolve.outputs.hf_defaults_path }}
tinygrad_ref: ${{ steps.resolve.outputs.tinygrad_ref }}
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- id: resolve
run: |
export PYTHONPATH=${{ github.workspace }}
if [ "${{ inputs.target }}" = "big" ]; then
NAME=$(python3 -c "from openpilot.sunnypilot.models.model_name import DEFAULT_BIG_MODEL; print(DEFAULT_BIG_MODEL)")
ONNX_PATH="openpilot/selfdrive/modeld/models/big_driving_supercombo.onnx"
HF_DEFAULTS_PATH="models/defaults/big"
elif [ "${{ inputs.target }}" = "dm" ]; then
ONNX_PATH="openpilot/selfdrive/modeld/models/dmonitoring_model.onnx"
HF_DEFAULTS_PATH="models/defaults/dm"
NAME="dmonitoring_model ($(git log -1 --format=%cd --date=format:'%B %d, %Y' -- "$ONNX_PATH"))"
else
NAME=$(python3 -c "from openpilot.sunnypilot.models.model_name import DEFAULT_MODEL; print(DEFAULT_MODEL)")
ONNX_PATH="openpilot/selfdrive/modeld/models/driving_supercombo.onnx"
HF_DEFAULTS_PATH="models/defaults/small"
fi
ONNX_REF=$(git log -1 --format='%H' -- "$ONNX_PATH")
TINYGRAD_REF=$(python3 openpilot/sunnypilot/models/tinygrad_ref.py)
if [ -z "$TINYGRAD_REF" ]; then
echo "::error::Failed to resolve tinygrad ref"
exit 1
fi
echo "model_name=${NAME}" >> $GITHUB_OUTPUT
echo "onnx_ref=${ONNX_REF}" >> $GITHUB_OUTPUT
echo "onnx_path=${ONNX_PATH}" >> $GITHUB_OUTPUT
echo "hf_defaults_path=${HF_DEFAULTS_PATH}" >> $GITHUB_OUTPUT
echo "tinygrad_ref=${TINYGRAD_REF}" >> $GITHUB_OUTPUT
build_small_model:
needs: resolve
if: ${{ inputs.target == 'small' }}
runs-on: [self-hosted, tici]
env:
SMALL_ONNX: openpilot/selfdrive/modeld/models/driving_supercombo.onnx
SMALL_PKL: openpilot/selfdrive/modeld/models/driving_tinygrad.pkl
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- name: Pull ONNX via LFS
run: git lfs pull -I "${{ env.SMALL_ONNX }}"
- name: Set environment variables
run: |
source /etc/profile
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export UV_PYTHON_PREFERENCE=managed
export UV_PYTHON_INSTALL_DIR=${HOME}/uv/python
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
uv sync --frozen
printenv >> $GITHUB_ENV
- name: Disable powersave
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
- name: Compile small model with stock compiler
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH="${PYTHONPATH}:${{ github.workspace }}/tinygrad_repo:${{ github.workspace }}"
MODEL_SIZE=$(python3 -c "from openpilot.common.transformations.model import MEDMODEL_INPUT_SIZE as s; print(f'{s[0]}x{s[1]}')")
CAMERA_RES=$(python3 -c "from openpilot.common.transformations.camera import _ar_ox_fisheye as a, _os_fisheye as o; print(f'{a.width}x{a.height} {o.width}x{o.height}')")
FRAME_SKIP=$(python3 -c "from openpilot.selfdrive.modeld.constants import ModelConstants as MC; print(MC.MODEL_RUN_FREQ // MC.MODEL_CONTEXT_FREQ)")
TG_FLAGS="DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1"
env ${TG_FLAGS} python3 \
${{ github.workspace }}/openpilot/selfdrive/modeld/compile_modeld.py \
--onnx ${{ github.workspace }}/${{ env.SMALL_ONNX }} \
--model-size $MODEL_SIZE \
--camera-resolutions $CAMERA_RES \
--frame-skip $FRAME_SKIP \
--output ${{ github.workspace }}/${{ env.SMALL_PKL }}
- name: Chunk small pkl
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH=${{ github.workspace }}
python3 -c "
from openpilot.common.file_chunker import chunk_file, get_chunk_targets
import os
pkl = '${{ github.workspace }}/${{ env.SMALL_PKL }}'
size = os.path.getsize(pkl)
targets = get_chunk_targets(pkl, size)
chunk_file(pkl, targets)
print(f'Chunked into {len(targets)} files')
"
- name: Prepare output
env:
MODEL_NAME: ${{ needs.resolve.outputs.model_name }}
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH=${{ github.workspace }}
MODELS_DIR="${{ github.workspace }}/openpilot/selfdrive/modeld/models"
OUTPUT_DIR="${{ github.workspace }}/small_output"
PKL_BASE="driving_tinygrad.pkl"
mkdir -p "$OUTPUT_DIR"
cp "$MODELS_DIR/${PKL_BASE}".chunk* "$OUTPUT_DIR/"
cp "$MODELS_DIR/${PKL_BASE}.chunkmanifest" "$OUTPUT_DIR/"
python3 "${{ github.workspace }}/release/ci/model_generator.py" \
--model-dir "$MODELS_DIR" \
--output-dir "$OUTPUT_DIR" \
--custom-name "$MODEL_NAME" \
--upstream-branch "${{ needs.resolve.outputs.onnx_ref }}"
echo "model-${MODEL_NAME}-${{ github.run_number }}" > "$OUTPUT_DIR/artifact_name.txt"
- name: Upload small model artifact
uses: actions/upload-artifact@v4
with:
name: model-${{ needs.resolve.outputs.model_name }}-${{ github.run_number }}
path: ${{ github.workspace }}/small_output/
- name: Upload artifact name file
uses: actions/upload-artifact@v4
with:
name: artifact-name-${{ needs.resolve.outputs.model_name }}
path: ${{ github.workspace }}/small_output/artifact_name.txt
- name: Re-enable powersave
if: always()
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
build_big_model:
needs: resolve
if: ${{ inputs.target == 'big' }}
runs-on: [self-hosted, chestnut]
env:
BIG_ONNX: openpilot/selfdrive/modeld/models/big_driving_supercombo.onnx
BIG_PKL: openpilot/selfdrive/modeld/models/big_driving_tinygrad.pkl
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- name: Pull big ONNX via LFS
run: git lfs pull -I "${{ env.BIG_ONNX }}"
- name: Set environment variables
run: |
source /etc/profile
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export UV_PYTHON_PREFERENCE=managed
export UV_PYTHON_INSTALL_DIR=${HOME}/uv/python
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
uv sync --frozen
printenv >> $GITHUB_ENV
- name: Disable powersave
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
- name: Wait for chestnut PCIe link
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH="${PYTHONPATH}:${{ github.workspace }}/tinygrad_repo:${{ github.workspace }}"
python3 -c "
import time
from openpilot.system.hardware.chestnut.flash import link_up
for i in range(10):
if link_up():
print(f'PCIe link up after {i+1} attempt(s)')
break
time.sleep(1)
else:
raise RuntimeError('Chestnut PCIe link not ready after 10 attempts')
"
- name: Compile big model with stock compiler
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH="${PYTHONPATH}:${{ github.workspace }}/tinygrad_repo:${{ github.workspace }}"
MODEL_SIZE=$(python3 -c "from openpilot.common.transformations.model import MEDMODEL_INPUT_SIZE as s; print(f'{s[0]}x{s[1]}')")
CAMERA_RES=$(python3 -c "from openpilot.common.transformations.camera import _ar_ox_fisheye as a, _os_fisheye as o; print(f'{a.width}x{a.height} {o.width}x{o.height}')")
FRAME_SKIP=$(python3 -c "from openpilot.selfdrive.modeld.constants import ModelConstants as MC; print(MC.MODEL_RUN_FREQ // MC.MODEL_CONTEXT_FREQ)")
TG_FLAGS="DEBUG=2 DEV=USB+AMD:LLVM WARP_DEV=QCOM FLOAT16=1 JIT_BATCH_SIZE=0 GMMU=0 TC_OPT=2"
env ${TG_FLAGS} python3 \
${{ github.workspace }}/openpilot/selfdrive/modeld/compile_modeld.py \
--onnx ${{ github.workspace }}/${{ env.BIG_ONNX }} \
--model-size $MODEL_SIZE \
--camera-resolutions $CAMERA_RES \
--frame-skip $FRAME_SKIP \
--output ${{ github.workspace }}/${{ env.BIG_PKL }}
- name: Chunk big pkl
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH=${{ github.workspace }}
python3 -c "
from openpilot.common.file_chunker import chunk_file, get_chunk_targets
import os
pkl = '${{ github.workspace }}/${{ env.BIG_PKL }}'
size = os.path.getsize(pkl)
targets = get_chunk_targets(pkl, size)
chunk_file(pkl, targets)
print(f'Chunked into {len(targets)} files')
"
- name: Prepare output
env:
MODEL_NAME: ${{ needs.resolve.outputs.model_name }}
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH=${{ github.workspace }}
MODELS_DIR="${{ github.workspace }}/openpilot/selfdrive/modeld/models"
OUTPUT_DIR="${{ github.workspace }}/big_output"
PKL_BASE="big_driving_tinygrad.pkl"
mkdir -p "$OUTPUT_DIR"
cp "$MODELS_DIR/${PKL_BASE}".chunk* "$OUTPUT_DIR/"
cp "$MODELS_DIR/${PKL_BASE}.chunkmanifest" "$OUTPUT_DIR/"
python3 "${{ github.workspace }}/release/ci/model_generator.py" \
--model-dir "$MODELS_DIR" \
--output-dir "$OUTPUT_DIR" \
--custom-name "$MODEL_NAME" \
--upstream-branch "${{ needs.resolve.outputs.onnx_ref }}"
echo "model-${MODEL_NAME}-${{ github.run_number }}" > "$OUTPUT_DIR/artifact_name.txt"
- name: Upload big model artifact
uses: actions/upload-artifact@v4
with:
name: model-${{ needs.resolve.outputs.model_name }}-${{ github.run_number }}
path: ${{ github.workspace }}/big_output/
- name: Upload artifact name file
uses: actions/upload-artifact@v4
with:
name: artifact-name-${{ needs.resolve.outputs.model_name }}
path: ${{ github.workspace }}/big_output/artifact_name.txt
- name: Re-enable powersave
if: always()
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
upload_defaults:
needs: [ resolve, build_small_model, build_big_model, build_dm_model ]
if: |
${{
!cancelled() &&
(inputs.target == 'big' && needs.build_big_model.result == 'success' ||
inputs.target == 'small' && needs.build_small_model.result == 'success' ||
inputs.target == 'dm' && needs.build_dm_model.result == 'success')
}}
runs-on: ubuntu-24.04
permissions:
id-token: write
contents: write
steps:
- uses: actions/checkout@v4
- name: Pull ONNX via LFS
run: git lfs pull -I "${{ needs.resolve.outputs.onnx_path }}"
- name: Install huggingface_hub
run: pip install --upgrade "huggingface_hub>=0.22.0"
- name: Download artifact name
if: ${{ inputs.target == 'small' || inputs.target == 'big' }}
uses: actions/download-artifact@v4
with:
name: artifact-name-${{ needs.resolve.outputs.model_name }}
path: artifact_name
- name: Read artifact name
if: ${{ inputs.target == 'small' || inputs.target == 'big' }}
id: artifact
run: |
ARTIFACT_NAME=$(cat artifact_name/artifact_name.txt)
echo "artifact_name=$ARTIFACT_NAME" >> $GITHUB_OUTPUT
- name: Download model artifact
if: ${{ inputs.target == 'small' || inputs.target == 'big' }}
uses: actions/download-artifact@v4
with:
name: ${{ steps.artifact.outputs.artifact_name }}
path: output
- name: Upload model to HF
if: ${{ inputs.target == 'small' || inputs.target == 'big' }}
env:
HF_OIDC_RESOURCE: datasets/${{ env.HF_REPO }}
ARTIFACT_NAME: ${{ steps.artifact.outputs.artifact_name }}
run: |
rm -f output/artifact_name.txt
export PYTHONPATH=$(pwd)
python3 release/ci/upload_default_model.py \
--hf-repo "${{ env.HF_REPO }}" \
--hf-defaults-path "${{ needs.resolve.outputs.hf_defaults_path }}" \
--artifact-name "$ARTIFACT_NAME" \
--model-dir output \
--onnx-path "${{ needs.resolve.outputs.onnx_path }}" \
--onnx-ref "${{ needs.resolve.outputs.onnx_ref }}" \
--model-name "${{ needs.resolve.outputs.model_name }}" \
--tinygrad-ref "${{ needs.resolve.outputs.tinygrad_ref }}" \
--run-number "${{ github.run_number }}"
- name: Download DM artifact
if: ${{ inputs.target == 'dm' }}
uses: actions/download-artifact@v4
with:
name: dm-model-${{ github.run_number }}
path: dm_output
- name: Generate DM metadata and upload to HF
if: ${{ inputs.target == 'dm' }}
env:
HF_OIDC_RESOURCE: datasets/${{ env.HF_REPO }}
run: |
export PYTHONPATH=$(pwd)
python3 -c "
import json, hashlib
from pathlib import Path
from datetime import datetime, UTC
dm_dir = Path('dm_output')
manifest = list(dm_dir.glob('*.chunkmanifest'))
assert manifest, 'No chunkmanifest found'
pkl_name = manifest[0].name.removesuffix('.chunkmanifest')
num_chunks = int(manifest[0].read_text().strip())
chunks = []
for i in range(num_chunks):
chunk = dm_dir / f'{pkl_name}.chunk{i+1:02d}of{num_chunks:02d}'
chunks.append({
'file_name': chunk.name,
'sha256': hashlib.sha256(chunk.read_bytes()).hexdigest()
})
digest = hashlib.sha256()
for c in chunks:
with open(dm_dir / c['file_name'], 'rb') as f:
while block := f.read(1024*1024):
digest.update(block)
metadata = {
'bundles': [{
'short_name': 'DMMODEL',
'display_name': '${{ needs.resolve.outputs.model_name }}',
'ref': '${{ needs.resolve.outputs.onnx_ref }}',
'runner': 'tinygrad',
'build_time': datetime.now(UTC).strftime('%Y-%m-%dT%H:%M:%SZ'),
'models': [{
'type': 'chunked',
'artifact': {
'file_name': pkl_name,
'download_uri': {'url': '', 'sha256': digest.hexdigest()},
'chunks': chunks
}
}]
}]
}
with open(dm_dir / 'metadata.json', 'w') as f:
json.dump(metadata, f, indent=2)
print('Generated DM metadata.json')
"
python3 release/ci/upload_default_model.py \
--hf-repo "${{ env.HF_REPO }}" \
--hf-defaults-path "${{ needs.resolve.outputs.hf_defaults_path }}" \
--artifact-name "dm-model-${{ github.run_number }}" \
--model-dir dm_output \
--onnx-path "${{ needs.resolve.outputs.onnx_path }}" \
--onnx-ref "${{ needs.resolve.outputs.onnx_ref }}" \
--model-name "${{ needs.resolve.outputs.model_name }}" \
--tinygrad-ref "${{ needs.resolve.outputs.tinygrad_ref }}" \
--run-number "${{ github.run_number }}"
build_dm_model:
needs: resolve
if: ${{ inputs.target == 'dm' }}
runs-on: [self-hosted, tici]
env:
DM_ONNX: openpilot/selfdrive/modeld/models/dmonitoring_model.onnx
DM_PKL: openpilot/selfdrive/modeld/models/dmonitoring_model_tinygrad.pkl
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- name: Pull DM ONNX via LFS
run: git lfs pull -I "${{ env.DM_ONNX }}"
- name: Set environment variables
run: |
source /etc/profile
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export UV_PYTHON_PREFERENCE=managed
export UV_PYTHON_INSTALL_DIR=${HOME}/uv/python
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
uv sync --frozen
printenv >> $GITHUB_ENV
- name: Disable powersave
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
- name: Compile DM model
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH="${PYTHONPATH}:${{ github.workspace }}/tinygrad_repo:${{ github.workspace }}"
TG_FLAGS="DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1"
taskset -c 7 env ${TG_FLAGS} python3 \
${{ github.workspace }}/tinygrad_repo/examples/openpilot/compile3.py \
${{ github.workspace }}/${{ env.DM_ONNX }} \
${{ github.workspace }}/${{ env.DM_PKL }}
- name: Chunk DM pkl
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH=${{ github.workspace }}
python3 -c "
from openpilot.common.file_chunker import chunk_file, get_chunk_targets
import os
pkl = '${{ github.workspace }}/${{ env.DM_PKL }}'
size = os.path.getsize(pkl)
targets = get_chunk_targets(pkl, size)
chunk_file(pkl, targets)
print(f'Chunked {pkl} into {len(targets)} chunks')
"
- name: Prepare DM output
run: |
mkdir -p dm_output
cp ${{ github.workspace }}/${{ env.DM_PKL }}.chunk* dm_output/
cp ${{ github.workspace }}/${{ env.DM_PKL }}.chunkmanifest dm_output/
- name: Upload DM artifact
uses: actions/upload-artifact@v4
with:
name: dm-model-${{ github.run_number }}
path: dm_output/
- name: Re-enable powersave
if: always()
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
@@ -30,7 +30,7 @@ on:
type: boolean
default: true
target_hardware:
description: 'Hardware target to compile for (qcom or usbgpu)'
description: 'Hardware target to compile for (qcom or chestnut)'
required: false
type: string
default: 'qcom'
@@ -101,7 +101,7 @@ on:
default: 'qcom'
options:
- qcom
- usbgpu
- chestnut
hf_repo:
description: 'Hugging Face dataset repository'
required: false
@@ -109,7 +109,7 @@ on:
default: 'sunnypilot/sunnypilot_models_v1'
env:
RECOMPILED_DIR: recompiled${{ inputs.recompiled_dir }}
JSON_FILE: docs/docs/driving_models_${{ inputs.target_hardware == 'usbgpu' && 'usbgpu_v' || 'v' }}${{ inputs.json_version }}.json
JSON_FILE: docs/docs/driving_models_${{ inputs.target_hardware == 'chestnut' && 'chestnut_v' || 'v' }}${{ inputs.json_version }}.json
jobs:
build_model:
@@ -0,0 +1,66 @@
name: Download HF model chunks
description: Resolve and download model chunks from HuggingFace in parallel
inputs:
hf_repo:
description: HuggingFace dataset repo
required: true
models:
description: 'JSON array of {hf_path, onnx_hash, canonical} objects'
required: true
dest_dir:
description: Destination directory for downloaded chunks
required: true
runs:
using: composite
steps:
- name: Download model chunks
shell: bash
env:
HF_REPO: ${{ inputs.hf_repo }}
MODELS_JSON: ${{ inputs.models }}
DEST_DIR: ${{ inputs.dest_dir }}
run: |
set -eo pipefail
DOWNLOAD_LIST=$(mktemp)
resolve_chunks() {
local HF_PATH="$1" ONNX_HASH="$2" CANONICAL="$3" DEST_DIR="$4"
local JSON_URL="https://huggingface.co/datasets/${HF_REPO}/resolve/main/${HF_PATH}/default_models.json"
local DEFAULTS BUNDLE ARTIFACT BASE_URL NUM_CHUNKS
DEFAULTS=$(curl -fsSL "$JSON_URL")
BUNDLE=$(echo "$DEFAULTS" | jq --arg hash "$ONNX_HASH" '.bundles[] | select(.onnx_sha256 == $hash)')
ARTIFACT=$(echo "$BUNDLE" | jq -r '.models[0].artifact')
BASE_URL=$(echo "$ARTIFACT" | jq -r '.download_uri.url' | sed 's|/[^/]*$||')
NUM_CHUNKS=$(echo "$ARTIFACT" | jq -r '.chunks | length')
mkdir -p "$DEST_DIR"
while IFS= read -r CHUNK_NAME; do
CHUNK_IDX=$(echo "$CHUNK_NAME" | grep -oP 'chunk\K[0-9]+of[0-9]+' || true)
if [ -z "$CHUNK_IDX" ]; then
echo "::error::Failed to parse chunk index from: $CHUNK_NAME"
return 1
fi
ENCODED_URL=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${BASE_URL}/${CHUNK_NAME}', safe=':/'))")
printf '%s\t%s\n' "$ENCODED_URL" "${DEST_DIR}/${CANONICAL}.chunk${CHUNK_IDX}" >> "$DOWNLOAD_LIST"
done < <(echo "$ARTIFACT" | jq -r '.chunks[].file_name')
echo "$NUM_CHUNKS" > "${DEST_DIR}/${CANONICAL}.chunkmanifest"
}
echo "$MODELS_JSON" | jq -c '.[]' | while IFS= read -r model; do
HF_PATH=$(echo "$model" | jq -r '.hf_path')
ONNX_HASH=$(echo "$model" | jq -r '.onnx_hash')
CANONICAL=$(echo "$model" | jq -r '.canonical')
resolve_chunks "$HF_PATH" "$ONNX_HASH" "$CANONICAL" "$DEST_DIR"
done
TOTAL=$(wc -l < "$DOWNLOAD_LIST")
echo "Downloading $TOTAL chunks with 8 parallel connections..."
xargs -P8 -d'\n' -I{} bash -c '
URL="${1%% *}"
DEST="${1#* }"
echo "Downloading $(basename "$DEST")"
curl -fsSL --retry 3 --retry-delay 5 -o "$DEST" "$URL"
' _ {} < "$DOWNLOAD_LIST"
rm -f "$DOWNLOAD_LIST"
+41 -23
View File
@@ -31,7 +31,7 @@ on:
type: string
default: ''
target_hardware:
description: 'Hardware target to compile for (qcom or usbgpu)'
description: 'Hardware target to compile for (qcom or chestnut)'
required: false
type: string
default: 'qcom'
@@ -57,7 +57,7 @@ on:
type: choice
options:
- qcom
- usbgpu
- chestnut
default: 'qcom'
@@ -102,21 +102,26 @@ jobs:
cat $GITHUB_OUTPUT
- run: |
cd ${{ github.workspace }}/openpilot/openpilot
if [ "${{ inputs.target_hardware }}" != "usbgpu" ]; then
git lfs pull -X "selfdrive/modeld/models/big_*.onnx" -X "selfdrive/modeld/models/dmonitoring_*.onnx"
if [ "${{ inputs.target_hardware }}" != "chestnut" ]; then
git lfs pull -X "**/selfdrive/modeld/models/big_*.onnx,**/selfdrive/modeld/models/dmonitoring_*.onnx"
rm -f selfdrive/modeld/models/big_*.onnx selfdrive/modeld/models/dmonitoring_*.onnx
else
git lfs pull -I "selfdrive/modeld/models/big_*.onnx"
git lfs pull -I "**/selfdrive/modeld/models/big_*.onnx" -X ""
find selfdrive/modeld/models -name "*.onnx" ! -name "big_*.onnx" -delete
fi
if grep -lIF "version https://git-lfs.github.com/spec/v1" selfdrive/modeld/models/*.onnx; then
echo "::error::the ONNX files above are still LFS pointers, not real models"
exit 1
fi
- name: 'Upload Artifact'
uses: actions/upload-artifact@v4
with:
name: models-${{ env.REF }}${{ inputs.artifact_suffix }}
path: ${{ github.workspace }}/openpilot/openpilot/selfdrive/modeld/models/*.onnx
if-no-files-found: error
build_model:
runs-on: [self-hosted, tici]
runs-on: [self-hosted, chestnut]
needs: get_model
env:
MODEL_NAME: ${{ inputs.custom_name || inputs.upstream_branch }} (${{ needs.get_model.outputs.model_date }})
@@ -127,7 +132,6 @@ jobs:
fetch-depth: 1
submodules: recursive
- run: git lfs pull
- name: Set environment variables
id: set-env
@@ -160,7 +164,7 @@ jobs:
fi
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
rm -rf ${{ env.MODELS_DIR }}/*.onnx
rm -rf ${{ env.MODELS_DIR }}/*.onnx*
- name: Download model artifacts
uses: actions/download-artifact@v4
@@ -180,34 +184,48 @@ jobs:
MODEL_SIZE=$(python3 -c "from openpilot.common.transformations.model import MEDMODEL_INPUT_SIZE as s; print(f'{s[0]}x{s[1]}')")
CAMERA_RES=$(python3 -c "from openpilot.common.transformations.camera import _ar_ox_fisheye as a, _os_fisheye as o; print(f'{a.width}x{a.height} {o.width}x{o.height}')")
if [ "${{ inputs.target_hardware }}" == "usbgpu" ]; then
echo "USBGPU build"
export USBGPU=1
TG_FLAGS="DEBUG=2 DEV=USB+AMD:LLVM WARP_DEV=QCOM FLOAT16=1 JIT_BATCH_SIZE=0 GMMU=0 TC_OPT=2"
TG_FLAGS_QCOM="DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1"
if [ "${{ inputs.target_hardware }}" == "chestnut" ]; then
echo "CHESTNUT build"
export CHESTNUT=1
TG_FLAGS="DEBUG=1 DEV=USB+AMD:LLVM WARP_DEV=QCOM FLOAT16=1 JIT_BATCH_SIZE=0 GMMU=0 TC_OPT=2"
OUTPUT_PKL="${{ env.MODELS_DIR }}/big_driving_tinygrad.pkl"
else
echo "QCOM build"
TG_FLAGS="DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1"
TG_FLAGS="$TG_FLAGS_QCOM"
OUTPUT_PKL="${{ env.MODELS_DIR }}/driving_tinygrad.pkl"
fi
# Generate metadata for all ONNX files
find "${{ env.MODELS_DIR }}" -maxdepth 1 -name '*.onnx' | while IFS= read -r onnx_file; do
echo "Generating metadata: $onnx_file"
env ${TG_FLAGS} python3 "${{ env.MODELS_DIR }}/../get_model_metadata.py" "$onnx_file" || true
env ${TG_FLAGS_QCOM} python3 "${{ env.MODELS_DIR }}/../get_model_metadata.py" "$onnx_file" || true
done
# Detect model type and build compile args
VISION_ONNX="${{ env.MODELS_DIR }}/driving_vision.onnx"
POLICY_ONNX="${{ env.MODELS_DIR }}/driving_policy.onnx"
OFF_POLICY_ONNX="${{ env.MODELS_DIR }}/driving_off_policy.onnx"
ON_POLICY_ONNX="${{ env.MODELS_DIR }}/driving_on_policy.onnx"
VISION_ONNX=""
for f in "${{ env.MODELS_DIR }}/driving_vision.onnx" "${{ env.MODELS_DIR }}/big_driving_vision.onnx"; do
[ -f "$f" ] && VISION_ONNX="$f" && break
done
POLICY_ONNX=""
for f in "${{ env.MODELS_DIR }}/driving_policy.onnx" "${{ env.MODELS_DIR }}/big_driving_policy.onnx"; do
[ -f "$f" ] && POLICY_ONNX="$f" && break
done
OFF_POLICY_ONNX=""
for f in "${{ env.MODELS_DIR }}/driving_off_policy.onnx" "${{ env.MODELS_DIR }}/big_driving_off_policy.onnx"; do
[ -f "$f" ] && OFF_POLICY_ONNX="$f" && break
done
ON_POLICY_ONNX=""
for f in "${{ env.MODELS_DIR }}/driving_on_policy.onnx" "${{ env.MODELS_DIR }}/big_driving_on_policy.onnx"; do
[ -f "$f" ] && ON_POLICY_ONNX="$f" && break
done
SUPERCOMBO_ONNX=""
for f in "${{ env.MODELS_DIR }}/supercombo.onnx" "${{ env.MODELS_DIR }}/driving_supercombo.onnx"; do
if [ -f "$f" ]; then
SUPERCOMBO_ONNX="$f"
break
fi
for f in "${{ env.MODELS_DIR }}/supercombo.onnx" "${{ env.MODELS_DIR }}/driving_supercombo.onnx" "${{ env.MODELS_DIR }}/big_supercombo.onnx" "${{ env.MODELS_DIR }}/big_driving_supercombo.onnx"; do
[ -f "$f" ] && SUPERCOMBO_ONNX="$f" && break
done
MODEL_TYPE="" ONNX_ARGS="" OUTPUT_NAME=""
@@ -36,8 +36,11 @@ jobs:
publish_concurrency_group: ${{ steps.strategy.outputs.publish_concurrency_group }}
is_stable_branch: ${{ steps.strategy.outputs.is_stable_branch }}
build: ${{ steps.strategy.outputs.build }}
include_big_model: ${{ steps.strategy.outputs.include_big_model }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Extract deploy strategy
id: strategy
run: |
@@ -78,6 +81,9 @@ jobs:
stable_version=$(cat openpilot/sunnypilot/common/version.h | grep SUNNYPILOT_VERSION | sed -e 's/[^0-9|.]//g');
echo "version=$([ "$is_stable_branch" = "true" ] && echo "$stable_version" || echo "$BUILD")" >> $GITHUB_OUTPUT
echo "extra_version_identifier=${environment}" >> $GITHUB_OUTPUT
include_big_model="$(echo "$CONFIG" | jq -r '.include_big_model // false')";
echo "include_big_model=$include_big_model" >> $GITHUB_OUTPUT
fi
echo "build=$BUILD" >> $GITHUB_OUTPUT
cat $GITHUB_OUTPUT
@@ -92,6 +98,8 @@ jobs:
}}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Wait for Tests
uses: ./.github/workflows/wait-for-action # Path to where you place the action
with:
@@ -115,6 +123,7 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
submodules: recursive
ref: ${{ env.SOURCE_BRANCH }}
repository: ${{ github.event.pull_request.head.repo.fork && github.event.pull_request.head.repo.full_name || github.repository }}
@@ -161,7 +170,7 @@ jobs:
scons -j1 cache_dir="$SCONS_CACHE" --minimal \
openpilot/selfdrive/locationd openpilot/sunnypilot/selfdrive/locationd
echo "Building rest of sunnypilot"
/usr/bin/time -v scons -j$(nproc) cache_dir="$SCONS_CACHE" --minimal
SKIP_TINYGRAD_COMPILE=1 /usr/bin/time -v scons -j$(nproc) cache_dir="$SCONS_CACHE" --minimal
touch ${BUILD_DIR}/prebuilt
if [[ "${{ runner.debug }}" == "1" ]]; then
ls -la ${BUILD_DIR}
@@ -203,22 +212,212 @@ jobs:
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
prepare_chestnut:
needs: [ prepare_strategy ]
runs-on: ubuntu-24.04
if: ${{ needs.prepare_strategy.outputs.include_big_model == 'true' }}
outputs:
onnx_sha256: ${{ steps.resolve.outputs.onnx_sha256 }}
env:
GH_REPO: ${{ github.repository }}
HF_REPO: sunnypilot/sunnypilot_models_v1
HF_DEFAULTS_PATH: models/defaults/big
steps:
- name: Resolve ONNX hash and tinygrad ref via API
id: resolve
run: |
REF="${{ github.head_ref || github.ref_name }}"
ONNX_HASH=$(gh api "repos/${GH_REPO}/contents/openpilot/selfdrive/modeld/models/big_driving_supercombo.onnx?ref=${REF}" --jq '.content' | base64 -d | grep '^oid sha256:' | cut -d: -f2)
echo "ONNX hash: $ONNX_HASH"
echo "onnx_sha256=$ONNX_HASH" >> $GITHUB_OUTPUT
TINYGRAD_REF=$(gh api "repos/${GH_REPO}/contents/tinygrad_repo?ref=${REF}" --jq '.sha')
echo "tinygrad ref: $TINYGRAD_REF"
JSON_URL="https://huggingface.co/datasets/${HF_REPO}/resolve/main/${HF_DEFAULTS_PATH}/default_models.json"
check_defaults() {
DEFAULTS=$(curl -fsSL "$JSON_URL" 2>/dev/null) || return 1
TINYGRAD_MATCH=$(echo "$DEFAULTS" | jq -r --arg ref "$TINYGRAD_REF" '.tinygrad_ref == $ref' 2>/dev/null)
[ "$TINYGRAD_MATCH" = "true" ] || return 1
BUNDLE=$(echo "$DEFAULTS" | jq --arg hash "$ONNX_HASH" '.bundles[] | select(.onnx_sha256 == $hash)' 2>/dev/null)
[ -n "$BUNDLE" ] && [ "$BUNDLE" != "null" ]
}
if check_defaults; then
echo "HF defaults match repo ONNX hash and tinygrad ref"
exit 0
fi
echo "No matching model on HF — dispatching build"
gh workflow run build-default-models.yaml --ref "$REF" -f target=big
echo "Polling HF for big model availability..."
for i in $(seq 1 90); do
sleep 30
if check_defaults; then
echo "Big model available on HF after $((i * 30))s"
exit 0
fi
echo "Poll $i/90: not yet available"
done
echo "::error::Big model not available on HF after 45 minutes"
exit 1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Cancel run on failure
if: failure()
run: gh run cancel ${{ github.run_id }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
prepare_small_model:
needs: [ prepare_strategy ]
runs-on: ubuntu-24.04
outputs:
driving_onnx_sha256: ${{ steps.resolve.outputs.driving_onnx_sha256 }}
env:
GH_REPO: ${{ github.repository }}
HF_REPO: sunnypilot/sunnypilot_models_v1
HF_DEFAULTS_PATH: models/defaults/small
steps:
- name: Resolve ONNX hash and tinygrad ref via API
id: resolve
run: |
REF="${{ github.head_ref || github.ref_name }}"
DRIVING_HASH=$(gh api "repos/${GH_REPO}/contents/openpilot/selfdrive/modeld/models/driving_supercombo.onnx?ref=${REF}" --jq '.content' | base64 -d | grep '^oid sha256:' | cut -d: -f2)
echo "Driving ONNX hash: $DRIVING_HASH"
echo "driving_onnx_sha256=$DRIVING_HASH" >> $GITHUB_OUTPUT
TINYGRAD_REF=$(gh api "repos/${GH_REPO}/contents/tinygrad_repo?ref=${REF}" --jq '.sha')
echo "tinygrad ref: $TINYGRAD_REF"
JSON_URL="https://huggingface.co/datasets/${HF_REPO}/resolve/main/${HF_DEFAULTS_PATH}/default_models.json"
check_defaults() {
DEFAULTS=$(curl -fsSL "$JSON_URL" 2>/dev/null) || return 1
TINYGRAD_MATCH=$(echo "$DEFAULTS" | jq -r --arg ref "$TINYGRAD_REF" '.tinygrad_ref == $ref' 2>/dev/null)
[ "$TINYGRAD_MATCH" = "true" ] || return 1
DRIVING=$(echo "$DEFAULTS" | jq --arg hash "$DRIVING_HASH" '.bundles[] | select(.onnx_sha256 == $hash)' 2>/dev/null)
[ -n "$DRIVING" ] && [ "$DRIVING" != "null" ] || return 1
}
if check_defaults; then
echo "HF defaults match repo ONNX hash and tinygrad ref"
exit 0
fi
echo "No matching model on HF — dispatching build"
gh workflow run build-default-models.yaml --ref "$REF" -f target=small
echo "Polling HF for model availability..."
for i in $(seq 1 60); do
sleep 30
if check_defaults; then
echo "Model available on HF after $((i * 30))s"
exit 0
fi
echo "Poll $i/60: not yet available"
done
echo "::error::Small driving model not available on HF after 30 minutes"
exit 1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Cancel run on failure
if: failure()
run: gh run cancel ${{ github.run_id }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
prepare_dm_model:
needs: [ prepare_strategy ]
runs-on: ubuntu-24.04
outputs:
dm_onnx_sha256: ${{ steps.resolve.outputs.dm_onnx_sha256 }}
env:
GH_REPO: ${{ github.repository }}
HF_REPO: sunnypilot/sunnypilot_models_v1
HF_DEFAULTS_PATH: models/defaults/dm
steps:
- name: Resolve ONNX hash and tinygrad ref via API
id: resolve
run: |
REF="${{ github.head_ref || github.ref_name }}"
DM_HASH=$(gh api "repos/${GH_REPO}/contents/openpilot/selfdrive/modeld/models/dmonitoring_model.onnx?ref=${REF}" --jq '.content' | base64 -d | grep '^oid sha256:' | cut -d: -f2)
echo "DM ONNX hash: $DM_HASH"
echo "dm_onnx_sha256=$DM_HASH" >> $GITHUB_OUTPUT
TINYGRAD_REF=$(gh api "repos/${GH_REPO}/contents/tinygrad_repo?ref=${REF}" --jq '.sha')
echo "tinygrad ref: $TINYGRAD_REF"
JSON_URL="https://huggingface.co/datasets/${HF_REPO}/resolve/main/${HF_DEFAULTS_PATH}/default_models.json"
check_defaults() {
DEFAULTS=$(curl -fsSL "$JSON_URL" 2>/dev/null) || return 1
TINYGRAD_MATCH=$(echo "$DEFAULTS" | jq -r --arg ref "$TINYGRAD_REF" '.tinygrad_ref == $ref' 2>/dev/null)
[ "$TINYGRAD_MATCH" = "true" ] || return 1
DM=$(echo "$DEFAULTS" | jq --arg hash "$DM_HASH" '.bundles[] | select(.onnx_sha256 == $hash)' 2>/dev/null)
[ -n "$DM" ] && [ "$DM" != "null" ] || return 1
}
if check_defaults; then
echo "HF defaults match DM ONNX hash and tinygrad ref"
exit 0
fi
echo "No matching DM model on HF — dispatching build"
gh workflow run build-default-models.yaml --ref "$REF" -f target=dm
echo "Polling HF for DM model availability..."
for i in $(seq 1 60); do
sleep 30
if check_defaults; then
echo "DM model available on HF after $((i * 30))s"
exit 0
fi
echo "Poll $i/60: not yet available"
done
echo "::error::DM model not available on HF after 30 minutes"
exit 1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Cancel run on failure
if: failure()
run: gh run cancel ${{ github.run_id }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
publish:
concurrency:
# We do a bit of a hack here to avoid canceling the publishing job if a new commit comes in while we're publishing by adding the sha to the group name.
# This means that if multiple commits come in while we're publishing, they will be queued up and publish one after the other.
# Otherwise, if a job is waiting to be published due to environment wait time, it would be canceled by a new commit and restart the wait time.
group: ${{ needs.prepare_strategy.outputs.publish_concurrency_group }}
cancel-in-progress: ${{ needs.prepare_strategy.outputs.cancel_publish_in_progress == 'true' }}
if: ${{ (always() && !cancelled() && !failure()) && needs.build.result == 'success' && needs.prepare_strategy.result == 'success' && (!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt')) }}
needs: [ build, prepare_strategy ]
if: ${{
always() && !cancelled() &&
needs.build.result == 'success' &&
needs.prepare_strategy.result == 'success' &&
needs.prepare_small_model.result == 'success' &&
needs.prepare_dm_model.result == 'success' &&
(!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt')) &&
(needs.prepare_strategy.outputs.include_big_model != 'true' || needs.prepare_chestnut.result == 'success')
}}
needs: [ build, prepare_strategy, prepare_chestnut, prepare_small_model, prepare_dm_model ]
runs-on: ubuntu-24.04
environment: ${{ needs.prepare_strategy.outputs.environment }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Download build artifacts
- name: Download prebuilt artifact
uses: actions/download-artifact@v4
with:
name: prebuilt
@@ -228,6 +427,17 @@ jobs:
mkdir -p ${{ env.OUTPUT_DIR }}
tar xzf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }}
- name: Download model chunks from HF
uses: ./.github/workflows/download-hf-model-chunks
with:
hf_repo: sunnypilot/sunnypilot_models_v1
dest_dir: ${{ env.OUTPUT_DIR }}/openpilot/selfdrive/modeld/models
models: |
[
{"hf_path": "models/defaults/small", "onnx_hash": "${{ needs.prepare_small_model.outputs.driving_onnx_sha256 }}", "canonical": "driving_tinygrad.pkl"},
{"hf_path": "models/defaults/dm", "onnx_hash": "${{ needs.prepare_dm_model.outputs.dm_onnx_sha256 }}", "canonical": "dmonitoring_model_tinygrad.pkl"}
]
- name: Configure Git
run: |
git config --global user.email "github-actions[bot]@users.noreply.github.com"
@@ -255,11 +465,77 @@ jobs:
git tag -f -a ${TAG} -m "${{ needs.prepare_strategy.outputs.environment }} @ ${{ needs.prepare_strategy.outputs.version }} of build ${{ needs.prepare_strategy.outputs.build }}."
git push -f origin ${TAG}
publish_chestnut:
concurrency:
group: ${{ needs.prepare_strategy.outputs.publish_concurrency_group }}-chestnut
cancel-in-progress: ${{ needs.prepare_strategy.outputs.cancel_publish_in_progress == 'true' }}
if: ${{
always() && !cancelled() &&
needs.build.result == 'success' &&
needs.prepare_strategy.result == 'success' &&
needs.prepare_small_model.result == 'success' &&
needs.prepare_dm_model.result == 'success' &&
needs.prepare_chestnut.result == 'success' &&
(!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
}}
needs: [ build, prepare_strategy, prepare_chestnut, prepare_small_model, prepare_dm_model ]
runs-on: ubuntu-24.04
environment: ${{ needs.prepare_strategy.outputs.environment }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Download prebuilt artifact
uses: actions/download-artifact@v4
with:
name: prebuilt
- name: Untar prebuilt
run: |
mkdir -p ${{ env.OUTPUT_DIR }}
tar xzf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }}
- name: Download model chunks from HF
uses: ./.github/workflows/download-hf-model-chunks
with:
hf_repo: sunnypilot/sunnypilot_models_v1
dest_dir: ${{ env.OUTPUT_DIR }}/openpilot/selfdrive/modeld/models
models: |
[
{"hf_path": "models/defaults/small", "onnx_hash": "${{ needs.prepare_small_model.outputs.driving_onnx_sha256 }}", "canonical": "driving_tinygrad.pkl"},
{"hf_path": "models/defaults/dm", "onnx_hash": "${{ needs.prepare_dm_model.outputs.dm_onnx_sha256 }}", "canonical": "dmonitoring_model_tinygrad.pkl"},
{"hf_path": "models/defaults/big", "onnx_hash": "${{ needs.prepare_chestnut.outputs.onnx_sha256 }}", "canonical": "big_driving_tinygrad.pkl"}
]
- name: Configure Git
run: |
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"
- name: Publish chestnut branch
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
CHESTNUT_BRANCH="${{ needs.prepare_strategy.outputs.new_branch }}-chestnut"
${{ env.CI_DIR }}/publish.sh \
"${{ github.workspace }}" \
"${{ env.OUTPUT_DIR }}" \
"$CHESTNUT_BRANCH" \
"${{ needs.prepare_strategy.outputs.version }}" \
"https://x-access-token:${{github.token}}@github.com/sunnypilot/sunnypilot.git" \
"${{ needs.prepare_strategy.outputs.extra_version_identifier }}"
notify:
needs:
- prepare_strategy
- build
- publish
- publish_chestnut
- prepare_chestnut
- prepare_small_model
- prepare_dm_model
runs-on: ubuntu-24.04
if: ${{ (always() && !cancelled() && !failure())
&& needs.publish.result == 'success'
@@ -267,6 +543,8 @@ jobs:
&& (fromJSON(vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES_V2)[github.head_ref || github.ref_name] != null) }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Prepare notification message
id: message
@@ -279,6 +557,7 @@ jobs:
export commit_short_sha="${commit_short_sha:0:7}"
export extra_version_identifier="${{ needs.prepare_strategy.outputs.extra_version_identifier || github.run_number }}"
export PUBLIC_REPO_URL="${{ env.PUBLIC_REPO_URL }}"
export chestnut_branch="${{ needs.prepare_chestnut.result == 'success' && format('{0}-chestnut', needs.prepare_strategy.outputs.new_branch) || '' }}"
MESSAGE=$(cat << 'EOF' | envsubst
${{ vars.DISCOURSE_GENERAL_UPDATE_NOTICE }}
+7
View File
@@ -21,5 +21,12 @@
</clean>
</configuration>
</target>
<target id="f2590b2b-9b93-49f9-8510-da3f3724a2ae" name="replay" defaultType="TOOL">
<configuration id="d475264f-6f4c-4092-9b4e-6773309f38b7" name="replay" toolchainName="Default">
<build type="TOOL">
<tool actionId="Tool_External Tools_uv build tools replay" />
</build>
</configuration>
</target>
</component>
</project>
+7
View File
@@ -20,4 +20,11 @@
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
<tool name="uv build tools replay" showInMainMenu="false" showInEditor="false" showInProject="false" showInSearchPopup="false" disabled="false" useConsole="true" showConsoleOnStdOut="false" showConsoleOnStdErr="false" synchronizeAfterRun="true">
<exec>
<option name="COMMAND" value="bash" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -u -j$(nproc) tools/replay/&quot;" />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
</toolSet>
+1 -1
View File
@@ -1,5 +1,5 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Build Debug" type="CLionExternalRunConfiguration" factoryName="Application" REDIRECT_INPUT="false" ELEVATE="false" USE_EXTERNAL_CONSOLE="false" EMULATE_TERMINAL="false" WORKING_DIR="file://$ProjectFileDir$/selfdrive/ui" PASS_PARENT_ENVS_2="true" PROJECT_NAME="sunnypilot" TARGET_NAME="uv Scons Build Debug" CONFIG_NAME="uv Scons Build Debug" RUN_PATH="ui">
<configuration default="false" name="Build Debug" type="CLionExternalRunConfiguration" factoryName="Application" REDIRECT_INPUT="false" ELEVATE="false" USE_EXTERNAL_CONSOLE="false" EMULATE_TERMINAL="false" WORKING_DIR="file://$ProjectFileDir$/selfdrive/ui" PASS_PARENT_ENVS_2="true" PROJECT_NAME="openpilot-special" TARGET_NAME="uv Scons Build Debug" CONFIG_NAME="uv Scons Build Debug" RUN_PATH="ui">
<envs>
<env name="QT_DBL_CLICK_DIST" value="150" />
</envs>
+27
View File
@@ -0,0 +1,27 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Debug Route Controls" type="PythonConfigurationType" factoryName="Python">
<module name="openpilot-special" />
<option name="ENV_FILES" value="" />
<option name="INTERPRETER_OPTIONS" value="" />
<option name="PARENT_ENVS" value="true" />
<envs>
<env name="PYTHONUNBUFFERED" value="1" />
<env name="FINGERPRINT" value="KIA_EV9" />
<env name="SKIP_FW_QUERY" value="1" />
</envs>
<option name="SDK_HOME" value="" />
<option name="WORKING_DIRECTORY" value="$PROJECT_DIR$/selfdrive/car" />
<option name="IS_MODULE_SDK" value="true" />
<option name="ADD_CONTENT_ROOTS" value="true" />
<option name="ADD_SOURCE_ROOTS" value="true" />
<EXTENSION ID="PythonCoverageRunConfigurationExtension" runner="coverage.py" />
<option name="SCRIPT_NAME" value="$PROJECT_DIR$/selfdrive/car/card.py" />
<option name="PARAMETERS" value="" />
<option name="SHOW_COMMAND_LINE" value="false" />
<option name="EMULATE_TERMINAL" value="true" />
<option name="MODULE_MODE" value="false" />
<option name="REDIRECT_INPUT" value="false" />
<option name="INPUT_FILE" value="" />
<method v="2" />
</configuration>
</component>
+7
View File
@@ -0,0 +1,7 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Replay for controls + ui" type="Multirun" separateTabs="false" reuseTabsWithFailures="false" startOneByOne="true" markFailedProcess="true" hideSuccessProcess="false" delayTime="0.0">
<runConfiguration name="replay for controls" type="Native Application" />
<runConfiguration name="Build Debug" type="Custom Build Application" />
<method v="2" />
</configuration>
</component>
+7
View File
@@ -0,0 +1,7 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="replay for controls" type="CLionNativeAppRunConfigurationType" focusToolWindowBeforeRun="true" PROGRAM_PARAMS="&quot;$Prompt$&quot; --block &quot;sendcan,carState,carParams,carOutput,liveTracks,carParamsSP,carStateSP,bookmarkButton&quot;" REDIRECT_INPUT="false" ELEVATE="false" USE_EXTERNAL_CONSOLE="false" EMULATE_TERMINAL="true" WORKING_DIR="file://$ProjectFileDir$/tools/replay" PASS_PARENT_ENVS_2="true" PROJECT_NAME="openpilot-special" TARGET_NAME="replay" CONFIG_NAME="replay" version="1" RUN_PATH="replay">
<method v="2">
<option name="CLION.COMPOUND.BUILD" enabled="true" />
</method>
</configuration>
</component>
+3 -1
View File
@@ -24,7 +24,9 @@ function agnos_init {
if $AGNOS_PY --verify $MANIFEST; then
sudo reboot
fi
$DIR/openpilot/common/hardware/comma/updater $AGNOS_PY $MANIFEST
while true; do
$DIR/openpilot/common/hardware/comma/updater $AGNOS_PY $MANIFEST
done
fi
}
+2
View File
@@ -131,6 +131,7 @@ struct ModelManagerSP @0xaedffd8f31e7b55d {
downloaded @2;
cached @3;
failed @4;
verifying @5;
}
struct DownloadProgress {
@@ -352,6 +353,7 @@ struct OnroadEventSP @0xda96579883444c35 {
speedLimitPending @22;
e2eChime @23;
laneChangeRoadEdge @24;
bigModelReady @25;
}
}
+10 -7
View File
@@ -59,7 +59,7 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"IsDriverViewEnabled", {CLEAR_ON_MANAGER_START, BOOL}},
{"IsEngaged", {PERSISTENT, BOOL}},
{"IsLdwEnabled", {PERSISTENT | BACKUP, BOOL}},
{"IsLiveStreaming", {CLEAR_ON_MANAGER_START, BOOL}},
{"IsLiveStreaming", {CLEAR_ON_MANAGER_START | CLEAR_ON_IGNITION_ON, BOOL}},
{"IsMetric", {PERSISTENT | BACKUP, BOOL}},
{"IsOffroad", {CLEAR_ON_MANAGER_START, BOOL}},
{"IsRhdDetected", {PERSISTENT, BOOL}},
@@ -130,8 +130,8 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"UpdaterLastFetchTime", {PERSISTENT, TIME}},
{"UptimeOffroad", {PERSISTENT, FLOAT, "0.0"}},
{"UptimeOnroad", {PERSISTENT, FLOAT, "0.0"}},
{"UsbGpuActive", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION | CLEAR_ON_IGNITION_ON, BOOL}},
{"UsbGpuLoading", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION | CLEAR_ON_IGNITION_ON, BOOL}},
{"ChestnutActive", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION | CLEAR_ON_IGNITION_ON, BOOL}},
{"ChestnutLoading", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION | CLEAR_ON_IGNITION_ON, BOOL}},
{"Version", {PERSISTENT, STRING}},
// --- sunnypilot params --- //
@@ -195,14 +195,16 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
// Model Manager params
{"ModelManager_ActiveBundle", {PERSISTENT, JSON}},
{"ModelManager_ActiveJson", {CLEAR_ON_MANAGER_START, STRING}},
{"ModelManager_ActiveBundleUSBGPU", {PERSISTENT, JSON}}, //TODO-SP: kept for migration, remove on next sync?
{"ModelManager_ActiveBundleChestnut", {PERSISTENT, JSON}},
{"ModelManager_ActiveJson", {CLEAR_ON_MANAGER_START, JSON}},
{"ModelManager_ClearCache", {CLEAR_ON_MANAGER_START, BOOL}},
{"ModelManager_DownloadIndex", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, INT}},
{"ModelManager_DownloadRef", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, STRING}},
{"ModelManager_Favs", {PERSISTENT | BACKUP, STRING}},
{"ModelManager_LastSyncTime", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, INT, "0"}},
{"ModelManager_LastSyncTime_USBGPU", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, INT, "0"}},
{"ModelManager_LastSyncTime_Chestnut", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, INT, "0"}},
{"ModelManager_ModelsCache", {PERSISTENT | BACKUP, JSON}},
{"ModelManager_ModelsCache_USBGPU", {PERSISTENT | BACKUP, JSON}},
{"ModelManager_ModelsCache_Chestnut", {PERSISTENT | BACKUP, JSON}},
// Neural Network Lateral Control
{"NeuralNetworkLateralControl", {PERSISTENT | BACKUP, BOOL, "0"}},
@@ -245,6 +247,7 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
// mapd
{"MapAdvisorySpeedLimit", {CLEAR_ON_ONROAD_TRANSITION, FLOAT}},
{"Mapd_ClearCache", {CLEAR_ON_MANAGER_START, BOOL}},
{"MapdVersion", {PERSISTENT, STRING}},
{"MapSpeedLimit", {CLEAR_ON_ONROAD_TRANSITION, FLOAT, "0.0"}},
{"NextMapSpeedLimit", {CLEAR_ON_ONROAD_TRANSITION, JSON}},
+11
View File
@@ -16,6 +16,15 @@ MASTER_SP_BRANCHES = ['master']
RELEASE_BRANCHES = ['release-tizi-staging', 'release-mici-staging', 'release-tizi', 'release-mici', 'nightly']
TESTED_BRANCHES = RELEASE_BRANCHES + ['devel-staging', 'nightly-dev'] + RELEASE_SP_BRANCHES + TESTED_SP_BRANCHES
CHESTNUT_BRANCHES = {
"staging": "staging-chestnut",
"dev": "dev-chestnut",
"release-mici": "release-chestnut",
"release-tizi": "release-chestnut",
"release-mici-staging": "release-chestnut-staging",
"release-tizi-staging": "release-chestnut-staging",
}
SP_BRANCH_MIGRATIONS = {
("tici", "staging-c3-new"): "staging-tici",
("tici", "dev-c3-new"): "staging-tici",
@@ -28,6 +37,8 @@ SP_BRANCH_MIGRATIONS = {
("tizi", "release3-staging"): "release-tizi-staging",
("mici", "release3"): "release-mici",
("mici", "release3-staging"): "release-mici-staging",
("tici", "hkg-angle-steering-2025"): "hkg-angle-steering-2025-tici",
("tici", "hkg-angle-steering-2025-prebuilt"): "hkg-angle-steering-2025-tici-prebuilt"
}
BUILD_METADATA_FILENAME = "build.json"
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:845c40ff0d37612e8f2f482a36845744b5ae91ce2fcfc8117990d7d278b59820
size 13079
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8a8c5fece2a1c7587feb41cbe04c6aee08e768ecd9b5d00da6af9832a4ccc842
size 2034
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7409c53d7c72681c24982fd83b56ce70f80797c9c0f936d9296a5c18557ac472
size 7279
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:58bd6155433f623b1f75d134bd8ca4745d9aa71f6767eb807cdbcf7deb3089a1
size 10876
+47 -45
View File
@@ -7,7 +7,7 @@ from openpilot.common.file_chunker import chunk_file, get_chunk_targets, get_exi
from openpilot.common.transformations.camera import _ar_ox_fisheye, _os_fisheye
from openpilot.common.transformations.model import MEDMODEL_INPUT_SIZE, DM_INPUT_SIZE
from openpilot.selfdrive.modeld.constants import ModelConstants
from openpilot.selfdrive.modeld.helpers import TG_INPUT_DEVICES_PATH, usbgpu_present, modeld_pkl_path
from openpilot.selfdrive.modeld.helpers import TG_INPUT_DEVICES_PATH, chestnut_present, modeld_pkl_path
CAMERA_CONFIGS = [
@@ -36,18 +36,18 @@ else:
tg_devices = { # which device to put jit inputs to at runtime
'openpilot.selfdrive.modeld.modeld': {
'default': {'WARP_DEV': tg_backend, 'QUEUE_DEV': tg_backend},
'usbgpu': {'WARP_DEV': tg_backend, 'QUEUE_DEV': 'AMD'}
'chestnut': {'WARP_DEV': tg_backend, 'QUEUE_DEV': 'AMD'}
},
'openpilot.selfdrive.modeld.dmonitoringmodeld': {
'default': {'DEV': tg_backend}
},
}
USBGPU = usbgpu_present()
if USBGPU:
usbgpu_tg_flags = f'DEBUG=2 DEV=USB+AMD:LLVM WARP_DEV={tg_backend} FLOAT16=1 JIT_BATCH_SIZE=0 GMMU=0 TC_OPT=2'
CHESTNUT = chestnut_present()
if CHESTNUT:
chestnut_tg_flags = f'DEBUG=2 DEV=USB+AMD:LLVM WARP_DEV={tg_backend} FLOAT16=1 JIT_BATCH_SIZE=0 GMMU=0 TC_OPT=2'
# the USB+AMD GPU takes an exclusive flock; serialize all targets that touch it
usbgpu_lock = File("models/.usb_gpu.lock").abspath
chestnut_lock = File("models/.chestnut.lock").abspath
def write_tg_devices(target, source, env):
with open(str(target[0]), "w") as f:
@@ -73,44 +73,45 @@ compile_modeld_script = [
model_w, model_h = MEDMODEL_INPUT_SIZE
frame_skip = ModelConstants.MODEL_RUN_FREQ // ModelConstants.MODEL_CONTEXT_FREQ
for usbgpu in [False, True] if USBGPU else [False]:
target_pkl_path = File(modeld_pkl_path(usbgpu)).abspath
# BIG_INTO_SMALL=1 builds the default target from the big model, e.g. to test it without a USB GPU
file_prefix, cmd_flags = ('big_', usbgpu_tg_flags) if usbgpu else ('big_' if os.getenv('BIG_INTO_SMALL') else '', tg_flags)
driving_onnx_deps = get_existing_chunks(File(f"models/{file_prefix}driving_supercombo.onnx").abspath)
camera_res_args = ' '.join(f'{cw}x{ch}' for cw, ch in CAMERA_CONFIGS)
# CPU 7 is isolated with isolcpus on AGNOS, so explicitly pin the compiler to it.
taskset = 'taskset -c 7 ' if arch == 'comma_arm64' else ''
cmd = (f'{cmd_flags} {mac_brew_string} {taskset}python3 {modeld_dir}/compile_modeld.py '
f'--model-size {model_w}x{model_h} '
f'--camera-resolutions {camera_res_args} '
f'--onnx {File(f"models/{file_prefix}driving_supercombo.onnx").abspath} '
f'--output {target_pkl_path} --frame-skip {frame_skip}')
onnx_sizes_sum = sum(os.path.getsize(f) for f in driving_onnx_deps)
chunk_targets = get_chunk_targets(target_pkl_path, estimate_pickle_max_size(onnx_sizes_sum))
def do_compile(target, source, env, command=cmd, pkl=target_pkl_path, chunks=chunk_targets):
from openpilot.system.hardware.chestnut.flash import link_up
# chestnut can enumerate before its PCIe link is up due to varying 12V power behavior across cars
for _ in range(10):
if link_up():
break
time.sleep(1)
else:
print("Chestnut not ready, skipping big model build")
return
if ret := env.Execute(command):
return ret
chunk_file(pkl, chunks)
def do_chunk(target, source, env, pkl=target_pkl_path, chunks=chunk_targets):
chunk_file(pkl, chunks)
actions = Action(do_compile, " [USBGPU] $TARGET") if usbgpu else [cmd, Action(do_chunk, " [CHUNK] $TARGET")]
node = lenv.Command(
chunk_targets,
tinygrad_files + compile_modeld_script + driving_onnx_deps + [Value(chunk_targets), chunker_file],
actions,
)
if usbgpu:
lenv.SideEffect(usbgpu_lock, node)
if not os.getenv('SKIP_TINYGRAD_COMPILE'):
for chestnut in [False, True] if CHESTNUT else [False]:
target_pkl_path = File(modeld_pkl_path(chestnut)).abspath
# BIG_INTO_SMALL=1 builds the default target from the big model, e.g. to test it without a chestnut
file_prefix, cmd_flags = ('big_', chestnut_tg_flags) if chestnut else ('big_' if os.getenv('BIG_INTO_SMALL') else '', tg_flags)
driving_onnx_deps = get_existing_chunks(File(f"models/{file_prefix}driving_supercombo.onnx").abspath)
camera_res_args = ' '.join(f'{cw}x{ch}' for cw, ch in CAMERA_CONFIGS)
# CPU 7 is isolated with isolcpus on AGNOS, so explicitly pin the compiler to it.
taskset = 'taskset -c 7 ' if arch == 'comma_arm64' else ''
cmd = (f'{cmd_flags} {mac_brew_string} {taskset}python3 {modeld_dir}/compile_modeld.py '
f'--model-size {model_w}x{model_h} '
f'--camera-resolutions {camera_res_args} '
f'--onnx {File(f"models/{file_prefix}driving_supercombo.onnx").abspath} '
f'--output {target_pkl_path} --frame-skip {frame_skip}')
onnx_sizes_sum = sum(os.path.getsize(f) for f in driving_onnx_deps)
chunk_targets = get_chunk_targets(target_pkl_path, estimate_pickle_max_size(onnx_sizes_sum))
def do_compile(target, source, env, command=cmd, pkl=target_pkl_path, chunks=chunk_targets):
from openpilot.system.hardware.chestnut.flash import link_up
# chestnut can enumerate before its PCIe link is up due to varying 12V power behavior across cars
for _ in range(10):
if link_up():
break
time.sleep(1)
else:
print("Chestnut not ready, skipping big model build")
return
if ret := env.Execute(command):
return ret
chunk_file(pkl, chunks)
def do_chunk(target, source, env, pkl=target_pkl_path, chunks=chunk_targets):
chunk_file(pkl, chunks)
actions = Action(do_compile, " [CHESTNUT] $TARGET") if chestnut else [cmd, Action(do_chunk, " [CHUNK] $TARGET")]
node = lenv.Command(
chunk_targets,
tinygrad_files + compile_modeld_script + driving_onnx_deps + [Value(chunk_targets), chunker_file],
actions,
)
if chestnut:
lenv.SideEffect(chestnut_lock, node)
# get model metadata
fn = File(f"models/dmonitoring_model").abspath
@@ -142,4 +143,5 @@ def tg_compile(flags, model_name):
Action(do_chunk, " [CHUNK] $TARGET")],
)
tg_compile(tg_flags, 'dmonitoring_model')
if not os.getenv('SKIP_TINYGRAD_COMPILE'):
tg_compile(tg_flags, 'dmonitoring_model')
@@ -29,7 +29,7 @@ class ModelState:
output: np.ndarray
def __init__(self, cam_w: int, cam_h: int):
self.DEV = get_tg_input_devices(PROCESS_NAME, usbgpu=False)['DEV']
self.DEV = get_tg_input_devices(PROCESS_NAME, chestnut=False)['DEV']
with open(METADATA_PATH, 'rb') as f:
model_metadata = pickle.load(f)
self.input_shapes = model_metadata['input_shapes']
+7 -7
View File
@@ -13,12 +13,12 @@ MODELS_DIR = Path(__file__).resolve().parent / 'models'
TG_INPUT_DEVICES_PATH = MODELS_DIR / 'tg_input_devices.json'
def get_tg_input_devices(process_name: str, usbgpu: bool):
def get_tg_input_devices(process_name: str, chestnut: bool):
with open(TG_INPUT_DEVICES_PATH) as f:
return json.load(f)[process_name]['default' if not usbgpu else 'usbgpu']
return json.load(f)[process_name]['default' if not chestnut else 'chestnut']
def modeld_pkl_path(usbgpu: bool):
prefix = 'big_' if usbgpu else ''
def modeld_pkl_path(chestnut: bool):
prefix = 'big_' if chestnut else ''
return MODELS_DIR / f'{prefix}driving_tinygrad.pkl'
def dump_oob(obj, f):
@@ -45,7 +45,7 @@ def load_oob(f):
yield pb
return pickle.load(io.BytesIO(opcodes), buffers=buffers())
def usbgpu_present() -> bool:
def chestnut_present() -> bool:
for d in USB_DEVICES_PATH.glob("*"):
try:
usb_id = (int((d / "idVendor").read_text(), 16), int((d / "idProduct").read_text(), 16))
@@ -56,5 +56,5 @@ def usbgpu_present() -> bool:
pass
return False
def usbgpu_compiled() -> bool:
return Path(get_manifest_path(modeld_pkl_path(usbgpu=True))).is_file()
def chestnut_compiled() -> bool:
return Path(get_manifest_path(modeld_pkl_path(chestnut=True))).is_file()
+32 -30
View File
@@ -1,7 +1,9 @@
#!/usr/bin/env python3
from collections.abc import Callable
import ctypes
from functools import cached_property
import os
os.environ['GMMU'] = '0' # for usbgpu fast loading, noop for qcom
os.environ['GMMU'] = '0' # for chestnut fast loading, noop for qcom
from tinygrad.tensor import Tensor
from tinygrad.device import Device
import struct
@@ -30,7 +32,7 @@ from openpilot.selfdrive.modeld.compile_modeld import make_input_queues, WARP_IN
from openpilot.selfdrive.modeld.fill_model_msg import fill_model_msg, fill_driving_model_data, fill_pose_msg, PublishState
from openpilot.common.file_chunker import open_file_chunked
from openpilot.selfdrive.modeld.constants import ModelConstants, Plan
from openpilot.selfdrive.modeld.helpers import usbgpu_present, usbgpu_compiled, modeld_pkl_path, get_tg_input_devices, load_oob
from openpilot.selfdrive.modeld.helpers import chestnut_present, chestnut_compiled, modeld_pkl_path, get_tg_input_devices, load_oob
from openpilot.sunnypilot.livedelay.helpers import get_lat_delay
from openpilot.sunnypilot.modeld_v2.modeld_base import ModelStateBase
@@ -94,8 +96,10 @@ class ChestnutState:
if self.big and "AMD" in Device._opened_devices and self.sends % 100 == 1:
try:
smu = Device["AMD"].iface.dev_impl.smu
metrics_t = smu.smu_mod.SmuMetricsExternal_t
smu._send_msg(smu.smu_mod.PPSMC_MSG_TransferTableSmu2Dram, smu.smu_mod.TABLE_SMU_METRICS, timeout=100)
metrics = smu.read_table(smu.smu_mod.SmuMetricsExternal_t, smu.smu_mod.TABLE_SMU_METRICS).SmuMetrics
metrics_buf = bytearray(smu.adev.vram.view(smu.driver_table_paddr, ctypes.sizeof(metrics_t))[:])
metrics = metrics_t.from_buffer(metrics_buf).SmuMetrics
self.metrics = {'tempC': metrics.AvgTemperature[smu.smu_mod.TEMP_HOTSPOT],
'memoryTempC': metrics.AvgTemperature[smu.smu_mod.TEMP_MEM],
'powerDrawW': metrics.AverageSocketPower,
@@ -141,18 +145,18 @@ class FrameMeta:
class ModelState(ModelStateBase):
prev_desire: np.ndarray # for tracking the rising edge of the pulse
def __init__(self, cam_w: int, cam_h: int, usbgpu: bool):
def __init__(self, cam_w: int, cam_h: int, chestnut: bool):
ModelStateBase.__init__(self)
input_devices = get_tg_input_devices(PROCESS_NAME, usbgpu)
input_devices = get_tg_input_devices(PROCESS_NAME, chestnut)
self.WARP_DEV, self.QUEUE_DEV = input_devices['WARP_DEV'], input_devices['QUEUE_DEV']
jits = load_oob(open_file_chunked(modeld_pkl_path(usbgpu)))
jits = load_oob(open_file_chunked(modeld_pkl_path(chestnut)))
metadata = jits['metadata']
self.input_shapes = metadata['input_shapes']
self.vision_input_names = [k for k in self.input_shapes if 'img' in k]
self.output_slices = metadata['output_slices']
self.prev_desire = np.zeros(ModelConstants.DESIRE_LEN, dtype=np.float32)
self.usbgpu = usbgpu
self.chestnut = chestnut
self.frame_skip = ModelConstants.MODEL_RUN_FREQ // ModelConstants.MODEL_CONTEXT_FREQ
self.input_queues, self.npy = make_input_queues(self.input_shapes, self.frame_skip, device=self.QUEUE_DEV)
@@ -168,7 +172,7 @@ class ModelState(ModelStateBase):
return parsed_model_outputs
def run(self, bufs: dict[str, VisionBuf], transforms: dict[str, np.ndarray],
inputs: dict[str, np.ndarray]) -> dict[str, np.ndarray] | None:
inputs: dict[str, np.ndarray], after_enqueue: Callable[[], None] | None = None) -> dict[str, np.ndarray]:
for key in bufs.keys():
ptr = np.frombuffer(bufs[key].data, dtype=np.uint8).ctypes.data
yuv_size = self.frame_buf_params[key][3]
@@ -192,11 +196,11 @@ class ModelState(ModelStateBase):
outs, = self.run_policy(
**{k: self.input_queues[k] for k in POLICY_INPUTS if k in self.input_queues}, warped=warped
)
if after_enqueue is not None:
after_enqueue()
model_output = outs.numpy()[0]
if self.usbgpu and not np.all(np.isfinite(model_output)):
# TODO remove with prev_feat
cloudlog.error("model output not finite, dropping frame")
return None
if self.chestnut and not np.all(np.isfinite(model_output)):
raise RuntimeError("model output not finite")
outputs_dict = self.parser.parse_outputs(self.slice_outputs(model_output, self.output_slices))
self.npy['prev_feat'][:] = model_output[self.output_slices['hidden_state']]
@@ -218,12 +222,12 @@ class ModelState(ModelStateBase):
def main(demo=False):
cloudlog.warning("modeld init")
USBGPU = usbgpu_present() and usbgpu_compiled()
if USBGPU:
CHESTNUT = chestnut_present() and chestnut_compiled()
if CHESTNUT:
os.environ['HCQDEV_WAIT_TIMEOUT_MS'] = '3000'
params = Params()
params.put_bool("UsbGpuLoading", USBGPU)
params.remove("UsbGpuActive")
params.put_bool("ChestnutLoading", CHESTNUT)
params.remove("ChestnutActive")
config_realtime_process(7, 54)
@@ -253,7 +257,7 @@ def main(demo=False):
st = time.monotonic()
cloudlog.warning("loading model")
model = None
if USBGPU:
if CHESTNUT:
big_model = None
def load_big():
nonlocal big_model
@@ -267,23 +271,23 @@ def main(demo=False):
loader.start()
loader.join(BIG_MODEL_TIMEOUT)
model = big_model
params.put_bool("UsbGpuActive", model is not None)
params.put_bool("ChestnutActive", model is not None)
small_model = ModelState(vipc_client_main.width, vipc_client_main.height, False) if model is None or USBGPU else None
small_model = ModelState(vipc_client_main.width, vipc_client_main.height, False) if model is None or CHESTNUT else None
if model is None:
model = small_model
params.put_bool("UsbGpuLoading", False)
params.put_bool("ChestnutLoading", False)
assert model is not None
cloudlog.warning(f"models loaded in {time.monotonic() - st:.1f}s, modeld starting")
# messaging
pub_socks = ["modelV2", "drivingModelData", "cameraOdometry", "modelDataV2SP"] + (["chestnutState"] if USBGPU else [])
pub_socks = ["modelV2", "drivingModelData", "cameraOdometry", "modelDataV2SP"] + (["chestnutState"] if CHESTNUT else [])
pm = PubMaster(pub_socks)
sm = SubMaster(["deviceState", "carState", "narrowRoadCameraState", "extrinsicsCalibration", "driverMonitoringState", "carControl", "lateralDelay"])
publish_state = PublishState()
params = Params()
chestnut_state = ChestnutState(pm, model.usbgpu) if USBGPU else None
chestnut_state = ChestnutState(pm, model.chestnut) if CHESTNUT else None
# setup filter to track dropped frames
frame_dropped_filter = FirstOrderFilter(0., 10., 1. / ModelConstants.MODEL_RUN_FREQ)
@@ -393,13 +397,15 @@ def main(demo=False):
mt1 = time.perf_counter()
try:
model_output = model.run(bufs, transforms, inputs)
send_chestnut = (chestnut_state is not None and
run_count % round(ModelConstants.MODEL_RUN_FREQ / SERVICE_LIST['chestnutState'].frequency) == 0)
model_output = model.run(bufs, transforms, inputs, chestnut_state.send if send_chestnut else None)
except Exception:
if not params.get_bool("UsbGpuActive"):
if not params.get_bool("ChestnutActive"):
raise
# fallback to small model
cloudlog.exception("big model failed, fall back to small")
params.put_bool("UsbGpuActive", False)
params.put_bool("ChestnutActive", False)
assert small_model is not None
model = small_model
if chestnut_state is not None:
@@ -419,7 +425,7 @@ def main(demo=False):
fill_model_msg(modelv2_send, model_output, action,
publish_state, meta_main.frame_id, meta_extra.frame_id, frame_id,
frame_drop_ratio, meta_main.timestamp_eof, model_execution_time, extrinsics_calibration_seen)
modelv2_send.modelV2.big = model.usbgpu
modelv2_send.modelV2.big = model.chestnut
desire_state = modelv2_send.modelV2.meta.desireState
l_lane_change_prob = desire_state[log.Desire.laneChangeLeft]
@@ -441,10 +447,6 @@ def main(demo=False):
pm.send('modelDataV2SP', mdv2sp_send)
last_vipc_frame_id = meta_main.frame_id
if chestnut_state is not None and run_count % round(ModelConstants.MODEL_RUN_FREQ / SERVICE_LIST['chestnutState'].frequency) == 0:
chestnut_state.send()
if __name__ == "__main__":
try:
import argparse
@@ -18,7 +18,7 @@
"_comment": "Set extra field to the failed reason."
},
"Offroad_ChestnutBranch": {
"text": "Chestnut detected! Switch to the release-chestnut branch to use chestnut-class models.",
"text": "Chestnut detected! Switch to the %1 branch to use chestnut-class models.",
"severity": 0
},
"Offroad_UnregisteredHardware": {
+5 -4
View File
@@ -195,17 +195,18 @@ class SelfdriveD(CruiseHelper):
self.events.add(EventName.joystickDebug)
self.startup_event = None
loading = self.params.get_bool("UsbGpuLoading")
loading = self.params.get_bool("ChestnutLoading")
if self.big_model_loading and not loading:
self.big_model_ready_t = time.monotonic()
self.events_sp.add(custom.OnroadEventSP.EventName.bigModelReady)
self.big_model_loading = loading
if self.big_model_loading:
self.events.add(EventName.bigModelLoading)
big_active = self.params.get("UsbGpuActive")
usbgpu_present = self.sm['deviceState'].chestnutPresent
big_active = self.params.get("ChestnutActive")
chestnut_present = self.sm['deviceState'].chestnutPresent
model_unavailable = big_active is True and self.sm.seen['modelV2'] and not self.sm.alive['modelV2']
big_failed = big_active is False or model_unavailable or (self.big_model_active and not usbgpu_present)
big_failed = big_active is False or model_unavailable or (self.big_model_active and not chestnut_present)
if big_failed and not self.big_model_failed:
self.events.add(EventName.bigModelFailed)
self.big_model_failed = big_failed
+8 -1
View File
@@ -168,9 +168,16 @@ class Sidebar(Widget, SidebarSP):
# Home/Flag button
flag_pressed = mouse_down and rl.check_collision_point_rec(mouse_pos, HOME_BTN)
button_img = self._flag_img if ui_state.started else self._home_img
button_pos = rl.Vector2(HOME_BTN.x, HOME_BTN.y)
icon_opacity = 1.0
if gui_app.sunnypilot_ui():
button_img, button_pos, icon_opacity = SidebarSP._get_home_icon(self, button_img)
tint = Colors.BUTTON_PRESSED if (ui_state.started and flag_pressed) else Colors.BUTTON_NORMAL
rl.draw_texture_ex(button_img, rl.Vector2(HOME_BTN.x, HOME_BTN.y), 0.0, 1.0, tint)
if icon_opacity < 1.0:
tint = rl.Color(tint[0], tint[1], tint[2], int(255 * icon_opacity))
rl.draw_texture_ex(button_img, button_pos, 0.0, 1.0, tint)
# Microphone button
if self._recording_audio:
+10 -7
View File
@@ -9,7 +9,7 @@ from openpilot.system.ui.widgets.layouts import HBoxLayout
from openpilot.system.ui.widgets.icon_widget import IconWidget
from openpilot.system.ui.widgets.label import UnifiedLabel, gui_label
from openpilot.system.ui.lib.application import gui_app, FontWeight, MousePos
from openpilot.selfdrive.ui.ui_state import ui_state
from openpilot.selfdrive.ui.ui_state import ui_state, ChestnutState
from openpilot.common.version import RELEASE_BRANCHES
HEAD_BUTTON_FONT_SIZE = 40
@@ -139,8 +139,8 @@ class MiciHomeLayout(Widget):
self._version_text = self._get_version_text()
self._experimental_icon = IconWidget("icons_mici/experimental_mode.png", (48, 48))
self._egpu_icon = IconWidget("icons_mici/egpu_green.png", (50, 37))
self._egpu_icon_gray = IconWidget("icons_mici/egpu_gray.png", (50, 37))
self._chestnut_icon = IconWidget("icons_mici/chestnut_green.png", (68, 40))
self._chestnut_failed_icon = IconWidget("icons_mici/chestnut_orange.png", (68, 40))
self._mic_icon = IconWidget("icons_mici/microphone.png", (32, 46))
self._body_icon = IconWidget("icons_mici/body.png", (54, 37))
@@ -150,8 +150,8 @@ class MiciHomeLayout(Widget):
IconWidget("icons_mici/settings.png", (48, 48), opacity=0.9),
NetworkIcon(),
self._experimental_icon,
self._egpu_icon,
self._egpu_icon_gray,
self._chestnut_icon,
self._chestnut_failed_icon,
self._body_icon,
self._mic_icon,
], spacing=18)
@@ -248,8 +248,11 @@ class MiciHomeLayout(Widget):
# ***** Center-aligned bottom section icons *****
self._experimental_icon.set_visible(ui_state.experimental_mode)
self._egpu_icon.set_visible(ui_state.sm["deviceState"].chestnutPresent and ui_state.usbgpu_compiled)
self._egpu_icon_gray.set_visible(ui_state.sm["deviceState"].chestnutPresent and not ui_state.usbgpu_compiled)
if gui_app.sunnypilot_ui():
self._set_chestnut_visibility()
else:
self._chestnut_icon.set_visible(ui_state.chestnut_state in (ChestnutState.READY, ChestnutState.LOADING, ChestnutState.ACTIVE))
self._chestnut_failed_icon.set_visible(ui_state.chestnut_state in (ChestnutState.UNCOMPILED, ChestnutState.FAILED))
self._mic_icon.set_visible(ui_state.recording_audio)
self._body_icon.set_visible(bool(ui_state.is_body))
@@ -3,7 +3,7 @@ import pyray as rl
from dataclasses import dataclass
from openpilot.common.constants import CV
from openpilot.selfdrive.ui.mici.onroad.torque_bar import TorqueBar
from openpilot.selfdrive.ui.ui_state import ui_state, UIStatus
from openpilot.selfdrive.ui.ui_state import ui_state, UIStatus, ChestnutState
from openpilot.system.ui.lib.application import gui_app, FontWeight
from openpilot.system.ui.lib.multilang import tr
from openpilot.system.ui.lib.text_measure import measure_text_cached
@@ -107,8 +107,7 @@ class HudRenderer(Widget):
self.speed: float = 0.0
self.v_ego_cluster_seen: bool = False
self._engaged: bool = False
self._small_model_engaged: bool = False
self._egpu_fade_time: float = 0
self._chestnut_fade_time: float = 0
self._can_draw_top_icons = True
self._show_wheel_critical = False
@@ -124,17 +123,15 @@ class HudRenderer(Widget):
self._txt_wheel: rl.Texture = gui_app.texture('icons_mici/wheel.png', 50, 50)
self._txt_wheel_critical: rl.Texture = gui_app.texture('icons_mici/wheel_critical.png', 50, 50)
self._txt_exclamation_point: rl.Texture = gui_app.texture('icons_mici/exclamation_point.png', 9, 44)
self._txt_egpu: rl.Texture = gui_app.texture('icons_mici/egpu.png', 60, 44)
self._txt_egpu_green: rl.Texture = gui_app.texture('icons_mici/egpu_green.png', 60, 44)
self._txt_egpu_orange: rl.Texture = gui_app.texture('icons_mici/egpu_orange.png', 60, 44)
self._txt_egpu_crossed: rl.Texture = gui_app.texture('icons_mici/egpu_crossed.png', 60, 52)
self._egpu_icon: rl.Texture | None = None
self._txt_chestnut: rl.Texture = gui_app.texture('icons_mici/chestnut.png', 60, 44)
self._txt_chestnut_green: rl.Texture = gui_app.texture('icons_mici/chestnut_green.png', 60, 44)
self._txt_chestnut_orange: rl.Texture = gui_app.texture('icons_mici/chestnut_orange.png', 75, 44)
self._chestnut_icon: rl.Texture | None = None
self._wheel_alpha_filter = FirstOrderFilter(0, 0.05, 1 / gui_app.target_fps)
self._wheel_y_filter = FirstOrderFilter(0, 0.1, 1 / gui_app.target_fps)
self._set_speed_alpha_filter = FirstOrderFilter(0.0, 0.1, 1 / gui_app.target_fps)
self._egpu_alpha_filter = FirstOrderFilter(0.0, 0.1, 1 / gui_app.target_fps)
self._chestnut_alpha_filter = FirstOrderFilter(0.0, 0.1, 1 / gui_app.target_fps)
def set_wheel_critical_icon(self, critical: bool):
"""Set the wheel icon to critical or normal state."""
@@ -165,13 +162,10 @@ class HudRenderer(Widget):
controls_state.deprecated.vCruise if v_cruise_cluster == 0.0 else v_cruise_cluster
)
engaged = sm['selfdriveState'].enabled
if (engaged and not self._engaged and not ui_state.usbgpu_loading and ui_state.usbgpu_active is not True and
ui_state.sm.recv_frame['modelV2'] > ui_state.started_frame):
self._small_model_engaged = True
if engaged != self._engaged:
self._egpu_fade_time = rl.get_time() if engaged else 0
if (set_speed != self.set_speed and engaged) or (engaged and not self._engaged):
self._set_speed_changed_time = rl.get_time()
if engaged != self._engaged:
self._chestnut_fade_time = rl.get_time() if engaged else 0
self._engaged = engaged
self.set_speed = set_speed
self.is_cruise_set = 0 < self.set_speed < SET_SPEED_NA
@@ -191,8 +185,7 @@ class HudRenderer(Widget):
if self.is_cruise_set:
self._draw_set_speed(rect)
if ui_state.usbgpu and ui_state.usbgpu_compiled:
self._draw_model_source(rect)
self._draw_model_source(rect)
self._draw_steering_wheel(rect)
@@ -200,30 +193,24 @@ class HudRenderer(Widget):
if ui_state.sm.recv_frame['selfdriveState'] < ui_state.started_frame:
return
big_failed = (ui_state.usbgpu_active is False or not ui_state.sm['deviceState'].chestnutPresent or
(ui_state.usbgpu_active is True and ui_state.sm.recv_frame['modelV2'] > ui_state.started_frame and
not ui_state.sm.alive['modelV2']) or
(ui_state.usbgpu_active is None and ui_state.sm.recv_frame['modelV2'] > ui_state.started_frame))
self._small_model_engaged &= big_failed
loading = ui_state.usbgpu_loading or (ui_state.usbgpu_active is None and not big_failed)
loading = ui_state.chestnut_state == ChestnutState.LOADING
if loading:
pulse = 0.5 - 0.5 * math.cos(rl.get_time() * 6.0)
icon = self._txt_egpu
opacity = 0.35 + 0.65 * pulse
elif self._small_model_engaged:
icon = self._txt_egpu_crossed
opacity = 0.65
elif big_failed:
icon = self._txt_egpu_orange
icon = self._txt_chestnut
opacity = 0.35 + 0.65 * (0.5 - 0.5 * math.cos(rl.get_time() * 6.0))
elif ui_state.chestnut_state in (ChestnutState.UNCOMPILED, ChestnutState.FAILED):
icon = self._txt_chestnut_orange
opacity = 1.0
elif ui_state.chestnut_state == ChestnutState.ACTIVE:
icon = self._txt_chestnut_green
opacity = 1.0
else:
icon = self._txt_egpu_green
opacity = 1.0
return
if icon is not self._egpu_icon:
self._egpu_fade_time = rl.get_time()
self._egpu_icon = icon
alpha = self._egpu_alpha_filter.update(loading or 0 < rl.get_time() - self._egpu_fade_time < SET_SPEED_PERSISTENCE)
if icon is not self._chestnut_icon:
self._chestnut_fade_time = rl.get_time()
self._chestnut_icon = icon
visible = loading or rl.get_time() - self._chestnut_fade_time < SET_SPEED_PERSISTENCE
alpha = self._chestnut_alpha_filter.update(visible)
if alpha < 1e-2:
return
+9 -11
View File
@@ -149,11 +149,15 @@ class BigButton(Widget):
def set_touch_valid_callback(self, touch_callback: Callable[[], bool]) -> None:
super().set_touch_valid_callback(lambda: touch_callback() and self._grow_animation_until is None)
def _width_hint(self) -> int:
# A value moves the title to the top, where it shares space with the icon.
def _title_width_hint(self) -> int:
# A value moves the title to the top, where it shares space with the icon
icon_size = self._txt_icon.width if self._txt_icon and self.value else 0
return int(self._rect.width - self.LABEL_HORIZONTAL_PADDING * 2 - icon_size)
def _subtitle_width_hint(self) -> int:
# Bottom aligned, so it sits below the icon
return int(self._rect.width - self.LABEL_HORIZONTAL_PADDING * 2)
def _get_label_font_size(self):
if len(self.text) <= 18:
return 48
@@ -228,14 +232,14 @@ class BigButton(Widget):
label_color = LABEL_COLOR if self.enabled else rl.Color(255, 255, 255, int(255 * 0.35))
self._label.set_color(label_color)
label_rect = rl.Rectangle(label_x, btn_y + self.LABEL_VERTICAL_PADDING, self._width_hint(),
label_rect = rl.Rectangle(label_x, btn_y + self.LABEL_VERTICAL_PADDING, self._title_width_hint(),
self._rect.height - self.LABEL_VERTICAL_PADDING * 2)
self._label.render(label_rect)
if self.value:
label_y = btn_y + self.LABEL_VERTICAL_PADDING + self._label.get_content_height(self._width_hint())
label_y = label_rect.y + self._label.get_content_height(int(label_rect.width))
sub_label_height = btn_y + self._rect.height - self.LABEL_VERTICAL_PADDING - label_y
sub_label_rect = rl.Rectangle(label_x, label_y, self._width_hint(), sub_label_height)
sub_label_rect = rl.Rectangle(label_x, label_y, self._subtitle_width_hint(), sub_label_height)
self._sub_label.render(sub_label_rect)
# ICON -------------------------------------------------------------------
@@ -312,9 +316,6 @@ class BigMultiToggle(BigToggle):
self.set_value(self._options[0])
def _width_hint(self) -> int:
return int(self._rect.width - self.LABEL_HORIZONTAL_PADDING * 2 - self._txt_enabled_toggle.width)
def _handle_mouse_release(self, mouse_pos: MousePos):
super()._handle_mouse_release(mouse_pos)
cur_idx = self._options.index(self.value)
@@ -363,9 +364,6 @@ class GreyBigButton(BigButton):
def LABEL_VERTICAL_PADDING(self):
return BigButton.LABEL_VERTICAL_PADDING if self._label.text else 18
def _width_hint(self) -> int:
return int(self._rect.width - self.LABEL_HORIZONTAL_PADDING * 2)
def _get_label_font_size(self):
return 36
@@ -192,7 +192,7 @@ class ModelRenderer(Widget, ChevronMetrics, ModelRendererSP):
max_idx = self._get_path_length_idx(path_x_array, max_distance)
self._path.projected_points = self._map_line_to_polygon(
self._path.raw_points, 0.9, self._path_offset_z, max_idx, max_distance, allow_invert=False
self._path.raw_points, self._get_path_half_width(), self._path_offset_z, max_idx, max_distance, allow_invert=False
)
self._update_experimental_gradient()
@@ -292,7 +292,7 @@ class ModelRenderer(Widget, ChevronMetrics, ModelRendererSP):
allow_throttle = sm['longitudinalPlan'].allowThrottle or not self._longitudinal_control
self._blend_filter.update(int(allow_throttle))
if ui_state.rainbow_path:
if ui_state.rainbow_path and self._lateral_active:
self.rainbow_path.draw_rainbow_path(self._rect, self._path)
return
@@ -10,9 +10,10 @@ import time
import pyray as rl
from openpilot.cereal import custom
from openpilot.sunnypilot.models.default_model import DEFAULT_MODEL
from openpilot.sunnypilot.models.helpers import ACTIVE_BUNDLE_KEYS, get_selected_bundle, resolve_bundle_by_ref
from openpilot.common.constants import CV
from openpilot.selfdrive.ui.ui_state import device, ui_state
from openpilot.selfdrive.ui.sunnypilot.model_info import big_model_state, bundles_for_source, carrying_model, default_model_name, queued_name
from openpilot.system.ui.lib.multilang import tr
from openpilot.system.ui.lib.application import gui_app
from openpilot.system.ui.widgets import DialogResult, Widget
@@ -36,7 +37,10 @@ class ModelsLayout(Widget):
super().__init__()
self.model_manager = None
self.model_dialog = None
self._selection_source = None
self._downloading = False
self._verifying = False
self._last_note = None
self.last_cache_calc_time = 0
self._initialize_items()
@@ -48,17 +52,24 @@ class ModelsLayout(Widget):
self._scroller = Scroller(self.items, line_separator=True, spacing=0)
def _initialize_items(self):
self.current_model_item = ListItemSP(
title=tr("Current Model"),
self.small_model_item = ListItemSP(
title=tr("Small Model"),
description="",
action_item=ScrollingButtonAction(tr("SELECT")),
callback=self._handle_current_model_clicked
callback=lambda: self._open_source_dialog("qcom")
)
self.big_model_item = ListItemSP(
title=tr("Big Model"),
action_item=ScrollingButtonAction(tr("SELECT")),
callback=lambda: self._open_source_dialog("chestnut")
)
self.download_item = download_status_item(lambda: tr("Download") if self._downloading else tr("Model Status"))
self.refresh_item = button_item(tr("Refresh Model List"), tr("REFRESH"), "",
lambda: (ui_state.params.put("ModelManager_LastSyncTime", 0),
ui_state.params.put("ModelManager_LastSyncTime_Chestnut", 0),
gui_app.push_widget(alert_dialog(tr("Fetching Latest Models")))))
self.clear_cache_item = ListItemSP(
@@ -68,7 +79,9 @@ class ModelsLayout(Widget):
callback=self._clear_cache
)
self.cancel_download_item = button_item(tr("Cancel Download"), tr("Cancel"), "", lambda: ui_state.params.remove("ModelManager_DownloadIndex"))
self.cancel_download_item = button_item(lambda: tr("Cancel Verification") if self._verifying else tr("Cancel Download"),
tr("Cancel"), "",
lambda: ui_state.params.remove("ModelManager_DownloadRef"))
self.lane_turn_value_control = option_item_sp(tr("Adjust Lane Turn Speed"), "LaneTurnValue", 500, 2000,
tr("Set the maximum speed for lane turn desires. Default is 19 mph."),
@@ -93,7 +106,7 @@ class ModelsLayout(Widget):
1, None, True, "", style.BUTTON_ACTION_WIDTH, None, True,
lambda v: f"{v / 100:.2f} m")
self.items = [self.current_model_item, self.cancel_download_item, self.download_item, self.refresh_item, self.clear_cache_item,
self.items = [self.small_model_item, self.big_model_item, self.cancel_download_item, self.download_item, self.refresh_item, self.clear_cache_item,
self.lane_turn_desire_toggle, self.lane_turn_value_control, self.lagd_toggle, self.delay_control, self.camera_offset]
def _update_lagd_description(self, lagd_toggle: bool):
@@ -107,16 +120,16 @@ class ModelsLayout(Widget):
desc += f"<br>{tr('Actuator Delay:')} {cp:.2f} s + {tr('Software Delay:')} {sw:.2f} s = {tr('Total Delay:')} {cp + sw:.2f} s"
self.lagd_toggle.set_description(desc)
def _is_downloading(self):
return (self.model_manager and self.model_manager.selectedBundle and
self.model_manager.selectedBundle.status == custom.ModelManagerSP.DownloadStatus.downloading)
@staticmethod
def calculate_cache_size():
cache_size = 0.0
if os.path.exists(CUSTOM_MODEL_PATH):
cache_size = sum(os.path.getsize(os.path.join(CUSTOM_MODEL_PATH, file)) for file in os.listdir(CUSTOM_MODEL_PATH)) / (1024**2)
return cache_size
for file in os.listdir(CUSTOM_MODEL_PATH):
try:
cache_size += os.path.getsize(os.path.join(CUSTOM_MODEL_PATH, file))
except OSError:
continue
return cache_size / (1024**2)
def _clear_cache(self):
def _callback(response):
@@ -129,36 +142,90 @@ class ModelsLayout(Widget):
gui_app.push_widget(dialog)
def _handle_bundle_download_progress(self):
self.download_item.set_visible(False)
self.cancel_download_item.set_visible(False)
self._downloading = False
if not self.model_manager or (not self.model_manager.selectedBundle and not self.model_manager.activeBundle):
return
bundle = self.model_manager.selectedBundle if self._is_downloading() or (
self.model_manager.selectedBundle and self.model_manager.selectedBundle.status == custom.ModelManagerSP.DownloadStatus.failed
) else self.model_manager.activeBundle
if not bundle:
return
self.cancel_download_item.set_visible(bool(self.model_manager.selectedBundle) and ui_state.params.get("ModelManager_DownloadIndex") is not None)
self._verifying = False
self.download_item.set_visible(True)
if (current_time := time.monotonic()) - self.last_cache_calc_time > 0.5:
self.last_cache_calc_time = current_time
self.clear_cache_item.action_item.set_value(f"{self.calculate_cache_size():.2f} MB")
bundle = self.model_manager.selectedBundle if self.model_manager else None
progresses = [model.artifact.downloadProgress for model in bundle.models if model.artifact.fileName] if bundle else []
if not progresses or bundle.status not in (custom.ModelManagerSP.DownloadStatus.downloading,
custom.ModelManagerSP.DownloadStatus.failed):
self.download_item.action_item.update(name="", segments=self._slot_segments())
return
self.cancel_download_item.set_visible(ui_state.params.get("ModelManager_DownloadRef") is not None)
if bundle.status == custom.ModelManagerSP.DownloadStatus.downloading:
device._reset_interactive_timeout()
# every bundle is a single chunked artifact now
progresses = [model.artifact.downloadProgress for model in bundle.models if model.artifact.fileName]
if not progresses:
return
self.download_item.set_visible(True)
self.download_item.action_item.update(**self._download_row_state(progresses, bundle.internalName))
state = self._download_row_state(progresses, bundle.internalName)
if queued := queued_name(bundle.ref):
state["name"] += f" | {queued} {tr('queued')}"
self.download_item.action_item.update(**state)
self._downloading = self.download_item.action_item.downloading
ds = custom.ModelManagerSP.DownloadStatus
self._verifying = any(getattr(p.status, 'raw', p.status) == ds.verifying for p in progresses)
def _slot_segments(self):
"""small and big slots side by side; green marks the slot whose pick is actually
driving (runner-matched, so a failed Default big greens neither slot), an empty
slot shows its default."""
big_state = big_model_state()
carry_source, carry_internal, _ = carrying_model()
segments = []
for source, label in (("qcom", tr("small")), ("chestnut", tr("big"))):
if segments:
segments.append(("|", rl.GRAY, None, None))
bundle = get_selected_bundle(ui_state.params, source)
name = bundle.internalName if bundle else default_model_name(source)
color = ON_COLOR if (source == carry_source and name == carry_internal) else rl.LIGHTGRAY
name = "" + name
if source == "chestnut":
if big_state == 'failed':
color = rl.RED
elif big_state == 'loading':
color = rl.GOLD
segments.append((label, rl.GRAY, None, None))
segments.append((name, color, None, None))
return segments
@staticmethod
def _set_item_note(item, text):
# a description renders only while shown; hide before clearing or the
# empty description keeps its visible state
if text:
item.set_description(text)
item.show_description(True)
else:
item.show_description(False)
item.set_description("")
def _status_note(self) -> str:
"""The failover story for the Model Status row. One-way big -> small, and the
fallback is runner-matched: a Default big can only fall back to the Default
small (stock modeld), a custom big has no automatic fallback yet."""
if not ui_state.chestnut_present:
return ""
big_bundle = get_selected_bundle(ui_state.params, "chestnut")
big_name = big_bundle.internalName if big_bundle else default_model_name("chestnut")
big_is_default = big_bundle is None
fallback_name = default_model_name("qcom")
state = big_model_state()
if state == 'failed':
if big_is_default:
return tr("Big model unavailable, {} is driving until the next drive.").format(fallback_name)
return tr("Big model unavailable until the next drive.")
if state == 'loading':
if big_is_default:
return tr("{} drives until the big model is ready.").format(fallback_name)
return tr("Getting the big model ready.")
if big_is_default:
return tr("{} will drive. If it fails during a drive, {} takes over until the next drive.").format(big_name, fallback_name)
return tr("{} will drive when the chestnut is ready.").format(big_name)
@staticmethod
def _download_row_state(progresses, name: str) -> dict:
@@ -171,6 +238,8 @@ class ModelsLayout(Widget):
if ds.failed in statuses:
# close.png is authored black and a tint cannot lift it, hence close2
return {"name": name, "status_text": tr("download failed"), "text_color": rl.RED, "icon": "icons/close2.png"}
if ds.verifying in statuses:
return {"name": name, "downloading": True, "progress": progress, "status_text": tr("verifying")}
if ds.downloading in statuses:
return {"name": name, "downloading": True, "progress": progress}
if statuses <= {ds.downloaded, ds.cached}:
@@ -178,64 +247,73 @@ class ModelsLayout(Widget):
# circled_slash is authored grey; tinting it again only darkens it
return {"name": name, "text_color": rl.GRAY, "icon": "icons/circled_slash.png", "icon_color": rl.WHITE}
@staticmethod
def _show_reset_params_dialog():
def _callback(response):
if response == DialogResult.CONFIRM:
ui_state.params.remove("CalibrationParams")
ui_state.params.remove("LiveTorqueParameters")
msg = tr("Model download has started in the background. We suggest resetting calibration. Would you like to do that now?")
dialog = ConfirmDialog(msg, tr("Reset Calibration"), callback=_callback)
gui_app.push_widget(dialog)
def _on_model_selected(self, result):
if result != DialogResult.CONFIRM:
self.model_dialog = None
return
selected_ref = self.model_dialog.selection_ref
if selected_ref == "Default":
ui_state.params.remove("ModelManager_ActiveBundle")
self._show_reset_params_dialog()
elif selected_bundle := next((bundle for bundle in self.model_manager.availableBundles if bundle.ref == selected_ref), None):
ui_state.params.put("ModelManager_DownloadIndex", selected_bundle.index)
if self.model_manager.activeBundle and selected_bundle.generation != self.model_manager.activeBundle.generation:
self._show_reset_params_dialog()
self.model_dialog = None
if selected_ref == "Default":
if self._selection_source in ACTIVE_BUNDLE_KEYS:
ui_state.params.remove(ACTIVE_BUNDLE_KEYS[self._selection_source])
return
if selected_bundle := self._resolve_selected_bundle(selected_ref):
ui_state.params.put("ModelManager_DownloadRef", selected_bundle.ref)
def _resolve_selected_bundle(self, ref):
source_bundles = {source: bundles_for_source(source) for source in ("qcom", "chestnut")}
resolved = resolve_bundle_by_ref(ref, source_bundles)
return resolved[0] if resolved else None
@staticmethod
def _bundle_to_node(bundle):
return TreeNode(bundle.ref, {'display_name': bundle.displayName, 'short_name': bundle.internalName})
def _get_folders(self, favorites):
bundles = self.model_manager.availableBundles
def _get_folders(self, favorites, bundles):
folders = {}
for bundle in bundles:
folders.setdefault(next((ov_ride.value for ov_ride in bundle.overrides if ov_ride.key == "folder"), ""), []).append(bundle)
folders_list = [TreeFolder("", [TreeNode("Default", {'display_name': f"{DEFAULT_MODEL} (Default)", 'short_name': "Default"})])]
folders_list = []
for folder, folder_bundles in sorted(folders.items(), key=lambda x: max((bundle.index for bundle in x[1]), default=-1), reverse=True):
folder_bundles.sort(key=lambda bundle: bundle.index, reverse=True)
name = folder + (f" - (Updated: {m.group(1)})" if folder_bundles and (m := re.search(r'\(([^)]*)\)[^(]*$', folder_bundles[0].displayName)) else "")
folders_list.append(TreeFolder(name, [self._bundle_to_node(bundle) for bundle in folder_bundles]))
if favorites and (fav_bundles := [bundle for bundle in bundles if bundle.ref in favorites]):
folders_list.insert(1, TreeFolder("Favorites", [self._bundle_to_node(bundle) for bundle in fav_bundles]))
folders_list.insert(0, TreeFolder("Favorites", [self._bundle_to_node(bundle) for bundle in fav_bundles]))
return folders_list
def _handle_current_model_clicked(self):
def _open_source_dialog(self, source):
self._selection_source = source
favs = ui_state.params.get("ModelManager_Favs")
favorites = set(favs.split(';')) if favs else set()
folders_list = self._get_folders(favorites)
active_ref = self.model_manager.activeBundle.ref if self.model_manager.activeBundle else "Default"
self.model_dialog = TreeOptionDialog(tr("Select a Model"), folders_list, active_ref, "ModelManager_Favs",
get_folders_fn=self._get_folders, on_exit=self._on_model_selected)
folders_list = self._source_folders(favorites, source)
if not folders_list:
gui_app.push_widget(alert_dialog(tr("No models are available for this hardware yet. Connect to the internet and refresh the model list.")))
return
self.model_dialog = TreeOptionDialog(tr("Select a Model"), folders_list, self._slot_active_ref(source), "ModelManager_Favs",
get_folders_fn=lambda favs: self._source_folders(favs, source), on_exit=self._on_model_selected)
gui_app.push_widget(self.model_dialog)
def _source_folders(self, favorites, source):
bundles = bundles_for_source(source)
if not bundles:
return []
folders_list = [TreeFolder("", [TreeNode("Default", {'display_name': default_model_name(source)})])]
folders_list.extend(self._get_folders(favorites, bundles))
return folders_list
@staticmethod
def _slot_active_ref(source: str) -> str:
bundle = get_selected_bundle(ui_state.params, source)
return bundle.ref if bundle else "Default"
def _update_state(self):
advanced_controls: bool = ui_state.params.get_bool("ShowAdvancedControls")
turn_desire: bool = ui_state.params.get_bool("LaneTurnDesire")
live_delay: bool = ui_state.params.get_bool("LagdToggle")
camera_offset: bool = ui_state.params.get("ModelManager_ActiveBundle") is not None
camera_offset: bool = ui_state.active_bundle is not None
self.lane_turn_desire_toggle.action_item.set_state(turn_desire)
self.lane_turn_value_control.set_visible(turn_desire and advanced_controls)
@@ -249,18 +327,27 @@ class ModelsLayout(Widget):
self._update_lagd_description(live_delay)
self.model_manager = ui_state.sm["modelManagerSP"]
self._handle_bundle_download_progress()
active_name = self.model_manager.activeBundle.displayName if self.model_manager and self.model_manager.activeBundle.ref else f"{DEFAULT_MODEL} (Default)"
self.current_model_item.action_item.set_value(active_name)
if not ui_state.is_offroad():
self.current_model_item.action_item.set_enabled(False)
self.current_model_item.set_description(tr("Only available when vehicle is off, or always offroad mode is on"))
else:
self.current_model_item.action_item.set_enabled(True)
self.current_model_item.set_description("")
carry_source, _, carry_display = carrying_model()
for item, item_source in ((self.small_model_item, "qcom"), (self.big_model_item, "chestnut")):
bundle = get_selected_bundle(ui_state.params, item_source)
name = bundle.displayName if bundle else default_model_name(item_source)
color = ON_COLOR if (item_source == carry_source and name == carry_display) else style.ITEM_TEXT_VALUE_COLOR
item.action_item.set_value(name, color)
note = self._status_note()
if note != self._last_note:
self._last_note = note
self._set_item_note(self.download_item, note)
offroad = ui_state.is_offroad()
self.small_model_item.action_item.set_enabled(offroad)
self.big_model_item.action_item.set_enabled(offroad)
self.small_model_item.set_description("" if offroad else tr("Only available when vehicle is off, or always offroad mode is on"))
def _render(self, rect):
self._scroller.render(rect)
def show_event(self):
self._scroller.show_event()
self._last_note = None # re-expand the failover note every time the page opens
@@ -8,7 +8,6 @@ import datetime
import os
import platform
import requests
import shutil
import threading
from pathlib import Path
from time import monotonic
@@ -75,22 +74,12 @@ class OSMLayout(Widget):
def _update_map_size(self):
threading.Thread(target=self.calculate_size, daemon=True).start()
def _do_delete_maps(self):
if MAP_PATH.exists():
shutil.rmtree(MAP_PATH)
for param in ("OsmDownloadedDate", "OsmLocal", "OsmLocationName", "OsmLocationTitle", "OsmStateName", "OsmStateTitle"):
ui_state.params.remove(param)
def _on_confirm_delete_maps(self):
ui_state.params.put_bool("Mapd_ClearCache", True)
self._delete_maps_btn.action_item.set_enabled(True)
self._delete_maps_btn.action_item.set_text(tr("DELETE"))
self._update_map_size()
def _on_confirm_delete_maps(self):
self._delete_maps_btn.action_item.set_enabled(False)
self._delete_maps_btn.action_item.set_text("DELETING...")
threading.Thread(target=self._do_delete_maps).start()
def _delete_maps(self):
self._show_confirm(tr("This will delete ALL downloaded maps\n\nAre you sure you want to delete all maps?"),
tr("Yes, delete all maps"), self._on_confirm_delete_maps)
@@ -4,11 +4,14 @@ Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
import math
import pyray as rl
import time
from dataclasses import dataclass
from openpilot.selfdrive.ui.ui_state import ui_state
from openpilot.selfdrive.ui.ui_state import ui_state, ChestnutState
from openpilot.sunnypilot.sunnylink.api import UNREGISTERED_SUNNYLINK_DONGLE_ID
from openpilot.system.ui.lib.application import gui_app
from openpilot.system.ui.lib.multilang import tr_noop
@@ -18,6 +21,9 @@ METRIC_MARGIN = 30
METRIC_START_Y = 300
HOME_BTN = rl.Rectangle(60, 860, 180, 180)
CHESTNUT_ICON_WIDTH = 180
CHESTNUT_ICON_HEIGHT = 133
# Color scheme
class Colors:
@@ -53,6 +59,9 @@ class MetricData:
class SidebarSP:
def __init__(self):
self._sunnylink_status = MetricData(tr_noop("SUNNYLINK"), tr_noop("OFFLINE"), Colors.WARNING)
self._chestnut_green_img = gui_app.texture("icons_mici/chestnut_green.png", CHESTNUT_ICON_WIDTH, CHESTNUT_ICON_HEIGHT)
self._chestnut_default_img = gui_app.texture("icons_mici/chestnut.png", CHESTNUT_ICON_WIDTH, CHESTNUT_ICON_HEIGHT)
self._chestnut_orange_img = gui_app.texture("icons_mici/chestnut_orange.png", CHESTNUT_ICON_WIDTH, CHESTNUT_ICON_HEIGHT)
def _update_sunnylink_status(self):
if not ui_state.params.get_bool("SunnylinkEnabled"):
@@ -78,6 +87,24 @@ class SidebarSP:
self._sunnylink_status.update(tr_noop("SUNNYLINK"), status, color)
def _get_home_icon(self, default_img: rl.Texture) -> tuple[rl.Texture, rl.Vector2, float]:
default_pos = rl.Vector2(HOME_BTN.x, HOME_BTN.y)
state = ui_state.chestnut_state
if state == ChestnutState.DISCONNECTED:
return default_img, default_pos, 1.0
if state == ChestnutState.LOADING:
icon = self._chestnut_default_img
opacity = 0.35 + 0.65 * (0.5 - 0.5 * math.cos(rl.get_time() * 6.0))
elif state in (ChestnutState.UNCOMPILED, ChestnutState.FAILED):
icon, opacity = self._chestnut_orange_img, 1.0
else:
icon, opacity = self._chestnut_green_img, 1.0
x = HOME_BTN.x + (HOME_BTN.width - icon.width) / 2
y = HOME_BTN.y + (HOME_BTN.height - icon.height) / 2
return icon, rl.Vector2(x, y), opacity
def _draw_metrics_w_sunnylink(self, rect: rl.Rectangle, _temp, _panda, _connect):
metrics = [_temp, _panda, _connect, self._sunnylink_status]
start_y = int(rect.y) + METRIC_START_Y
@@ -4,8 +4,14 @@ Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
import math
import pyray as rl
from openpilot.selfdrive.ui.mici.layouts.home import MiciHomeLayout
from openpilot.selfdrive.ui.ui_state import ui_state, ChestnutState
from openpilot.system.ui.lib.application import FontWeight
from openpilot.system.ui.widgets.icon_widget import IconWidget
from openpilot.system.ui.widgets.label import UnifiedLabel
@@ -13,3 +19,16 @@ class MiciHomeLayoutSP(MiciHomeLayout):
def __init__(self):
super().__init__()
self._openpilot_label = UnifiedLabel("sunnypilot", font_size=88, font_weight=FontWeight.AUDIOWIDE, max_width=480, wrap_text=False)
self._chestnut_loading_icon = IconWidget("icons_mici/chestnut.png", (68, 40))
self._chestnut_loading_icon.set_visible(False)
failed_idx = self._status_bar_layout.widgets.index(self._chestnut_failed_icon)
self._status_bar_layout.widgets.insert(failed_idx + 1, self._chestnut_loading_icon)
def _set_chestnut_visibility(self):
# stock has no loading tier: it shows green from the moment a big model is available. keep the
# pulse so the status bar and the onroad HUD agree on what loading looks like.
loading = ui_state.chestnut_state == ChestnutState.LOADING
self._chestnut_loading_icon._opacity = 0.35 + 0.65 * (0.5 - 0.5 * math.cos(rl.get_time() * 6.0))
self._chestnut_loading_icon.set_visible(loading)
self._chestnut_icon.set_visible(not loading and ui_state.chestnut_state in (ChestnutState.READY, ChestnutState.ACTIVE))
self._chestnut_failed_icon.set_visible(ui_state.chestnut_state in (ChestnutState.UNCOMPILED, ChestnutState.FAILED))
@@ -7,16 +7,37 @@ See the LICENSE.md file in the root directory for more details.
import pyray as rl
from openpilot.cereal import custom
from openpilot.sunnypilot.models.default_model import DEFAULT_MODEL
from openpilot.selfdrive.ui.mici.widgets.dialog import BigDialog
from openpilot.sunnypilot.models.helpers import ACTIVE_BUNDLE_KEYS, get_selected_bundle
from openpilot.selfdrive.ui.mici.widgets.button import BigButton
from openpilot.selfdrive.ui.sunnypilot.layouts.settings.models import ModelsLayout
from openpilot.selfdrive.ui.ui_state import ui_state, device
from openpilot.selfdrive.ui.sunnypilot.model_info import (active_source, big_model_state, bundles_for_source, carrying_model,
default_model_name, model_info, queued_name)
from openpilot.system.ui.lib.application import FontWeight, gui_app
from openpilot.system.ui.lib.multilang import tr
from openpilot.system.ui.widgets import Widget
from openpilot.system.ui.widgets.label import UnifiedLabel
from openpilot.system.ui.widgets.scroller import NavScroller
def _model_info() -> tuple[str, str, str]:
"""(active model, info header, info text) for the panel. Runner-matched: the
active line names what actually drives, and a notable big-model state takes
the info pair."""
source, active_name, other_name = model_info()
state = big_model_state()
_, _, carry_display = carrying_model()
if carry_display is None:
big = get_selected_bundle(ui_state.params, "chestnut")
carry_display = big.displayName if big else default_model_name("chestnut")
active_text = (carry_display or active_name).lower()
if state == 'failed':
return active_text, tr("big model"), tr("unavailable")
if state == 'loading':
return active_text, tr("big model"), tr("getting ready")
header = tr("small model") if source == "chestnut" else tr("big model")
return active_text, header, other_name.lower()
class CurrentModelInfo(Widget):
def __init__(self):
super().__init__()
@@ -26,12 +47,12 @@ class CurrentModelInfo(Widget):
header_color = rl.Color(255, 255, 255, int(255 * 0.9))
subheader_color = rl.Color(255, 255, 255, int(255 * 0.9 * 0.65))
max_width = int(self._rect.width - 20)
active_text, info_header, info_text = _model_info()
self.current_model_header = UnifiedLabel(tr("active model"), 48, max_width=max_width, text_color=header_color, font_weight=FontWeight.DISPLAY)
default_text = f"{DEFAULT_MODEL} (Default)".lower()
self.current_model_text = UnifiedLabel(default_text, 32, max_width=max_width, text_color=subheader_color, font_weight=FontWeight.ROMAN, scroll=True)
self.current_model_text = UnifiedLabel(active_text, 32, max_width=max_width, text_color=subheader_color, font_weight=FontWeight.ROMAN, scroll=True)
self.info_header = UnifiedLabel("cache size", 48, max_width=max_width, text_color=header_color, font_weight=FontWeight.DISPLAY)
self.info_text = UnifiedLabel("0 mb", 32, max_width=max_width, text_color=subheader_color, font_weight=FontWeight.ROMAN)
self.info_header = UnifiedLabel(info_header, 48, max_width=max_width, text_color=header_color, font_weight=FontWeight.DISPLAY)
self.info_text = UnifiedLabel(info_text, 32, max_width=max_width, text_color=subheader_color, font_weight=FontWeight.ROMAN, scroll=True)
def _render(self, _):
self.current_model_header.set_position(self._rect.x + 20, self._rect.y - 10)
@@ -55,12 +76,13 @@ class ModelsLayoutMici(NavScroller):
self._download_progress = "."
self._download_frame = 0
self._was_downloading = False
self._selection_source: str | None = None
self.select_model_btn = BigButton(tr("select model"))
self.select_model_btn.set_click_callback(self._show_folders)
self.cancel_download_btn = BigButton(tr("cancel download"))
self.cancel_download_btn.set_click_callback(lambda: ui_state.params.remove("ModelManager_DownloadIndex"))
self.cancel_download_btn.set_click_callback(lambda: ui_state.params.remove("ModelManager_DownloadRef"))
self.main_items = [self.current_model_info, self.select_model_btn, self.cancel_download_btn]
self._scroller.add_widgets(self.main_items)
@@ -69,8 +91,7 @@ class ModelsLayoutMici(NavScroller):
def model_manager(self):
return ui_state.sm["modelManagerSP"]
def _get_grouped_bundles(self, favorites = None):
bundles = self.model_manager.availableBundles
def _get_grouped_bundles(self, bundles, favorites = None):
folders = {}
for bundle in bundles:
folder = next((override.value for override in bundle.overrides if override.key == "folder"), "")
@@ -90,47 +111,70 @@ class ModelsLayoutMici(NavScroller):
def _show_folders(self):
self.focused_widget = self.select_model_btn
hardware_btns = []
active = active_source()
for source, label in (("qcom", tr("small models")), ("chestnut", tr("big models"))):
bundle = get_selected_bundle(ui_state.params, source)
value = (bundle.internalName if bundle else default_model_name(source)).lower()
if source == active:
value += f" ({tr('active')})"
btn = BigButton(label.lower(), value=value)
btn.set_click_callback(lambda s=source: self._select_hardware(s))
hardware_btns.append(btn)
self._push_selection_view(hardware_btns)
def _select_hardware(self, source):
self._selection_source = source
favs = ui_state.params.get("ModelManager_Favs")
favorites = set(favs.split(';')) if favs else set()
folders = self._get_grouped_bundles(favorites)
bundles = bundles_for_source(source)
if not bundles:
gui_app.push_widget(BigDialog(title=tr("No models available"),
description=tr("No models are available for this hardware yet. Connect to the internet and refresh the model list.")))
return
folders = self._get_grouped_bundles(bundles, favorites)
folder_buttons = []
default_btn = BigButton(f"{DEFAULT_MODEL} (Default)".lower())
default_btn.set_click_callback(self._select_default)
default_btn = BigButton(default_model_name(source).lower())
default_btn.set_click_callback(lambda s=source: self._select_default(s))
folder_buttons.append(default_btn)
for folder in sorted(folders.keys(), key=lambda f: max((bundle.index for bundle in folders[f]), default=-1), reverse=True):
if folder.lower() in ["release models", "master models", "favorites"]:
btn = BigButton(folder.lower())
btn.set_click_callback(lambda f=folder: self._select_folder(f))
if folder.lower() == "favorites":
folder_buttons.insert(0, btn)
else:
folder_buttons.append(btn)
btn = BigButton(folder.lower())
btn.set_click_callback(lambda f=folder: self._select_folder(f))
if folder.lower() == "favorites":
folder_buttons.insert(0, btn)
else:
folder_buttons.append(btn)
self._push_selection_view(folder_buttons)
def _pop_to_main(self):
gui_app.pop_widgets_to(self)
self._scroller.scroll_panel.set_offset(0.0)
def _select_model(self, bundle):
ui_state.params.put("ModelManager_DownloadIndex", bundle.index)
ui_state.params.put("ModelManager_DownloadRef", bundle.ref)
self._pop_to_main()
def _select_default(self):
ui_state.params.remove("ModelManager_ActiveBundle")
def _select_default(self, source):
ui_state.params.remove(ACTIVE_BUNDLE_KEYS[source])
self._pop_to_main()
def _select_folder(self, folder_name):
source = self._selection_source
if source is None: # folders are only reachable after picking a hardware
return
favs = ui_state.params.get("ModelManager_Favs")
favorites = set(favs.split(';')) if favs else set()
folders = self._get_grouped_bundles(favorites)
folders = self._get_grouped_bundles(bundles_for_source(source), favorites)
bundles = sorted(folders.get(folder_name, []), key=lambda b: b.index, reverse=True)
btns = []
for bundle in bundles:
txt = bundle.displayName.lower()
btn = BigButton(txt)
btn = BigButton(bundle.displayName.lower())
btn.set_click_callback(lambda b=bundle: self._select_model(b))
btns.append(btn)
self._push_selection_view(btns)
@@ -162,10 +206,10 @@ class ModelsLayoutMici(NavScroller):
self._was_downloading = is_downloading
self.current_model_info.current_model_header.set_text(tr("active model"))
model_text = manager.activeBundle.displayName.lower() if manager.activeBundle.ref else f"{DEFAULT_MODEL} (Default)".lower()
self.current_model_info.current_model_text.set_text(model_text)
self.current_model_info.info_header.set_text(tr("cache size"))
self.current_model_info.info_text.set_text(f"{ModelsLayout.calculate_cache_size():.2f} MB")
active_text, info_header, info_text = _model_info()
self.current_model_info.current_model_text.set_text(active_text)
self.current_model_info.info_header.set_text(info_header)
self.current_model_info.info_text.set_text(info_text)
if manager.selectedBundle and manager.selectedBundle.status == custom.ModelManagerSP.DownloadStatus.failed:
self.current_model_info.info_header.set_text(tr("error") + self._download_progress)
@@ -176,19 +220,29 @@ class ModelsLayoutMici(NavScroller):
device.set_override_interactive_timeout(5)
progress = 0.0
count = 0
verifying = False
for model in manager.selectedBundle.models:
count += 1
p = model.artifact.downloadProgress
if p.status == custom.ModelManagerSP.DownloadStatus.downloading:
if p.status in (custom.ModelManagerSP.DownloadStatus.downloading,
custom.ModelManagerSP.DownloadStatus.verifying):
progress += p.progress
verifying = verifying or p.status == custom.ModelManagerSP.DownloadStatus.verifying
elif p.status in (custom.ModelManagerSP.DownloadStatus.downloaded,
custom.ModelManagerSP.DownloadStatus.cached):
progress += 100.0
self.current_model_info.current_model_header.set_text(tr("downloading"))
self.current_model_info.current_model_header.set_text(tr("verifying") if verifying else tr("downloading"))
self.cancel_download_btn.set_text(tr("cancel verification") if verifying else tr("cancel download"))
self.current_model_info.current_model_header._shimmer = True
self.current_model_info.current_model_text.set_text(f"{manager.selectedBundle.internalName.lower()}")
name_text = manager.selectedBundle.internalName.lower()
if queued := queued_name(manager.selectedBundle.ref):
name_text += f" | {queued.lower()} {tr('queued')}"
self.current_model_info.current_model_text.set_text(name_text)
self.current_model_info.info_header.set_text(tr("progress") + self._download_progress)
self.current_model_info.info_header._shimmer = True
self.current_model_info.info_text.set_text(f"{progress/count:.2f}%")
elif manager.selectedBundle and manager.selectedBundle.status == custom.ModelManagerSP.DownloadStatus.downloaded:
self.current_model_info.info_header.set_text(tr("downloaded"))
self.current_model_info.info_text.set_text(tr("downloaded"))
@@ -12,13 +12,23 @@ from openpilot.selfdrive.ui.mici.widgets.dialog import BigConfirmationDialog, Bi
from openpilot.selfdrive.ui.sunnypilot.mici.layouts.sunnylink import SunnylinkLayoutMici
from openpilot.selfdrive.ui.sunnypilot.mici.layouts.models import ModelsLayoutMici
from openpilot.selfdrive.ui.ui_state import ui_state
from openpilot.system.ui.lib.application import gui_app
from openpilot.system.ui.lib.application import gui_app, FontWeight
from openpilot.system.ui.lib.multilang import tr
ICON_SIZE = 70
BIG_ICON_SIZE = 110
class SunnylinkBigButton(SettingsBigButton):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._label.set_font_weight(FontWeight.AUDIOWIDE)
def _get_label_font_size(self):
# Audiowide runs wider than Inter: "sunnylink" wraps to two lines at 64
return 56
class SettingsLayoutSP(OP.SettingsLayout):
def __init__(self):
OP.SettingsLayout.__init__(self)
@@ -33,7 +43,7 @@ class SettingsLayoutSP(OP.SettingsLayout):
self.icon_offroad_slider = gui_app.texture("icons_mici/settings/device/lkas.png", BIG_ICON_SIZE, BIG_ICON_SIZE)
sunnylink_panel = SunnylinkLayoutMici()
sunnylink_btn = SettingsBigButton(tr("sunnylink"), "", gui_app.texture("icons_mici/settings/developer/ssh.png", 55, 55))
sunnylink_btn = SunnylinkBigButton(tr("sunnylink"), "", gui_app.texture("../../sunnypilot/selfdrive/assets/icons_mici/sunnylink.png", 76, 44))
sunnylink_btn.set_click_callback(lambda: gui_app.push_widget(sunnylink_panel))
models_panel = ModelsLayoutMici()
@@ -56,8 +66,8 @@ class SettingsLayoutSP(OP.SettingsLayout):
items = self._scroller._items.copy()
items.insert(1, sunnylink_btn)
items.insert(2, models_btn)
items.insert(1, models_btn)
items.insert(5, sunnylink_btn)
# front slots (only one ever visible at a time): exit-always-offroad, then enable-onroad
items.insert(0, self._enable_offroad_btn_onroad)
@@ -0,0 +1,85 @@
"""
Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
from openpilot.selfdrive.ui.ui_state import ui_state, ChestnutState
from openpilot.sunnypilot.models.fetcher import get_cached_bundles
from openpilot.sunnypilot.models.helpers import get_active_source, get_selected_bundle, resolve_bundle_by_ref
from openpilot.sunnypilot.models.model_name import DEFAULT_BIG_MODEL, DEFAULT_MODEL
def active_source() -> str:
return get_active_source(chestnut=ui_state.chestnut_present,
chestnut_active=ui_state.chestnut_active, chestnut_loading=ui_state.chestnut_loading,
offroad=ui_state.is_offroad())
def bundles_for_source(source: str):
if source == active_source():
return ui_state.sm["modelManagerSP"].availableBundles
return get_cached_bundles(ui_state.params, source)
def default_model(source: str) -> str:
return DEFAULT_BIG_MODEL if source == 'chestnut' else DEFAULT_MODEL
def default_model_name(source: str) -> str:
return f"{default_model(source)} (Default)"
def big_model_state() -> str | None:
"""'failed' | 'loading' | None, from the same state the icons render."""
return {ChestnutState.UNCOMPILED: 'failed',
ChestnutState.FAILED: 'failed',
ChestnutState.LOADING: 'loading'}.get(ui_state.chestnut_state)
def carrying_model() -> tuple[str | None, str | None, str | None]:
"""(source, internal name, display name) of what actually drives. Runner-matched:
when a Default big cannot carry, stock modeld runs the Default small, never the
small slot's pick; a custom big has no automatic fallback yet -> (None, None, None)."""
source = active_source()
if source == "chestnut":
bundle = get_selected_bundle(ui_state.params, "chestnut")
if bundle:
return "chestnut", bundle.internalName, bundle.displayName
name = default_model_name("chestnut")
return "chestnut", name, name
if ui_state.chestnut_present:
if get_selected_bundle(ui_state.params, "chestnut") is None:
name = default_model_name("qcom")
return "qcom", name, name
return None, None, None
bundle = get_selected_bundle(ui_state.params, "qcom")
if bundle:
return "qcom", bundle.internalName, bundle.displayName
name = default_model_name("qcom")
return "qcom", name, name
def queued_name(current_ref) -> str | None:
ref = ui_state.params.get("ModelManager_DownloadRef")
if ref and ref != current_ref:
source_bundles = {source: bundles_for_source(source) for source in ("qcom", "chestnut")}
if resolved := resolve_bundle_by_ref(ref, source_bundles):
return resolved[0].internalName
return None
def model_info() -> tuple[str, str, str]:
"""returns (active source, active model name, other model name)
Names come from the params slots, never modelManagerSP.activeBundle — the
manager republishes a tick after a chestnut change, so the stale bundle
would flash the wrong model."""
source = active_source()
other = "qcom" if source == "chestnut" else "chestnut"
active_bundle = get_selected_bundle(ui_state.params, source)
other_bundle = get_selected_bundle(ui_state.params, other)
active_name = active_bundle.displayName if active_bundle else default_model_name(source)
other_name = other_bundle.displayName if other_bundle else default_model_name(other)
return source, active_name, other_name
@@ -4,11 +4,29 @@ Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
from openpilot.common.filter_simple import FirstOrderFilter
from openpilot.selfdrive.ui.ui_state import ui_state, UIStatus
from openpilot.selfdrive.ui.sunnypilot.onroad.chevron_metrics import ChevronMetrics
from openpilot.selfdrive.ui.sunnypilot.onroad.rainbow_path import RainbowPath
from openpilot.selfdrive.ui.sunnypilot.ui_state import MADSState
from openpilot.system.ui.lib.application import gui_app
class ModelRendererSP:
def __init__(self):
self.rainbow_path = RainbowPath()
self.chevron_metrics = ChevronMetrics()
self._width_filter = FirstOrderFilter(0.9, 0.1, 1 / gui_app.target_fps)
@property
def _lateral_active(self) -> bool:
sm = ui_state.sm
if sm.valid["selfdriveStateSP"]:
mads = sm["selfdriveStateSP"].mads
if mads.available:
return mads.enabled and mads.state != MADSState.paused
return ui_state.status in (UIStatus.ENGAGED, UIStatus.LAT_ONLY)
def _get_path_half_width(self) -> float:
target = 0.9 if self._lateral_active else 0.40
return self._width_filter.update(target)
@@ -10,6 +10,7 @@ from openpilot.cereal import messaging, log, custom
from opendbc.car.structs import car
from openpilot.common.params import Params
from openpilot.selfdrive.ui.sunnypilot.layouts.settings.display import OnroadBrightness
from openpilot.sunnypilot.models.helpers import ACTIVE_BUNDLE_KEYS, get_active_source
from openpilot.sunnypilot.sunnylink.sunnylink_state import SunnylinkState
from openpilot.system.ui.lib.application import gui_app
from openpilot.system.ui.sunnypilot.widgets.screen_saver import ScreenSaverSP
@@ -43,6 +44,7 @@ class UIStateSP:
self.screensaver_enabled: bool = False
self.active_bundle = None
self.model_runner_tinygrad: bool = False
self.blindspot: bool = False
self.chevron_metrics = None
self.custom_interactive_timeout: int = 0
@@ -150,7 +152,13 @@ class UIStateSP:
self.has_icbm = self.CP_SP.intelligentCruiseButtonManagementAvailable and self.params.get_bool("IntelligentCruiseButtonManagement")
self._enforce_constraints()
self.active_bundle = self.params.get("ModelManager_ActiveBundle")
source = get_active_source(chestnut=self.chestnut_present, chestnut_active=self.chestnut_active,
chestnut_loading=self.chestnut_loading, offroad=self.is_offroad())
self.active_bundle = self.params.get(ACTIVE_BUNDLE_KEYS[source])
self.model_runner_tinygrad = self.active_bundle is not None and self.active_bundle.get("runner") == "tinygrad"
# stock only counts the default big model's compiled pkl. a downloaded big bundle runs on the
# chestnut just the same, so ChestnutState has to see it as available too.
self.chestnut_compiled = self.chestnut_compiled or self.model_runner_tinygrad
self.blindspot = self.params.get_bool("BlindSpot")
self.chevron_metrics = self.params.get("ChevronInfo")
self.custom_interactive_timeout = self.params.get("InteractivityTimeout", return_default=True)
+43 -11
View File
@@ -12,7 +12,7 @@ from openpilot.common.swaglog import cloudlog
from openpilot.selfdrive.ui.lib.prime_state import PrimeState
from openpilot.system.ui.lib.application import gui_app
from openpilot.common.hardware import HARDWARE, PC
from openpilot.selfdrive.modeld.helpers import usbgpu_compiled
from openpilot.selfdrive.modeld.helpers import chestnut_compiled
from openpilot.selfdrive.ui.sunnypilot.ui_state import UIStateSP, DeviceSP
@@ -28,6 +28,15 @@ class UIStatus(Enum):
LONG_ONLY = "long_only"
class ChestnutState(Enum):
DISCONNECTED = "disconnected"
UNCOMPILED = "uncompiled"
READY = "ready"
LOADING = "loading"
ACTIVE = "active"
FAILED = "failed"
class UIState(UIStateSP):
_instance: 'UIState | None' = None
@@ -82,10 +91,11 @@ class UIState(UIStateSP):
self.always_on_dm: bool = self.params.get_bool("AlwaysOnDM")
self.experimental_mode: bool = self.params.get_bool("ExperimentalMode")
self.experimental_mode_confirmed: bool = self.params.get_bool("ExperimentalModeConfirmed")
self.usbgpu: bool = False
self.usbgpu_compiled: bool = usbgpu_compiled()
self.usbgpu_active: bool | None = self.params.get("UsbGpuActive")
self.usbgpu_loading: bool = self.params.get_bool("UsbGpuLoading")
self.chestnut_present: bool = False
self.chestnut_compiled: bool = chestnut_compiled()
self.chestnut_active: bool | None = None
self.chestnut_loading: bool = False
self.chestnut_state = ChestnutState.DISCONNECTED
self.started: bool = False
self.ignition: bool = False
self.recording_audio: bool = False
@@ -131,6 +141,7 @@ class UIState(UIStateSP):
self.sm.update(0)
self._update_state()
self._update_status()
self._update_chestnut_state()
device.update()
UIStateSP.update(self)
@@ -194,12 +205,35 @@ class UIState(UIStateSP):
self.status = UIStatus.DISENGAGED
self.started_frame = self.sm.frame
self.started_time = time.monotonic()
self.chestnut_present = self.sm["deviceState"].chestnutPresent
for callback in self._offroad_transition_callbacks:
callback()
self._started_prev = self.started
def _update_chestnut_state(self) -> None:
detected = self.sm["deviceState"].chestnutPresent
if not self.started:
self.chestnut_present = detected
self.chestnut_state = (ChestnutState.READY if detected and self.chestnut_compiled else
ChestnutState.UNCOMPILED if detected else ChestnutState.DISCONNECTED)
return
model_seen = self.sm.recv_frame["modelV2"] > self.started_frame
if not self.chestnut_present:
self.chestnut_state = ChestnutState.DISCONNECTED
elif not self.chestnut_compiled:
self.chestnut_state = ChestnutState.UNCOMPILED
elif self.chestnut_state == ChestnutState.FAILED or not detected or (model_seen and (not self.sm.alive["modelV2"] or not self.sm["modelV2"].big)):
self.chestnut_state = ChestnutState.FAILED
elif self.chestnut_loading or not model_seen:
self.chestnut_state = ChestnutState.LOADING
elif self.chestnut_active is False:
self.chestnut_state = ChestnutState.FAILED
else:
self.chestnut_state = ChestnutState.ACTIVE
def update_params(self) -> None:
# For slower operations
# Update longitudinal control state
@@ -216,12 +250,10 @@ class UIState(UIStateSP):
self.always_on_dm = self.params.get_bool("AlwaysOnDM")
self.experimental_mode = self.params.get_bool("ExperimentalMode")
self.experimental_mode_confirmed = self.params.get_bool("ExperimentalModeConfirmed")
# keep usbgpu UI active until offroad transition when gpu disappears
self.usbgpu = self.sm["deviceState"].chestnutPresent or (self.usbgpu and self.started)
if not self.usbgpu_compiled:
self.usbgpu_compiled = usbgpu_compiled()
self.usbgpu_active = self.params.get("UsbGpuActive")
self.usbgpu_loading = self.params.get_bool("UsbGpuLoading")
if not self.chestnut_compiled:
self.chestnut_compiled = chestnut_compiled()
self.chestnut_active = self.params.get("ChestnutActive")
self.chestnut_loading = self.params.get_bool("ChestnutLoading")
UIStateSP.update_params(self)
+17
View File
@@ -55,6 +55,19 @@ def cleanup_old_osm_data(files_to_remove: list[str]) -> None:
shutil.rmtree(file, ignore_errors=False)
def clear_downloaded_maps() -> None:
"""Deletes downloaded OSM map data and resets params."""
path = f"{Paths.mapd_root()}/offline"
if os.path.exists(path):
shutil.rmtree(path, ignore_errors=True)
for param in ("OsmDownloadedDate", "OsmLocal", "OsmLocationName", "OsmLocationTitle",
"OsmStateName", "OsmStateTitle"):
params.remove(param)
cloudlog.info("mapd: downloaded maps cleared")
def request_refresh_osm_location_data(nations: list[str], states: list[str] | None = None) -> None:
params.put("OsmDownloadedDate", str(datetime.now().timestamp()), block=True)
params.put_bool("OsmDbUpdatesCheck", False, block=True)
@@ -131,6 +144,10 @@ def main_thread():
show_alert = bool(get_files_for_cleanup() and params.get_bool("OsmLocal"))
set_offroad_alert("Offroad_OSMUpdateRequired", show_alert, "This alert will be cleared when new maps are downloaded.")
if params.get("Mapd_ClearCache"):
clear_downloaded_maps()
params.remove("Mapd_ClearCache")
update_osm_db()
live_map_sp.tick()
rk.keep_time()
@@ -7,6 +7,7 @@ See the LICENSE.md file in the root directory for more details.
"""
import argparse
import math
import os
import tempfile
import time
@@ -66,14 +67,15 @@ def get_policy_npy_shapes(input_shapes: dict, is_supercombo: bool = False) -> tu
if desire_key:
shapes['desire'] = (input_shapes[desire_key][2],)
if is_supercombo and 'features_buffer' in input_shapes:
fb = input_shapes['features_buffer']
shapes['prev_feat'] = (fb[0], fb[2])
for key, shape in input_shapes.items():
if key not in (desire_key, 'features_buffer') and 'img' not in key:
shapes[key] = tuple(shape)
if is_supercombo and 'features_buffer' in input_shapes:
fb = input_shapes['features_buffer']
feat_dim = math.prod(fb[2:])
shapes['prev_feat'] = (fb[0], feat_dim)
sizes = [int(np.prod(size)) for size in shapes.values()]
return shapes, sizes
@@ -117,8 +119,9 @@ def generate_queues_and_npy(input_shapes: dict, frame_skip: int, device: str = D
}
if features_buffer:
feat_dim = math.prod(features_buffer[2:])
feat_q_len = frame_skip * features_buffer[1] if is_supercombo else frame_skip * (features_buffer[1] - 1) + 1
queues['feat_q'] = Tensor(np.zeros((feat_q_len, features_buffer[0], features_buffer[2]),
queues['feat_q'] = Tensor(np.zeros((feat_q_len, features_buffer[0], feat_dim),
dtype=np.float32), device=device).contiguous().realize()
queues.update({key: Tensor(value, device='NPY').realize() for key, value in npy_arrays.items() if key in ('tfm', 'big_tfm')})
@@ -183,14 +186,14 @@ def make_run_policy(vision_runner, policy_runners: list, features_slice: slice,
warped_dev = warped.to(Device.DEFAULT)
Tensor.realize(packed_npy_inputs_dev, warped_dev)
img = shift_and_sample(img_q, warped_dev[0:1], sample_skip_fn).realize()
big_img = shift_and_sample(big_img_q, warped_dev[1:2], sample_skip_fn).realize()
img = shift_and_sample(img_q, warped_dev[0:1], sample_skip_fn)
big_img = shift_and_sample(big_img_q, warped_dev[1:2], sample_skip_fn)
unpacked_tensors = [tensor.reshape(shape) for tensor, shape in zip(packed_npy_inputs_dev.split(npy_sizes), npy_shapes.values(), strict=True)]
unpacked_dict = dict(zip(npy_shapes.keys(), unpacked_tensors, strict=True))
desire_dev = unpacked_dict['desire']
desire_buf = shift_and_sample(desire_q, desire_dev.reshape(1, 1, -1), sample_desire_fn).realize()
desire_buf = shift_and_sample(desire_q, desire_dev.reshape(1, 1, -1), sample_desire_fn)
inputs = {desire_key: desire_buf}
for key, tensor_val in unpacked_dict.items():
@@ -199,7 +202,7 @@ def make_run_policy(vision_runner, policy_runners: list, features_slice: slice,
if 'prev_feat' in unpacked_dict:
prev_feat_dev = unpacked_dict['prev_feat']
inputs['features_buffer'] = shift_and_sample(feat_q, prev_feat_dev.reshape(1, 1, -1), sample_skip_fn).realize()
inputs['features_buffer'] = shift_and_sample(feat_q, prev_feat_dev.reshape(1, 1, -1), sample_skip_fn).reshape(input_shapes['features_buffer'])
if vision_runner:
vision_out_cast = next(iter(vision_runner({road_key: img, wide_key: big_img}).values())).cast('float32').realize()
@@ -211,7 +214,7 @@ def make_run_policy(vision_runner, policy_runners: list, features_slice: slice,
inputs.update({road_key: img, wide_key: big_img})
if 'features_buffer' not in inputs:
inputs['features_buffer'] = sample_skip_fn(feat_q)
inputs['features_buffer'] = sample_skip_fn(feat_q).reshape(input_shapes['features_buffer'])
policy_out = next(iter(policy_runners[0](inputs).values())).cast('float32').realize()
if 'features_buffer' not in inputs and features_slice is not None:
@@ -272,18 +275,17 @@ def _parse_size(size_str: str) -> tuple[int, int]:
return int(width), int(height)
def read_file_chunked_to_shm(path):
def read_file_chunked_to_disk(path):
if not path:
return None
import atexit
import shutil
from openpilot.common.file_chunker import open_file_chunked
from openpilot.common.hardware.hw import Paths
shm_path = os.path.join(Paths.shm_path(), os.path.basename(path))
atexit.register(lambda: os.path.exists(shm_path) and os.remove(shm_path))
with open(shm_path, 'wb') as dst, open_file_chunked(path) as src:
shutil.copyfileobj(src, dst)
return shm_path
tmp_path = f'{path}.unchunked'
with open(tmp_path, 'wb') as f, open_file_chunked(path) as src:
shutil.copyfileobj(src, f)
atexit.register(lambda: os.path.exists(tmp_path) and os.remove(tmp_path))
return tmp_path
def _load_policy_runners(args: argparse.Namespace) -> tuple[list, list]:
@@ -296,7 +298,7 @@ def _load_policy_runners(args: argparse.Namespace) -> tuple[list, list]:
if __name__ == "__main__":
if 'USB' in os.getenv('DEV', '') or os.getenv('USBGPU'):
if 'USB' in os.getenv('DEV', '') or os.getenv('CHESTNUT'):
from openpilot.system.hardware.chestnut.flash import link_up
for _ in range(10):
if link_up():
@@ -327,11 +329,11 @@ if __name__ == "__main__":
model_w, model_h = args.model_size
output_data = {}
args.vision_onnx = read_file_chunked_to_shm(args.vision_onnx)
args.policy_onnx = read_file_chunked_to_shm(args.policy_onnx)
args.off_policy_onnx = read_file_chunked_to_shm(args.off_policy_onnx)
args.on_policy_onnx = read_file_chunked_to_shm(args.on_policy_onnx)
args.supercombo_onnx = read_file_chunked_to_shm(args.supercombo_onnx)
args.vision_onnx = read_file_chunked_to_disk(args.vision_onnx)
args.policy_onnx = read_file_chunked_to_disk(args.policy_onnx)
args.off_policy_onnx = read_file_chunked_to_disk(args.off_policy_onnx)
args.on_policy_onnx = read_file_chunked_to_disk(args.on_policy_onnx)
args.supercombo_onnx = read_file_chunked_to_disk(args.supercombo_onnx)
vision_runner = OnnxRunner(args.vision_onnx) if args.vision_onnx else None
+56 -42
View File
@@ -8,22 +8,22 @@ See the LICENSE.md file in the root directory for more details.
import os
os.environ['GMMU'] = '0'
from openpilot.common.hardware import COMMA_HARDWARE
from openpilot.selfdrive.modeld.helpers import usbgpu_present, load_oob
import time
import numpy as np
import threading
import time
from setproctitle import setproctitle
from tinygrad.tensor import Tensor
import openpilot.cereal.messaging as messaging
from openpilot.common.hardware import COMMA_HARDWARE
from openpilot.selfdrive.modeld.helpers import chestnut_present, load_oob
from openpilot.cereal import log
from opendbc.car.structs import car
from openpilot.cereal.services import SERVICE_LIST
from setproctitle import setproctitle
from openpilot.cereal.messaging import PubMaster, SubMaster
from openpilot.cereal.visionipc import VisionStreamType
from msgq.visionipc import VisionIpcClient, VisionBuf
from opendbc.car.car_helpers import get_demo_car_params
from tinygrad.tensor import Tensor
from openpilot.common.file_chunker import open_file_chunked
from openpilot.common.swaglog import cloudlog
from openpilot.common.params import Params
@@ -42,13 +42,13 @@ from openpilot.sunnypilot.modeld_v2.constants import Plan
from openpilot.sunnypilot.modeld_v2.meta_helper import load_meta_constants
from openpilot.sunnypilot.modeld_v2.camera_offset_helper import CameraOffsetHelper
from openpilot.sunnypilot.modeld_v2.compile_modeld import derive_frame_skip, make_split_input_queues, make_supercombo_input_queues, WARP_INPUTS, POLICY_INPUTS
from openpilot.sunnypilot.livedelay.helpers import get_lat_delay
from openpilot.sunnypilot.modeld_v2.modeld_base import ModelStateBase
from openpilot.sunnypilot.models.helpers import get_active_bundle
from openpilot.sunnypilot.selfdrive.controls.lib.relc import RoadEdgeLaneChangeController
PROCESS_NAME = "openpilot.selfdrive.modeld.modeld_tinygrad"
BIG_MODEL_TIMEOUT = 60
def _pkl_exists(path):
@@ -68,6 +68,7 @@ def _find_driving_pkl(bundle):
pkl_path = os.path.join(model_root, pkl_name)
if _pkl_exists(pkl_path):
return pkl_path
return None
class FrameMeta:
@@ -84,14 +85,14 @@ class ModelState(ModelStateBase):
inputs: dict[str, np.ndarray]
prev_desire: np.ndarray
def __init__(self, cam_w: int, cam_h: int, usbgpu: bool = False):
def __init__(self, cam_w: int, cam_h: int, chestnut: bool = False):
ModelStateBase.__init__(self)
env_pkl = os.environ.get('COMBINED_MODEL_PKL')
if env_pkl and os.path.exists(env_pkl):
model_bundle = None
else:
model_bundle = get_active_bundle()
model_bundle = get_active_bundle(chestnut=chestnut)
self.generation = model_bundle.generation if model_bundle is not None else None
overrides = {override.key: override.value for override in model_bundle.overrides} if model_bundle else {}
@@ -99,10 +100,10 @@ class ModelState(ModelStateBase):
self.LONG_SMOOTH_SECONDS = float(overrides.get('long', ".0"))
self.MIN_LAT_CONTROL_SPEED = 0.3
self.PLANPLUS_CONTROL: float = 1.0
self.usbgpu = usbgpu
self.chestnut = chestnut
pkl_path = _find_driving_pkl(model_bundle)
assert pkl_path is not None, "No driving pkl found — all models must be compiled with compile_modeld.py"
assert pkl_path is not None, f"No driving pkl found for {'chestnut' if chestnut else 'small model'} — all models must be compiled with compile_modeld.py"
self._init_combined(pkl_path, cam_w, cam_h, model_bundle)
def _init_combined(self, pkl_path, cam_w, cam_h, bundle):
@@ -110,7 +111,7 @@ class ModelState(ModelStateBase):
jits = load_oob(open_file_chunked(pkl_path))
self.WARP_DEV = 'QCOM' if COMMA_HARDWARE else 'CPU'
self.DEV = 'AMD' if self.usbgpu else self.WARP_DEV
self.DEV = 'AMD' if self.chestnut else self.WARP_DEV
self.QUEUE_DEV = self.DEV
metadata = jits['metadata']
@@ -185,9 +186,6 @@ class ModelState(ModelStateBase):
else:
self.warp(**{k: self.input_queues[k] for k in WARP_INPUTS}, frame=frame_tensor, big_frame=big_frame_tensor)
if self.usbgpu:
self.warmup()
def warmup(self) -> None:
dummy_frames = {k: np.zeros(self.frame_buf_params[k][3], dtype=np.uint8) for k in self._vision_input_names}
transforms = {k: np.eye(3, dtype=np.float32) for k in [self._road_key, self._wide_key] if k}
@@ -287,9 +285,8 @@ class ModelState(ModelStateBase):
buf[0, :-1] = buf[0, 1:]
buf[0, -1, :] = outputs['desired_curvature'][0, :] if not self.mlsim else 0
if self.usbgpu and not np.all(np.isfinite(outputs.get('plan', np.array([0.])))):
cloudlog.error("model output not finite, dropping frame")
return None
if self.chestnut and not np.all(np.isfinite(outputs.get('plan', np.array([0.])))):
raise RuntimeError("model output not finite")
return outputs
@@ -327,13 +324,13 @@ def main(demo=False):
setproctitle(PROCESS_NAME)
config_realtime_process(7, 54)
USBGPU = usbgpu_present()
if USBGPU:
CHESTNUT = chestnut_present()
if CHESTNUT:
os.environ['HCQDEV_WAIT_TIMEOUT_MS'] = '3000'
params = Params()
params.put_bool("UsbGpuLoading", USBGPU)
params.remove("UsbGpuActive")
params.put_bool("ChestnutLoading", CHESTNUT)
params.remove("ChestnutActive")
# visionipc clients
while True:
@@ -362,31 +359,36 @@ def main(demo=False):
st = time.monotonic()
model = None
if USBGPU:
import threading
def load():
nonlocal model
model = ModelState(cam_w=vipc_client_main.width, cam_h=vipc_client_main.height, usbgpu=True)
t = threading.Thread(target=load, daemon=True)
t.start()
t.join(60)
if model is None:
params.put_bool("UsbGpuActive", False)
raise RuntimeError("eGPU model load failed or timed out (60s)")
params.put_bool("UsbGpuActive", True)
else:
model = ModelState(cam_w=vipc_client_main.width, cam_h=vipc_client_main.height, usbgpu=False)
if CHESTNUT:
big_model = None
def load_big():
nonlocal big_model
try:
m = ModelState(cam_w=vipc_client_main.width, cam_h=vipc_client_main.height, chestnut=True)
m.warmup()
big_model = m
except Exception:
cloudlog.exception("chestnut load failed")
loader = threading.Thread(target=load_big, daemon=True)
loader.start()
loader.join(BIG_MODEL_TIMEOUT)
model = big_model
params.put_bool("ChestnutActive", model is not None)
params.put_bool("UsbGpuLoading", False)
small_model = ModelState(cam_w=vipc_client_main.width, cam_h=vipc_client_main.height, chestnut=False) if model is None or CHESTNUT else None
if model is None:
model = small_model
params.put_bool("ChestnutLoading", False)
assert model is not None
cloudlog.warning(f"models loaded in {time.monotonic() - st:.1f}s, modeld starting")
# messaging
pub_socks = ["modelV2", "drivingModelData", "cameraOdometry", "modelDataV2SP"] + (["chestnutState"] if USBGPU else [])
pub_socks = ["modelV2", "drivingModelData", "cameraOdometry", "modelDataV2SP"] + (["chestnutState"] if CHESTNUT else [])
pm = PubMaster(pub_socks)
sm = SubMaster(["deviceState", "carState", "narrowRoadCameraState", "extrinsicsCalibration", "driverMonitoringState", "carControl", "lateralDelay"])
publish_state = PublishState()
chestnut_state = ChestnutState(pm, USBGPU) if USBGPU else None
chestnut_state = ChestnutState(pm, model.chestnut) if CHESTNUT else None
# setup filter to track dropped frames
frame_dropped_filter = FirstOrderFilter(0., 10., 1. / model.constants.MODEL_FREQ)
@@ -509,7 +511,19 @@ def main(demo=False):
inputs['action_t'] = np.array([lat_action_t, long_action_t], dtype=np.float32)
mt1 = time.perf_counter()
model_output = model.run(bufs, transforms, inputs, prepare_only)
try:
model_output = model.run(bufs, transforms, inputs, prepare_only)
except Exception:
if not params.get_bool("ChestnutActive"):
raise
cloudlog.exception("chestnut failed, falling back to small")
params.put_bool("ChestnutActive", False)
assert small_model is not None
model = small_model
if chestnut_state is not None:
chestnut_state.big = False
run_count = 0
model_output = None
mt2 = time.perf_counter()
model_execution_time = mt2 - mt1
@@ -524,7 +538,7 @@ def main(demo=False):
fill_model_msg(drivingdata_send, modelv2_send, model_output, action,
publish_state, meta_main.frame_id, meta_extra.frame_id, frame_id,
frame_drop_ratio, meta_main.timestamp_eof, model_execution_time, live_calib_seen, meta_constants)
modelv2_send.modelV2.big = model.usbgpu
modelv2_send.modelV2.big = model.chestnut
desire_state = modelv2_send.modelV2.meta.desireState
l_lane_change_prob = desire_state[log.Desire.laneChangeLeft]
@@ -190,8 +190,8 @@ def tmp_path():
def patch_modeld(monkeypatch):
def _patch(bundle):
monkeypatch.setattr(helpers, 'get_active_bundle', lambda params=None: bundle)
monkeypatch.setattr(modeld_module, 'get_active_bundle', lambda params=None: bundle)
monkeypatch.setattr(helpers, 'get_active_bundle', lambda params=None, *, chestnut=None: bundle)
monkeypatch.setattr(modeld_module, 'get_active_bundle', lambda params=None, *, chestnut=None: bundle)
return _patch
@@ -59,8 +59,8 @@ class TestFindDrivingPkl(OpenpilotTestCase):
class TestModelStateCombinedInit(OpenpilotTestCase):
def test_asserts_when_no_pkl(self, monkeypatch):
bundle = DummyBundle(models=[], is_20hz=True)
monkeypatch.setattr(helpers, 'get_active_bundle', lambda params=None: bundle)
monkeypatch.setattr(modeld_module, 'get_active_bundle', lambda params=None: bundle)
monkeypatch.setattr(helpers, 'get_active_bundle', lambda params=None, *, chestnut=None: bundle)
monkeypatch.setattr(modeld_module, 'get_active_bundle', lambda params=None, *, chestnut=None: bundle)
with self.assertRaisesRegex(AssertionError, "No driving pkl found"):
ModelState(cam_w=CAM_W, cam_h=CAM_H)
@@ -5,10 +5,15 @@ This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
import os
import tempfile
from pathlib import Path
import numpy as np
from openpilot.common.parameterized import parameterized
from openpilot.sunnypilot.modeld_v2.compile_modeld import derive_frame_skip, _detect_desire_key
from openpilot.common.file_chunker import chunk_file, get_chunk_targets
from openpilot.sunnypilot.modeld_v2.compile_modeld import derive_frame_skip, _detect_desire_key, read_file_chunked_to_disk
from openpilot.common.test import OpenpilotTestCase
@@ -160,3 +165,115 @@ class TestOutputSlicePreservation(OpenpilotTestCase):
policy_slices = {'plan': slice(0, 495), 'meta': slice(495, 550)}
assert set(vision_slices.keys()) & set(policy_slices.keys()) == set(), \
"vision and policy slices should not overlap in keys"
class TestReadFileChunkedToDisk(OpenpilotTestCase):
def test_none_passthrough(self):
assert read_file_chunked_to_disk(None) is None
def test_unchunked_source_staged_on_disk(self):
with tempfile.TemporaryDirectory() as d:
src = Path(d) / "driving_supercombo.onnx"
payload = os.urandom(1024)
src.write_bytes(payload)
out = Path(read_file_chunked_to_disk(str(src)))
assert out.parent == Path(d)
assert out.name == "driving_supercombo.onnx.unchunked"
assert out.read_bytes() == payload
def test_chunked_source_reassembled_on_disk(self):
with tempfile.TemporaryDirectory() as d:
src = Path(d) / "driving_supercombo.onnx"
payload = os.urandom(4096)
src.write_bytes(payload)
chunk_file(str(src), get_chunk_targets(str(src), len(payload)))
assert not src.exists()
out = Path(read_file_chunked_to_disk(str(src)))
assert out.parent == Path(d)
assert out.read_bytes() == payload
class Test4DFeaturesBuffer(OpenpilotTestCase):
def test_get_policy_npy_shapes_4d(self):
from openpilot.sunnypilot.modeld_v2.compile_modeld import get_policy_npy_shapes
input_shapes = {
'desire_pulse': (1, 25, 8),
'features_buffer': (1, 24, 32, 512), # compare 4d to 3d for regression
'traffic_convention': (1, 2),
'action_t': (1, 2)
}
shapes, sizes = get_policy_npy_shapes(input_shapes, is_supercombo=True)
assert shapes['prev_feat'] == (1, 16384)
assert sizes == [8, 2, 2, 16384]
def test_get_policy_npy_shapes_3d(self):
from openpilot.sunnypilot.modeld_v2.compile_modeld import get_policy_npy_shapes
input_shapes = {
'desire_pulse': (1, 25, 8),
'features_buffer': (1, 24, 512),
'traffic_convention': (1, 2),
'action_t': (1, 2)
}
shapes, sizes = get_policy_npy_shapes(input_shapes, is_supercombo=True)
assert shapes['prev_feat'] == (1, 512)
assert sizes == [8, 2, 2, 512]
class TestStockCompileModeldEquivalence(OpenpilotTestCase):
def test_get_policy_npy_shapes_matches_stock(self):
from openpilot.selfdrive.modeld.compile_modeld import get_policy_npy_shapes as stock_get_policy_npy_shapes
from openpilot.sunnypilot.modeld_v2.compile_modeld import get_policy_npy_shapes as sunny_get_policy_npy_shapes
stock_input_shapes = {
'desire_pulse': (1, 25, 8),
'features_buffer': (1, 24, 512), # see below comment
'traffic_convention': (1, 2),
'action_t': (1, 2),
}
stock_shapes, stock_sizes = stock_get_policy_npy_shapes(stock_input_shapes)
sunny_shapes, sunny_sizes = sunny_get_policy_npy_shapes(stock_input_shapes, is_supercombo=True)
assert sunny_shapes == stock_shapes
assert sunny_sizes == stock_sizes
assert sunny_shapes['prev_feat'] == (1, 512)
def test_make_input_queues_full_stock_equivalence(self):
from openpilot.selfdrive.modeld.compile_modeld import make_input_queues as stock_make_input_queues
from openpilot.sunnypilot.modeld_v2.compile_modeld import make_supercombo_input_queues as sunny_make_supercombo_input_queues
input_shapes = {
'img': (1, 12, 128, 256),
'desire_pulse': (1, 25, 8),
'features_buffer': (1, 24, 512), # when https://github.com/commaai/openpilot/pull/38681 merges, update to 1,24,32,512
'traffic_convention': (1, 2),
'action_t': (1, 2),
}
frame_skip = 4
stock_queues, stock_npy = stock_make_input_queues(input_shapes, frame_skip, device='NPY')
sunny_queues, sunny_npy = sunny_make_supercombo_input_queues(input_shapes, frame_skip, device='NPY')
assert set(sunny_queues.keys()) == set(stock_queues.keys())
for key in stock_queues:
assert sunny_queues[key].shape == stock_queues[key].shape, \
f"Queue shape mismatch for {key}: sunny {sunny_queues[key].shape} != stock {stock_queues[key].shape}"
assert set(sunny_npy.keys()) == set(stock_npy.keys())
for key in stock_npy:
assert sunny_npy[key].shape == stock_npy[key].shape, \
f"Numpy array shape mismatch for {key}: sunny {sunny_npy[key].shape} != stock {stock_npy[key].shape}"
def test_make_warp_queues_stock_equivalence(self):
from openpilot.selfdrive.modeld.compile_modeld import make_warp_input_queues as stock_make_warp_queues
from openpilot.sunnypilot.modeld_v2.compile_modeld import make_warp_queues as sunny_make_warp_queues
stock_vision_shapes = {'img': (1, 12, 128, 256)} # for now?
stock_queues, stock_npy = stock_make_warp_queues(stock_vision_shapes, frame_skip=4, device='NPY')
sunny_queues, sunny_npy = sunny_make_warp_queues(device='NPY')
assert set(sunny_npy.keys()) == set(stock_npy.keys()) == {'tfm', 'big_tfm'}
for key in sunny_npy:
assert sunny_npy[key].shape == stock_npy[key].shape == (3, 3)
@@ -0,0 +1,62 @@
"""
Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
import io
import requests
from openpilot.common.file_chunker import get_chunk_name
from openpilot.common.hardware import hw
from openpilot.common.test import OpenpilotTestCase
from openpilot.selfdrive.modeld.helpers import dump_oob
import openpilot.sunnypilot.modeld_v2.modeld as modeld_module
from openpilot.sunnypilot.modeld_v2.tests import helpers as tests_helpers
from openpilot.sunnypilot.modeld_v2.tests.helpers import DummyModel, DummyBundle, CAM_W, CAM_H
from openpilot.sunnypilot.models.fetcher import ModelParser, ModelFetcher
tmp_path = tests_helpers.tmp_path
class TestFallback(OpenpilotTestCase):
def test_find_dual_model_in_bundle(self, tmp_path, monkeypatch):
lebowski_file = 'driving_lebowski.pkl'
tsfdo_file = 'driving_tsfdo.pkl'
(tmp_path / lebowski_file).write_bytes(b'fkasdjfkljf')
(tmp_path / tsfdo_file).write_bytes(b'dskfajklsdjlsfka')
monkeypatch.setattr(hw.Paths, 'model_root', staticmethod(lambda: str(tmp_path)))
big_bundle = DummyBundle(models=[DummyModel('supercombo', lebowski_file)])
small_bundle = DummyBundle(models=[DummyModel('supercombo', tsfdo_file)])
big_pkl = modeld_module._find_driving_pkl(big_bundle)
small_pkl = modeld_module._find_driving_pkl(small_bundle)
assert big_pkl is not None and lebowski_file in big_pkl
assert small_pkl is not None and tsfdo_file in small_pkl
def test_download_models_and_init_modelstate_fallback(self, tmp_path, monkeypatch):
monkeypatch.setattr(hw.Paths, 'model_root', staticmethod(lambda: str(tmp_path)))
big_json = requests.get(ModelFetcher.MODEL_URL_CHESTNUT).json()
big_bundle = ModelParser.parse_models(big_json)[-1]
small_json = requests.get(ModelFetcher.MODEL_URL).json()
small_bundle = ModelParser.parse_models(small_json)[-1]
buf = io.BytesIO()
dump_oob(tests_helpers.make_pkl_data(tests_helpers.ARCHETYPES['supercombo_non20hz']), buf)
oob_bytes = buf.getvalue()
for bundle in (big_bundle, small_bundle):
artifact = bundle.models[0].artifact
for i in range(len(artifact.chunks)):
(tmp_path / get_chunk_name(artifact.fileName, i, len(artifact.chunks))).write_bytes(oob_bytes if i == 0 else b"")
monkeypatch.setattr(modeld_module, 'get_active_bundle', lambda params=None, *, chestnut=None: small_bundle)
assert modeld_module.ModelState(CAM_W, CAM_H, chestnut=False).chestnut is False
monkeypatch.setattr(modeld_module, 'get_active_bundle', lambda params=None, *, chestnut=None: big_bundle)
try:
assert modeld_module.ModelState(CAM_W, CAM_H, chestnut=True).chestnut is True
except Exception as e:
assert "AMD" in str(e) or "device" in str(e).lower()
+42 -28
View File
@@ -1,10 +1,22 @@
import argparse
import os
import hashlib
import requests
import re
from openpilot.common.basedir import BASEDIR
from openpilot.selfdrive.ui.ui_state import ui_state
from openpilot.sunnypilot import get_file_hash
from openpilot.sunnypilot.models.model_name import DEFAULT_MODEL
from openpilot.sunnypilot.models.model_name import DEFAULT_MODEL, DEFAULT_BIG_MODEL
from openpilot.sunnypilot.models.fetcher import ModelFetcher
def get_default_model() -> str:
show_big_model = (ui_state.chestnut_present
and (ui_state.chestnut_active or ui_state.chestnut_loading or ui_state.is_offroad()))
return DEFAULT_BIG_MODEL if show_big_model else DEFAULT_MODEL
DEFAULT_MODEL_NAME_PATH = os.path.join(BASEDIR, "openpilot", "sunnypilot", "models", "model_name.py")
MODEL_HASH_PATH = os.path.join(BASEDIR, "openpilot", "sunnypilot", "models", "tests", "model_hash")
@@ -13,7 +25,6 @@ SUPERCOMBO_ONNX_PATH = os.path.join(BASEDIR, "openpilot", "selfdrive", "modeld",
def update_model_hash():
supercombo_hash = get_file_hash(SUPERCOMBO_ONNX_PATH)
combined_hash = hashlib.sha256(supercombo_hash.encode()).hexdigest()
with open(MODEL_HASH_PATH, "w") as f:
@@ -22,40 +33,43 @@ def update_model_hash():
print(f"Generated and updated new combined model hash to {MODEL_HASH_PATH}")
def get_current_default_model_name():
print("[GET DEFAULT MODEL NAME]")
name = DEFAULT_MODEL
print(f'Current default model name: "{name}"')
return name
def get_ref_for_name(url: str, name: str) -> str:
response = requests.get(url, timeout=10)
if response.status_code == 200:
bundles = response.json()["bundles"]
matching = [b for b in bundles if re.search(name, f"{b['short_name']} {b['display_name']}", re.IGNORECASE)]
if matching:
return max(matching, key=lambda b: int(b["index"]))["ref"]
return ""
def update_default_model_name(name: str):
print("[CHANGE DEFAULT MODEL NAME]")
def update_default_model_names(default_model_name: str, default_big_model_name: str):
print("[CHANGE DEFAULT MODEL NAMES]")
small_ref = get_ref_for_name(ModelFetcher.MODEL_URL, default_model_name)
big_ref = get_ref_for_name(ModelFetcher.MODEL_URL_CHESTNUT, default_big_model_name)
with open(DEFAULT_MODEL_NAME_PATH, "w") as f:
f.write(f'DEFAULT_MODEL = "{name}"\n')
print(f'New default model name: "{name}"')
f.write(f'DEFAULT_MODEL = "{default_model_name}"\n')
f.write(f'DEFAULT_MODEL_REF = "{small_ref}"\n')
f.write(f'DEFAULT_BIG_MODEL = "{default_big_model_name}"\n')
f.write(f'DEFAULT_BIG_MODEL_REF = "{big_ref}"\n')
print(f'New default small model name: "{default_model_name}" (ref: {small_ref})')
print(f'New default big model name: "{default_big_model_name}" (ref: {big_ref})')
print("[DONE]")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Update default model name and hash")
parser.add_argument("--new_name", type=str, help="New default model name")
parser = argparse.ArgumentParser(description="Update default model names and hash")
parser.add_argument("--new_small_model_name", type=str, help="New default small model name")
parser.add_argument("--new_big_model_name", type=str, help="New default big model name")
args = parser.parse_args()
if not args.new_name:
print("Warning: No new default model name provided. Use --new_name to specify")
print("Default model name and hash will not be updated! (aborted)")
exit(0)
if args.new_small_model_name is None and args.new_big_model_name is None:
new_name = input(f'Enter new default small model name (current: "{DEFAULT_MODEL}", leave empty to keep): ').strip()
new_big_model_name = input(f'Enter new default big model name (current: "{DEFAULT_BIG_MODEL}", leave empty to keep): ').strip()
else:
new_name, new_big_model_name = args.new_small_model_name, args.new_big_model_name
current_name = get_current_default_model_name()
new_name = args.new_name
if current_name == new_name:
print(f'Proposed default model name: "{new_name}"')
confirm = input("Proposed default model name is the same as the current default model name. Confirm? (y/n): ").upper().strip()
if confirm != "Y":
print("Default model name and hash will not be updated! (aborted)")
exit(0)
update_default_model_name(new_name)
update_default_model_names(new_name or DEFAULT_MODEL, new_big_model_name or DEFAULT_BIG_MODEL)
update_model_hash()
+83 -32
View File
@@ -13,8 +13,6 @@ from openpilot.common.params import Params
from openpilot.common.swaglog import cloudlog
from openpilot.common.hardware.hw import Paths
from openpilot.sunnypilot.models.helpers import is_bundle_version_compatible
from openpilot.selfdrive.modeld.helpers import usbgpu_present
from openpilot.cereal import custom
@@ -140,45 +138,53 @@ class ModelCache:
class ModelFetcher:
"""Handles fetching and caching of model data from remote source"""
MODEL_URL = "https://raw.githubusercontent.com/sunnypilot/sunnypilot-models/refs/heads/gh-pages/docs/driving_models_v20.json"
MODEL_URL_USBGPU = "https://raw.githubusercontent.com/sunnypilot/sunnypilot-models/refs/heads/gh-pages/docs/driving_models_usbgpu_v20.json"
MODEL_URL = "https://raw.githubusercontent.com/sunnypilot/sunnypilot-models/refs/heads/gh-pages/docs/driving_models_v21.json"
MODEL_URL_CHESTNUT = "https://raw.githubusercontent.com/sunnypilot/sunnypilot-models/refs/heads/gh-pages/docs/driving_models_chestnut_v22.json"
MODEL_SOURCES = {
"qcom": (MODEL_URL, ""),
"chestnut": (MODEL_URL_CHESTNUT, "_Chestnut"),
}
def __init__(self, params: Params):
self.params = params
self.model_parser = ModelParser()
self._is_usbgpu: bool | None = None
self.model_cache = ModelCache(params)
self.model_url = self.MODEL_URL
self._update_model_source()
self.model_caches = {
source: ModelCache(params, suffix=suffix)
for source, (_, suffix) in self.MODEL_SOURCES.items()
}
self._refetched: set[str] = set()
self.params.put("ModelManager_ActiveJson", {
"qcom": self.MODEL_URL,
"chestnut": self.MODEL_URL_CHESTNUT,
}, block=True)
def _update_model_source(self) -> None:
"""Updates what json to use based on usbgpu availability"""
is_usbgpu = usbgpu_present()
if is_usbgpu != self._is_usbgpu:
self._is_usbgpu = is_usbgpu
self.model_cache = ModelCache(self.params, suffix="_USBGPU" if is_usbgpu else "")
self.model_url = self.MODEL_URL_USBGPU if is_usbgpu else self.MODEL_URL
self.params.put("ModelManager_ActiveJson", self.model_url, block=True)
@staticmethod
def active_source(chestnut_present: bool) -> str:
return "chestnut" if chestnut_present else "qcom"
def _fetch_and_cache_models(self) -> list[custom.ModelManagerSP.ModelBundle] | None:
def _fetch_and_cache_models(self, source: str) -> list[custom.ModelManagerSP.ModelBundle] | None:
"""Fetches fresh model data from remote and updates cache.
Returns None on transport errors. Raises on 404 and other fatal HTTP errors.
"""
model_url, _ = self.MODEL_SOURCES[source]
try:
response = requests.get(self.model_url, timeout=10)
response = requests.get(model_url, timeout=10)
# Explicitly handle 404 differently
if response.status_code == 404:
cloudlog.error(f"Models URL returned 404 Not Found: {self.model_url}")
raise HTTPError(f"404 Not Found: {self.model_url}", response=response)
cloudlog.error(f"Models URL returned 404 Not Found: {model_url}")
raise HTTPError(f"404 Not Found: {model_url}", response=response)
# Raise for any other 4xx/5xx
response.raise_for_status()
json_data = response.json()
self.model_cache.set(json_data)
cloudlog.debug("Successfully updated models cache")
return self.model_parser.parse_models(json_data)
parsed = self.model_parser.parse_models(json_data)
if parsed:
self.model_caches[source].set(json_data)
cloudlog.debug(f"Successfully updated models cache for {source}")
return parsed
except ConnectionError as e:
cloudlog.warning(f"DNS/connection error while fetching models: {e}")
@@ -191,16 +197,40 @@ class ModelFetcher:
return None
def get_available_bundles(self) -> list[custom.ModelManagerSP.ModelBundle]:
"""Gets the list of available models, with smart cache handling"""
self._update_model_source()
cached_data, is_expired = self.model_cache.get()
@staticmethod
def _cache_matches_source(source: str, cached_data: dict) -> bool:
bundles = cached_data.get("bundles", [])
if source == "chestnut":
return any(bundle.get("is_big") is True for bundle in bundles)
return not any(bundle.get("is_big") is True for bundle in bundles)
def get_bundles_for_source(self, source: str) -> list[custom.ModelManagerSP.ModelBundle]:
if source not in self.MODEL_SOURCES:
cloudlog.warning(f"Unknown model source: {source}")
return []
cached_data, is_expired = self.model_caches[source].get()
if cached_data and not is_expired:
cloudlog.debug("Using valid cached models data")
return self.model_parser.parse_models(cached_data)
# a source is refetched over a mismatch at most once per process: if the fresh
# manifest still mismatches, the URL is authoritative and the cache is trusted
if self._cache_matches_source(source, cached_data) or source in self._refetched:
try:
parsed = self.model_parser.parse_models(cached_data)
except Exception:
cloudlog.warning(f"Failed to parse cached models for {source}; refetching", exc_info=True)
else:
if parsed:
cloudlog.debug(f"Using valid cached models data for source {source}")
return parsed
# a source-matching cache that yields no valid bundles is stale (e.g. an old
# manifest version) - do not trust it, refetch so the source is repopulated
cloudlog.warning(f"Cached models for {source} have no valid bundles; refetching")
else:
self._refetched.add(source)
cloudlog.warning(f"Cached models for {source} not valid; refetching once")
fetched_bundles = self._fetch_and_cache_models()
fetched_bundles = self._fetch_and_cache_models(source)
if fetched_bundles is not None:
return fetched_bundles
@@ -208,12 +238,33 @@ class ModelFetcher:
cloudlog.warning("Failed to fetch fresh data and no cache available")
cloudlog.warning("Failed to fetch fresh data. Using expired cache as fallback")
return self.model_parser.parse_models(cached_data)
try:
return self.model_parser.parse_models(cached_data)
except Exception:
return []
def get_cached_bundles(params: Params, source: str) -> list[custom.ModelManagerSP.ModelBundle]:
if source not in ModelFetcher.MODEL_SOURCES:
cloudlog.warning(f"Unknown model source: {source}")
return []
_, suffix = ModelFetcher.MODEL_SOURCES[source]
cached_data = params.get(f"ModelManager_ModelsCache{suffix}")
if not cached_data:
return []
try:
return ModelParser.parse_models(cached_data)
except Exception as e:
cloudlog.warning(f"Failed to parse cached models for source {source}: {e}")
return []
if __name__ == "__main__":
from openpilot.selfdrive.modeld.helpers import chestnut_present
params = Params()
model_fetcher = ModelFetcher(params)
bundles = model_fetcher.get_available_bundles()
bundles = model_fetcher.get_bundles_for_source(ModelFetcher.active_source(chestnut_present()))
for bundle in bundles:
for model in bundle.models:
model_overrides = {override.key: override.value for override in bundle.overrides}
+73 -35
View File
@@ -16,14 +16,20 @@ from openpilot.common.params import Params
from openpilot.common.swaglog import cloudlog
from openpilot.sunnypilot.models.constants import Meta, MetaSimPose, MetaTombRaider
from openpilot.common.hardware.hw import Paths
from openpilot.selfdrive.modeld.helpers import chestnut_present
# SET ME TO THE EXACT JSON VERSION WE SET IN SUNNYPILOT_MODELS REPO
REQUIRED_JSON_VERSION = 17
REQUIRED_JSON_VERSION = 18
CUSTOM_MODEL_PATH = Paths.model_root()
METADATA_PATH = Path(__file__).parent / '../models/supercombo_metadata.pkl'
ModelManager = custom.ModelManagerSP
_LAST_VALIDATED_RAW = None
ACTIVE_BUNDLE_KEYS = {
"qcom": "ModelManager_ActiveBundle",
"chestnut": "ModelManager_ActiveBundleChestnut",
}
_LAST_VALIDATED_RAW: dict[str, dict | None] = {}
def _compute_hash(file_path: str) -> str | None:
@@ -86,11 +92,11 @@ def _bundle_needs_reset(active_bundle: custom.ModelManagerSP.ModelBundle, availa
if available_bundles is not None:
matching_bundle = None
for bundle in available_bundles:
if getattr(active_bundle, 'ref', None) and getattr(bundle, 'ref', None):
if active_bundle.ref and bundle.ref:
if active_bundle.ref == bundle.ref:
matching_bundle = bundle
break
elif getattr(active_bundle, 'internalName', None) == getattr(bundle, 'internalName', None):
elif active_bundle.internalName == bundle.internalName:
matching_bundle = bundle
break
@@ -98,49 +104,81 @@ def _bundle_needs_reset(active_bundle: custom.ModelManagerSP.ModelBundle, availa
return True
if active_bundle.minimumSelectorVersion != matching_bundle.minimumSelectorVersion:
return True
active_runner = getattr(active_bundle, 'runner', None)
matching_runner = getattr(matching_bundle, 'runner', None)
if active_runner is not None and matching_runner is not None:
if getattr(active_runner, 'raw', active_runner) != getattr(matching_runner, 'raw', matching_runner):
return True
if active_bundle.runner != matching_bundle.runner:
return True
if set(_bundle_artifacts(active_bundle)) != set(_bundle_artifacts(matching_bundle)):
return True
return not _bundle_is_valid_locally(active_bundle)
def validate_active_bundle(params: Params, available_bundles: list[custom.ModelManagerSP.ModelBundle] | None = None) -> None:
global _LAST_VALIDATED_RAW
raw_bundle = params.get("ModelManager_ActiveBundle")
if not raw_bundle:
return
if raw_bundle == _LAST_VALIDATED_RAW:
return
active_bundle = get_active_bundle(params, raw_bundle_dict=raw_bundle)
if active_bundle is None or _bundle_needs_reset(active_bundle, available_bundles):
cloudlog.warning("Active model bundle invalid; resetting to default")
params.remove("ModelManager_ActiveBundle")
params.put("ModelRunnerTypeCache", int(custom.ModelManagerSP.Runner.stock), block=True)
_LAST_VALIDATED_RAW = None
else:
_LAST_VALIDATED_RAW = raw_bundle
def get_active_bundle(params: Params | None = None, raw_bundle_dict: dict | bytes | None = None) -> "custom.ModelManagerSP.ModelBundle | None":
params = params or Params()
def _parse_active_bundle(raw_bundle) -> "custom.ModelManagerSP.ModelBundle | None":
try:
active_bundle_dict = raw_bundle_dict if raw_bundle_dict is not None else (params.get("ModelManager_ActiveBundle") or {})
if isinstance(active_bundle_dict, dict) and active_bundle_dict and is_bundle_version_compatible(active_bundle_dict):
return custom.ModelManagerSP.ModelBundle(**active_bundle_dict)
if isinstance(raw_bundle, dict) and raw_bundle and is_bundle_version_compatible(raw_bundle):
return custom.ModelManagerSP.ModelBundle(**raw_bundle)
except Exception:
pass
return None
def get_selected_bundle(params: Params | None = None, source: str = "qcom") -> "custom.ModelManagerSP.ModelBundle | None":
params = params or Params()
return _parse_active_bundle(params.get(ACTIVE_BUNDLE_KEYS[source]))
def get_active_source(chestnut: bool | None = None, chestnut_active: bool | None = None,
chestnut_loading: bool | None = None, offroad: bool | None = None) -> str:
if chestnut is None:
chestnut = chestnut_present()
state_valid = chestnut_active is not None or chestnut_loading is not None or offroad is not None
big_active = chestnut and (not state_valid or chestnut_active or chestnut_loading or offroad)
return "chestnut" if big_active else "qcom"
def get_active_bundle(params: Params | None = None, *, chestnut: bool | None = None) -> "custom.ModelManagerSP.ModelBundle | None":
# no cross-slot fallback: an empty active slot means the hardware default, which
# only stock modeld can run - modeld_v2 requires a real bundle
params = params or Params()
return get_selected_bundle(params, get_active_source(chestnut=chestnut))
def resolve_bundle_by_ref(
ref: str, source_bundles: dict[str, list[custom.ModelManagerSP.ModelBundle]],
) -> "tuple[custom.ModelManagerSP.ModelBundle, str] | None":
for source, bundles in source_bundles.items():
for bundle in bundles:
if bundle.ref == ref:
return bundle, source
return None
def _validate_active_bundle(params: Params, source: str, available_bundles: list[custom.ModelManagerSP.ModelBundle] | None = None) -> None:
global _LAST_VALIDATED_RAW
key = ACTIVE_BUNDLE_KEYS[source]
raw_bundle = params.get(key)
if not raw_bundle:
return
if _LAST_VALIDATED_RAW.get(key) == raw_bundle:
return
active_bundle = _parse_active_bundle(raw_bundle)
if active_bundle is None or _bundle_needs_reset(active_bundle, available_bundles):
cloudlog.warning(f"Active model bundle invalid for {source}; resetting to default")
params.remove(key)
_LAST_VALIDATED_RAW[key] = None
else:
_LAST_VALIDATED_RAW[key] = raw_bundle
def validate_active_bundles(params: Params, source_bundles: dict[str, list[custom.ModelManagerSP.ModelBundle]]) -> None:
# an empty list means the fetch failed, not that the catalog dropped the bundle
for source, bundles in source_bundles.items():
_validate_active_bundle(params, source, bundles or None)
get_active_model_runner(params, force_check=True)
def get_active_model_runner(params: Params | None = None, force_check: bool = False) -> int:
params = params or Params()
cached_runner_type = params.get("ModelRunnerTypeCache")
+110 -41
View File
@@ -17,12 +17,17 @@ from openpilot.common.hardware.hw import Paths
from openpilot.cereal import messaging, custom
from openpilot.sunnypilot.models.fetcher import ModelFetcher
from openpilot.sunnypilot.models.helpers import get_active_bundle, validate_active_bundle, verify_file
from openpilot.sunnypilot.models.helpers import (ACTIVE_BUNDLE_KEYS, get_active_bundle, get_selected_bundle,
resolve_bundle_by_ref, validate_active_bundles, verify_file)
# (connect, read) seconds. read is per-request inactivity, not a total cap
DOWNLOAD_TIMEOUT = (30, 30)
class DownloadCancelled(Exception):
pass
class ModelManagerSP:
"""Manages model downloads and status reporting"""
@@ -30,11 +35,25 @@ class ModelManagerSP:
self.params = Params()
self.model_fetcher = ModelFetcher(self.params)
self.pm = messaging.PubMaster(["modelManagerSP"])
self.sm = messaging.SubMaster(["deviceState"])
self.chestnut_present = False
self.available_models: list[custom.ModelManagerSP.ModelBundle] = []
self.source_models: dict[str, list[custom.ModelManagerSP.ModelBundle]] = {}
self.selected_bundle: custom.ModelManagerSP.ModelBundle = None
self.active_bundle: custom.ModelManagerSP.ModelBundle = get_active_bundle(self.params)
self.active_bundle: custom.ModelManagerSP.ModelBundle = get_active_bundle(self.params, chestnut=self.chestnut_present)
self._chunk_size = 128 * 1000 # 128 KB chunks
self._download_start_times: dict[str, float] = {} # Track start time per model
self._download_ref: bytes | str | None = None
def _download_interrupted(self) -> bool:
# only removal cancels: a different ref is a queued selection that
# _release_download_ref leaves in place for the next tick
return self.params.get("ModelManager_DownloadRef") is None
def _release_download_ref(self) -> None:
if self.params.get("ModelManager_DownloadRef") == self._download_ref:
self.params.remove("ModelManager_DownloadRef")
self._download_ref = None
def _sync_artifact_progress(self, source_artifact) -> None:
"""Mirror download progress to all artifacts sharing the same filename in the selected bundle."""
@@ -76,8 +95,8 @@ class ModelManagerSP:
f.write(chunk)
bytes_downloaded += len(chunk)
if self.params.get("ModelManager_DownloadIndex") is None:
raise Exception("Download cancelled")
if self._download_interrupted():
raise DownloadCancelled("Download cancelled")
if total_size > 0:
progress = (bytes_downloaded / total_size) * 100
@@ -90,7 +109,7 @@ class ModelManagerSP:
# Clean up start time after download completes
del self._download_start_times[model.fileName]
async def _download_chunked(self, base_url: str, base_path: str, artifact) -> None:
async def _download_chunked(self, base_url: str, base_path: str, artifact, skip: frozenset[int] | set[int] = frozenset()) -> None:
from openpilot.common.file_chunker import get_chunk_name, get_manifest_path
num_chunks = len(artifact.chunks)
@@ -102,8 +121,11 @@ class ModelManagerSP:
# Shared connection saves a TCP+TLS handshake per chunk.
# Keep sequential: the link saturates on one stream and Session is not thread-safe.
completed = len(skip)
with requests.Session() as session:
for i, _ in enumerate(artifact.chunks):
if i in skip:
continue
chunk_url = get_chunk_name(base_url, i, num_chunks)
chunk_path = get_chunk_name(base_path, i, num_chunks)
chunk_downloaded = 0
@@ -114,15 +136,16 @@ class ModelManagerSP:
for data in response.iter_content(chunk_size=self._chunk_size):
f.write(data)
chunk_downloaded += len(data)
if self.params.get("ModelManager_DownloadIndex") is None:
raise Exception("Download cancelled")
if self._download_interrupted():
raise DownloadCancelled("Download cancelled")
intra = chunk_downloaded / max(chunk_size, 1)
progress = min(99.0, ((i + intra) / num_chunks) * 100)
progress = min(99.0, ((completed + intra) / num_chunks) * 100)
artifact.downloadProgress.status = custom.ModelManagerSP.DownloadStatus.downloading
artifact.downloadProgress.progress = progress
artifact.downloadProgress.eta = self._calculate_eta(artifact.fileName, progress)
self._sync_artifact_progress(artifact)
self._report_status()
completed += 1
with open(manifest_path, 'w') as f: # noqa: ASYNC230
f.write(str(num_chunks))
@@ -133,6 +156,8 @@ class ModelManagerSP:
async def _process_artifact(self, artifact, destination_path: str) -> None:
if not artifact.downloadUri.uri:
return None
if self._download_interrupted():
raise DownloadCancelled("Download cancelled")
url = artifact.downloadUri.uri
expected_hash = artifact.downloadUri.sha256
@@ -140,17 +165,23 @@ class ModelManagerSP:
full_path = os.path.join(destination_path, filename)
try:
# progress counts only valid chunks so a resumed download continues the
# bar from where verification left it, instead of falling back to zero
is_cached = False
valid_chunks: set[int] = set()
if len(artifact.chunks) > 0:
from openpilot.common.file_chunker import get_chunk_name
chunks_valid = True
num_chunks = len(artifact.chunks)
for i, chunk in enumerate(artifact.chunks):
chunk_path = get_chunk_name(full_path, i, len(artifact.chunks))
if not await verify_file(chunk_path, chunk.sha256):
chunks_valid = False
break
if chunks_valid and len(artifact.chunks) > 0:
is_cached = True
if self._download_interrupted():
raise DownloadCancelled("Download cancelled")
if await verify_file(get_chunk_name(full_path, i, num_chunks), chunk.sha256):
valid_chunks.add(i)
artifact.downloadProgress.status = custom.ModelManagerSP.DownloadStatus.verifying
artifact.downloadProgress.progress = (len(valid_chunks) / num_chunks) * 100
self._sync_artifact_progress(artifact)
self._report_status()
is_cached = len(valid_chunks) == num_chunks
else:
if await verify_file(full_path, expected_hash):
is_cached = True
@@ -164,7 +195,7 @@ class ModelManagerSP:
return
if len(artifact.chunks) > 0:
await self._download_chunked(url, full_path, artifact)
await self._download_chunked(url, full_path, artifact, skip=valid_chunks)
from openpilot.common.file_chunker import get_chunk_name
for i, chunk in enumerate(artifact.chunks):
chunk_path = get_chunk_name(full_path, i, len(artifact.chunks))
@@ -181,6 +212,17 @@ class ModelManagerSP:
self._sync_artifact_progress(artifact)
self._report_status()
except DownloadCancelled:
# a cancel keeps whatever is on disk: complete chunks resume the next attempt
self._download_start_times.pop(artifact.fileName, None)
artifact.downloadProgress.status = custom.ModelManagerSP.DownloadStatus.failed
artifact.downloadProgress.eta = 0
self._sync_artifact_progress(artifact)
if self.selected_bundle:
self.selected_bundle.status = custom.ModelManagerSP.DownloadStatus.failed
self._report_status()
raise
except Exception as e:
cloudlog.error(f"Error downloading {filename}: {str(e)}")
for f in [full_path] + [p for p in (os.path.join(destination_path, f) for f in os.listdir(destination_path)) if filename in p]:
@@ -212,10 +254,12 @@ class ModelManagerSP:
model_manager_state.availableBundles = self.available_models
self.pm.send('modelManagerSP', msg)
async def _download_bundle(self, model_bundle: custom.ModelManagerSP.ModelBundle, destination_path: str) -> None:
"""Downloads all models in a bundle"""
async def _download_bundle(self, model_bundle: custom.ModelManagerSP.ModelBundle, destination_path: str, source: str) -> None:
self.selected_bundle = model_bundle
self.selected_bundle.status = custom.ModelManagerSP.DownloadStatus.downloading
for model in self.selected_bundle.models:
model.artifact.downloadProgress.status = custom.ModelManagerSP.DownloadStatus.downloading
self._report_status()
os.makedirs(destination_path, exist_ok=True)
try:
@@ -232,10 +276,11 @@ class ModelManagerSP:
seen_artifacts.add(artifact.fileName)
await self._process_artifact(artifact, destination_path)
self.active_bundle = self.selected_bundle
self.active_bundle.status = custom.ModelManagerSP.DownloadStatus.downloaded
self.params.put("ModelManager_ActiveBundle", self.active_bundle.to_dict(), block=True)
self.selected_bundle = None
if self._download_interrupted():
raise DownloadCancelled("Download cancelled")
self.selected_bundle.status = custom.ModelManagerSP.DownloadStatus.downloaded
self.params.put(ACTIVE_BUNDLE_KEYS[source], model_bundle.to_dict(), block=True)
self.active_bundle = get_active_bundle(self.params, chestnut=self.chestnut_present)
except Exception:
if self.selected_bundle is not None:
@@ -245,9 +290,30 @@ class ModelManagerSP:
finally:
self._report_status()
def download(self, model_bundle: custom.ModelManagerSP.ModelBundle, destination_path: str) -> None:
def download(self, model_bundle: custom.ModelManagerSP.ModelBundle, destination_path: str, source: str) -> None:
"""Main entry point for downloading a model bundle"""
asyncio.run(self._download_bundle(model_bundle, destination_path))
asyncio.run(self._download_bundle(model_bundle, destination_path, source))
def _process_download_requests(self) -> None:
# loops so a ref queued during a download starts in the same tick, without
# the bar dropping to idle for a tick between the two transfers
last_ref = None
while (ref_to_download := self.params.get("ModelManager_DownloadRef")) is not None:
if ref_to_download == last_ref: # a repeating ref falls back to the next tick instead of spinning
return
last_ref = ref_to_download
resolved = resolve_bundle_by_ref(ref_to_download, self.source_models)
if not resolved:
return
model_to_download, source = resolved
self._download_ref = ref_to_download
try:
self.download(model_to_download, Paths.model_root(), source)
except Exception as e:
cloudlog.exception(e)
finally:
self._release_download_ref()
self.selected_bundle = None
def main_thread(self) -> None:
"""Main thread for model management"""
@@ -255,19 +321,20 @@ class ModelManagerSP:
while True:
try:
self.available_models = self.model_fetcher.get_available_bundles()
validate_active_bundle(self.params, self.available_models)
self.active_bundle = get_active_bundle(self.params)
self.sm.update(0)
self.chestnut_present = self.sm['deviceState'].chestnutPresent
self.source_models = {source: self.model_fetcher.get_bundles_for_source(source) for source in ModelFetcher.MODEL_SOURCES}
self.available_models = self.source_models[ModelFetcher.active_source(self.chestnut_present)]
validate_active_bundles(self.params, self.source_models)
self.active_bundle = get_active_bundle(self.params, chestnut=self.chestnut_present)
if (index_to_download := self.params.get("ModelManager_DownloadIndex")) is not None:
if model_to_download := next((model for model in self.available_models if model.index == index_to_download), None):
try:
self.download(model_to_download, Paths.model_root())
except Exception as e:
cloudlog.exception(e)
finally:
self.params.remove("ModelManager_DownloadIndex")
self.selected_bundle = None
if get_selected_bundle(self.params, "chestnut") is not None and get_selected_bundle(self.params, "qcom") is None:
if self.params.get("ModelManager_DownloadRef") is None:
from openpilot.sunnypilot.models.model_name import DEFAULT_MODEL_REF
if DEFAULT_MODEL_REF:
self.params.put("ModelManager_DownloadRef", DEFAULT_MODEL_REF)
self._process_download_requests()
if self.params.get("ModelManager_ClearCache"):
self.clear_model_cache()
@@ -285,12 +352,14 @@ class ModelManagerSP:
Clears the model cache directory of all files except those in the active model bundle.
"""
# Get list of files used by active model bundle
# Get list of files used by both slots' selected bundles (either may become
# the truly active bundle depending on hardware availability)
active_files = []
if self.active_bundle is not None: # When the default model is active
for model in self.active_bundle.models:
if hasattr(model, 'artifact') and model.artifact.fileName:
active_files.append(model.artifact.fileName)
for source in ACTIVE_BUNDLE_KEYS:
if selected_bundle := get_selected_bundle(self.params, source):
for model in selected_bundle.models:
if model.artifact.fileName:
active_files.append(model.artifact.fileName)
# Remove all files except active ones (including their chunk files)
model_dir = Paths.model_root()
@@ -1 +1,4 @@
DEFAULT_MODEL = "CD210"
DEFAULT_MODEL_REF = "5b6436a90cf6902b8aaa71c2b6f3d7164d8ae391"
DEFAULT_BIG_MODEL = "Lebowski"
DEFAULT_BIG_MODEL_REF = "fa0c6876d3cf070e91e25e5353ceadc68a5b3285"
@@ -20,4 +20,4 @@ class TestDefaultModel(OpenpilotTestCase):
with open(MODEL_HASH_PATH) as f:
current_hash = f.read().strip()
assert combined_hash == current_hash, "Run sunnypilot/models/default_model.py to update the default model name and hash"
assert combined_hash == current_hash, "Run openpilot/sunnypilot/models/default_model.py to update the default model name and hash"
@@ -11,6 +11,7 @@ import http.server
import os
import tempfile
import threading
import time
import unittest
from typing import Any
from unittest import mock
@@ -23,6 +24,10 @@ from openpilot.common.test import OpenpilotTestCase
from openpilot.common.file_chunker import get_chunk_name, get_manifest_path
from openpilot.selfdrive.test.helpers import http_server_context
from openpilot.sunnypilot.models import manager as manager_module
from openpilot.sunnypilot.models.fetcher import ModelFetcher, get_cached_bundles
from openpilot.sunnypilot.models import helpers
from openpilot.sunnypilot.models.helpers import (get_active_bundle, get_active_source, get_selected_bundle,
resolve_bundle_by_ref, validate_active_bundles)
from openpilot.sunnypilot.models.manager import ModelManagerSP
CHUNK_BODIES = [b'A' * 5000, b'B' * 5000, b'C' * 3000]
@@ -98,11 +103,13 @@ class ManagerDownloadTestBase(OpenpilotTestCase):
self.manager = ModelManagerSP.__new__(ModelManagerSP)
self.manager.params = mock.MagicMock()
self.manager.params.get.return_value = b'0' # not cancelled
self.manager._download_ref = b'0'
self.manager.pm = mock.MagicMock()
self.manager.pm.send.side_effect = self._record_progress
self.manager.selected_bundle = None
self.manager.active_bundle = None
self.manager.available_models = []
self.manager.chestnut_present = False
self.manager._chunk_size = 1024
self.manager._download_start_times = {}
@@ -249,6 +256,166 @@ class TestManagerDownload(ManagerDownloadTestBase):
assert self.manager._download_start_times == {}
self.run_with_server(body)
def test_download_ref_present_keeps_download_alive(self):
"""A pending download request (DownloadRef set) must not be cancelled mid-transfer."""
def body():
artifact = self.make_artifact(chunked=True)
base_path = os.path.join(self.dest, artifact.fileName)
self.manager.params.get.side_effect = lambda key: b"ref" if key == "ModelManager_DownloadRef" else None
self.manager._download_ref = b"ref"
asyncio.run(self.manager._download_chunked(artifact.downloadUri.uri, base_path, artifact))
assert os.path.isfile(get_manifest_path(base_path))
self.run_with_server(body)
def test_cancellation_via_download_ref(self):
"""Removing DownloadRef mid-transfer cancels the download."""
def body():
artifact = self.make_artifact(chunked=True)
base_path = os.path.join(self.dest, artifact.fileName)
checks = {"n": 0}
def get(key):
if key == "ModelManager_DownloadRef":
checks["n"] += 1
return b"ref" if checks["n"] <= 2 else None
return b"0"
self.manager.params.get.side_effect = get
self.manager._download_ref = b"ref"
with self.assertRaises(Exception) as ctx:
asyncio.run(self.manager._download_chunked(artifact.downloadUri.uri, base_path, artifact))
assert 'cancelled' in str(ctx.exception).lower()
assert not os.path.isfile(get_manifest_path(base_path))
self.run_with_server(body)
def test_replaced_download_ref_queues_instead_of_cancelling(self):
"""Selecting another model mid-transfer lets the running download finish."""
def body():
artifact = self.make_artifact(chunked=True)
base_path = os.path.join(self.dest, artifact.fileName)
self.manager.params.get.side_effect = lambda key: b"other-ref" if key == "ModelManager_DownloadRef" else None
self.manager._download_ref = b"ref"
asyncio.run(self.manager._download_chunked(artifact.downloadUri.uri, base_path, artifact))
assert os.path.isfile(get_manifest_path(base_path))
self.run_with_server(body)
def test_replaced_download_ref_is_kept(self):
"""A selection made during a download must survive that download's cleanup."""
self.manager.params.get.return_value = b"new-ref"
self.manager._download_ref = b"old-ref"
self.manager._release_download_ref()
self.manager.params.remove.assert_not_called()
def test_own_download_ref_is_released(self):
self.manager.params.get.return_value = b"ref"
self.manager._download_ref = b"ref"
self.manager._release_download_ref()
self.manager.params.remove.assert_called_once_with("ModelManager_DownloadRef")
def test_cached_bundle_cancel_skips_slot_write(self):
"""A cancel must stop an already-on-disk bundle before it is applied to the slot."""
def body():
artifact = self.make_artifact(chunked=True)
base_path = os.path.join(self.dest, artifact.fileName)
for i, data in enumerate(CHUNK_BODIES):
with open(get_chunk_name(base_path, i, len(CHUNK_BODIES)), 'wb') as f:
f.write(data)
self._bundle.ref = "test-ref"
params, store = self._make_params_with_store()
store["ModelManager_DownloadRef"] = None # removed -> cancelled
self.manager.params = params
self.manager._download_ref = b"ref"
with self.assertRaises(Exception) as ctx:
asyncio.run(self.manager._download_bundle(self._bundle, self.dest, "qcom"))
assert 'cancelled' in str(ctx.exception).lower()
assert "ModelManager_ActiveBundle" not in store
assert all(os.path.isfile(p) for p in self.chunk_paths(base_path)), "cancel must not delete cached chunks"
self.run_with_server(body)
def test_resume_skips_valid_chunks(self):
"""A chunk already on disk is kept and not re-downloaded; progress starts above its share."""
def body():
artifact = self.make_artifact(chunked=True)
base_path = os.path.join(self.dest, artifact.fileName)
with open(get_chunk_name(base_path, 0, len(CHUNK_BODIES)), 'wb') as f:
f.write(CHUNK_BODIES[0])
asyncio.run(self.manager._process_artifact(artifact, self.dest))
chunk0_suffix = get_chunk_name('', 0, len(CHUNK_BODIES))
assert not any(p.endswith(chunk0_suffix) for p in DownloadHandler.request_paths), "valid chunk was re-downloaded"
for i, expected in enumerate(CHUNK_BODIES):
with open(get_chunk_name(base_path, i, len(CHUNK_BODIES)), 'rb') as f:
assert f.read() == expected
assert os.path.isfile(get_manifest_path(base_path))
assert min(self.reported) >= (1 / len(CHUNK_BODIES)) * 100 - 1, "progress must not restart below the resumed share"
self.run_with_server(body)
def test_verify_reports_valid_fraction_then_cached(self):
"""A fully cached bundle publishes climbing verify progress and ends cached."""
def body():
artifact = self.make_artifact(chunked=True)
base_path = os.path.join(self.dest, artifact.fileName)
for i, data in enumerate(CHUNK_BODIES):
with open(get_chunk_name(base_path, i, len(CHUNK_BODIES)), 'wb') as f:
f.write(data)
asyncio.run(self.manager._process_artifact(artifact, self.dest))
assert DownloadHandler.request_paths == [], "cached bundle must not hit the network"
assert [round(p) for p in self.reported[:3]] == [33, 67, 100]
assert artifact.downloadProgress.status == custom.ModelManagerSP.DownloadStatus.cached
self.run_with_server(body)
def _make_params_with_store(self):
params = mock.MagicMock()
store = {}
def get(key, *args, **kwargs):
return store.get(key, b"0") # b"0" -> download not cancelled
def put(key, value, *args, **kwargs):
store[key] = value
params.get.side_effect = get
params.put.side_effect = put
return params, store
def test_download_writes_qcom_slot(self):
"""A download resolved to the qcom source writes the qcom active bundle slot only."""
def body():
artifact = self.make_artifact(chunked=True)
self._bundle.ref = "test-ref"
self._bundle.minimumSelectorVersion = 18
params, store = self._make_params_with_store()
self.manager.params = params
asyncio.run(self.manager._download_bundle(self._bundle, self.dest, "qcom"))
assert "ModelManager_ActiveBundle" in store, "qcom download must write the qcom slot"
assert "ModelManager_ActiveBundleChestnut" not in store, "qcom download must not touch the chestnut slot"
assert self.manager.selected_bundle.status == custom.ModelManagerSP.DownloadStatus.downloaded
assert self.manager.active_bundle is not None and self.manager.active_bundle.ref == "test-ref"
assert self.manager.active_bundle.status == custom.ModelManagerSP.DownloadStatus.downloaded
chunk_names = [get_chunk_name(artifact.fileName, i, len(artifact.chunks)) for i in range(len(artifact.chunks))]
missing = [c for c in chunk_names if not os.path.isfile(os.path.join(self.dest, c))]
assert missing == [], f"chunks missing from the cache: {missing}"
self.run_with_server(body)
def test_download_writes_chestnut_slot(self):
"""A download resolved to the chestnut source writes the chestnut active bundle slot only."""
def body():
self.make_artifact(chunked=True)
self._bundle.ref = "big-ref"
self._bundle.minimumSelectorVersion = 18
params, store = self._make_params_with_store()
self.manager.params = params
asyncio.run(self.manager._download_bundle(self._bundle, self.dest, "chestnut"))
assert "ModelManager_ActiveBundleChestnut" in store, "chestnut download must write the chestnut slot"
assert "ModelManager_ActiveBundle" not in store, "chestnut download must not touch the qcom slot"
assert self.manager.selected_bundle.status == custom.ModelManagerSP.DownloadStatus.downloaded
self.run_with_server(body)
class TestManagerImports(OpenpilotTestCase):
"""Catches undeclared dependencies. aiohttp lived only in the AGNOS venv; 19.6 dropped
@@ -267,6 +434,352 @@ class TestManagerImports(OpenpilotTestCase):
assert connect > 0 and read > 0, "requests defaults to no timeout; downloads would hang forever"
class TestResolveBundleByRef(OpenpilotTestCase):
"""A ref resolves to (bundle, source) across both hardware manifests. Refs are
unique per manifest and never overlap across sources, so a ref maps to exactly
one slot. Shared by the manager's download flow and the settings UI."""
@staticmethod
def _bundle(ref: str):
bundle = custom.ModelManagerSP.ModelBundle.new_message()
bundle.ref = ref
return bundle
def test_qcom_ref_resolves_to_qcom_slot(self):
small = self._bundle("small")
assert resolve_bundle_by_ref("small", {"qcom": [small], "chestnut": []}) == (small, "qcom")
def test_chestnut_ref_resolves_to_chestnut_slot(self):
big = self._bundle("big")
assert resolve_bundle_by_ref("big", {"qcom": [], "chestnut": [big]}) == (big, "chestnut")
def test_unknown_ref_returns_none(self):
source_bundles = {"qcom": [self._bundle("small")], "chestnut": []}
assert resolve_bundle_by_ref("nope", source_bundles) is None
def manifest_bundle(short_name: str, ref: str, index: int = 0, is_big: bool = False) -> dict:
"""Minimal manifest bundle dict, version-compatible (no chunks to avoid disk side effects).
Big (chestnut) bundles carry `is_big: true` in the manifest JSON."""
return {
"index": index,
"short_name": short_name,
"display_name": short_name.upper(),
"generation": 1,
"environment": "release",
"runner": "tinygrad",
"is_big": is_big,
"minimum_selector_version": "18",
"ref": ref,
"models": [{
"type": "supercombo",
"artifact": {
"file_name": f"{short_name}.pkl",
"download_uri": {"url": f"https://example.com/{short_name}.pkl", "sha256": "s"},
},
}],
}
def fresh_sync_time() -> int:
return int(time.monotonic() * 1e9)
class TestModelFetcherSources(OpenpilotTestCase):
"""Both manifests are always maintained: get_bundles_for_source exposes either
source by name, and active_source picks which one matches the attached hardware."""
def _make_params(self, qcom_manifest, chestnut_manifest):
params = mock.MagicMock()
def get(key):
if key == "ModelManager_ModelsCache":
return qcom_manifest
if key == "ModelManager_ModelsCache_Chestnut":
return chestnut_manifest
if key in ("ModelManager_LastSyncTime", "ModelManager_LastSyncTime_Chestnut"):
return fresh_sync_time()
return None
params.get.side_effect = get
return params
def test_active_source_follows_chestnut_presence(self):
assert ModelFetcher.active_source(False) == "qcom"
assert ModelFetcher.active_source(True) == "chestnut"
def test_get_bundles_for_source_returns_each_source(self):
params = self._make_params({"bundles": [manifest_bundle("small", "aaa")]},
{"bundles": [manifest_bundle("big", "bbb", is_big=True)]})
fetcher = ModelFetcher(params)
assert [bundle.ref for bundle in fetcher.get_bundles_for_source("qcom")] == ["aaa"]
assert [bundle.ref for bundle in fetcher.get_bundles_for_source("chestnut")] == ["bbb"]
def test_get_bundles_for_source_unknown(self):
assert ModelFetcher(mock.MagicMock()).get_bundles_for_source("bogus") == []
def test_get_cached_bundles_parses_source(self):
params = self._make_params({"bundles": [manifest_bundle("small", "aaa")]},
{"bundles": [manifest_bundle("big", "bbb", is_big=True)]})
qcom_bundles = get_cached_bundles(params, "qcom")
chestnut_bundles = get_cached_bundles(params, "chestnut")
assert [b.ref for b in qcom_bundles] == ["aaa"]
assert [b.ref for b in chestnut_bundles] == ["bbb"]
assert qcom_bundles[0].displayName == "SMALL"
def test_get_cached_bundles_empty_when_missing(self):
params = mock.MagicMock()
params.get.return_value = None
assert get_cached_bundles(params, "qcom") == []
assert get_cached_bundles(params, "chestnut") == []
def test_get_cached_bundles_unknown_source(self):
assert get_cached_bundles(mock.MagicMock(), "bogus") == []
def test_active_json_has_both_urls(self):
params = mock.MagicMock()
ModelFetcher(params)
active_json_calls = [call for call in params.put.call_args_list if call.args[0] == "ModelManager_ActiveJson"]
assert active_json_calls, "expected ModelManager_ActiveJson to be written"
assert active_json_calls[-1].args[1] == {
"qcom": ModelFetcher.MODEL_URL,
"chestnut": ModelFetcher.MODEL_URL_CHESTNUT,
}
class TestSourceCacheIntegrity(OpenpilotTestCase):
"""Each source's cached manifest must contain only that source's models; the
`is_big` flag in the JSON marks the big (chestnut) models. A mismatched cache is
legacy data from before the per-source split (the active manifest was cached
under the unsuffixed key regardless of hardware) and is refetched. This
replaces the old one-time bundle migration."""
def _make_params(self, qcom_manifest, chestnut_manifest):
params = mock.MagicMock()
def get(key):
if key == "ModelManager_ModelsCache":
return qcom_manifest
if key == "ModelManager_ModelsCache_Chestnut":
return chestnut_manifest
if key in ("ModelManager_LastSyncTime", "ModelManager_LastSyncTime_Chestnut"):
return fresh_sync_time()
return None
params.get.side_effect = get
return params
def _fetched(self, *bundles):
return ModelFetcher(mock.MagicMock()).model_parser.parse_models({"bundles": list(bundles)})
def test_qcom_cache_with_big_models_is_refetched(self):
"""Legacy: the unsuffixed cache holds the big manifest. is_big confirms it is
the wrong set for qcom, so a fresh fetch replaces it."""
params = self._make_params({"bundles": [manifest_bundle("big", "bbb", is_big=True)]},
{"bundles": [manifest_bundle("big2", "ccc", is_big=True)]})
fetcher = ModelFetcher(params)
fetched = self._fetched(manifest_bundle("small", "aaa"))
with mock.patch.object(fetcher, "_fetch_and_cache_models", return_value=fetched):
bundles = fetcher.get_bundles_for_source("qcom")
assert [bundle.ref for bundle in bundles] == ["aaa"]
def test_chestnut_cache_without_big_models_is_refetched(self):
params = self._make_params({"bundles": [manifest_bundle("small", "aaa")]},
{"bundles": [manifest_bundle("big2", "ccc")]})
fetcher = ModelFetcher(params)
fetched = self._fetched(manifest_bundle("big", "bbb", is_big=True))
with mock.patch.object(fetcher, "_fetch_and_cache_models", return_value=fetched):
bundles = fetcher.get_bundles_for_source("chestnut")
assert [bundle.ref for bundle in bundles] == ["bbb"]
def test_matching_caches_are_used_without_fetch(self):
params = self._make_params({"bundles": [manifest_bundle("small", "aaa")]},
{"bundles": [manifest_bundle("big", "bbb", is_big=True)]})
fetcher = ModelFetcher(params)
with mock.patch.object(fetcher, "_fetch_and_cache_models", side_effect=AssertionError("cache should be used")):
assert [bundle.ref for bundle in fetcher.get_bundles_for_source("qcom")] == ["aaa"]
assert [bundle.ref for bundle in fetcher.get_bundles_for_source("chestnut")] == ["bbb"]
def test_stale_version_cache_is_refetched(self):
"""A source-matching cache whose bundles are all filtered by the selector version
check parses to zero valid bundles; it is stale (e.g. an old manifest) and must be
refetched instead of silently returning an empty list forever."""
stale = manifest_bundle("small", "aaa")
stale["minimum_selector_version"] = "16"
params = self._make_params({"bundles": [stale]},
{"bundles": [manifest_bundle("big", "bbb", is_big=True)]})
fetcher = ModelFetcher(params)
fetched = self._fetched(manifest_bundle("small2", "ddd"))
with mock.patch.object(fetcher, "_fetch_and_cache_models", return_value=fetched) as fetch:
bundles = fetcher.get_bundles_for_source("qcom")
fetch.assert_called_once_with("qcom")
assert [bundle.ref for bundle in bundles] == ["ddd"]
def test_mismatched_refetch_happens_once(self):
"""If the fresh manifest still fails the source check, the URL is authoritative:
trust it instead of refetching at 1 Hz forever."""
params = self._make_params({"bundles": [manifest_bundle("big", "bbb", is_big=True)]},
{"bundles": [manifest_bundle("big2", "ccc", is_big=True)]})
fetcher = ModelFetcher(params)
fetched = self._fetched(manifest_bundle("big", "bbb", is_big=True))
with mock.patch.object(fetcher, "_fetch_and_cache_models", return_value=fetched) as fetch:
first = fetcher.get_bundles_for_source("qcom")
second = fetcher.get_bundles_for_source("qcom")
fetch.assert_called_once_with("qcom")
assert [bundle.ref for bundle in first] == ["bbb"]
assert [bundle.ref for bundle in second] == ["bbb"]
def test_corrupt_cache_is_refetched(self):
"""A cache that fails to parse (e.g. truncated/foreign JSON) must trigger a
refetch instead of raising every loop and never recovering."""
corrupt = {"bundles": [{"short_name": "broken"}]} # missing required fields
params = self._make_params(corrupt, {"bundles": [manifest_bundle("big", "bbb", is_big=True)]})
fetcher = ModelFetcher(params)
fetched = self._fetched(manifest_bundle("small", "aaa"))
with mock.patch.object(fetcher, "_fetch_and_cache_models", return_value=fetched) as fetch:
bundles = fetcher.get_bundles_for_source("qcom")
fetch.assert_called_once_with("qcom")
assert [bundle.ref for bundle in bundles] == ["aaa"]
class TestActiveBundleValidation(OpenpilotTestCase):
"""Validation is per-slot: a failed fetch (empty bundle list) must not reset a slot,
and resetting one slot must not stomp the runner cache derived from the other."""
def setUp(self):
super().setUp()
helpers._LAST_VALIDATED_RAW.clear()
@staticmethod
def _raw_bundle(ref: str, runner: int | None = None) -> dict:
bundle = custom.ModelManagerSP.ModelBundle.new_message()
bundle.ref = ref
bundle.minimumSelectorVersion = 18
if runner is not None:
bundle.runner = runner
return bundle.to_dict()
def _params(self, qcom=None, chestnut=None):
params = mock.MagicMock()
def get(key, *args, **kwargs):
return {"ModelManager_ActiveBundle": qcom, "ModelManager_ActiveBundleChestnut": chestnut}.get(key)
params.get.side_effect = get
return params
def test_empty_catalog_does_not_reset_slot(self):
params = self._params(qcom=self._raw_bundle("small"))
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=False):
validate_active_bundles(params, {"qcom": [], "chestnut": []})
params.remove.assert_not_called()
def test_reset_recomputes_runner_from_surviving_slot(self):
tinygrad = int(custom.ModelManagerSP.Runner.tinygrad)
big_raw = self._raw_bundle("big", runner=tinygrad)
params = self._params(qcom=self._raw_bundle("gone"), chestnut=big_raw)
catalog = {"qcom": [custom.ModelManagerSP.ModelBundle(**self._raw_bundle("other"))],
"chestnut": [custom.ModelManagerSP.ModelBundle(**big_raw)]}
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=True):
validate_active_bundles(params, catalog)
params.remove.assert_called_once_with("ModelManager_ActiveBundle")
runner_puts = [call for call in params.put.call_args_list if call.args[0] == "ModelRunnerTypeCache"]
assert [call.args[1] for call in runner_puts] == [tinygrad]
class TestActiveBundleSelection(OpenpilotTestCase):
"""The effective active bundle is the active source's slot: chestnut when a GPU is
present, qcom otherwise. An empty active slot means the hardware default (stock
runner), never the other slot's pick - modeld_v2 requires a real bundle."""
@staticmethod
def _raw_bundle(ref: str) -> dict:
bundle = custom.ModelManagerSP.ModelBundle.new_message()
bundle.ref = ref
bundle.minimumSelectorVersion = 18
return bundle.to_dict()
def _params(self, qcom=None, chestnut=None):
params = mock.MagicMock()
def get(key, *args, **kwargs):
if key == "ModelManager_ActiveBundle":
return qcom
if key == "ModelManager_ActiveBundleChestnut":
return chestnut
return None
params.get.side_effect = get
return params
def test_selected_bundle_is_per_slot(self):
params = self._params(qcom=self._raw_bundle("small"), chestnut=self._raw_bundle("big"))
assert get_selected_bundle(params, "qcom").ref == "small"
assert get_selected_bundle(params, "chestnut").ref == "big"
def test_no_gpu_uses_qcom_slot(self):
params = self._params(qcom=self._raw_bundle("small"), chestnut=self._raw_bundle("big"))
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=False):
assert get_active_bundle(params).ref == "small"
def test_gpu_uses_chestnut_slot(self):
params = self._params(qcom=self._raw_bundle("small"), chestnut=self._raw_bundle("big"))
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=True):
assert get_active_bundle(params).ref == "big"
def test_gpu_without_big_selection_is_hardware_default(self):
params = self._params(qcom=self._raw_bundle("small"), chestnut=None)
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=True):
assert get_active_bundle(params) is None
class TestEffectiveSource(OpenpilotTestCase):
"""One gate decides the active source. With no flags it is runtime truth (GPU
attached); display callers (mici) pass the ui_state flags, which additionally
require the big model to be loading, active, or the device offroad. The active
bundle is simply the selected bundle of that source."""
@staticmethod
def _raw_bundle(ref: str) -> dict:
bundle = custom.ModelManagerSP.ModelBundle.new_message()
bundle.ref = ref
bundle.minimumSelectorVersion = 18
return bundle.to_dict()
def test_runtime_no_gpu(self):
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=False):
assert get_active_source() == "qcom"
def test_runtime_gpu_present(self):
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=True):
assert get_active_source() == "chestnut"
def test_display_offroad_gpu_present_shows_big(self):
assert get_active_source(chestnut=True, chestnut_active=False, chestnut_loading=False, offroad=True) == "chestnut"
def test_display_onroad_gpu_loading_shows_big(self):
assert get_active_source(chestnut=True, chestnut_active=False, chestnut_loading=True, offroad=False) == "chestnut"
def test_display_onroad_gpu_active_shows_big(self):
assert get_active_source(chestnut=True, chestnut_active=True, chestnut_loading=False, offroad=False) == "chestnut"
def test_display_onroad_gpu_idle_shows_small(self):
assert get_active_source(chestnut=True, chestnut_active=False, chestnut_loading=False, offroad=False) == "qcom"
def test_display_active_none_is_idle(self):
assert get_active_source(chestnut=True, chestnut_active=None, chestnut_loading=False, offroad=False) == "qcom"
def test_active_bundle_follows_source(self):
params = mock.MagicMock()
params.get.side_effect = lambda key: {"ModelManager_ActiveBundle": self._raw_bundle("small"),
"ModelManager_ActiveBundleChestnut": self._raw_bundle("big")}.get(key)
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=False):
assert get_active_bundle(params).ref == "small"
assert get_selected_bundle(params, get_active_source(chestnut=True, chestnut_active=False,
chestnut_loading=False, offroad=True)).ref == "big"
@unittest.skipUnless(os.environ.get('RUN_INTEGRATION_TESTS'), 'requires external network')
class TestLiveModelManifest(OpenpilotTestCase):
"""Every artifact and chunk URL in the published manifest must resolve."""
@@ -1,13 +1,11 @@
import requests
from openpilot.common.params import Params
from openpilot.sunnypilot.models.tinygrad_ref import get_tinygrad_ref
from openpilot.sunnypilot.models.fetcher import ModelFetcher
from openpilot.common.test import OpenpilotTestCase
def fetch_tinygrad_ref():
fetcher = ModelFetcher(Params())
response = requests.get(fetcher.model_url, timeout=10)
response = requests.get(ModelFetcher.MODEL_URL, timeout=10)
response.raise_for_status()
json_data = response.json()
return json_data.get("tinygrad_ref")
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:447099e93e303b29e7b3eac237bb0f27f8c5e12786991139aee2432532a75f58
size 12310
@@ -83,9 +83,14 @@ class TestLocationdProc(OpenpilotTestCase):
self.pm.send(msg.which(), msg)
if msg.which() == "cameraOdometry":
self.pm.wait_for_readers_to_update(msg.which(), timeout=1, dt=0.005)
time.sleep(1) # wait for async params write
for _ in range(50):
val = self.params.get('LastGPSPositionLLK')
if val is not None:
break
time.sleep(0.1)
lastGPS = json.loads(self.params.get('LastGPSPositionLLK'))
self.assertIsNotNone(val, "LastGPSPositionLLK not written within 5s")
lastGPS = json.loads(val)
self.assertAlmostEqual(lastGPS['latitude'], self.lat, delta=0.001)
self.assertAlmostEqual(lastGPS['longitude'], self.lon, delta=0.001)
self.assertAlmostEqual(lastGPS['altitude'], self.alt, delta=0.001)
@@ -252,4 +252,12 @@ EVENTS_SP: dict[int, dict[str, Alert | AlertCallbackType]] = {
AlertStatus.userPrompt, AlertSize.small,
Priority.LOW, VisualAlert.none, AudibleAlert.prompt, 0.1),
},
EventNameSP.bigModelReady: {
ET.PERMANENT: Alert(
"Big Model Ready",
"",
AlertStatus.normal, AlertSize.small,
Priority.LOW, VisualAlert.none, AudibleAlert.prompt, 2.),
},
}
@@ -28,7 +28,7 @@ from websocket import (ABNF, WebSocket, WebSocketException, WebSocketTimeoutExce
create_connection, WebSocketConnectionClosedException)
import openpilot.cereal.messaging as messaging
from openpilot.sunnypilot.models.default_model import DEFAULT_MODEL
from openpilot.sunnypilot.models.model_name import DEFAULT_MODEL, DEFAULT_BIG_MODEL
from openpilot.sunnypilot.selfdrive.car.sync_sunnylink_params import update_car_list_param
from openpilot.sunnypilot.sunnylink.api import SunnylinkApi
from openpilot.sunnypilot.sunnylink.utils import sunnylink_need_register, sunnylink_ready, get_param_as_byte, save_param_from_base64_encoded_string
@@ -182,6 +182,8 @@ def getParamsMetadata() -> str:
schema["capabilities"] = generate_capabilities()
schema["capability_labels"] = CAPABILITY_LABELS
schema["default_model"] = DEFAULT_MODEL
schema["default_big_model"] = DEFAULT_BIG_MODEL
schema["chestnut_active"] = params.get_bool("ChestnutActive")
raw = json.dumps(schema, separators=(",", ":")).encode("utf-8")
return base64.b64encode(gzip.compress(raw)).decode("utf-8")
except Exception:
+1
View File
@@ -65,6 +65,7 @@ def sp_stats(end_event):
'MadsSteeringMode',
'MadsUnifiedEngagementMode',
'ModelManager_ActiveBundle',
'ModelManager_ActiveBundleChestnut',
'ModelManager_Favs',
'EnableSunnylinkUploader',
'SunnylinkEnabled',
@@ -84,6 +84,25 @@ def _migrate_tesla_mads_screen_button(_params):
cloudlog.exception(f"Error migrating TeslaMadsScreenButton: {e}")
def _migrate_model_bundle_slots(_params):
# Pre-split, a chestnut user's big-model selection lived in the single
# ActiveBundle. Seed both slots; validation drops whichever does not match
# its own manifest.
try:
if _params.get("ModelManager_ActiveBundleChestnut") is not None:
return
if (chestnut_bundle := _params.get("ModelManager_ActiveBundleUSBGPU")) is not None:
_params.put("ModelManager_ActiveBundleChestnut", chestnut_bundle, block=True)
cloudlog.info("params_migration: seeded ModelManager_ActiveBundleChestnut from ModelManager_ActiveBundleUSBGPU")
return
if (bundle := _params.get("ModelManager_ActiveBundle")) is None:
return
_params.put("ModelManager_ActiveBundleChestnut", bundle, block=True)
cloudlog.info("params_migration: seeded ModelManager_ActiveBundleChestnut from ModelManager_ActiveBundle")
except Exception as e:
cloudlog.exception(f"Error migrating model bundle slots: {e}")
def run_migration(_params):
# migrate OnroadScreenOffBrightness
if _params.get("OnroadScreenOffBrightnessMigrated") != ONROAD_BRIGHTNESS_MIGRATION_VERSION:
@@ -120,3 +139,6 @@ def run_migration(_params):
# seed TeslaMadsScreenButton for existing Tesla installs
_migrate_tesla_mads_screen_button(_params)
# seed the chestnut model slot from the pre-split single slot
_migrate_model_bundle_slots(_params)
@@ -0,0 +1,36 @@
"""
Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
from openpilot.common.params import Params
from openpilot.common.test import OpenpilotTestCase
from openpilot.sunnypilot.system.params_migration import _migrate_model_bundle_slots
class TestModelBundleSlotMigration(OpenpilotTestCase):
"""Pre-split, a chestnut user's big-model selection lived in the single ActiveBundle.
The migration seeds both slots; per-source validation later drops whichever does not
match its own manifest."""
def test_seeds_chestnut_slot_from_active_bundle(self):
params = Params()
bundle = {"ref": "big", "minimumSelectorVersion": 18}
params.put("ModelManager_ActiveBundle", bundle, block=True)
_migrate_model_bundle_slots(params)
assert params.get("ModelManager_ActiveBundleChestnut") == bundle
assert params.get("ModelManager_ActiveBundle") == bundle
def test_noop_when_chestnut_slot_already_set(self):
params = Params()
params.put("ModelManager_ActiveBundle", {"ref": "small"}, block=True)
params.put("ModelManager_ActiveBundleChestnut", {"ref": "big"}, block=True)
_migrate_model_bundle_slots(params)
assert params.get("ModelManager_ActiveBundleChestnut") == {"ref": "big"}
def test_noop_when_no_selection(self):
params = Params()
_migrate_model_bundle_slots(params)
assert params.get("ModelManager_ActiveBundleChestnut") is None
+6 -4
View File
@@ -828,20 +828,22 @@ def startStream(sdp: str, enabled: bool) -> dict:
bridge_services_in = []
# stale car params case taken care of by webrtcd being shut off on ignition
cp_bytes = Params().get("CarParamsPersistent")
cp_bytes = params.get("CarParamsPersistent")
if cp_bytes is not None:
with car.CarParams.from_bytes(cp_bytes) as CP:
if CP.notCar:
bridge_services_in.append("testJoystick")
else:
raise Exception("failed to get CarParamsPersistent")
if params.get_bool("IsOffroad"):
# manager owns camerad/stream_encoderd/webrtcd; flip the param and let it bring them up.
# webrtcd clears IsLiveStreaming when the session ends
params.put_bool("IsLiveStreaming", True)
# wait for webrtcd end points to wake up
wait_for_webrtcd()
try:
wait_for_webrtcd()
except TimeoutError:
cloudlog.event("athena.startStream.webrtcd_offroad_start_timeout", error=True)
raise
return post_stream_request(StreamRequestBody(sdp, ["wideRoad"], enabled, bridge_services_in, ["carState", "deviceState"]))
+8 -4
View File
@@ -16,7 +16,7 @@ from openpilot.common.utils import strip_deprecated_keys
from openpilot.common.filter_simple import FirstOrderFilter
from openpilot.common.params import Params
from openpilot.common.realtime import DT_HW
from openpilot.selfdrive.modeld.helpers import MODELS_DIR, usbgpu_compiled
from openpilot.selfdrive.modeld.helpers import MODELS_DIR, chestnut_compiled
from openpilot.selfdrive.selfdrived.alertmanager import set_offroad_alert
from openpilot.common.hardware import HARDWARE, COMMA_HARDWARE
from openpilot.common.basedir import BASEDIR
@@ -27,7 +27,7 @@ from openpilot.common.swaglog import cloudlog
from openpilot.sunnypilot.system.statsd import statlog
from openpilot.system.hardware.power_monitoring import PowerMonitoring
from openpilot.system.hardware.fan_controller import FanController
from openpilot.common.version import terms_version, training_version, get_build_metadata, terms_version_sp
from openpilot.common.version import terms_version, training_version, get_build_metadata, terms_version_sp, CHESTNUT_BRANCHES
ThermalStatus = log.DeviceState.ThermalStatus
@@ -239,7 +239,7 @@ def hardware_thread(end_event, hw_queue) -> None:
fan_controller = FanController(int(1./DT_HW))
chestnut = Chestnut()
big_model_available = (MODELS_DIR / 'big_driving_supercombo.onnx').is_file() or usbgpu_compiled()
big_model_available = (MODELS_DIR / 'big_driving_supercombo.onnx').is_file() or chestnut_compiled()
while not end_event.is_set():
sm.update(PANDA_STATES_TIMEOUT)
@@ -301,7 +301,11 @@ def hardware_thread(end_event, hw_queue) -> None:
set_usb_state(msg.deviceState, last_hw_state.usb_state)
chestnut.update(started_ts is None, last_hw_state.usb_state)
set_offroad_alert_if_changed("Offroad_ChestnutBranch", msg.deviceState.chestnutPresent and not big_model_available)
current_channel = get_build_metadata().channel
chestnut_target = CHESTNUT_BRANCHES.get(current_channel)
chestnut_needs_switch = msg.deviceState.chestnutPresent and not big_model_available and chestnut_target is not None
set_offroad_alert_if_changed("Offroad_ChestnutBranch", chestnut_needs_switch,
extra_text=chestnut_target if chestnut_needs_switch else None)
# this subset is only used for offroad
temp_sources = [
-40
View File
@@ -1,40 +0,0 @@
#!/usr/bin/env bash
# Define the service name
SERVICE_NAME="actions.runner.sunnypilot.$(uname -n)"
# Function to control the service
control_service() {
local action=$1 # Store the function argument in a local variable
sudo systemctl $action ${SERVICE_NAME}
}
service_exists_and_is_loaded() {
sudo systemctl status ${SERVICE_NAME} &>/dev/null
if [[ $? -ne 4 ]]; then
return 0 # Service is known to systemd (i.e., loaded)
else
return 1 # Service is unknown to systemd (i.e., not loaded)
fi
}
# Check for required argument
if [[ -z $1 ]] || { [[ $1 != "start" ]] && [[ $1 != "stop" ]]; }; then
echo "Usage: $0 {start|stop}"
exit 1
fi
# Store the script argument in a descriptive variable
ACTION=$1
# Trap EXIT signal (Ctrl+C) and stop the service
trap 'control_service stop ; exit' SIGINT SIGKILL EXIT
# Enter the main loop
while true; do
# Check if the service is actually present on the system
if service_exists_and_is_loaded; then
control_service $ACTION # Call the function with the specified action
fi
sleep 1 # Pause before the next iteration
done
+2 -13
View File
@@ -68,10 +68,6 @@ def only_offroad(started: bool, params: Params, CP: car.CarParams) -> bool:
def livestream(started: bool, params: Params, CP: car.CarParams) -> bool:
return params.get_bool("IsLiveStreaming")
def use_github_runner(started, params, CP: car.CarParams) -> bool:
return not PC and params.get_bool("EnableGithubRunner") and (
not params.get_bool("NetworkMetered") and not params.get_bool("GithubRunnerSufficientVoltage"))
def use_copyparty(started, params, CP: car.CarParams) -> bool:
return bool(params.get_bool("EnableCopyparty"))
@@ -110,15 +106,12 @@ def or_(*fns):
def and_(*fns):
return lambda *args: operator.and_(*(fn(*args) for fn in fns))
def not_(*fns):
return lambda *args: operator.not_(*(fn(*args) for fn in fns))
procs = [
DaemonProcess("manage_athenad", "openpilot.system.athena.manage_athenad", "AthenadPid"),
NativeProcess("loggerd", "openpilot/system/loggerd", ["./loggerd"], logging),
NativeProcess("encoderd", "openpilot/system/loggerd", ["./encoderd"], only_onroad),
NativeProcess("stream_encoderd", "openpilot/system/loggerd", ["./encoderd", "--stream"], or_(and_(livestream, not_(iscar)), notcar)),
NativeProcess("stream_encoderd", "openpilot/system/loggerd", ["./encoderd", "--stream"], or_(livestream, notcar)),
PythonProcess("logmessaged", "openpilot.system.logmessaged", always_run),
NativeProcess("camerad", "openpilot/system/camerad", ["./camerad"], or_(driverview, livestream), enabled=not WEBCAM),
@@ -163,7 +156,7 @@ procs = [
# debug procs
NativeProcess("bridge", "openpilot/cereal/messaging", ["./bridge"], notcar),
PythonProcess("webrtcd", "openpilot.system.webrtc.webrtcd", or_(and_(livestream, not_(iscar)), notcar)),
PythonProcess("webrtcd", "openpilot.system.webrtc.webrtcd", or_(livestream, notcar)),
PythonProcess("joystick", "openpilot.tools.joystick.joystick_control", and_(joystick, iscar)),
# sunnylink <3
@@ -189,10 +182,6 @@ procs += [
NativeProcess("locationd_llk", "openpilot/sunnypilot/selfdrive/locationd", ["./locationd"], only_onroad),
]
if os.path.exists("./github_runner.sh"):
procs += [NativeProcess("github_runner_start", "openpilot/system/manager",
["./github_runner.sh", "start"], and_(only_offroad, use_github_runner), sigkill=False)]
if os.path.exists("../../sunnypilot/sunnylink/uploader.py"):
procs += [PythonProcess("sunnylink_uploader", "openpilot.sunnypilot.sunnylink.uploader", use_sunnylink_uploader_shim)]
+19 -7
View File
@@ -8,12 +8,26 @@ from collections.abc import Callable
import pyray as rl
from openpilot.system.ui.lib.application import FontWeight
from openpilot.system.ui.lib.application import gui_app, FontWeight
from openpilot.system.ui.sunnypilot.lib.styles import style
from openpilot.system.ui.sunnypilot.widgets.list_view import ButtonActionSP
from openpilot.system.ui.widgets.label import UnifiedLabel
from openpilot.system.ui.widgets.label import ScrollState, UnifiedLabel
from openpilot.system.ui.widgets.list_view import BUTTON_WIDTH, BUTTON_HEIGHT, TEXT_PADDING, _resolve_value
SCROLL_SPEED = 1.2 # stock is 0.8, boosted 50% to compensate for larger font (50 vs 32)
SCROLL_REFERENCE_FPS = 60.
class UnifiedLabelSP(UnifiedLabel):
# stock scroll formula (0.8 / 60 * fps) is inverted — pre-correct so speed is constant px/sec
def _render(self, _):
if self._needs_scroll and self._scroll_state == ScrollState.SCROLLING:
fps = gui_app.target_fps
wrong_step = 0.8 / SCROLL_REFERENCE_FPS * fps
correct_step = SCROLL_SPEED * SCROLL_REFERENCE_FPS / fps
self._scroll_offset -= (correct_step - wrong_step)
super()._render(_)
class NoElideButtonAction(ButtonActionSP):
def get_width_hint(self):
@@ -21,14 +35,12 @@ class NoElideButtonAction(ButtonActionSP):
class ScrollingButtonAction(ButtonActionSP):
"""ButtonActionSP whose value scrolls instead of eliding when it doesn't fit."""
def __init__(self, text: str | Callable[[], str], width: int = style.BUTTON_ACTION_WIDTH,
enabled: bool | Callable[[], bool] = True):
super().__init__(text=text, width=width, enabled=enabled)
self._value_label = UnifiedLabel("", font_size=style.ITEM_TEXT_FONT_SIZE, font_weight=FontWeight.NORMAL,
text_color=self._value_color, scroll=True,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_MIDDLE)
self._value_label = UnifiedLabelSP("", font_size=style.ITEM_TEXT_FONT_SIZE, font_weight=FontWeight.NORMAL,
text_color=self._value_color, scroll=True,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_MIDDLE)
def set_value(self, value: str | Callable[[], str], color: rl.Color = style.ITEM_TEXT_VALUE_COLOR):
if self.value != _resolve_value(value, ""):
@@ -16,6 +16,7 @@ from openpilot.system.ui.lib.text_measure import measure_text_cached
from openpilot.system.ui.sunnypilot.lib.styles import style
from openpilot.system.ui.sunnypilot.widgets.list_view import ListItemSP
from openpilot.system.ui.widgets.label import UnifiedLabel
from openpilot.system.ui.sunnypilot.lib.utils import UnifiedLabelSP
from openpilot.system.ui.widgets.list_view import ItemAction
FONT_SIZE = style.ITEM_TEXT_FONT_SIZE
@@ -24,6 +25,8 @@ ICON_PADDING = 12
BAR_WIDTH = 1100
BAR_HEIGHT = 20
SEGMENT_GAP = 24
SEGMENT_NAME_MAX_WIDTH = 380
BAR_GAP = 16
BAR_RADIUS = BAR_HEIGHT / 2
CAPSULE_POINTS = 24
@@ -45,6 +48,8 @@ class DownloadStatusAction(ItemAction):
super().__init__(width=BAR_WIDTH)
self.name = ""
self.status_text = ""
self.segments: list[tuple[str, rl.Color, str | None, rl.Color | None]] | None = None
self._segment_labels: list[UnifiedLabelSP] = []
self.downloading = False
self.text_color = rl.GRAY
self.icon: str | None = None
@@ -62,7 +67,8 @@ class DownloadStatusAction(ItemAction):
alignment=rl.GuiTextAlignment.TEXT_ALIGN_RIGHT,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_MIDDLE)
def update(self, name, downloading=False, progress=0.0, status_text="", text_color=rl.GRAY, icon=None, icon_color=None):
def update(self, name, downloading=False, progress=0.0, status_text="", text_color=rl.GRAY, icon=None, icon_color=None, segments=None):
self.segments = segments
if downloading and not self.downloading:
self._name_label.reset_shimmer()
self._progress.x = progress
@@ -85,11 +91,22 @@ class DownloadStatusAction(ItemAction):
def get_width_hint(self) -> float:
if self.downloading:
return BAR_WIDTH
if self.segments:
return sum(total for _, _, total in self._measured_segments())
width = measure_text_cached(self._font, self._idle_text, FONT_SIZE).x
if self.icon:
width += ICON_SIZE + ICON_PADDING
return width
def _measured_segments(self):
"""[(segment, text width, total width incl. icon and gap)]"""
out = []
for i, seg in enumerate(self.segments or []):
text_width = min(measure_text_cached(self._font, seg[0], FONT_SIZE).x, SEGMENT_NAME_MAX_WIDTH)
total = text_width + (ICON_PADDING + ICON_SIZE if seg[2] else 0) + (SEGMENT_GAP if i else 0)
out.append((seg, text_width, total))
return out
def _render(self, rect: rl.Rectangle):
if self.downloading:
self._render_downloading(rect)
@@ -134,6 +151,8 @@ class DownloadStatusAction(ItemAction):
def _render_downloading(self, rect: rl.Rectangle):
percent = f"{int(self._progress.x)}%"
if self.status_text:
percent = f"{self.status_text} {percent}"
text_height = measure_text_cached(self._font, percent, FONT_SIZE).y
top = rect.y + (rect.height - (text_height + BAR_GAP + BAR_HEIGHT)) / 2
@@ -148,6 +167,9 @@ class DownloadStatusAction(ItemAction):
self._draw_fill(rail, max(0.0, min(rect.width, rect.width * (self._progress.x / 100.0))))
def _render_idle(self, rect: rl.Rectangle):
if self.segments:
self._render_segments(rect)
return
text = self._idle_text
text_size = measure_text_cached(self._font, text, FONT_SIZE)
right = rect.x + rect.width
@@ -161,6 +183,29 @@ class DownloadStatusAction(ItemAction):
rl.draw_text_ex(self._font, text, rl.Vector2(right - text_size.x, rect.y + (rect.height - text_size.y) / 2),
FONT_SIZE, 0, self.text_color)
def _render_segments(self, rect: rl.Rectangle):
measured = self._measured_segments()
while len(self._segment_labels) < len(measured):
self._segment_labels.append(UnifiedLabelSP("", font_size=FONT_SIZE, max_width=SEGMENT_NAME_MAX_WIDTH,
scroll=True, wrap_text=False))
x = rect.x + rect.width - sum(total for _, _, total in measured)
for i, ((text, color, icon, icon_color), text_width, _) in enumerate(measured):
if i:
x += SEGMENT_GAP
label = self._segment_labels[i]
if label.text != text:
label.set_text(text)
label.set_text_color(color)
text_height = measure_text_cached(self._font, text, FONT_SIZE).y
label.set_position(x, rect.y + (rect.height - text_height) / 2)
label.render()
x += text_width
if icon:
texture = gui_app.texture(icon, ICON_SIZE, ICON_SIZE, keep_aspect_ratio=True)
rl.draw_texture_v(texture, rl.Vector2(x + ICON_PADDING, rect.y + (rect.height - texture.height) / 2),
icon_color or color)
x += ICON_PADDING + ICON_SIZE
def download_status_item(title):
return ListItemSP(title=title, action_item=DownloadStatusAction(), title_color=style.ITEM_TEXT_COLOR)
+3 -3
View File
@@ -23,9 +23,9 @@ def post_stream_request(body: StreamRequestBody) -> dict:
ret["time"] = (t_end - t_start) * 1000
return ret
except requests.ConnectTimeout as e:
raise Exception("webrtc took too long to respond.") from e
raise Exception("device took too long to respond.") from e
except requests.ConnectionError as e:
raise Exception("webrtc server on device is not running.") from e
raise Exception("turn car ignition off to use livestreaming.") from e
def wait_for_webrtcd(max_retries: float = 10) -> None:
@@ -37,4 +37,4 @@ def wait_for_webrtcd(max_retries: float = 10) -> None:
except requests.ConnectionError:
attempts += 1
time.sleep(0.5)
raise TimeoutError("webrtcd did not initialize in time.")
raise TimeoutError("livestreaming service did not initialize in time.")
+27 -4
View File
@@ -21,10 +21,16 @@ from typing import Any
from openpilot.system.webrtc.helpers import StreamRequestBody
from openpilot.system.webrtc.schema import generate_field
from openpilot.common.params import Params
from openpilot.common.swaglog import cloudlog
from openpilot.cereal import messaging, log
SESSION_TIMEOUT_SECONDS = 300
# ice candidate parser for logging
def _ice_candidates(sdp: str) -> list[str]:
return [line.removeprefix("a=") for line in sdp.splitlines() if line.startswith("a=candidate:")]
# socket trick: route lookup for 8.8.8.8 (nothing is sent or actually connected to)
# return the source interfaces IP which is the default interface of the device
def _default_route_ip() -> str | None:
@@ -253,7 +259,7 @@ class StreamSession:
self._cleanup_lock = asyncio.Lock()
self._cleanup_done = False
self.logger = logging.getLogger("webrtcd")
self.logger.info(
cloudlog.warning(
"New stream session (%s), video cameras %s, video enabled %s, incoming services %s, outgoing services %s",
self.identifier, [t.id for t in self.video_tracks], body.enabled, body.bridge_services_in, body.bridge_services_out,
)
@@ -329,9 +335,12 @@ class StreamSession:
async def run(self):
try:
self.params.put("LivestreamRequestKeyframe", True)
# avoid datachannel race by adding messange_handler immediately
self.stream.set_message_handler(self.message_handler)
await asyncio.wait_for(self.stream.wait_for_connection(), timeout=15)
if self.stream.has_messaging_channel():
self.stream.set_message_handler(self.message_handler)
if self.incoming_bridge is not None:
await self.shared_pub_master.add_services_if_needed(self.incoming_bridge_services)
if self.outgoing_bridge is not None:
@@ -341,14 +350,18 @@ class StreamSession:
if self.bitrate_controller is not None:
self.bitrate_controller.start()
self.logger.info("Stream session (%s) connected", self.identifier)
with cloudlog.ctx(session_id=self.identifier):
cloudlog.warning("webrtcd.session.connected")
if self.is_body:
await self.run_body_session()
else:
await self.run_normal_session()
self.logger.info("Stream session (%s) ended", self.identifier)
with cloudlog.ctx(session_id=self.identifier):
cloudlog.warning("webrtcd.session.ended")
except Exception:
self.logger.exception("Stream session failure")
with cloudlog.ctx(session_id=self.identifier):
cloudlog.exception("webrtcd.session.exception")
finally:
await self.post_run_cleanup()
@@ -422,15 +435,25 @@ async def handle_get_stream(state: ServerState, raw_body: bytes, content_type: s
stream_dict[session.identifier] = session
try:
answer = await asyncio.wait_for(session.get_answer(), timeout=30)
cloudlog.event(
"webrtcd.session.ice_candidates",
session_id=session.identifier,
offer_candidates=_ice_candidates(body.sdp),
answer_candidates=_ice_candidates(answer.sdp),
)
except TimeoutError:
await session.stop()
stream_dict.pop(session.identifier, None)
logging.getLogger("webrtcd").exception("Timed out creating stream answer")
with cloudlog.ctx(session_id=session.identifier):
cloudlog.warning("webrtcd.session.answer_timeout")
raise
except Exception:
await session.stop()
stream_dict.pop(session.identifier, None)
logging.getLogger("webrtcd").exception("Failed to create stream answer")
with cloudlog.ctx(session_id=session.identifier):
cloudlog.exception("webrtcd.session.answer_exception")
raise
session.start()
-260
View File
@@ -1,260 +0,0 @@
#!/usr/bin/env bash
set -e
# Default values
DEFAULT_REPO_URL="https://github.com/sunnypilot"
START_AT_BOOT=false
RESTORE_MODE=false
RUNNER_VERSION="2.325.0"
# Parse command line arguments
while [[ $# -gt 0 ]]; do
case $1 in
--start-at-boot)
START_AT_BOOT=true
shift
;;
--token)
GITHUB_TOKEN="$2"
shift 2
;;
--repo)
REPO_URL="$2"
shift 2
;;
--restore)
RESTORE_MODE=true
shift
;;
*)
if [ -z "$GITHUB_TOKEN" ]; then
GITHUB_TOKEN="$1"
elif [ -z "$REPO_URL" ]; then
REPO_URL="$1"
fi
shift
;;
esac
done
# Determine BASE_DIR based on mount point
if mountpoint -q /data/media; then
BASE_DIR="/data/media/0/github"
else
BASE_DIR="/data/github"
fi
# Constants
RUNNER_USER="github-runner"
USER_GROUPS="comma,gpu,gpio,sudo"
RUNNER_DIR="${BASE_DIR}/runner"
BUILDS_DIR="${BASE_DIR}/builds"
LOGS_DIR="${BASE_DIR}/logs"
CACHE_DIR="${BASE_DIR}/cache"
OPENPILOT_DIR="${BASE_DIR}/openpilot"
# Basic utility functions (no dependencies)
remount_rw() {
sudo mount -o remount,rw /
}
remount_ro() {
sync || true # Try to sync but continue even if it fails
sudo mount -o remount,ro / # Always try to remount as read-only
}
# Always ensure we try to remount as read-only on exit
trap remount_ro EXIT
setup_runner_user() {
sudo useradd --comment 'GitHub Runner' --create-home --home-dir ${BASE_DIR} ${RUNNER_USER} --shell /bin/bash -G ${USER_GROUPS} || sudo usermod -aG ${USER_GROUPS} ${RUNNER_USER}
}
create_sudoers_entry() {
sudo grep -qxF "${RUNNER_USER} ALL=(ALL) NOPASSWD: ALL" /etc/sudoers || echo "${RUNNER_USER} ALL=(ALL) NOPASSWD: ALL" | sudo tee -a /etc/sudoers
}
set_directory_permissions() {
sudo chown -R ${RUNNER_USER}:comma "$BASE_DIR"
sudo chmod -R g+rwx "$BASE_DIR"
sudo find "$BASE_DIR" -type d -exec chmod g+s {} +
}
setup_directories() {
echo "Creating necessary directories..."
sudo mkdir -p "$RUNNER_DIR" "$BUILDS_DIR" "$LOGS_DIR" "$CACHE_DIR" "$OPENPILOT_DIR"
mkdir -p "/data/openpilot"
sudo chown -R comma:comma "/data/openpilot"
sync
}
wipe_bash_logout() {
export BASE_DIR
sudo -u ${RUNNER_USER} bash -c "touch ${BASE_DIR}/.bash_logout"
sudo -u ${RUNNER_USER} bash -c "truncate -s 0 '${BASE_DIR}/.bash_logout'"
}
# System configuration functions (depends on basic utility functions)
setup_system_configs() {
echo "Setting up system configurations..."
remount_rw
setup_runner_user
create_sudoers_entry
remount_ro
set_directory_permissions
wipe_bash_logout
}
# Runner setup functions
install_runner() {
echo "Downloading and setting up runner..."
cd "$RUNNER_DIR"
curl -o actions-runner-linux-arm64-${RUNNER_VERSION}.tar.gz -L https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-arm64-${RUNNER_VERSION}.tar.gz
sudo -u ${RUNNER_USER} tar -xzf ./actions-runner-linux-arm64-${RUNNER_VERSION}.tar.gz
sudo rm ./actions-runner-linux-arm64-${RUNNER_VERSION}.tar.gz
sudo chmod +x ./config.sh
}
configure_runner() {
remount_rw
echo "Configuring runner..."
cd "$RUNNER_DIR"
sudo -u ${RUNNER_USER} ./config.sh --url "$REPO_URL" --token "$GITHUB_TOKEN" --name $(hostname) --runnergroup "tici-tizi" --labels "tici" --work "$BUILDS_DIR" --unattended
remount_ro
}
create_service_template() {
echo "Creating service template..."
cat <<EOL > "$RUNNER_DIR/bin/actions.runner.service.template"
[Unit]
Description={{Description}}
After=network-online.target nss-lookup.target time-sync.target
Wants=network-online.target nss-lookup.target time-sync.target
StartLimitInterval=5
StartLimitBurst=10
[Service]
Type=simple
User=root
ExecStart=/usr/bin/unshare -m -- /bin/bash -c 'mount --bind ${OPENPILOT_DIR} /data/openpilot && setpriv --reuid={{User}} --regid={{User}} --init-groups env HOME=${BASE_DIR} USER={{User}} LOGNAME={{User}} MAIL=/var/mail/{{User}} {{RunnerRoot}}/runsvc.sh'
WorkingDirectory={{RunnerRoot}}
KillMode=process
KillSignal=SIGTERM
TimeoutStopSec=5min
Restart=always
RestartSec=120
[Install]
WantedBy=multi-user.target
EOL
}
install_service() {
local service_name
if [ -f "${RUNNER_DIR}/.service" ]; then
service_name=$(cat "${RUNNER_DIR}/.service")
else
service_name="actions.runner.sunnypilot.$(uname -n)"
fi
create_service_template
remount_rw
local service_path="/etc/systemd/system/${service_name}"
echo "Installing systemd service..."
if [ -f "${service_path}" ]; then
echo "Service ${service_path} found in systemd, we will delete it"
sudo rm -f "${service_path}"
fi
cd "$RUNNER_DIR"
sudo ./svc.sh install $RUNNER_USER
if [ "$START_AT_BOOT" = false ]; then
sudo systemctl disable "${service_name}"
fi
remount_ro
}
check_restore_prerequisites() {
local can_restore=false
local service_name=""
# Check if base runner directory exists
if [ ! -d "${RUNNER_DIR}" ]; then
echo "ERROR: Runner directory ${RUNNER_DIR} does not exist"
echo "This directory is required for restore operations"
exit 1
fi
# First check if we have the required files for restoration
if [ -f "${RUNNER_DIR}/.credentials" ] && [ -f "${RUNNER_DIR}/.service" ]; then
can_restore=true
service_name=$(cat "${RUNNER_DIR}/.service")
echo "Found required runner configuration files"
else
echo "Missing required runner configuration files"
echo "Required: .credentials and .service files in ${RUNNER_DIR}"
exit 1
fi
if ! id "${RUNNER_USER}" &>/dev/null; then
echo "User ${RUNNER_USER} does not exist"
fi
# Only proceed if we can restore AND need to restore
if [ "$can_restore" = true ]; then
echo "Restoration is possible"
return 0
else
echo "No restoration possible"
exit 0
fi
}
perform_restore() {
echo "Starting runner restoration..."
setup_directories
setup_system_configs
install_service
echo "Runner restoration completed successfully"
}
perform_install() {
echo "Starting fresh installation..."
setup_directories
setup_system_configs
install_runner
set_directory_permissions
configure_runner
install_service
echo "Installation completed successfully"
}
main() {
if [ "$RESTORE_MODE" = true ]; then
echo "Running in restore mode - will only restore system configurations..."
check_restore_prerequisites
perform_restore
else
# Check required arguments for normal installation
if [ -z "$GITHUB_TOKEN" ]; then
echo "Usage: $0 [--start-at-boot] [--token <github_token>] [--repo <repository_url>] [--restore]"
echo "Required argument (except for --restore): github_token"
echo "Optional arguments:"
echo " --start-at-boot Enable auto-start at boot (default: false)"
echo " --repo Repository URL (default: ${DEFAULT_REPO_URL})"
echo " --restore Restore existing runner configuration"
exit 1
fi
# Set repository URL if not provided
REPO_URL="${REPO_URL:-$DEFAULT_REPO_URL}"
perform_install
fi
echo "Starting runner service..."
cd "$RUNNER_DIR"
sudo ./svc.sh start
}
main
+37 -12
View File
@@ -53,24 +53,28 @@ def create_pkl_name(full_name: str) -> str:
return pkl
def _read_pkl_bytes(pkl_path: Path) -> bytes:
def _hash_pkl(pkl_path: Path) -> str:
manifest = Path(f"{pkl_path}.chunkmanifest")
if manifest.exists():
num_chunks = int(manifest.read_text().strip())
parts = []
for i in range(num_chunks):
chunk = Path(f"{pkl_path}.chunk{i + 1:02d}of{num_chunks:02d}")
parts.append(chunk.read_bytes())
return b''.join(parts)
return pkl_path.read_bytes()
paths = [Path(f"{pkl_path}.chunk{i + 1:02d}of{num_chunks:02d}") for i in range(num_chunks)]
else:
paths = [pkl_path]
digest = hashlib.sha256()
for path in paths:
with path.open('rb') as f:
while block := f.read(1024 * 1024):
digest.update(block)
return digest.hexdigest()
def _find_driving_pkl(output_path: Path) -> Path | None:
for pattern in ('driving_tinygrad.pkl', 'driving_*_tinygrad.pkl'):
for pattern in ('*driving_tinygrad.pkl', '*driving_*_tinygrad.pkl'):
matches = sorted(output_path.glob(pattern))
if matches:
return matches[0]
for pattern in ('driving_tinygrad.pkl.chunkmanifest', 'driving_*_tinygrad.pkl.chunkmanifest'):
for pattern in ('*driving_tinygrad.pkl.chunkmanifest', '*driving_*_tinygrad.pkl.chunkmanifest'):
matches = sorted(output_path.glob(pattern))
if matches:
return Path(str(matches[0]).removesuffix('.chunkmanifest'))
@@ -86,8 +90,20 @@ def _rename_pkl_with_chunks(old_pkl: Path, new_pkl: Path) -> Path:
return old_pkl.rename(new_pkl)
def _hash_onnx_files(model_dir: Path) -> str | None:
onnx_files = sorted(model_dir.glob("*.onnx"))
if not onnx_files:
return None
digest = hashlib.sha256()
for f in onnx_files:
with f.open('rb') as fh:
while block := fh.read(1024 * 1024):
digest.update(block)
return digest.hexdigest()
def generate_chunked_model(driving_pkl: Path) -> dict:
tinygrad_hash = hashlib.sha256(_read_pkl_bytes(driving_pkl)).hexdigest()
tinygrad_hash = _hash_pkl(driving_pkl)
chunks_config = []
manifest_file = Path(f"{driving_pkl}.chunkmanifest")
@@ -119,7 +135,8 @@ def generate_chunked_model(driving_pkl: Path) -> dict:
}
def create_metadata_json(models: list, output_dir: Path, custom_name=None, short_name=None, is_20hz=False, upstream_branch="unknown") -> None:
def create_metadata_json(models: list, output_dir: Path, custom_name=None, short_name=None, is_20hz=False, upstream_branch="unknown",
onnx_sha256=None, is_big=False) -> None:
bundle_json = {
"short_name": short_name,
"display_name": custom_name or upstream_branch,
@@ -132,9 +149,13 @@ def create_metadata_json(models: list, output_dir: Path, custom_name=None, short
"generation": "-1",
"build_time": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ"),
"overrides": {},
"is_big": is_big,
"models": models,
}
if onnx_sha256:
bundle_json["onnx_sha256"] = onnx_sha256
# Write metadata to output_dir
metadata_json = {
"bundles": [bundle_json]
@@ -166,6 +187,8 @@ if __name__ == "__main__":
print(f"No driving_tinygrad.pkl found in {_output_dir}", file=sys.stderr)
sys.exit(1)
is_big = _driving_pkl.name.startswith('big_')
if _pkl:
new_pkl = _output_dir / f"driving_{_pkl}_tinygrad.pkl"
if not new_pkl.exists():
@@ -174,4 +197,6 @@ if __name__ == "__main__":
_driving_pkl = new_pkl
_model_metadata = generate_chunked_model(_driving_pkl)
create_metadata_json([_model_metadata], _output_dir, args.custom_name, _short_name, args.is_20hz, args.upstream_branch)
_onnx_sha256 = _hash_onnx_files(Path(args.model_dir))
create_metadata_json([_model_metadata], _output_dir, args.custom_name, _short_name, args.is_20hz, args.upstream_branch,
onnx_sha256=_onnx_sha256, is_big=is_big)
+1 -1
View File
@@ -47,7 +47,7 @@ git rm -rf $OUTPUT_DIR/.git || true # Doing cleanup, but it might fail if the .g
git remote remove origin || true # ensure cleanup
git remote add origin $GIT_ORIGIN
#git push origin -d $DEV_BRANCH || true # Ensuring we delete the remote branch if it exists as we are wiping it out
git fetch origin $DEV_BRANCH || (git checkout -b $DEV_BRANCH && git commit --allow-empty -m "sunnypilot v$VERSION release" && git push -u origin $DEV_BRANCH)
git fetch --depth 1 origin $DEV_BRANCH || (git checkout -b $DEV_BRANCH && git commit --allow-empty -m "sunnypilot v$VERSION release" && git push -u origin $DEV_BRANCH)
echo "[-] committing version $VERSION T=$SECONDS"
git add -f .
-66
View File
@@ -1,66 +0,0 @@
#!/usr/bin/env bash
# Determine BASE_DIR based on mount point
if mountpoint -q /data/media; then
GITHUB_BASE_DIR="/data/media/0/github"
else
GITHUB_BASE_DIR="/data/github"
fi
# Define directories and user
BIN_DIR="$GITHUB_BASE_DIR/bin"
BUILDS_DIR="$GITHUB_BASE_DIR/builds"
OPENPILOT_DIR="$GITHUB_BASE_DIR/openpilot"
LOGS_DIR="$GITHUB_BASE_DIR/logs"
CACHE_DIR="$GITHUB_BASE_DIR/cache"
RUNNER_USERNAME="github-runner"
# Define the systemd service name
SERVICE_NAME="github-runner"
USER_GROUPS="comma,gpu,gpio,sudo"
# Function to stop and disable the systemd service
stop_and_uninstall_service() {
cd $GITHUB_BASE_DIR/runner
sudo ./svc.sh stop
sudo ./svc.sh uninstall
}
# Function to remove the systemd service file
remove_runner() {
cd $GITHUB_BASE_DIR/runner
sudo rm .runner
sudo su -c './config.sh remove' github-runner
}
# Function to delete the Github Runner directories
delete_directories() {
sudo rm -rf "$BIN_DIR/github-runner"
sudo rm -rf "$GITHUB_BASE_DIR" "$BIN_DIR" "$BUILDS_DIR" "$LOGS_DIR" "$CACHE_DIR" "$OPENPILOT_DIR"
}
# Function to remove the Github Runner user
delete_user() {
for group in ${USER_GROUPS//,/ }
do
sudo gpasswd -d ${RUNNER_USERNAME} ${group}
done
sudo userdel -r ${RUNNER_USERNAME}
}
# Function to remove sudoers entry
remove_sudoers_entry() {
sudo sed -i.bak "/${RUNNER_USERNAME} ALL=(ALL) NOPASSWD: ALL/d" /etc/sudoers
}
# Make filesystem writable
sudo mount -o remount rw /
# Ensure filesystem is remounted as read-only on script exit
trap "sudo mount -o remount ro /" EXIT
# Call functions
stop_and_uninstall_service
remove_runner
delete_directories
delete_user
remove_sudoers_entry
# End of uninstall script
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env python3
"""
Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
import argparse
import hashlib
import json
import tempfile
from huggingface_hub import HfApi, hf_hub_download
def hash_file(path: str) -> str:
digest = hashlib.sha256()
with open(path, 'rb') as f:
while block := f.read(1024 * 1024):
digest.update(block)
return digest.hexdigest()
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--hf-repo", required=True)
parser.add_argument("--hf-defaults-path", required=True)
parser.add_argument("--artifact-name", required=True)
parser.add_argument("--model-dir", required=True)
parser.add_argument("--onnx-path", required=True)
parser.add_argument("--onnx-ref", required=True)
parser.add_argument("--model-name", required=True)
parser.add_argument("--tinygrad-ref", required=True)
parser.add_argument("--run-number", required=True)
args = parser.parse_args()
api = HfApi()
onnx_sha256 = hash_file(args.onnx_path)
short_ref = args.onnx_ref[:8]
folder_name = f"model-{args.model_name}-{short_ref}-{args.run_number}"
print(f"ONNX hash: {onnx_sha256}")
print(f"ONNX ref: {args.onnx_ref} (short: {short_ref})")
print(f"Folder: {folder_name}")
metadata_path = f"{args.model_dir}/metadata.json"
with open(metadata_path) as f:
metadata = json.load(f)
bundle = metadata['bundles'][0]
bundle['display_name'] = args.model_name
bundle['onnx_sha256'] = onnx_sha256
bundle['onnx_ref'] = args.onnx_ref
artifact = bundle['models'][0]['artifact']
hf_base = f"https://huggingface.co/datasets/{args.hf_repo}/resolve/main/{args.hf_defaults_path}/{folder_name}"
artifact['download_uri']['url'] = f"{hf_base}/{artifact['file_name']}"
for chunk in artifact.get('chunks', []):
chunk['url'] = f"{hf_base}/{chunk['file_name']}"
print(f"Uploading model to {args.hf_defaults_path}/{folder_name}/")
api.upload_folder(
folder_path=args.model_dir,
path_in_repo=f"{args.hf_defaults_path}/{folder_name}",
repo_id=args.hf_repo,
repo_type="dataset",
)
json_filename = f"{args.hf_defaults_path}/default_models.json"
try:
local_path = hf_hub_download(repo_id=args.hf_repo, repo_type='dataset', filename=json_filename)
with open(local_path) as f:
defaults_json = json.load(f)
except Exception:
defaults_json = {"tinygrad_ref": args.tinygrad_ref, "bundles": []}
defaults_json['tinygrad_ref'] = args.tinygrad_ref
existing_idx = next((i for i, b in enumerate(defaults_json['bundles'])
if b.get('onnx_sha256') == onnx_sha256), None)
if existing_idx is not None:
defaults_json['bundles'][existing_idx] = bundle
else:
defaults_json['bundles'].append(bundle)
print(json.dumps(defaults_json, indent=2))
with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f:
json.dump(defaults_json, f, indent=2)
tmp_path = f.name
api.upload_file(
path_or_fileobj=tmp_path,
path_in_repo=json_filename,
repo_id=args.hf_repo,
repo_type="dataset",
)
print(f"Updated {json_filename}")
if __name__ == "__main__":
main()
File diff suppressed because it is too large Load Diff
File diff suppressed because one or more lines are too long