Compare commits

...

302 Commits

Author SHA1 Message Date
royjr 5360beb1ef Merge branch 'sync-20260902' into ccnc-port-sync-20260902 2026-09-02 13:24:57 -04:00
Jason Wen d0ab77cf2e Merge commit '6249f4d5b0e63c05f08bce12ca3afebda9f764a3' into sync-20260902
# Conflicts:
#	openpilot/selfdrive/modeld/SConscript
#	openpilot/selfdrive/modeld/modeld.py
#	openpilot/selfdrive/pandad/pandad.cc
#	openpilot/selfdrive/selfdrived/alerts_offroad.json
#	openpilot/selfdrive/ui/layouts/onboarding.py
#	openpilot/selfdrive/ui/mici/layouts/home.py
#	openpilot/system/hardware/hardwared.py
#	panda
#	tinygrad_repo
2026-09-02 13:17:53 -04:00
Jason Wen 47db84ebfb models: add big model ONNX hash tracking (#1982) 2026-09-02 01:28:43 -04:00
Jason Wen 68be777395 bump tg 2026-09-01 22:14:43 -04:00
github-actions[bot] ab389498a8 [bot] Update Python packages (#1950)
* Update Python packages

* bump tg

* bump

* ci: route build_model runner by target_hardware instead of hardcoding chestnut

* hack, remove before merge

* Revert build-model runner hack and uv.lock update

* why were they hard coded

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2026-09-01 22:12:54 -04:00
Daniel Koepping 6249f4d5b0 AGNOS 19.7 (#38750) 2026-09-01 18:32:59 -07:00
Daniel Koepping 8b88f7dd6e ui: show one GPU status (#38748)
ui: show one GPU status icon
2026-09-01 18:32:39 -07:00
Harald Schäfer 79658800ce cereal: log big model in drivingModelData (#38747) 2026-09-01 17:15:13 -07:00
Daniel Koepping 36561258fa ui: show usb connection (#38745)
* ui: show USB status

* ui: resize USB icon

* ui: classify USB device once

* ui: debounce USB disconnect
2026-09-01 15:45:57 -07:00
YassineYousfi cb85ac1f0e amd warp (#38684)
* modeld: fuse warp and policy TinyJit

* bump tg

* fix?

* this simple trick...

* debug 1

* bump tg

* pack all

* wips

* fix

* BIG_INTO_SMALL remove

* slower
2026-09-01 13:59:55 -07:00
Daniel Koepping c9f1602040 Revert "monitor chestnut USB in hardwared (#38741)" (#38744)
This reverts commit 7d5596d5c3.
2026-09-01 11:13:30 -07:00
Daniel Koepping 06af2abe67 modeld: wait for stable chestnut (#38742)
modeld: wait for stable chestnut
2026-09-01 07:20:56 -07:00
Daniel Koepping 7d5596d5c3 monitor chestnut USB in hardwared (#38741)
hardwared: monitor chestnut USB independently
2026-09-01 05:59:02 -07:00
YassineYousfi a2e422eee0 TGC (#38739)
* 23e6a04e-e6e5-462b-a0bb-e4088275ee43/12864 tgc

* here
2026-08-31 22:30:28 -07:00
Jason Wen 51987a62d0 ci: route build_model runner by hardware type 2026-09-01 01:16:04 -04:00
XiaoXX e10c0fd960 modem.py: accept hex chars in ICCID (#38735)
E.118 specifies decimal digits, but many real SIMs carry hex characters
in EF_ICCID (e.g. China Mobile's 898600B5... range, some MVNO/IoT SIMs).
AT+QCCID returns them verbatim, and the strict isdigit() check blanked
the ICCID, leaving the modem daemon stuck in INITIALIZING forever and
cellular dead. ModemManager parses ICCID as hex for the same reason.

Verified on a comma four with a China Mobile SIM (EG916Q-GL): previously
stuck retrying 'identity read incomplete', now dials and passes traffic.
2026-08-31 21:35:56 -07:00
James Vecellio-Grant 98ed8111f6 modeld_v2: big to small model fallback (#1974) 2026-09-01 00:15:23 -04:00
Trey Moen da8ce858ec ui(mici): name updater signal constants (#38731)
* mici: name updater signal constants

* drop SIGNAL_ prefix

* self contained

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
2026-08-31 15:55:16 -07:00
Trey Moen 9fa7ef3d17 ui: clarify branch switcher error message (#38732) 2026-08-31 15:46:52 -07:00
Harald Schäfer 4adbb85742 BMRLNAP (#38681) 2026-08-31 09:25:32 -07:00
Robbe Derks 70df7f227b bump panda (new health packet) (#38736)
pandad: support compact health packet
2026-08-31 14:01:20 +02:00
royjr 50d7f75bfc Merge branch 'master' into ccnc-port 2026-08-30 16:21:17 -04:00
royjr de197ba6fa chestnut: alert when big model ready (#1947)
egpu: alert when big model ready

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2026-08-30 16:20:16 -04:00
Trey Moen 0e32059484 replay: capture downloader's stderr so download progress is reported again (#38734) 2026-08-30 09:32:36 -07:00
Trey Moen 7cf55c3b7a common: fix OpenpilotPrefix cleanup on macOS (#38728)
The destructor built its cleanup commands as "rm %s -rf", with the flags
after the operand. GNU rm permutes arguments so this works on device and
in CI, but BSD rm on macOS stops option parsing at the first operand and
treats "-rf" as a second filename:

  $ mkdir -p /tmp/rmtest/sub && rm /tmp/rmtest -rf
  rm: /tmp/rmtest: is a directory
  rm: -rf: No such file or directory
  exit=1

So nothing is removed, and each of the four calls prints two errors plus
"system command failed (256)" from check_system. Every run of a tool that
owns an OpenpilotPrefix (replay, cabana) leaks its params dir, its
comma_home and its /tmp/msgq_ dir; 33 of each had accumulated on my
machine.

Pass the flags first.
2026-08-28 22:11:52 -07:00
royjr 15c7f52e40 Merge branch 'master' into ccnc-port 2026-08-28 20:28:33 -04:00
Daniel Koepping 682b6a20df add chestnut offroad alerts (#38706)
* system: add chestnut offroad alerts

* system: refine chestnut offroad alerts

* system: refine chestnut power alerts

* system: confirm chestnut power recovery from PCIe

* system: detect missing chestnut power from INA voltage
2026-08-28 15:46:56 -07:00
Daniel Koepping a67cdf9a51 ui: sync gpu loading to offroad (#38727)
ui: sync gpu loading state
2026-08-28 15:08:18 -07:00
Trey Moen e571e21d14 ui: check for update on target branch switch (#38693) 2026-08-28 12:07:15 -07:00
Trey Moen 839d3f5004 ui: guard branch switcher before internet connected (#38692) 2026-08-28 12:06:33 -07:00
Trey Moen 5645370f84 cabana: split utils/util into Qt-free util and qtutil (#38723) 2026-08-28 11:37:09 -07:00
Trey Moen 633d17cd12 ui: fix install update button overflow (#38696) 2026-08-28 11:30:13 -07:00
Trey Moen 5419f57b3a cabana: de-QT streams (#38718) 2026-08-28 10:18:10 -07:00
Jason Wen 1dd5a7c91d Sync: commaai/openpilot:mastersunnypilot/sunnypilot:master (#1973) 2026-08-28 12:58:26 -04:00
Trey Moen 46f612224c cabana: split comma API route fetching out of RoutesDialog (#38721) 2026-08-28 09:57:47 -07:00
Trey Moen 6e0f4f4630 cabana: split SettingsDialog out of settings (#38719)
cabana: split SettingsDialog out of settings.{h,cc}
2026-08-28 09:55:28 -07:00
Trey Moen 0f9c753e6e cabana: remove Qt from livestream (#38722) 2026-08-28 09:46:45 -07:00
nayan acb784d207 Merge commit '4a13639cfd122ccb9113a4d6ce225dcbd8e61914' into sync-20260827
# Conflicts:
#	openpilot/selfdrive/modeld/SConscript
#	openpilot/selfdrive/modeld/modeld.py
#	openpilot/selfdrive/ui/mici/layouts/home.py
#	openpilot/selfdrive/ui/ui_state.py
#	tinygrad_repo
2026-08-28 12:39:53 -04:00
Trey Moen 131e473f37 cabana: move stream open widgets into streamselector (#38715) 2026-08-28 09:36:26 -07:00
Trey Moen 30f358eb59 cabana: use std::string in RoutesDialog API results (#38717) 2026-08-28 07:25:34 -07:00
Trey Moen 9b9e3ea604 cabana: string helpers in utils return std::string (#38720) 2026-08-28 07:25:11 -07:00
Trey Moen 7cc48b5bc9 cabana: move RoutesDialog out of streams/ (#38716) 2026-08-27 22:00:24 -07:00
Trey Moen cbf750de20 cabana: replace custom non-view Qt signals w/ plain observer (#38713) 2026-08-27 18:54:06 -07:00
Trey Moen 318257fa3b bump raylib (#38712) 2026-08-27 11:38:53 -07:00
Daniel Koepping 4cdc16031f log chestnut supply fault (#38711)
* log chestnut INA supply fault

* ci
2026-08-27 11:21:56 -07:00
Trey Moen 31ea1850f7 ui: remove raygui usage (#38708)
* ui: remove raygui usage

* match previous gui_text_box line spacing

* Revert "match previous gui_text_box line spacing"

This reverts commit ffd2fe31725c6d50bffaebc621c1e170d0926c66.

* Reapply "match previous gui_text_box line spacing"

This reverts commit d41404f09607e225f43868f7747f22dc0bb2cf16.
2026-08-27 10:54:52 -07:00
Nayan 4075befc5e osm: support map deletion via sunnylink (#1971)
delete delete
2026-08-27 11:23:26 -04:00
Jason Wen 9f43d2477d [MICI] ui: move and restyle the sunnylink pill in settings (#1972) 2026-08-27 03:52:57 -04:00
Nayan 2d6cc4c065 models: Model Selector upgrades (#1953)
* uh, i did not commit anything all this time

* slideee to the left, cha cha

* lint lint

* ui: unify model source predicate and per-source bundle lookup in model_info

* [TIZI/TICI] ui: disable the other-model row onroad like the active row

* [TIZI/TICI] ui: drop docstring that restates the function name

* [TIZI/TICI] ui: keep Favorites as the first model folder in the picker

* ui: record why model names read the params slots and not modelManagerSP

* ui: show the default model's name on the picker Default entries

* models: bind a download to its ref so cancel and reselect work everywhere

* models: resume partial chunked downloads and verify silently

* models: publish a verifying status so cached checks read as verification, not a stuck download

* [TIZI/TICI] ui: move download status onto each model's own row

* [TIZI/TICI] ui: show the row status description while it has text

* [TIZI/TICI] ui: restore the Model Status bar row

* models: a cancel interrupts verification immediately and keeps on-disk chunks

* models: a selection made mid-download queues instead of cancelling the transfer

* [TIZI/TICI] ui: Model Status shows both slots idle and the queued pick while busy

* [TIZI/TICI] ui: label the Model Status slots small and big and scroll long names

* models: start a queued download in the same tick and label empty slots (Default)

* ui: scroll Model Status names at the corrected speed

* [TIZI/TICI] ui: Model Status shows the big model failing over to small

* [TIZI/TICI] ui: stable model rows and a runner-matched failover note on Model Status

* [TIZI/TICI] ui: model rows show full names and the failover note reopens with the page

* ui: name the actually driving model runner-matched and bring mici to state parity

* fix ugly

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
Co-authored-by: James Vecellio-Grant <159560811+Discountchubbs@users.noreply.github.com>
2026-08-27 02:03:53 -04:00
Daniel Koepping 4a13639cfd reduce chestnut states (#38705)
ui: unify chestnut status presentation
2026-08-26 19:12:49 -07:00
YassineYousfi fa75fdd852 chestnut stats: overlap with gpu work (#38704)
* chestnut stats: overlap with gpu work

* ci

---------

Co-authored-by: elkoled <elkoled@gmail.com>
2026-08-26 17:57:55 -07:00
Daniel Koepping 5cfdb2f4da rename usbgpu to chestnut (#38703)
chestnut: rename eGPU interfaces
2026-08-26 15:42:59 -07:00
Daniel Koepping 63548ce10d bump tinygrad (#38702) 2026-08-26 15:20:04 -07:00
Daniel Koepping 980fb79c1a update orange GPU icon (#38701)
mici: update failed eGPU icon
2026-08-26 12:21:14 -07:00
Harald Schäfer d40df6f829 modeld: fall back on invalid big model outputs (#38700) 2026-08-26 12:06:53 -07:00
Nayan da28afca91 models: dual-slot backend (qcom/usbgpu) with ref-based downloads (#1966)
* models: dual-slot backend (qcom/usbgpu) with ref-based downloads

* models: restore get_active_source and the usbgpu-to-qcom fallback

* models: fix per-slot validation and cap mismatched-source refetches

* ui/models: select models by ref and seed the usbgpu slot on migration

* models: drop defensive attribute guards on capnp bundles

* models: remove vestigial fetcher state and dead fallbacks

* models: resolve the active bundle from the active source slot only

* models: pass the usbgpu kwarg through the modeld test stubs

* models: resolve the displayed model from the active slot in ui_state

* models: correct the validation memo type hint

* models: drop docstrings that restate the function name

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2026-08-26 02:34:02 -04:00
Jason Wen 15f201caed ui: use full big model failure detection for sidebar and home eGPU icons (#1969) 2026-08-25 21:07:44 -04:00
Jason Wen 1d4558c067 [TIZI/TICI] sidebar: show eGPU icon when chestnut is present (#1968)
* [tizi/tici] sidebar: show eGPU icon when chestnut is present

* matchy match

* fix
2026-08-25 20:47:52 -04:00
Jason Wen 78a766eb61 ui: fix scrolling label speed at non-60fps refresh rates (#1967)
* ui: fix scrolling label speed at non-60fps refresh rates

* send it

* nope

* more
2026-08-25 20:38:02 -04:00
Jason Wen b742b96c44 [MICI] ui: four-state eGPU icon for non-default big models (#1945)
* ui: four-state eGPU icon for non-default big models

* oops

* try this out

* align
2026-08-25 12:04:39 -04:00
Jason Wen 25c25047b8 models: persist model selection per catalog across chestnut state changes (#1960) 2026-08-25 01:12:12 -04:00
Jason Wen cefe5737b9 models: fix current model not updating on chestnut status (#1959)
* models: preserve user model selection across reboots and power cycles

* no

* again

* idk

* over
2026-08-25 00:41:31 -04:00
Jason Wen 760c19d3f9 ui/models: handle missing files during cache size calculation (#1958) 2026-08-24 23:31:52 -04:00
James Vecellio-Grant 45814e3313 modeld_v2: spatial features (#1934)
* modeld_v2: spatial features

* Update fetcher.py

* dont reshape non 4 dim arrays

* realize for non compiled

* Update compile_modeld.py

* god dammit it was realize()

* it was fucking frozen tinygrad. just need to recompile

* bump

* ci: add is_big flag to metadata.json to support backward compat

* Update model_generator.py

* Update sunnypilot-build-model.yaml

* Update helpers.py

* Revert "Update helpers.py"

This reverts commit 3a955ca11a.

* Reapply "Update helpers.py"

This reverts commit ca9c6e1933.

* models: use less strict chestnut detection state

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2026-08-24 22:52:29 -04:00
Jason Wen 2ba91d2be5 ci: add tinygrad ref check to prepare_chestnut and even faster prebuilt stages (#1957)
* ci: faster prebuilt stages

* tg check chestnut

* zoomer!
2026-08-24 22:42:45 -04:00
Jason Wen 19f83b274f ci: identical environment for publish_chestnut prebuilt 2026-08-24 22:06:20 -04:00
Jason Wen d14d0b1dd0 ci: parallelize models chunk downloads and split branch publishing (#1955)
* ci: parallelize model chunk downloads and better publish

* ci: download all model chunks in parallel with xargs -P8

* split split

* ew

* must require
2026-08-24 21:48:53 -04:00
Jason Wen 6cc5f3aad8 ci: fix DM model build, separate HF defaults paths, nuke build races (#1956)
* ci: fix DM model build, separate HF defaults paths, nuke build races

* more split!

* name

* ci: download driving and DM model chunks into chestnut prebuilt output
2026-08-24 20:02:48 -04:00
Jason Wen 8e16c9babb ci: offload small model compilation (#1952)
* ci: compile default big model with stock modeld

* Revert "Revert big RL model (#38627)"

This reverts commit 516ec1e682.

* ci: compile default small model with stock modeld compiler

* ci: offload small model compilation

* Reapply "Revert big RL model (#38627)"

This reverts commit d06cfabb62.

* Reapply "Revert big RL model (#38627)"

This reverts commit d06cfabb62.
2026-08-24 16:24:19 -04:00
Jason Wen 2bcfed5c71 ci: compile default models with stock modeld (#1954)
* ci: compile default big model with stock modeld

* Revert "Revert big RL model (#38627)"

This reverts commit 516ec1e682.

* ci: compile default small model with stock modeld compiler

* Reapply "Revert big RL model (#38627)"

This reverts commit d06cfabb62.
2026-08-24 15:37:46 -04:00
royjr 8dfe04a318 Merge branch 'master' into ccnc-port 2026-08-24 14:19:36 -04:00
Jason Wen 66cf334067 ci: unify default model build into single workflow (#1951)
* ci: unify default model build into single workflow

* ci: consolidate upload jobs and add tinygrad ref validation
2026-08-24 12:35:37 -04:00
Jason Wen 94ed0608e6 models: use less strict chestnut detection state (#1948) 2026-08-24 01:40:31 -04:00
royjr 9648ac2f04 Merge branch 'master' into ccnc-port 2026-08-23 21:48:17 -04:00
Jason Wen 0fbca979df models: show big model list when Chestnut present (#1943) 2026-08-23 20:19:57 -04:00
Jason Wen dcddb2a0bd models: revert icon override from this branch scope 2026-08-23 19:53:46 -04:00
royjr a03a11e333 Merge branch 'master' into ccnc-port 2026-08-23 19:36:45 -04:00
Jason Wen 699eaf7957 include them! 2026-08-23 19:16:15 -04:00
Jason Wen c246e6318a Merge branch 'master' into models-good-detect 2026-08-23 16:56:44 -04:00
Jason Wen 718db8c62e Sync: commaai/openpilot:mastersunnypilot/sunnypilot:master (#1944) 2026-08-23 16:56:00 -04:00
Jason Wen c2214d4c32 Merge commit '084747c75d2cbd23af65ab7a9e770bbd7b98bac9' into sync-20260823
# Conflicts:
#	openpilot/common/params_keys.h
2026-08-23 15:36:54 -04:00
Jason Wen 0de7fbf33d new 2026-08-23 15:06:53 -04:00
royjr 53b1070b09 Merge branch 'master' into ccnc-port 2026-08-23 11:05:40 -04:00
Jason Wen 211f990f6b models: fix sunnylink default model display and false big model re-downloading (#1941)
* big needs small

* no download

* actually

* send it
2026-08-23 04:04:46 -04:00
Jason Wen 97468e4fa4 [TIZI/TICI] ui: remove calibration reset dialog on model change (#1942) 2026-08-23 03:48:52 -04:00
Jason Wen 6c6fba9a14 ci: fix flaky LLK test (#1940) 2026-08-23 02:57:02 -04:00
Jason Wen 34621cf816 ci: refactor big model chunk handling (#1939) 2026-08-23 02:48:10 -04:00
Jason Wen 086530b7c6 [TIZI/TICI] ui: fix path width during gas and steering override (#1938) 2026-08-22 21:47:38 -04:00
royjr 68bf6d8162 Merge branch 'master' into ccnc-port 2026-08-22 13:19:05 -04:00
Jason Wen 4f46433e2b alerts: add branch metadata to chestnut offroad warning (#1936)
* alerts: add branch metadata to chestnut offroad warning

* all branches
2026-08-22 10:20:06 -04:00
Jason Wen 5a8567e3e7 ci: chestnut prebuilt branches (#1935)
* ci: chestnut prebuilt branches

* fix

* nope

* big

* try again

* diff

* malformed

* auth

* more
2026-08-22 03:41:48 -04:00
Jason Wen 07558166c8 ci: only check default model on dispatch 2026-08-22 00:32:35 -04:00
Jason Wen ca9338812e ci: prep for chestnut prebuilts 2026-08-22 00:16:10 -04:00
granolaFPV 4667241fe7 [TIZI/TICI] ui: dynamic path width color (#1926)
* Fix UI path color and thickness based on lateral steering state (Issue #1441)

* Fix UI path color and thickness based on lateral control engagement (Issue #1441)

* Fix UI path width and color based on MADS lateral engagement (Issue #1441)

* Fix UI path width and color based on MADS lateral engagement (Issue #1441)

* move to ModelRendererSP

* match torque bar

* same behavior across the board

* simplify

---------

Co-authored-by: Brennan Browne <brennanbrowne@google.com>
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2026-08-21 20:51:43 -04:00
Shane Smiskol 084747c75d Fix button label widths (#38680)
* Revert "ui: fix text and icon overlap on button (#38628)"

This reverts commit d9c4120f89.

* simple

* can do this

* fix eliding

* Revert "fix eliding"

This reverts commit b271a350182ad87f9942d7363383ee8ec72d0e36.

* clean up

* clean up
2026-08-21 15:41:12 -07:00
royjr 8b472291d0 Merge branch 'master' into ccnc-port 2026-08-21 17:12:13 -04:00
royjr 797c8d4293 Update opendbc_repo 2026-08-21 17:09:31 -04:00
Marceline Milligan a49c260927 ui: show default big model name when eGPU present/active (#1930)
* Name the big default model in the device UI

Build on the default big-model metadata from #1929 and resolve the displayed model from cached capability and modeld runtime state. Keep model selection behavior unchanged.

Assisted-by: GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove get_default_model_label

* simplify

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: nayan <nayan8teen@gmail.com>
2026-08-21 13:26:52 -04:00
Jason Wen 5ad2bfdb75 ci: deprecate GitHub runners (#1933) 2026-08-21 00:35:38 -04:00
Jason Wen b742557d62 sunnylink: add model resolver (#1931)
* models: add get_default_model resolver for sunnylink

* models: move get_default_model to default_model.py
2026-08-20 21:56:57 -04:00
Nayan 5ecd05aedf models: add big model to default model resolution (#1929)
* device

* sunnylink

* lint

* lfs?

* Revert "lfs?"

This reverts commit bcdaec6b4c.

* update path

* Scope the default big model down to the sunnylink schema

* Drop the mock-only default model test

* Move the default model resolver out to separate PR

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2026-08-20 21:31:32 -04:00
Jason Wen 5ae100aa1d models fetcher: bump big model to v21 2026-08-20 19:19:59 -04:00
Jason Wen be76a88b80 ci override LFS fetch exclude for real ONNX file retrieval (#1928)
ci: override lfs.fetchexclude so the model fetch pulls real ONNX files instead of pointers
2026-08-20 19:12:59 -04:00
James Vecellio-Grant 049d225d5a ci: Dedicated Model Runner (#1922)
* ci: Dedicated Model Runner

* recurse

* not needed

* fix wrapper

* whoops

* bypass

* modeld_v2: restore chestnut link check before big model build

* modeld_v2: stage onnx to disk instead of shared memory

* ci: clear unchunked onnx temps before model build

* ci: stream the pkl hash instead of loading it into memory

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2026-08-20 16:18:17 -04:00
stef 555f48c5d2 params: remove livestream param on ignition (#38679)
* remove livestream param on ignition

* simplify process config
2026-08-19 14:21:59 -07:00
stef dcf9d25bf3 webrtcd: more descriptive errors (#38677)
more descriptive errors
2026-08-19 14:02:03 -07:00
stef a8d1a280c6 webrtcd/athenad: we don't have to fail on no car params (#38678)
we don't have to fail on no car params
2026-08-19 13:49:19 -07:00
royjr 7568528507 Merge branch 'master' into ccnc-port 2026-08-19 12:53:59 -04:00
stef 5b36799eec webrtc: fix message handler race (#38675)
open message handler early
2026-08-18 21:56:33 -07:00
stef 20fdc3d824 webrtcd: cloud logging (#38674)
* logging

* remove test

* get rid of redudant try except

* fix logger context
2026-08-18 21:13:57 -07:00
Shane Smiskol 7bd6cad821 Re-open agnos updater UI if crash (#38672)
loop if crash
2026-08-18 19:06:48 -07:00
royjr 6665c3acf6 Merge branch 'master' into ccnc-port 2026-08-17 10:14:33 -04:00
royjr 3b13eb4d0d Update opendbc_repo 2026-08-17 10:14:19 -04:00
royjr f7eb4c2561 Update opendbc_repo 2026-08-15 01:39:27 -04:00
royjr be9255358c Update opendbc_repo 2026-08-15 01:35:33 -04:00
royjr f48ccb5d7a Merge branch 'master' into ccnc-port 2026-08-14 20:42:45 -04:00
royjr 99559d6749 Update opendbc_repo 2026-08-14 20:42:36 -04:00
royjr 4b6f0ffb46 Merge branch 'master' into ccnc-port 2026-08-11 11:56:46 -04:00
royjr 55415382ce Update opendbc_repo 2026-08-11 11:56:38 -04:00
royjr 9f30901eb6 Merge branch 'master' into ccnc-port 2026-08-10 16:33:50 -04:00
royjr d2045d24fb Update opendbc_repo 2026-08-10 16:33:37 -04:00
royjr 6488a11e4d Merge branch 'master' into ccnc-port 2026-08-02 14:27:34 -04:00
royjr 7b22b65313 Merge branch 'master' into ccnc-port 2026-07-27 15:29:04 -04:00
royjr b59a50101e Update opendbc_repo 2026-07-27 15:28:51 -04:00
royjr 3e30962bd3 Merge branch 'master' into ccnc-port 2026-07-25 03:04:23 -04:00
royjr 3274043063 Update opendbc_repo 2026-07-25 03:04:12 -04:00
royjr c624550d2e Merge branch 'master' into ccnc-port 2026-07-20 15:52:41 -04:00
royjr 6759038671 Merge branch 'master' into ccnc-port 2026-07-20 09:18:18 -04:00
royjr aa636e75c8 Merge branch 'master' into ccnc-port 2026-07-16 00:23:49 -04:00
royjr 5ff3c3bd8d Update opendbc_repo 2026-07-16 00:23:43 -04:00
royjr 1e11b52290 Update opendbc_repo 2026-07-14 00:40:02 -04:00
royjr 76e0204025 Update opendbc_repo 2026-07-14 00:37:13 -04:00
royjr cc88f2cbd6 Update opendbc_repo 2026-07-10 22:00:25 -04:00
royjr ee0ac199a4 Update opendbc_repo 2026-07-03 08:08:29 -04:00
royjr d5ed828eaa Update opendbc_repo 2026-07-01 19:30:06 -04:00
royjr 9fca585f2a Merge branch 'master' into ccnc-port 2026-06-30 22:14:33 -04:00
royjr bb1259303e Update opendbc_repo 2026-06-30 22:14:23 -04:00
royjr e135051ca8 Merge branch 'master' into ccnc-port 2026-06-13 22:38:22 -04:00
royjr 12bef55d8a Update opendbc_repo 2026-06-13 22:12:00 -04:00
royjr 2f7a45e6c8 Merge branch 'master' into ccnc-port 2026-06-08 22:01:57 -04:00
royjr 936ebfc12b Update opendbc_repo 2026-06-08 22:01:44 -04:00
royjr aa0c9dc0eb Merge branch 'master' into ccnc-port 2026-06-04 09:52:35 -04:00
royjr 7476a866e7 Update opendbc_repo 2026-06-04 09:52:06 -04:00
royjr 610d857e33 Merge branch 'master' into ccnc-port 2026-05-28 08:30:40 -04:00
royjr d2f47407d0 Update opendbc_repo 2026-05-12 21:03:18 -04:00
royjr db75ec76ea Merge branch 'master' into ccnc-port 2026-05-12 20:44:57 -04:00
royjr 24066465d7 Update opendbc_repo 2026-05-12 20:44:48 -04:00
royjr a7abbd6e25 Update opendbc_repo 2026-04-20 18:22:12 -04:00
royjr 878982447c Merge branch 'master' into ccnc-port 2026-04-19 12:06:46 -04:00
royjr 576527a36b Merge branch 'master' into ccnc-port 2026-04-17 05:42:35 -04:00
royjr ef8c35da24 Update opendbc_repo 2026-04-17 05:41:21 -04:00
royjr 85688b1040 Merge branch 'master' into ccnc-port 2026-04-16 19:48:39 -04:00
royjr 0fb2199130 Update opendbc_repo 2026-04-16 19:47:57 -04:00
royjr d48d756c1d Update opendbc_repo 2026-04-08 00:56:54 -04:00
royjr 2ed298a0c9 Merge branch 'master' into ccnc-port 2026-04-08 00:51:13 -04:00
royjr d68f038949 Update opendbc_repo 2026-04-08 00:51:11 -04:00
royjr 7231571e57 Merge branch 'master' into ccnc-port 2026-04-03 23:34:00 -04:00
royjr b37f1419d3 Update opendbc_repo 2026-04-03 23:33:19 -04:00
royjr cd85a66790 Merge branch 'master' into ccnc-port 2026-03-26 00:53:07 -04:00
royjr 305ea87daf Update opendbc_repo 2026-03-26 00:52:44 -04:00
royjr 4bbfc793e0 Merge branch 'master' into ccnc-port 2026-03-15 15:44:59 -04:00
royjr d5d983676e Update opendbc_repo 2026-03-13 16:41:05 -04:00
royjr de8a96a398 Merge branch 'master' into ccnc-port 2026-03-13 16:41:00 -04:00
royjr 0cbf45f699 Merge branch 'master' into ccnc-port 2026-03-11 23:28:59 -04:00
royjr 0d68a3a2ab Merge branch 'master' into ccnc-port 2026-03-09 19:58:32 -04:00
royjr 9e85a85059 Update opendbc_repo 2026-03-09 19:58:18 -04:00
royjr 0373c327c0 Update opendbc_repo 2026-03-02 10:10:38 -05:00
royjr efe9e5c200 Update opendbc_repo 2026-03-02 02:15:05 -05:00
royjr 8a249a45dc Update opendbc_repo 2026-03-02 02:06:16 -05:00
royjr bdbefe67f6 Update opendbc_repo 2026-03-02 01:40:31 -05:00
royjr 675bb166ad Merge branch 'master' into ccnc-port 2026-03-01 17:18:11 -05:00
royjr 1b717a7e88 Merge branch 'master' into ccnc-port 2026-03-01 13:23:30 -05:00
royjr 86f55a8ba9 Update opendbc_repo 2026-03-01 13:23:24 -05:00
royjr 629392d2f7 Update opendbc_repo 2026-02-27 16:31:59 -05:00
royjr bc414bdc8b Update opendbc_repo 2026-02-26 23:53:26 -05:00
royjr 7ca5649f2c Merge branch 'master' into ccnc-port 2026-02-26 23:52:46 -05:00
royjr 641ee8fa87 Update opendbc_repo 2026-02-26 23:52:27 -05:00
royjr 56c276158c Merge branch 'master' into ccnc-port 2026-02-24 14:12:12 -05:00
royjr c65308a8bd Update opendbc_repo 2026-02-24 14:12:01 -05:00
royjr 994e526460 Merge branch 'master' into ccnc-port 2026-02-18 12:06:05 -05:00
royjr 1defae36b7 Update opendbc_repo 2026-02-18 12:05:53 -05:00
royjr 8f029fd0ef Merge branch 'master' into ccnc-port 2026-02-13 23:01:41 -05:00
royjr ddb46284dc Update opendbc_repo 2026-02-13 23:01:16 -05:00
royjr 9effc754d9 Merge branch 'master' into ccnc-port 2026-02-06 01:16:06 -05:00
royjr e49ffc2a2d Update opendbc_repo 2026-02-06 01:15:59 -05:00
royjr 2cacd0b3e5 Merge branch 'master' into ccnc-port 2026-01-24 12:50:18 -05:00
royjr c4b8859dff Update opendbc_repo 2026-01-24 12:50:11 -05:00
royjr 8fb0953205 Merge branch 'master' into ccnc-port 2026-01-11 22:17:38 -05:00
royjr 63d1c8835f Merge branch 'master' into ccnc-port 2026-01-10 13:03:48 -05:00
royjr 17a185606d Merge branch 'master' into ccnc-port 2026-01-09 16:40:32 -05:00
royjr da10131392 Merge branch 'master' into ccnc-port 2025-12-28 17:18:51 -05:00
royjr 7107c2ba14 Merge branch 'master' into ccnc-port 2025-12-23 12:13:48 -05:00
royjr 95b6e877ac Update opendbc_repo 2025-12-23 12:13:29 -05:00
royjr eb02c6570e Update opendbc_repo 2025-12-21 15:43:04 -05:00
royjr 1be8ae31c4 Merge branch 'master' into ccnc-port 2025-12-19 01:04:42 -05:00
royjr 04dcd38856 Update opendbc_repo 2025-12-19 01:04:30 -05:00
royjr 22ccf0d72f Merge branch 'master' into ccnc-port 2025-12-15 17:02:50 -05:00
royjr 3c969bb627 Merge branch 'master' into ccnc-port 2025-12-13 23:22:22 -05:00
royjr 20f8011feb Update opendbc_repo 2025-12-13 23:22:11 -05:00
royjr 9cf17e74a1 Merge branch 'master' into ccnc-port 2025-12-12 23:19:56 -05:00
royjr 2c4efdf557 Merge branch 'master' into ccnc-port 2025-12-07 13:29:48 -05:00
royjr 4cd3d3c16c Merge branch 'master' into ccnc-port 2025-12-02 12:56:21 -05:00
royjr 637f3ae9c8 Merge branch 'master' into ccnc-port 2025-12-01 14:41:03 -05:00
royjr 464ee80f71 Merge branch 'master' into ccnc-port 2025-11-26 00:27:53 -05:00
royjr 2743a04613 Merge branch 'master' into ccnc-port 2025-11-24 18:44:13 -05:00
royjr 7f9978d001 Merge branch 'master' into ccnc-port 2025-11-22 00:21:15 -05:00
royjr 4b83961c67 Merge branch 'master' into ccnc-port 2025-11-21 16:23:22 -05:00
royjr c00eaf428a Update opendbc_repo 2025-11-21 16:23:01 -05:00
royjr 0a9993e8d4 Merge branch 'master' into ccnc-port 2025-11-19 16:49:59 -05:00
royjr 0af214a985 Update opendbc_repo 2025-11-19 16:49:51 -05:00
royjr af43385e3a Merge branch 'master' into ccnc-port 2025-11-11 10:19:51 -05:00
royjr 0ab2b8c590 Update opendbc_repo 2025-11-07 19:59:50 -05:00
royjr 67ab18a0de Merge branch 'master' into ccnc-port 2025-11-07 19:23:51 -05:00
royjr e87dc15b30 Update opendbc_repo 2025-11-07 19:23:37 -05:00
royjr 192d08516c Merge branch 'master' into ccnc-port 2025-11-02 19:23:00 -05:00
royjr 3cf001c59c Update opendbc_repo 2025-11-02 19:22:49 -05:00
royjr f2ccd021da Merge branch 'master' into ccnc-port 2025-11-02 14:07:54 -05:00
royjr c9fc900f64 Update opendbc_repo 2025-11-02 14:07:44 -05:00
royjr 3c37c5ce5d Update opendbc_repo 2025-10-30 11:28:49 -04:00
royjr 7c45889e4e Merge branch 'master' into ccnc-port 2025-10-30 11:27:50 -04:00
royjr 2aabb7aee8 Merge branch 'master' into ccnc-port 2025-10-24 14:16:09 -04:00
royjr 3859e9962f Update opendbc_repo 2025-10-24 14:15:51 -04:00
royjr 810efbab72 Merge branch 'master' into ccnc-port 2025-10-18 07:33:11 -04:00
royjr ec27bec326 Update opendbc_repo 2025-10-18 07:32:33 -04:00
royjr 250d553157 Merge branch 'master' into ccnc-port 2025-10-14 21:57:32 -04:00
royjr cea54a0ca8 Update opendbc_repo 2025-10-14 21:57:26 -04:00
royjr 8e72d783bd Update opendbc_repo 2025-10-13 22:41:21 -04:00
royjr 1b0dc103dc Merge branch 'master' into ccnc-port 2025-10-11 23:51:46 -04:00
royjr 6c364d292b Update opendbc_repo 2025-10-11 23:51:31 -04:00
royjr bcdec2ce84 Merge branch 'master' into ccnc-port 2025-10-10 17:29:05 -04:00
royjr 3deaeb3759 Merge branch 'master' into ccnc-port 2025-10-10 15:02:32 -04:00
royjr c669f0984a Update opendbc_repo 2025-10-10 15:02:17 -04:00
royjr 46dd946740 Merge branch 'master' into ccnc-port 2025-10-07 01:36:06 -04:00
royjr 9da4b3653e Update opendbc_repo 2025-10-07 01:35:58 -04:00
royjr 4e21ae7c50 Update opendbc_repo 2025-10-05 06:21:56 -04:00
royjr bb91e92237 Update opendbc_repo 2025-10-05 06:01:15 -04:00
royjr 14b4c4f85b Update opendbc_repo 2025-10-02 09:20:32 -04:00
royjr 0660b542c3 Merge branch 'master' into ccnc-port 2025-10-01 16:01:32 -04:00
royjr 2b893b90c9 Update opendbc_repo 2025-10-01 16:01:26 -04:00
royjr f5139178ed Merge branch 'master' into ccnc-port 2025-09-30 14:39:57 -04:00
royjr fb43b755f2 Update opendbc_repo 2025-09-30 14:39:50 -04:00
royjr 07f5b967d8 Merge branch 'master' into ccnc-port 2025-09-24 21:30:29 -04:00
royjr ea19c7d3bb Update opendbc_repo 2025-09-24 21:30:17 -04:00
royjr e461842cbb Merge branch 'master' into ccnc-port 2025-09-23 05:55:02 -04:00
royjr a73c9659d5 Update opendbc_repo 2025-09-23 05:54:50 -04:00
royjr cb796fbc76 Merge branch 'master' into ccnc-port 2025-09-18 20:10:43 -04:00
royjr 6bf75fc557 Update opendbc_repo 2025-09-18 20:10:37 -04:00
royjr 9a1fc28819 Merge branch 'master' into ccnc-port 2025-09-18 13:54:36 -04:00
royjr 0741d05e92 Update opendbc_repo 2025-09-18 13:54:23 -04:00
royjr 1ad008107d Merge branch 'master' into ccnc-port 2025-09-15 01:40:27 -04:00
royjr feebd9df93 Reapply "UI: Developer UI (#1233)"
This reverts commit 15e5d2efb9.
2025-09-15 01:40:21 -04:00
royjr c2e5ced3e5 Update opendbc_repo 2025-09-15 01:39:51 -04:00
royjr 15e5d2efb9 Revert "UI: Developer UI (#1233)"
This reverts commit 1bb4ca2547.
2025-09-12 02:10:29 -04:00
royjr a3929d0b54 Merge branch 'master' into ccnc-port 2025-09-12 01:40:29 -04:00
royjr 794f8f9991 Update opendbc_repo 2025-09-08 09:27:31 -04:00
royjr 68fa5e3f21 Merge branch 'master' into ccnc-port 2025-09-07 13:13:37 -04:00
royjr 86c6cc1f48 Merge branch 'master' into ccnc-port 2025-09-03 22:22:12 -04:00
royjr eb7ffbf093 Update opendbc_repo 2025-09-03 10:14:58 -04:00
royjr 3919095752 Update opendbc_repo 2025-09-03 10:05:35 -04:00
royjr 74d63be1c3 Merge branch 'master' into ccnc-port 2025-09-03 09:50:55 -04:00
royjr 8894486a1a Update opendbc_repo 2025-09-03 09:50:49 -04:00
royjr 810599315d Merge branch 'master' into ccnc-port 2025-08-31 16:53:30 -04:00
royjr 6f3ab810c8 Update opendbc_repo 2025-08-31 16:53:24 -04:00
royjr 230f78b8d3 Merge branch 'master' into ccnc-port 2025-08-26 12:14:46 -04:00
royjr f1affec088 Update opendbc_repo 2025-08-26 12:14:22 -04:00
royjr 97d8ef242c Merge branch 'master' into ccnc-port 2025-08-24 15:12:57 -04:00
royjr a63fff9b45 Update opendbc_repo 2025-08-24 15:12:46 -04:00
royjr cb3893daaa Merge branch 'master' into ccnc-port 2025-08-23 10:33:24 -04:00
royjr 29f60df74b Merge branch 'master' into ccnc-port 2025-08-22 11:18:27 -04:00
royjr c6c072e1f4 Update opendbc_repo 2025-08-22 11:18:18 -04:00
royjr d101cbb83e Update opendbc_repo 2025-08-13 16:00:04 -04:00
royjr 1536d59633 Update opendbc_repo 2025-08-13 15:28:37 -04:00
royjr dc99b865ae Merge branch 'master' into ccnc-port 2025-08-13 12:31:14 -04:00
royjr e59bc027ff Merge branch 'master' into ccnc-port 2025-08-13 11:58:05 -04:00
royjr cf7e5efaca Update opendbc_repo 2025-08-13 11:57:57 -04:00
royjr 4b44f2eb31 Merge branch 'master' into ccnc-port 2025-08-10 09:18:24 -04:00
royjr 107d2ab400 Update opendbc_repo 2025-08-10 09:18:15 -04:00
royjr 5432d9062c Merge branch 'master' into ccnc-port 2025-08-04 11:52:29 -04:00
royjr f533f6c843 Merge branch 'master' into ccnc-port 2025-08-02 06:53:54 -04:00
royjr 58e9ac763c Update opendbc_repo 2025-08-02 06:53:43 -04:00
royjr cb50d54169 Merge branch 'master-new' into ccnc-port 2025-07-24 20:23:46 -04:00
royjr bd5de4ed0a Merge branch 'master-new' into ccnc-port 2025-07-20 23:49:36 -04:00
royjr 0d4073fadb Merge branch 'master-new' into ccnc-port 2025-07-19 23:18:26 -04:00
royjr ebc70dcb52 Merge branch 'master-new' into ccnc-port 2025-07-19 14:37:17 -04:00
royjr 4d0426999e Update opendbc_repo 2025-07-19 14:36:59 -04:00
royjr 286da42573 Merge branch 'master-new' into ccnc-port 2025-07-16 23:48:23 -04:00
royjr 8a836710a9 Update opendbc_repo 2025-07-16 23:48:06 -04:00
royjr 5d515bcf33 Merge branch 'master-new' into ccnc-port 2025-07-07 05:35:55 -04:00
royjr 1c7f6d5133 Update opendbc_repo 2025-07-03 21:24:01 -04:00
royjr 05d57c7aeb Update opendbc_repo 2025-07-01 18:30:32 -04:00
royjr e4b0eaf352 Update opendbc_repo 2025-06-28 19:30:28 -04:00
royjr a710276472 Merge branch 'master-new' into ccnc-port 2025-06-28 19:22:59 -04:00
royjr af086db671 Merge branch 'master-new' into ccnc-port 2025-06-28 12:13:26 -04:00
royjr 0d9eb0e25e Update opendbc_repo submodule to latest commit
Advanced the opendbc_repo submodule to commit d309f7ec96e37267c94d12fc4bfe2672ad505b06. This pulls in the latest changes from the opendbc repository.
2025-06-26 14:03:56 -04:00
royjr 0616caed6d Merge branch 'master-new' into ccnc-port 2025-06-25 19:33:53 -04:00
royjr 095337b3c1 Update opendbc_repo 2025-06-25 19:33:38 -04:00
royjr 1edec2d22c Update opendbc_repo 2025-06-14 16:14:02 -04:00
royjr affabb9ee0 Update opendbc_repo 2025-06-14 15:55:07 -04:00
royjr dc27e8711c Update opendbc_repo 2025-06-14 14:54:32 -04:00
royjr cf7329a264 Merge branch 'master-new' into ccnc-port 2025-06-11 21:36:06 -04:00
royjr 5ee5ecd820 Merge branch 'master-new' into ccnc-port 2025-06-08 23:25:13 -04:00
royjr b064f730dd Update opendbc_repo 2025-06-08 17:54:43 -04:00
191 changed files with 5620 additions and 2897 deletions
@@ -8,13 +8,13 @@ on:
required: true
type: string
target_hardware:
description: 'Hardware target to compile for (qcom or usbgpu)'
description: 'Hardware target to compile for (qcom or chestnut)'
required: true
type: choice
default: 'qcom'
options:
- qcom
- usbgpu
- chestnut
hf_repo:
description: 'Hugging Face dataset repository'
required: false
@@ -59,7 +59,7 @@ jobs:
id: get-json
run: |
cd docs/docs
PREFIX="driving_models_${{ inputs.target_hardware == 'usbgpu' && 'usbgpu_' || '' }}v"
PREFIX="driving_models_${{ inputs.target_hardware == 'chestnut' && 'chestnut_' || '' }}v"
latest=$(ls ${PREFIX}*.json | sed -E "s/${PREFIX}([0-9]+)\.json/\1/" | sort -n | tail -1)
next=$((latest+1))
json_file="${PREFIX}${next}.json"
+501
View File
@@ -0,0 +1,501 @@
name: Build default models
on:
workflow_dispatch:
inputs:
target:
description: 'Model target to build'
required: true
type: choice
options:
- small
- big
- dm
workflow_call:
inputs:
target:
description: 'Model target to build (small, big, or dm)'
required: true
type: string
concurrency:
group: build-default-models-${{ inputs.target }}
cancel-in-progress: false
env:
HF_REPO: sunnypilot/sunnypilot_models_v1
jobs:
resolve:
runs-on: ubuntu-24.04
outputs:
model_name: ${{ steps.resolve.outputs.model_name }}
onnx_ref: ${{ steps.resolve.outputs.onnx_ref }}
onnx_path: ${{ steps.resolve.outputs.onnx_path }}
hf_defaults_path: ${{ steps.resolve.outputs.hf_defaults_path }}
tinygrad_ref: ${{ steps.resolve.outputs.tinygrad_ref }}
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- id: resolve
run: |
export PYTHONPATH=${{ github.workspace }}
if [ "${{ inputs.target }}" = "big" ]; then
NAME=$(python3 -c "from openpilot.sunnypilot.models.model_name import DEFAULT_BIG_MODEL; print(DEFAULT_BIG_MODEL)")
ONNX_PATH="openpilot/selfdrive/modeld/models/big_driving_supercombo.onnx"
HF_DEFAULTS_PATH="models/defaults/big"
elif [ "${{ inputs.target }}" = "dm" ]; then
ONNX_PATH="openpilot/selfdrive/modeld/models/dmonitoring_model.onnx"
HF_DEFAULTS_PATH="models/defaults/dm"
NAME="dmonitoring_model ($(git log -1 --format=%cd --date=format:'%B %d, %Y' -- "$ONNX_PATH"))"
else
NAME=$(python3 -c "from openpilot.sunnypilot.models.model_name import DEFAULT_MODEL; print(DEFAULT_MODEL)")
ONNX_PATH="openpilot/selfdrive/modeld/models/driving_supercombo.onnx"
HF_DEFAULTS_PATH="models/defaults/small"
fi
ONNX_REF=$(git log -1 --format='%H' -- "$ONNX_PATH")
TINYGRAD_REF=$(python3 openpilot/sunnypilot/models/tinygrad_ref.py)
if [ -z "$TINYGRAD_REF" ]; then
echo "::error::Failed to resolve tinygrad ref"
exit 1
fi
echo "model_name=${NAME}" >> $GITHUB_OUTPUT
echo "onnx_ref=${ONNX_REF}" >> $GITHUB_OUTPUT
echo "onnx_path=${ONNX_PATH}" >> $GITHUB_OUTPUT
echo "hf_defaults_path=${HF_DEFAULTS_PATH}" >> $GITHUB_OUTPUT
echo "tinygrad_ref=${TINYGRAD_REF}" >> $GITHUB_OUTPUT
build_small_model:
needs: resolve
if: ${{ inputs.target == 'small' }}
runs-on: [self-hosted, tici]
env:
SMALL_ONNX: openpilot/selfdrive/modeld/models/driving_supercombo.onnx
SMALL_PKL: openpilot/selfdrive/modeld/models/driving_tinygrad.pkl
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- name: Pull ONNX via LFS
run: git lfs pull -I "${{ env.SMALL_ONNX }}"
- name: Set environment variables
run: |
source /etc/profile
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export UV_PYTHON_PREFERENCE=managed
export UV_PYTHON_INSTALL_DIR=${HOME}/uv/python
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
uv sync --frozen
printenv >> $GITHUB_ENV
- name: Disable powersave
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
- name: Compile small model with stock compiler
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH="${PYTHONPATH}:${{ github.workspace }}/tinygrad_repo:${{ github.workspace }}"
MODEL_SIZE=$(python3 -c "from openpilot.common.transformations.model import MEDMODEL_INPUT_SIZE as s; print(f'{s[0]}x{s[1]}')")
CAMERA_RES=$(python3 -c "from openpilot.common.transformations.camera import _ar_ox_fisheye as a, _os_fisheye as o; print(f'{a.width}x{a.height} {o.width}x{o.height}')")
FRAME_SKIP=$(python3 -c "from openpilot.selfdrive.modeld.constants import ModelConstants as MC; print(MC.MODEL_RUN_FREQ // MC.MODEL_CONTEXT_FREQ)")
TG_FLAGS="DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1"
env ${TG_FLAGS} python3 \
${{ github.workspace }}/openpilot/selfdrive/modeld/compile_modeld.py \
--onnx ${{ github.workspace }}/${{ env.SMALL_ONNX }} \
--model-size $MODEL_SIZE \
--camera-resolutions $CAMERA_RES \
--frame-skip $FRAME_SKIP \
--output ${{ github.workspace }}/${{ env.SMALL_PKL }}
- name: Chunk small pkl
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH=${{ github.workspace }}
python3 -c "
from openpilot.common.file_chunker import chunk_file, get_chunk_targets
import os
pkl = '${{ github.workspace }}/${{ env.SMALL_PKL }}'
size = os.path.getsize(pkl)
targets = get_chunk_targets(pkl, size)
chunk_file(pkl, targets)
print(f'Chunked into {len(targets)} files')
"
- name: Prepare output
env:
MODEL_NAME: ${{ needs.resolve.outputs.model_name }}
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH=${{ github.workspace }}
MODELS_DIR="${{ github.workspace }}/openpilot/selfdrive/modeld/models"
OUTPUT_DIR="${{ github.workspace }}/small_output"
PKL_BASE="driving_tinygrad.pkl"
mkdir -p "$OUTPUT_DIR"
cp "$MODELS_DIR/${PKL_BASE}".chunk* "$OUTPUT_DIR/"
cp "$MODELS_DIR/${PKL_BASE}.chunkmanifest" "$OUTPUT_DIR/"
python3 "${{ github.workspace }}/release/ci/model_generator.py" \
--model-dir "$MODELS_DIR" \
--output-dir "$OUTPUT_DIR" \
--custom-name "$MODEL_NAME" \
--upstream-branch "${{ needs.resolve.outputs.onnx_ref }}"
echo "model-${MODEL_NAME}-${{ github.run_number }}" > "$OUTPUT_DIR/artifact_name.txt"
- name: Upload small model artifact
uses: actions/upload-artifact@v4
with:
name: model-${{ needs.resolve.outputs.model_name }}-${{ github.run_number }}
path: ${{ github.workspace }}/small_output/
- name: Upload artifact name file
uses: actions/upload-artifact@v4
with:
name: artifact-name-${{ needs.resolve.outputs.model_name }}
path: ${{ github.workspace }}/small_output/artifact_name.txt
- name: Re-enable powersave
if: always()
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
build_big_model:
needs: resolve
if: ${{ inputs.target == 'big' }}
runs-on: [self-hosted, chestnut]
env:
BIG_ONNX: openpilot/selfdrive/modeld/models/big_driving_supercombo.onnx
BIG_PKL: openpilot/selfdrive/modeld/models/big_driving_tinygrad.pkl
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- name: Pull big ONNX via LFS
run: git lfs pull -I "${{ env.BIG_ONNX }}"
- name: Set environment variables
run: |
source /etc/profile
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export UV_PYTHON_PREFERENCE=managed
export UV_PYTHON_INSTALL_DIR=${HOME}/uv/python
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
uv sync --frozen
printenv >> $GITHUB_ENV
- name: Disable powersave
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
- name: Wait for chestnut PCIe link
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH="${PYTHONPATH}:${{ github.workspace }}/tinygrad_repo:${{ github.workspace }}"
python3 -c "
import time
from openpilot.system.hardware.chestnut.flash import link_up
for i in range(10):
if link_up():
print(f'PCIe link up after {i+1} attempt(s)')
break
time.sleep(1)
else:
raise RuntimeError('Chestnut PCIe link not ready after 10 attempts')
"
- name: Compile big model with stock compiler
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH="${PYTHONPATH}:${{ github.workspace }}/tinygrad_repo:${{ github.workspace }}"
MODEL_SIZE=$(python3 -c "from openpilot.common.transformations.model import MEDMODEL_INPUT_SIZE as s; print(f'{s[0]}x{s[1]}')")
CAMERA_RES=$(python3 -c "from openpilot.common.transformations.camera import _ar_ox_fisheye as a, _os_fisheye as o; print(f'{a.width}x{a.height} {o.width}x{o.height}')")
FRAME_SKIP=$(python3 -c "from openpilot.selfdrive.modeld.constants import ModelConstants as MC; print(MC.MODEL_RUN_FREQ // MC.MODEL_CONTEXT_FREQ)")
TG_FLAGS="DEBUG=2 DEV=USB+AMD:LLVM WARP_DEV=QCOM FLOAT16=1 JIT_BATCH_SIZE=0 GMMU=0 TC_OPT=2"
env ${TG_FLAGS} python3 \
${{ github.workspace }}/openpilot/selfdrive/modeld/compile_modeld.py \
--onnx ${{ github.workspace }}/${{ env.BIG_ONNX }} \
--model-size $MODEL_SIZE \
--camera-resolutions $CAMERA_RES \
--frame-skip $FRAME_SKIP \
--output ${{ github.workspace }}/${{ env.BIG_PKL }}
- name: Chunk big pkl
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH=${{ github.workspace }}
python3 -c "
from openpilot.common.file_chunker import chunk_file, get_chunk_targets
import os
pkl = '${{ github.workspace }}/${{ env.BIG_PKL }}'
size = os.path.getsize(pkl)
targets = get_chunk_targets(pkl, size)
chunk_file(pkl, targets)
print(f'Chunked into {len(targets)} files')
"
- name: Prepare output
env:
MODEL_NAME: ${{ needs.resolve.outputs.model_name }}
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH=${{ github.workspace }}
MODELS_DIR="${{ github.workspace }}/openpilot/selfdrive/modeld/models"
OUTPUT_DIR="${{ github.workspace }}/big_output"
PKL_BASE="big_driving_tinygrad.pkl"
mkdir -p "$OUTPUT_DIR"
cp "$MODELS_DIR/${PKL_BASE}".chunk* "$OUTPUT_DIR/"
cp "$MODELS_DIR/${PKL_BASE}.chunkmanifest" "$OUTPUT_DIR/"
python3 "${{ github.workspace }}/release/ci/model_generator.py" \
--model-dir "$MODELS_DIR" \
--output-dir "$OUTPUT_DIR" \
--custom-name "$MODEL_NAME" \
--upstream-branch "${{ needs.resolve.outputs.onnx_ref }}"
echo "model-${MODEL_NAME}-${{ github.run_number }}" > "$OUTPUT_DIR/artifact_name.txt"
- name: Upload big model artifact
uses: actions/upload-artifact@v4
with:
name: model-${{ needs.resolve.outputs.model_name }}-${{ github.run_number }}
path: ${{ github.workspace }}/big_output/
- name: Upload artifact name file
uses: actions/upload-artifact@v4
with:
name: artifact-name-${{ needs.resolve.outputs.model_name }}
path: ${{ github.workspace }}/big_output/artifact_name.txt
- name: Re-enable powersave
if: always()
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
upload_defaults:
needs: [ resolve, build_small_model, build_big_model, build_dm_model ]
if: |
${{
!cancelled() &&
(inputs.target == 'big' && needs.build_big_model.result == 'success' ||
inputs.target == 'small' && needs.build_small_model.result == 'success' ||
inputs.target == 'dm' && needs.build_dm_model.result == 'success')
}}
runs-on: ubuntu-24.04
permissions:
id-token: write
contents: write
steps:
- uses: actions/checkout@v4
- name: Pull ONNX via LFS
run: git lfs pull -I "${{ needs.resolve.outputs.onnx_path }}"
- name: Install huggingface_hub
run: pip install --upgrade "huggingface_hub>=0.22.0"
- name: Download artifact name
if: ${{ inputs.target == 'small' || inputs.target == 'big' }}
uses: actions/download-artifact@v4
with:
name: artifact-name-${{ needs.resolve.outputs.model_name }}
path: artifact_name
- name: Read artifact name
if: ${{ inputs.target == 'small' || inputs.target == 'big' }}
id: artifact
run: |
ARTIFACT_NAME=$(cat artifact_name/artifact_name.txt)
echo "artifact_name=$ARTIFACT_NAME" >> $GITHUB_OUTPUT
- name: Download model artifact
if: ${{ inputs.target == 'small' || inputs.target == 'big' }}
uses: actions/download-artifact@v4
with:
name: ${{ steps.artifact.outputs.artifact_name }}
path: output
- name: Upload model to HF
if: ${{ inputs.target == 'small' || inputs.target == 'big' }}
env:
HF_OIDC_RESOURCE: datasets/${{ env.HF_REPO }}
ARTIFACT_NAME: ${{ steps.artifact.outputs.artifact_name }}
run: |
rm -f output/artifact_name.txt
export PYTHONPATH=$(pwd)
python3 release/ci/upload_default_model.py \
--hf-repo "${{ env.HF_REPO }}" \
--hf-defaults-path "${{ needs.resolve.outputs.hf_defaults_path }}" \
--artifact-name "$ARTIFACT_NAME" \
--model-dir output \
--onnx-path "${{ needs.resolve.outputs.onnx_path }}" \
--onnx-ref "${{ needs.resolve.outputs.onnx_ref }}" \
--model-name "${{ needs.resolve.outputs.model_name }}" \
--tinygrad-ref "${{ needs.resolve.outputs.tinygrad_ref }}" \
--run-number "${{ github.run_number }}"
- name: Download DM artifact
if: ${{ inputs.target == 'dm' }}
uses: actions/download-artifact@v4
with:
name: dm-model-${{ github.run_number }}
path: dm_output
- name: Generate DM metadata and upload to HF
if: ${{ inputs.target == 'dm' }}
env:
HF_OIDC_RESOURCE: datasets/${{ env.HF_REPO }}
run: |
export PYTHONPATH=$(pwd)
python3 -c "
import json, hashlib
from pathlib import Path
from datetime import datetime, UTC
dm_dir = Path('dm_output')
manifest = list(dm_dir.glob('*.chunkmanifest'))
assert manifest, 'No chunkmanifest found'
pkl_name = manifest[0].name.removesuffix('.chunkmanifest')
num_chunks = int(manifest[0].read_text().strip())
chunks = []
for i in range(num_chunks):
chunk = dm_dir / f'{pkl_name}.chunk{i+1:02d}of{num_chunks:02d}'
chunks.append({
'file_name': chunk.name,
'sha256': hashlib.sha256(chunk.read_bytes()).hexdigest()
})
digest = hashlib.sha256()
for c in chunks:
with open(dm_dir / c['file_name'], 'rb') as f:
while block := f.read(1024*1024):
digest.update(block)
metadata = {
'bundles': [{
'short_name': 'DMMODEL',
'display_name': '${{ needs.resolve.outputs.model_name }}',
'ref': '${{ needs.resolve.outputs.onnx_ref }}',
'runner': 'tinygrad',
'build_time': datetime.now(UTC).strftime('%Y-%m-%dT%H:%M:%SZ'),
'models': [{
'type': 'chunked',
'artifact': {
'file_name': pkl_name,
'download_uri': {'url': '', 'sha256': digest.hexdigest()},
'chunks': chunks
}
}]
}]
}
with open(dm_dir / 'metadata.json', 'w') as f:
json.dump(metadata, f, indent=2)
print('Generated DM metadata.json')
"
python3 release/ci/upload_default_model.py \
--hf-repo "${{ env.HF_REPO }}" \
--hf-defaults-path "${{ needs.resolve.outputs.hf_defaults_path }}" \
--artifact-name "dm-model-${{ github.run_number }}" \
--model-dir dm_output \
--onnx-path "${{ needs.resolve.outputs.onnx_path }}" \
--onnx-ref "${{ needs.resolve.outputs.onnx_ref }}" \
--model-name "${{ needs.resolve.outputs.model_name }}" \
--tinygrad-ref "${{ needs.resolve.outputs.tinygrad_ref }}" \
--run-number "${{ github.run_number }}"
build_dm_model:
needs: resolve
if: ${{ inputs.target == 'dm' }}
runs-on: [self-hosted, tici]
env:
DM_ONNX: openpilot/selfdrive/modeld/models/dmonitoring_model.onnx
DM_PKL: openpilot/selfdrive/modeld/models/dmonitoring_model_tinygrad.pkl
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- name: Pull DM ONNX via LFS
run: git lfs pull -I "${{ env.DM_ONNX }}"
- name: Set environment variables
run: |
source /etc/profile
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export UV_PYTHON_PREFERENCE=managed
export UV_PYTHON_INSTALL_DIR=${HOME}/uv/python
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
uv sync --frozen
printenv >> $GITHUB_ENV
- name: Disable powersave
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
- name: Compile DM model
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH="${PYTHONPATH}:${{ github.workspace }}/tinygrad_repo:${{ github.workspace }}"
TG_FLAGS="DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1"
taskset -c 7 env ${TG_FLAGS} python3 \
${{ github.workspace }}/tinygrad_repo/examples/openpilot/compile3.py \
${{ github.workspace }}/${{ env.DM_ONNX }} \
${{ github.workspace }}/${{ env.DM_PKL }}
- name: Chunk DM pkl
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
export PYTHONPATH=${{ github.workspace }}
python3 -c "
from openpilot.common.file_chunker import chunk_file, get_chunk_targets
import os
pkl = '${{ github.workspace }}/${{ env.DM_PKL }}'
size = os.path.getsize(pkl)
targets = get_chunk_targets(pkl, size)
chunk_file(pkl, targets)
print(f'Chunked {pkl} into {len(targets)} chunks')
"
- name: Prepare DM output
run: |
mkdir -p dm_output
cp ${{ github.workspace }}/${{ env.DM_PKL }}.chunk* dm_output/
cp ${{ github.workspace }}/${{ env.DM_PKL }}.chunkmanifest dm_output/
- name: Upload DM artifact
uses: actions/upload-artifact@v4
with:
name: dm-model-${{ github.run_number }}
path: dm_output/
- name: Re-enable powersave
if: always()
run: |
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
@@ -30,7 +30,7 @@ on:
type: boolean
default: true
target_hardware:
description: 'Hardware target to compile for (qcom or usbgpu)'
description: 'Hardware target to compile for (qcom or chestnut)'
required: false
type: string
default: 'qcom'
@@ -101,7 +101,7 @@ on:
default: 'qcom'
options:
- qcom
- usbgpu
- chestnut
hf_repo:
description: 'Hugging Face dataset repository'
required: false
@@ -109,7 +109,7 @@ on:
default: 'sunnypilot/sunnypilot_models_v1'
env:
RECOMPILED_DIR: recompiled${{ inputs.recompiled_dir }}
JSON_FILE: docs/docs/driving_models_${{ inputs.target_hardware == 'usbgpu' && 'usbgpu_v' || 'v' }}${{ inputs.json_version }}.json
JSON_FILE: docs/docs/driving_models_${{ inputs.target_hardware == 'chestnut' && 'chestnut_v' || 'v' }}${{ inputs.json_version }}.json
jobs:
build_model:
@@ -0,0 +1,66 @@
name: Download HF model chunks
description: Resolve and download model chunks from HuggingFace in parallel
inputs:
hf_repo:
description: HuggingFace dataset repo
required: true
models:
description: 'JSON array of {hf_path, onnx_hash, canonical} objects'
required: true
dest_dir:
description: Destination directory for downloaded chunks
required: true
runs:
using: composite
steps:
- name: Download model chunks
shell: bash
env:
HF_REPO: ${{ inputs.hf_repo }}
MODELS_JSON: ${{ inputs.models }}
DEST_DIR: ${{ inputs.dest_dir }}
run: |
set -eo pipefail
DOWNLOAD_LIST=$(mktemp)
resolve_chunks() {
local HF_PATH="$1" ONNX_HASH="$2" CANONICAL="$3" DEST_DIR="$4"
local JSON_URL="https://huggingface.co/datasets/${HF_REPO}/resolve/main/${HF_PATH}/default_models.json"
local DEFAULTS BUNDLE ARTIFACT BASE_URL NUM_CHUNKS
DEFAULTS=$(curl -fsSL "$JSON_URL")
BUNDLE=$(echo "$DEFAULTS" | jq --arg hash "$ONNX_HASH" '.bundles[] | select(.onnx_sha256 == $hash)')
ARTIFACT=$(echo "$BUNDLE" | jq -r '.models[0].artifact')
BASE_URL=$(echo "$ARTIFACT" | jq -r '.download_uri.url' | sed 's|/[^/]*$||')
NUM_CHUNKS=$(echo "$ARTIFACT" | jq -r '.chunks | length')
mkdir -p "$DEST_DIR"
while IFS= read -r CHUNK_NAME; do
CHUNK_IDX=$(echo "$CHUNK_NAME" | grep -oP 'chunk\K[0-9]+of[0-9]+' || true)
if [ -z "$CHUNK_IDX" ]; then
echo "::error::Failed to parse chunk index from: $CHUNK_NAME"
return 1
fi
ENCODED_URL=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${BASE_URL}/${CHUNK_NAME}', safe=':/'))")
printf '%s\t%s\n' "$ENCODED_URL" "${DEST_DIR}/${CANONICAL}.chunk${CHUNK_IDX}" >> "$DOWNLOAD_LIST"
done < <(echo "$ARTIFACT" | jq -r '.chunks[].file_name')
echo "$NUM_CHUNKS" > "${DEST_DIR}/${CANONICAL}.chunkmanifest"
}
echo "$MODELS_JSON" | jq -c '.[]' | while IFS= read -r model; do
HF_PATH=$(echo "$model" | jq -r '.hf_path')
ONNX_HASH=$(echo "$model" | jq -r '.onnx_hash')
CANONICAL=$(echo "$model" | jq -r '.canonical')
resolve_chunks "$HF_PATH" "$ONNX_HASH" "$CANONICAL" "$DEST_DIR"
done
TOTAL=$(wc -l < "$DOWNLOAD_LIST")
echo "Downloading $TOTAL chunks with 8 parallel connections..."
xargs -P8 -d'\n' -I{} bash -c '
URL="${1%% *}"
DEST="${1#* }"
echo "Downloading $(basename "$DEST")"
curl -fsSL --retry 3 --retry-delay 5 -o "$DEST" "$URL"
' _ {} < "$DOWNLOAD_LIST"
rm -f "$DOWNLOAD_LIST"
+41 -23
View File
@@ -31,7 +31,7 @@ on:
type: string
default: ''
target_hardware:
description: 'Hardware target to compile for (qcom or usbgpu)'
description: 'Hardware target to compile for (qcom or chestnut)'
required: false
type: string
default: 'qcom'
@@ -57,7 +57,7 @@ on:
type: choice
options:
- qcom
- usbgpu
- chestnut
default: 'qcom'
@@ -102,21 +102,26 @@ jobs:
cat $GITHUB_OUTPUT
- run: |
cd ${{ github.workspace }}/openpilot/openpilot
if [ "${{ inputs.target_hardware }}" != "usbgpu" ]; then
git lfs pull -X "selfdrive/modeld/models/big_*.onnx" -X "selfdrive/modeld/models/dmonitoring_*.onnx"
if [ "${{ inputs.target_hardware }}" != "chestnut" ]; then
git lfs pull -X "**/selfdrive/modeld/models/big_*.onnx,**/selfdrive/modeld/models/dmonitoring_*.onnx"
rm -f selfdrive/modeld/models/big_*.onnx selfdrive/modeld/models/dmonitoring_*.onnx
else
git lfs pull -I "selfdrive/modeld/models/big_*.onnx"
git lfs pull -I "**/selfdrive/modeld/models/big_*.onnx" -X ""
find selfdrive/modeld/models -name "*.onnx" ! -name "big_*.onnx" -delete
fi
if grep -lIF "version https://git-lfs.github.com/spec/v1" selfdrive/modeld/models/*.onnx; then
echo "::error::the ONNX files above are still LFS pointers, not real models"
exit 1
fi
- name: 'Upload Artifact'
uses: actions/upload-artifact@v4
with:
name: models-${{ env.REF }}${{ inputs.artifact_suffix }}
path: ${{ github.workspace }}/openpilot/openpilot/selfdrive/modeld/models/*.onnx
if-no-files-found: error
build_model:
runs-on: [self-hosted, tici]
runs-on: [self-hosted, "${{ inputs.target_hardware == 'chestnut' && 'chestnut' || 'tici' }}"]
needs: get_model
env:
MODEL_NAME: ${{ inputs.custom_name || inputs.upstream_branch }} (${{ needs.get_model.outputs.model_date }})
@@ -127,7 +132,6 @@ jobs:
fetch-depth: 1
submodules: recursive
- run: git lfs pull
- name: Set environment variables
id: set-env
@@ -160,7 +164,7 @@ jobs:
fi
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
rm -rf ${{ env.MODELS_DIR }}/*.onnx
rm -rf ${{ env.MODELS_DIR }}/*.onnx*
- name: Download model artifacts
uses: actions/download-artifact@v4
@@ -180,34 +184,48 @@ jobs:
MODEL_SIZE=$(python3 -c "from openpilot.common.transformations.model import MEDMODEL_INPUT_SIZE as s; print(f'{s[0]}x{s[1]}')")
CAMERA_RES=$(python3 -c "from openpilot.common.transformations.camera import _ar_ox_fisheye as a, _os_fisheye as o; print(f'{a.width}x{a.height} {o.width}x{o.height}')")
if [ "${{ inputs.target_hardware }}" == "usbgpu" ]; then
echo "USBGPU build"
export USBGPU=1
TG_FLAGS="DEBUG=2 DEV=USB+AMD:LLVM WARP_DEV=QCOM FLOAT16=1 JIT_BATCH_SIZE=0 GMMU=0 TC_OPT=2"
TG_FLAGS_QCOM="DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1"
if [ "${{ inputs.target_hardware }}" == "chestnut" ]; then
echo "CHESTNUT build"
export CHESTNUT=1
TG_FLAGS="DEBUG=1 DEV=USB+AMD:LLVM WARP_DEV=QCOM FLOAT16=1 JIT_BATCH_SIZE=0 GMMU=0 TC_OPT=2"
OUTPUT_PKL="${{ env.MODELS_DIR }}/big_driving_tinygrad.pkl"
else
echo "QCOM build"
TG_FLAGS="DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1"
TG_FLAGS="$TG_FLAGS_QCOM"
OUTPUT_PKL="${{ env.MODELS_DIR }}/driving_tinygrad.pkl"
fi
# Generate metadata for all ONNX files
find "${{ env.MODELS_DIR }}" -maxdepth 1 -name '*.onnx' | while IFS= read -r onnx_file; do
echo "Generating metadata: $onnx_file"
env ${TG_FLAGS} python3 "${{ env.MODELS_DIR }}/../get_model_metadata.py" "$onnx_file" || true
env ${TG_FLAGS_QCOM} python3 "${{ env.MODELS_DIR }}/../get_model_metadata.py" "$onnx_file" || true
done
# Detect model type and build compile args
VISION_ONNX="${{ env.MODELS_DIR }}/driving_vision.onnx"
POLICY_ONNX="${{ env.MODELS_DIR }}/driving_policy.onnx"
OFF_POLICY_ONNX="${{ env.MODELS_DIR }}/driving_off_policy.onnx"
ON_POLICY_ONNX="${{ env.MODELS_DIR }}/driving_on_policy.onnx"
VISION_ONNX=""
for f in "${{ env.MODELS_DIR }}/driving_vision.onnx" "${{ env.MODELS_DIR }}/big_driving_vision.onnx"; do
[ -f "$f" ] && VISION_ONNX="$f" && break
done
POLICY_ONNX=""
for f in "${{ env.MODELS_DIR }}/driving_policy.onnx" "${{ env.MODELS_DIR }}/big_driving_policy.onnx"; do
[ -f "$f" ] && POLICY_ONNX="$f" && break
done
OFF_POLICY_ONNX=""
for f in "${{ env.MODELS_DIR }}/driving_off_policy.onnx" "${{ env.MODELS_DIR }}/big_driving_off_policy.onnx"; do
[ -f "$f" ] && OFF_POLICY_ONNX="$f" && break
done
ON_POLICY_ONNX=""
for f in "${{ env.MODELS_DIR }}/driving_on_policy.onnx" "${{ env.MODELS_DIR }}/big_driving_on_policy.onnx"; do
[ -f "$f" ] && ON_POLICY_ONNX="$f" && break
done
SUPERCOMBO_ONNX=""
for f in "${{ env.MODELS_DIR }}/supercombo.onnx" "${{ env.MODELS_DIR }}/driving_supercombo.onnx"; do
if [ -f "$f" ]; then
SUPERCOMBO_ONNX="$f"
break
fi
for f in "${{ env.MODELS_DIR }}/supercombo.onnx" "${{ env.MODELS_DIR }}/driving_supercombo.onnx" "${{ env.MODELS_DIR }}/big_supercombo.onnx" "${{ env.MODELS_DIR }}/big_driving_supercombo.onnx"; do
[ -f "$f" ] && SUPERCOMBO_ONNX="$f" && break
done
MODEL_TYPE="" ONNX_ARGS="" OUTPUT_NAME=""
@@ -36,8 +36,11 @@ jobs:
publish_concurrency_group: ${{ steps.strategy.outputs.publish_concurrency_group }}
is_stable_branch: ${{ steps.strategy.outputs.is_stable_branch }}
build: ${{ steps.strategy.outputs.build }}
include_big_model: ${{ steps.strategy.outputs.include_big_model }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Extract deploy strategy
id: strategy
run: |
@@ -78,6 +81,9 @@ jobs:
stable_version=$(cat openpilot/sunnypilot/common/version.h | grep SUNNYPILOT_VERSION | sed -e 's/[^0-9|.]//g');
echo "version=$([ "$is_stable_branch" = "true" ] && echo "$stable_version" || echo "$BUILD")" >> $GITHUB_OUTPUT
echo "extra_version_identifier=${environment}" >> $GITHUB_OUTPUT
include_big_model="$(echo "$CONFIG" | jq -r '.include_big_model // false')";
echo "include_big_model=$include_big_model" >> $GITHUB_OUTPUT
fi
echo "build=$BUILD" >> $GITHUB_OUTPUT
cat $GITHUB_OUTPUT
@@ -92,6 +98,8 @@ jobs:
}}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Wait for Tests
uses: ./.github/workflows/wait-for-action # Path to where you place the action
with:
@@ -115,6 +123,7 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
submodules: recursive
ref: ${{ env.SOURCE_BRANCH }}
repository: ${{ github.event.pull_request.head.repo.fork && github.event.pull_request.head.repo.full_name || github.repository }}
@@ -161,7 +170,7 @@ jobs:
scons -j1 cache_dir="$SCONS_CACHE" --minimal \
openpilot/selfdrive/locationd openpilot/sunnypilot/selfdrive/locationd
echo "Building rest of sunnypilot"
/usr/bin/time -v scons -j$(nproc) cache_dir="$SCONS_CACHE" --minimal
SKIP_TINYGRAD_COMPILE=1 /usr/bin/time -v scons -j$(nproc) cache_dir="$SCONS_CACHE" --minimal
touch ${BUILD_DIR}/prebuilt
if [[ "${{ runner.debug }}" == "1" ]]; then
ls -la ${BUILD_DIR}
@@ -203,22 +212,212 @@ jobs:
source ${UV_PROJECT_ENVIRONMENT}/bin/activate
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
prepare_chestnut:
needs: [ prepare_strategy ]
runs-on: ubuntu-24.04
if: ${{ needs.prepare_strategy.outputs.include_big_model == 'true' }}
outputs:
onnx_sha256: ${{ steps.resolve.outputs.onnx_sha256 }}
env:
GH_REPO: ${{ github.repository }}
HF_REPO: sunnypilot/sunnypilot_models_v1
HF_DEFAULTS_PATH: models/defaults/big
steps:
- name: Resolve ONNX hash and tinygrad ref via API
id: resolve
run: |
REF="${{ github.head_ref || github.ref_name }}"
ONNX_HASH=$(gh api "repos/${GH_REPO}/contents/openpilot/selfdrive/modeld/models/big_driving_supercombo.onnx?ref=${REF}" --jq '.content' | base64 -d | grep '^oid sha256:' | cut -d: -f2)
echo "ONNX hash: $ONNX_HASH"
echo "onnx_sha256=$ONNX_HASH" >> $GITHUB_OUTPUT
TINYGRAD_REF=$(gh api "repos/${GH_REPO}/contents/tinygrad_repo?ref=${REF}" --jq '.sha')
echo "tinygrad ref: $TINYGRAD_REF"
JSON_URL="https://huggingface.co/datasets/${HF_REPO}/resolve/main/${HF_DEFAULTS_PATH}/default_models.json"
check_defaults() {
DEFAULTS=$(curl -fsSL "$JSON_URL" 2>/dev/null) || return 1
TINYGRAD_MATCH=$(echo "$DEFAULTS" | jq -r --arg ref "$TINYGRAD_REF" '.tinygrad_ref == $ref' 2>/dev/null)
[ "$TINYGRAD_MATCH" = "true" ] || return 1
BUNDLE=$(echo "$DEFAULTS" | jq --arg hash "$ONNX_HASH" '.bundles[] | select(.onnx_sha256 == $hash)' 2>/dev/null)
[ -n "$BUNDLE" ] && [ "$BUNDLE" != "null" ]
}
if check_defaults; then
echo "HF defaults match repo ONNX hash and tinygrad ref"
exit 0
fi
echo "No matching model on HF — dispatching build"
gh workflow run build-default-models.yaml --ref "$REF" -f target=big
echo "Polling HF for big model availability..."
for i in $(seq 1 90); do
sleep 30
if check_defaults; then
echo "Big model available on HF after $((i * 30))s"
exit 0
fi
echo "Poll $i/90: not yet available"
done
echo "::error::Big model not available on HF after 45 minutes"
exit 1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Cancel run on failure
if: failure()
run: gh run cancel ${{ github.run_id }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
prepare_small_model:
needs: [ prepare_strategy ]
runs-on: ubuntu-24.04
outputs:
driving_onnx_sha256: ${{ steps.resolve.outputs.driving_onnx_sha256 }}
env:
GH_REPO: ${{ github.repository }}
HF_REPO: sunnypilot/sunnypilot_models_v1
HF_DEFAULTS_PATH: models/defaults/small
steps:
- name: Resolve ONNX hash and tinygrad ref via API
id: resolve
run: |
REF="${{ github.head_ref || github.ref_name }}"
DRIVING_HASH=$(gh api "repos/${GH_REPO}/contents/openpilot/selfdrive/modeld/models/driving_supercombo.onnx?ref=${REF}" --jq '.content' | base64 -d | grep '^oid sha256:' | cut -d: -f2)
echo "Driving ONNX hash: $DRIVING_HASH"
echo "driving_onnx_sha256=$DRIVING_HASH" >> $GITHUB_OUTPUT
TINYGRAD_REF=$(gh api "repos/${GH_REPO}/contents/tinygrad_repo?ref=${REF}" --jq '.sha')
echo "tinygrad ref: $TINYGRAD_REF"
JSON_URL="https://huggingface.co/datasets/${HF_REPO}/resolve/main/${HF_DEFAULTS_PATH}/default_models.json"
check_defaults() {
DEFAULTS=$(curl -fsSL "$JSON_URL" 2>/dev/null) || return 1
TINYGRAD_MATCH=$(echo "$DEFAULTS" | jq -r --arg ref "$TINYGRAD_REF" '.tinygrad_ref == $ref' 2>/dev/null)
[ "$TINYGRAD_MATCH" = "true" ] || return 1
DRIVING=$(echo "$DEFAULTS" | jq --arg hash "$DRIVING_HASH" '.bundles[] | select(.onnx_sha256 == $hash)' 2>/dev/null)
[ -n "$DRIVING" ] && [ "$DRIVING" != "null" ] || return 1
}
if check_defaults; then
echo "HF defaults match repo ONNX hash and tinygrad ref"
exit 0
fi
echo "No matching model on HF — dispatching build"
gh workflow run build-default-models.yaml --ref "$REF" -f target=small
echo "Polling HF for model availability..."
for i in $(seq 1 60); do
sleep 30
if check_defaults; then
echo "Model available on HF after $((i * 30))s"
exit 0
fi
echo "Poll $i/60: not yet available"
done
echo "::error::Small driving model not available on HF after 30 minutes"
exit 1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Cancel run on failure
if: failure()
run: gh run cancel ${{ github.run_id }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
prepare_dm_model:
needs: [ prepare_strategy ]
runs-on: ubuntu-24.04
outputs:
dm_onnx_sha256: ${{ steps.resolve.outputs.dm_onnx_sha256 }}
env:
GH_REPO: ${{ github.repository }}
HF_REPO: sunnypilot/sunnypilot_models_v1
HF_DEFAULTS_PATH: models/defaults/dm
steps:
- name: Resolve ONNX hash and tinygrad ref via API
id: resolve
run: |
REF="${{ github.head_ref || github.ref_name }}"
DM_HASH=$(gh api "repos/${GH_REPO}/contents/openpilot/selfdrive/modeld/models/dmonitoring_model.onnx?ref=${REF}" --jq '.content' | base64 -d | grep '^oid sha256:' | cut -d: -f2)
echo "DM ONNX hash: $DM_HASH"
echo "dm_onnx_sha256=$DM_HASH" >> $GITHUB_OUTPUT
TINYGRAD_REF=$(gh api "repos/${GH_REPO}/contents/tinygrad_repo?ref=${REF}" --jq '.sha')
echo "tinygrad ref: $TINYGRAD_REF"
JSON_URL="https://huggingface.co/datasets/${HF_REPO}/resolve/main/${HF_DEFAULTS_PATH}/default_models.json"
check_defaults() {
DEFAULTS=$(curl -fsSL "$JSON_URL" 2>/dev/null) || return 1
TINYGRAD_MATCH=$(echo "$DEFAULTS" | jq -r --arg ref "$TINYGRAD_REF" '.tinygrad_ref == $ref' 2>/dev/null)
[ "$TINYGRAD_MATCH" = "true" ] || return 1
DM=$(echo "$DEFAULTS" | jq --arg hash "$DM_HASH" '.bundles[] | select(.onnx_sha256 == $hash)' 2>/dev/null)
[ -n "$DM" ] && [ "$DM" != "null" ] || return 1
}
if check_defaults; then
echo "HF defaults match DM ONNX hash and tinygrad ref"
exit 0
fi
echo "No matching DM model on HF — dispatching build"
gh workflow run build-default-models.yaml --ref "$REF" -f target=dm
echo "Polling HF for DM model availability..."
for i in $(seq 1 60); do
sleep 30
if check_defaults; then
echo "DM model available on HF after $((i * 30))s"
exit 0
fi
echo "Poll $i/60: not yet available"
done
echo "::error::DM model not available on HF after 30 minutes"
exit 1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Cancel run on failure
if: failure()
run: gh run cancel ${{ github.run_id }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
publish:
concurrency:
# We do a bit of a hack here to avoid canceling the publishing job if a new commit comes in while we're publishing by adding the sha to the group name.
# This means that if multiple commits come in while we're publishing, they will be queued up and publish one after the other.
# Otherwise, if a job is waiting to be published due to environment wait time, it would be canceled by a new commit and restart the wait time.
group: ${{ needs.prepare_strategy.outputs.publish_concurrency_group }}
cancel-in-progress: ${{ needs.prepare_strategy.outputs.cancel_publish_in_progress == 'true' }}
if: ${{ (always() && !cancelled() && !failure()) && needs.build.result == 'success' && needs.prepare_strategy.result == 'success' && (!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt')) }}
needs: [ build, prepare_strategy ]
if: ${{
always() && !cancelled() &&
needs.build.result == 'success' &&
needs.prepare_strategy.result == 'success' &&
needs.prepare_small_model.result == 'success' &&
needs.prepare_dm_model.result == 'success' &&
(!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt')) &&
(needs.prepare_strategy.outputs.include_big_model != 'true' || needs.prepare_chestnut.result == 'success')
}}
needs: [ build, prepare_strategy, prepare_chestnut, prepare_small_model, prepare_dm_model ]
runs-on: ubuntu-24.04
environment: ${{ needs.prepare_strategy.outputs.environment }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Download build artifacts
- name: Download prebuilt artifact
uses: actions/download-artifact@v4
with:
name: prebuilt
@@ -228,6 +427,17 @@ jobs:
mkdir -p ${{ env.OUTPUT_DIR }}
tar xzf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }}
- name: Download model chunks from HF
uses: ./.github/workflows/download-hf-model-chunks
with:
hf_repo: sunnypilot/sunnypilot_models_v1
dest_dir: ${{ env.OUTPUT_DIR }}/openpilot/selfdrive/modeld/models
models: |
[
{"hf_path": "models/defaults/small", "onnx_hash": "${{ needs.prepare_small_model.outputs.driving_onnx_sha256 }}", "canonical": "driving_tinygrad.pkl"},
{"hf_path": "models/defaults/dm", "onnx_hash": "${{ needs.prepare_dm_model.outputs.dm_onnx_sha256 }}", "canonical": "dmonitoring_model_tinygrad.pkl"}
]
- name: Configure Git
run: |
git config --global user.email "github-actions[bot]@users.noreply.github.com"
@@ -255,11 +465,77 @@ jobs:
git tag -f -a ${TAG} -m "${{ needs.prepare_strategy.outputs.environment }} @ ${{ needs.prepare_strategy.outputs.version }} of build ${{ needs.prepare_strategy.outputs.build }}."
git push -f origin ${TAG}
publish_chestnut:
concurrency:
group: ${{ needs.prepare_strategy.outputs.publish_concurrency_group }}-chestnut
cancel-in-progress: ${{ needs.prepare_strategy.outputs.cancel_publish_in_progress == 'true' }}
if: ${{
always() && !cancelled() &&
needs.build.result == 'success' &&
needs.prepare_strategy.result == 'success' &&
needs.prepare_small_model.result == 'success' &&
needs.prepare_dm_model.result == 'success' &&
needs.prepare_chestnut.result == 'success' &&
(!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
}}
needs: [ build, prepare_strategy, prepare_chestnut, prepare_small_model, prepare_dm_model ]
runs-on: ubuntu-24.04
environment: ${{ needs.prepare_strategy.outputs.environment }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Download prebuilt artifact
uses: actions/download-artifact@v4
with:
name: prebuilt
- name: Untar prebuilt
run: |
mkdir -p ${{ env.OUTPUT_DIR }}
tar xzf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }}
- name: Download model chunks from HF
uses: ./.github/workflows/download-hf-model-chunks
with:
hf_repo: sunnypilot/sunnypilot_models_v1
dest_dir: ${{ env.OUTPUT_DIR }}/openpilot/selfdrive/modeld/models
models: |
[
{"hf_path": "models/defaults/small", "onnx_hash": "${{ needs.prepare_small_model.outputs.driving_onnx_sha256 }}", "canonical": "driving_tinygrad.pkl"},
{"hf_path": "models/defaults/dm", "onnx_hash": "${{ needs.prepare_dm_model.outputs.dm_onnx_sha256 }}", "canonical": "dmonitoring_model_tinygrad.pkl"},
{"hf_path": "models/defaults/big", "onnx_hash": "${{ needs.prepare_chestnut.outputs.onnx_sha256 }}", "canonical": "big_driving_tinygrad.pkl"}
]
- name: Configure Git
run: |
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"
- name: Publish chestnut branch
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
CHESTNUT_BRANCH="${{ needs.prepare_strategy.outputs.new_branch }}-chestnut"
${{ env.CI_DIR }}/publish.sh \
"${{ github.workspace }}" \
"${{ env.OUTPUT_DIR }}" \
"$CHESTNUT_BRANCH" \
"${{ needs.prepare_strategy.outputs.version }}" \
"https://x-access-token:${{github.token}}@github.com/sunnypilot/sunnypilot.git" \
"${{ needs.prepare_strategy.outputs.extra_version_identifier }}"
notify:
needs:
- prepare_strategy
- build
- publish
- publish_chestnut
- prepare_chestnut
- prepare_small_model
- prepare_dm_model
runs-on: ubuntu-24.04
if: ${{ (always() && !cancelled() && !failure())
&& needs.publish.result == 'success'
@@ -267,6 +543,8 @@ jobs:
&& (fromJSON(vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES_V2)[github.head_ref || github.ref_name] != null) }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Prepare notification message
id: message
@@ -279,6 +557,7 @@ jobs:
export commit_short_sha="${commit_short_sha:0:7}"
export extra_version_identifier="${{ needs.prepare_strategy.outputs.extra_version_identifier || github.run_number }}"
export PUBLIC_REPO_URL="${{ env.PUBLIC_REPO_URL }}"
export chestnut_branch="${{ needs.prepare_chestnut.result == 'success' && format('{0}-chestnut', needs.prepare_strategy.outputs.new_branch) || '' }}"
MESSAGE=$(cat << 'EOF' | envsubst
${{ vars.DISCOURSE_GENERAL_UPDATE_NOTICE }}
+3 -1
View File
@@ -24,7 +24,9 @@ function agnos_init {
if $AGNOS_PY --verify $MANIFEST; then
sudo reboot
fi
$DIR/openpilot/common/hardware/comma/updater $AGNOS_PY $MANIFEST
while true; do
$DIR/openpilot/common/hardware/comma/updater $AGNOS_PY $MANIFEST
done
fi
}
+1 -1
View File
@@ -16,7 +16,7 @@ export VECLIB_MAXIMUM_THREADS=1
export QCOM_PRIORITY=12
if [ -z "$AGNOS_VERSION" ]; then
export AGNOS_VERSION="19.6"
export AGNOS_VERSION="19.7"
fi
export STAGING_ROOT="/data/safe_staging"
+2
View File
@@ -131,6 +131,7 @@ struct ModelManagerSP @0xaedffd8f31e7b55d {
downloaded @2;
cached @3;
failed @4;
verifying @5;
}
struct DownloadProgress {
@@ -352,6 +353,7 @@ struct OnroadEventSP @0xda96579883444c35 {
speedLimitPending @22;
e2eChime @23;
laneChangeRoadEdge @24;
bigModelReady @25;
}
}
+2
View File
@@ -725,6 +725,7 @@ struct ChestnutState {
pcieLtssm @7 :UInt8;
supplyVoltage @8 :UInt16; # mV
supplyCurrent @9 :Int16; # mA
supplyFault @10 :Bool;
}
struct RadarState @0x9a185389d6fdd05f {
@@ -1004,6 +1005,7 @@ struct DrivingModelData {
frameIdExtra @1 :UInt32;
frameDropPerc @6 :Float32;
modelExecutionTime @7 :Float32;
big @8 :Bool;
action @2 :ModelDataV2.Action;
+11 -11
View File
@@ -56,29 +56,29 @@
},
{
"name": "boot",
"url": "https://commadist.azureedge.net/agnosupdate/boot-b30f5eef65ec3878f3aa3dcaf2cc95c09e2c1e661cd3a38e94da37dee76f68bd.img.xz",
"hash": "b30f5eef65ec3878f3aa3dcaf2cc95c09e2c1e661cd3a38e94da37dee76f68bd",
"hash_raw": "b30f5eef65ec3878f3aa3dcaf2cc95c09e2c1e661cd3a38e94da37dee76f68bd",
"url": "https://commadist.azureedge.net/agnosupdate/boot-6ecf6f987cd11968104abcccabbe268485d329cdb73012dfd3c381a6b8deb27d.img.xz",
"hash": "6ecf6f987cd11968104abcccabbe268485d329cdb73012dfd3c381a6b8deb27d",
"hash_raw": "6ecf6f987cd11968104abcccabbe268485d329cdb73012dfd3c381a6b8deb27d",
"size": 46897152,
"sparse": false,
"full_check": true,
"has_ab": true,
"ondevice_hash": "6650e4c46df99ae6dfd6ee895a34b8a2a3cc490a8ce18e16cc3c451c3f822b6e"
"ondevice_hash": "d12e1e5b9455b62a1464558716493b33e470d7a7e88da1c4105a3b21d0961808"
},
{
"name": "system",
"url": "https://commadist.azureedge.net/agnosupdate/system-5b6ce7965904a157fd3a134ccfcb854f9ca5c1cc2a26b7cb80a4fa4e1cc4aaa3.img.xz",
"hash": "b134fd04e9da27fa1d359ea0f2742c216fa21a08b5c47e9be22ab3b0563d9b9b",
"hash_raw": "5b6ce7965904a157fd3a134ccfcb854f9ca5c1cc2a26b7cb80a4fa4e1cc4aaa3",
"url": "https://commadist.azureedge.net/agnosupdate/system-3c271e2b3d20d2f0a8bf6555a1319f3efb12845490967d6151195174a01e912f.img.xz",
"hash": "74ffc9c551e1f29cda897ace8a69080fe644f8039977c6885f2b48362e39b744",
"hash_raw": "3c271e2b3d20d2f0a8bf6555a1319f3efb12845490967d6151195174a01e912f",
"size": 4718592000,
"sparse": true,
"full_check": false,
"has_ab": true,
"ondevice_hash": "91242772af771ae96fe2eebc105f2b80a7e1dbaaf6003c2574b62d51b806f468",
"ondevice_hash": "6a992680183685eea9db99d915219a37935f45989330d9b619e880450257f448",
"alt": {
"hash": "5b6ce7965904a157fd3a134ccfcb854f9ca5c1cc2a26b7cb80a4fa4e1cc4aaa3",
"url": "https://commadist.azureedge.net/agnosupdate/system-5b6ce7965904a157fd3a134ccfcb854f9ca5c1cc2a26b7cb80a4fa4e1cc4aaa3.img",
"hash": "3c271e2b3d20d2f0a8bf6555a1319f3efb12845490967d6151195174a01e912f",
"url": "https://commadist.azureedge.net/agnosupdate/system-3c271e2b3d20d2f0a8bf6555a1319f3efb12845490967d6151195174a01e912f.img",
"size": 4718592000
}
}
]
]
+2 -1
View File
@@ -5,6 +5,7 @@ import logging
import os
import select
import signal
import string
import struct
import subprocess
import tempfile
@@ -354,7 +355,7 @@ class Modem:
imei = ""
iccid = (self._atv("AT+QCCID", "+QCCID:") or "").rstrip("F")
if not iccid.isdigit():
if not all(c in string.hexdigits for c in iccid):
iccid = ""
imsi = first_line("AT+CIMI")
+7 -1
View File
@@ -4,11 +4,17 @@ from pathlib import Path
CHESTNUT_FW_VERSION = "ed4e39b7"
CHESTNUT_USB_IDS = ((0xADD1, 0x0001), (0x3801, 0x0001))
CHESTNUT_ROM_USB_IDS = ((0x174C, 0x2464), (0x174C, 0x2463))
CHESTNUT_USB_PRODUCT = f"custom {CHESTNUT_FW_VERSION}-CLEAN"
USB_DEVICES_PATH = Path("/sys/bus/usb/devices")
TYPEC_CC_ORIENTATION_PATH = Path("/sys/class/power_supply/usb/typec_cc_orientation")
PRIMARY_USB_CONTROLLER = "a600000.ssusb"
def is_chestnut_usb_id(vendor_id: int, product_id: int, include_bootloader: bool = False) -> bool:
ids = CHESTNUT_USB_IDS + CHESTNUT_ROM_USB_IDS if include_bootloader else CHESTNUT_USB_IDS
return (vendor_id, product_id) in ids
def get_usb_topology() -> set[str]:
try:
return set(os.listdir(USB_DEVICES_PATH))
@@ -81,7 +87,7 @@ def set_usb_state(device_state, devices: list[dict]) -> None:
entry.linkErrorCount = device["linkErrorCount"]
entry.usb3Lane = device.get("usb3Lane", "unknown")
if (entry.vendorId, entry.productId) in CHESTNUT_USB_IDS:
if is_chestnut_usb_id(entry.vendorId, entry.productId):
chestnut_present = True
device_state.chestnutPresent = chestnut_present
+17 -7
View File
@@ -59,7 +59,7 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"IsDriverViewEnabled", {CLEAR_ON_MANAGER_START, BOOL}},
{"IsEngaged", {PERSISTENT, BOOL}},
{"IsLdwEnabled", {PERSISTENT | BACKUP, BOOL}},
{"IsLiveStreaming", {CLEAR_ON_MANAGER_START, BOOL}},
{"IsLiveStreaming", {CLEAR_ON_MANAGER_START | CLEAR_ON_IGNITION_ON, BOOL}},
{"IsMetric", {PERSISTENT | BACKUP, BOOL}},
{"IsOffroad", {CLEAR_ON_MANAGER_START, BOOL}},
{"IsRhdDetected", {PERSISTENT, BOOL}},
@@ -92,6 +92,12 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"ObdMultiplexingEnabled", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, BOOL}},
{"Offroad_CarUnrecognized", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, JSON}},
{"Offroad_ChestnutBranch", {CLEAR_ON_MANAGER_START, JSON}},
{"Offroad_ChestnutNotDetected", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, JSON}},
{"Offroad_ChestnutOverheated", {CLEAR_ON_MANAGER_START, JSON}},
{"Offroad_ChestnutPcieUnavailable", {CLEAR_ON_MANAGER_START, JSON}},
{"Offroad_ChestnutUncompiled", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, JSON}},
{"Offroad_ChestnutUpdateFailed", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, JSON}},
{"Offroad_ChestnutUsbSlow", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, JSON}},
{"Offroad_ConnectivityNeeded", {CLEAR_ON_MANAGER_START, JSON}},
{"Offroad_ConnectivityNeededPrompt", {CLEAR_ON_MANAGER_START, JSON}},
{"Offroad_ExcessiveActuation", {PERSISTENT, JSON}},
@@ -130,8 +136,9 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"UpdaterLastFetchTime", {PERSISTENT, TIME}},
{"UptimeOffroad", {PERSISTENT, FLOAT, "0.0"}},
{"UptimeOnroad", {PERSISTENT, FLOAT, "0.0"}},
{"UsbGpuActive", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION | CLEAR_ON_IGNITION_ON, BOOL}},
{"UsbGpuLoading", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION | CLEAR_ON_IGNITION_ON, BOOL}},
{"ChestnutActive", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION | CLEAR_ON_IGNITION_ON, BOOL}},
{"ChestnutLoading", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION | CLEAR_ON_IGNITION_ON, BOOL}},
{"ChestnutModelError", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION | CLEAR_ON_IGNITION_ON, BOOL}},
{"Version", {PERSISTENT, STRING}},
// --- sunnypilot params --- //
@@ -195,14 +202,16 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
// Model Manager params
{"ModelManager_ActiveBundle", {PERSISTENT, JSON}},
{"ModelManager_ActiveJson", {CLEAR_ON_MANAGER_START, STRING}},
{"ModelManager_ActiveBundleUSBGPU", {PERSISTENT, JSON}}, //TODO-SP: kept for migration, remove on next sync?
{"ModelManager_ActiveBundleChestnut", {PERSISTENT, JSON}},
{"ModelManager_ActiveJson", {CLEAR_ON_MANAGER_START, JSON}},
{"ModelManager_ClearCache", {CLEAR_ON_MANAGER_START, BOOL}},
{"ModelManager_DownloadIndex", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, INT}},
{"ModelManager_DownloadRef", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, STRING}},
{"ModelManager_Favs", {PERSISTENT | BACKUP, STRING}},
{"ModelManager_LastSyncTime", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, INT, "0"}},
{"ModelManager_LastSyncTime_USBGPU", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, INT, "0"}},
{"ModelManager_LastSyncTime_Chestnut", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, INT, "0"}},
{"ModelManager_ModelsCache", {PERSISTENT | BACKUP, JSON}},
{"ModelManager_ModelsCache_USBGPU", {PERSISTENT | BACKUP, JSON}},
{"ModelManager_ModelsCache_Chestnut", {PERSISTENT | BACKUP, JSON}},
// Neural Network Lateral Control
{"NeuralNetworkLateralControl", {PERSISTENT | BACKUP, BOOL, "0"}},
@@ -245,6 +254,7 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
// mapd
{"MapAdvisorySpeedLimit", {CLEAR_ON_ONROAD_TRANSITION, FLOAT}},
{"Mapd_ClearCache", {CLEAR_ON_MANAGER_START, BOOL}},
{"MapdVersion", {PERSISTENT, STRING}},
{"MapSpeedLimit", {CLEAR_ON_ONROAD_TRANSITION, FLOAT, "0.0"}},
{"NextMapSpeedLimit", {CLEAR_ON_ONROAD_TRANSITION, JSON}},
+4 -4
View File
@@ -27,14 +27,14 @@ public:
auto param_path = Params().getParamPath();
if (util::file_exists(param_path)) {
std::string real_path = util::readlink(param_path);
util::check_system(util::string_format("rm %s -rf", real_path.c_str()));
util::check_system(util::string_format("rm -rf %s", real_path.c_str()));
unlink(param_path.c_str());
}
if (getenv("COMMA_CACHE") == nullptr) {
util::check_system(util::string_format("rm %s -rf", Path::download_cache_root().c_str()));
util::check_system(util::string_format("rm -rf %s", Path::download_cache_root().c_str()));
}
util::check_system(util::string_format("rm %s -rf", Path::comma_home().c_str()));
util::check_system(util::string_format("rm %s -rf", msgq_path.c_str()));
util::check_system(util::string_format("rm -rf %s", Path::comma_home().c_str()));
util::check_system(util::string_format("rm -rf %s", msgq_path.c_str()));
unsetenv("OPENPILOT_PREFIX");
}
+9
View File
@@ -16,6 +16,15 @@ MASTER_SP_BRANCHES = ['master']
RELEASE_BRANCHES = ['release-tizi-staging', 'release-mici-staging', 'release-tizi', 'release-mici', 'nightly']
TESTED_BRANCHES = RELEASE_BRANCHES + ['devel-staging', 'nightly-dev'] + RELEASE_SP_BRANCHES + TESTED_SP_BRANCHES
CHESTNUT_BRANCHES = {
"staging": "staging-chestnut",
"dev": "dev-chestnut",
"release-mici": "release-chestnut",
"release-tizi": "release-chestnut",
"release-mici-staging": "release-chestnut-staging",
"release-tizi-staging": "release-chestnut-staging",
}
SP_BRANCH_MIGRATIONS = {
("tici", "staging-c3-new"): "staging-tici",
("tici", "dev-c3-new"): "staging-tici",
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:845c40ff0d37612e8f2f482a36845744b5ae91ce2fcfc8117990d7d278b59820
size 13079
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8a8c5fece2a1c7587feb41cbe04c6aee08e768ecd9b5d00da6af9832a4ccc842
size 2034
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7409c53d7c72681c24982fd83b56ce70f80797c9c0f936d9296a5c18557ac472
size 7279
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:58bd6155433f623b1f75d134bd8ca4745d9aa71f6767eb807cdbcf7deb3089a1
size 10876
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:07bda2fe5d6be0b2854044053c384fe002e96406da119863a443b9344258b500
size 1544
+51 -54
View File
@@ -7,14 +7,9 @@ from openpilot.common.file_chunker import chunk_file, get_chunk_targets, get_exi
from openpilot.common.transformations.camera import _ar_ox_fisheye, _os_fisheye
from openpilot.common.transformations.model import MEDMODEL_INPUT_SIZE, DM_INPUT_SIZE
from openpilot.selfdrive.modeld.constants import ModelConstants
from openpilot.selfdrive.modeld.helpers import TG_INPUT_DEVICES_PATH, usbgpu_present, modeld_pkl_path
from openpilot.selfdrive.modeld.helpers import TG_INPUT_DEVICES_PATH, chestnut_present, modeld_pkl_path
CAMERA_CONFIGS = [
(_ar_ox_fisheye.width, _ar_ox_fisheye.height), # tici: 1928x1208
(_os_fisheye.width, _os_fisheye.height), # mici: 1344x760
]
Import('env', 'arch')
chunker_file = File("#openpilot/common/file_chunker.py")
lenv = env.Clone()
@@ -24,30 +19,32 @@ tinygrad_files = ["#"+x for x in glob.glob(env.Dir("#tinygrad_repo").relpath + "
if 'pycache' not in x and os.path.isfile(os.path.join(tinygrad_root, x))]
def estimate_pickle_max_size(onnx_size):
return 1.2 * onnx_size + 10 * 1024 * 1024 # 20% + 10MB is plenty
# QCOM programs for models with spatial recurrent features can approach 2x
# the ONNX size. Overestimating only adds an empty trailing chunk.
return 2.0 * onnx_size + 10 * 1024 * 1024
if arch == 'comma_arm64':
from openpilot.common.hardware import HARDWARE
camera = _os_fisheye if HARDWARE.get_device_type() == "mici" else _ar_ox_fisheye
camera_configs = [(camera.width, camera.height)]
tg_backend = 'QCOM'
tg_flags = f'DEV={tg_backend} IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1'
else:
camera_configs = [(c.width, c.height) for c in (_ar_ox_fisheye, _os_fisheye)]
tg_backend = 'CPU'
tg_flags = f'DEV=CPU' if arch == 'Darwin' else 'DEV=CPU:LLVM'
tg_devices = { # which device to put jit inputs to at runtime
'openpilot.selfdrive.modeld.modeld': {
'default': {'WARP_DEV': tg_backend, 'QUEUE_DEV': tg_backend},
'usbgpu': {'WARP_DEV': tg_backend, 'QUEUE_DEV': 'AMD'}
},
'openpilot.selfdrive.modeld.dmonitoringmodeld': {
'default': {'DEV': tg_backend}
},
}
USBGPU = usbgpu_present()
if USBGPU:
usbgpu_tg_flags = f'DEBUG=2 DEV=USB+AMD:LLVM WARP_DEV={tg_backend} FLOAT16=1 JIT_BATCH_SIZE=0 GMMU=0 TC_OPT=2'
CHESTNUT = chestnut_present()
if CHESTNUT:
chestnut_tg_flags = 'DEBUG=1 DEV=USB+AMD:LLVM FRAME_DEV=CPU FLOAT16=1 JIT_BATCH_SIZE=0 GMMU=0 TC_OPT=2 TC_OCCUPANCY_OPT=1'
# the USB+AMD GPU takes an exclusive flock; serialize all targets that touch it
usbgpu_lock = File("models/.usb_gpu.lock").abspath
chestnut_lock = File("models/.chestnut.lock").abspath
def write_tg_devices(target, source, env):
with open(str(target[0]), "w") as f:
@@ -73,44 +70,44 @@ compile_modeld_script = [
model_w, model_h = MEDMODEL_INPUT_SIZE
frame_skip = ModelConstants.MODEL_RUN_FREQ // ModelConstants.MODEL_CONTEXT_FREQ
for usbgpu in [False, True] if USBGPU else [False]:
target_pkl_path = File(modeld_pkl_path(usbgpu)).abspath
# BIG_INTO_SMALL=1 builds the default target from the big model, e.g. to test it without a USB GPU
file_prefix, cmd_flags = ('big_', usbgpu_tg_flags) if usbgpu else ('big_' if os.getenv('BIG_INTO_SMALL') else '', tg_flags)
driving_onnx_deps = get_existing_chunks(File(f"models/{file_prefix}driving_supercombo.onnx").abspath)
camera_res_args = ' '.join(f'{cw}x{ch}' for cw, ch in CAMERA_CONFIGS)
# CPU 7 is isolated with isolcpus on AGNOS, so explicitly pin the compiler to it.
taskset = 'taskset -c 7 ' if arch == 'comma_arm64' else ''
cmd = (f'{cmd_flags} {mac_brew_string} {taskset}python3 {modeld_dir}/compile_modeld.py '
f'--model-size {model_w}x{model_h} '
f'--camera-resolutions {camera_res_args} '
f'--onnx {File(f"models/{file_prefix}driving_supercombo.onnx").abspath} '
f'--output {target_pkl_path} --frame-skip {frame_skip}')
onnx_sizes_sum = sum(os.path.getsize(f) for f in driving_onnx_deps)
chunk_targets = get_chunk_targets(target_pkl_path, estimate_pickle_max_size(onnx_sizes_sum))
def do_compile(target, source, env, command=cmd, pkl=target_pkl_path, chunks=chunk_targets):
from openpilot.system.hardware.chestnut.flash import link_up
# chestnut can enumerate before its PCIe link is up due to varying 12V power behavior across cars
for _ in range(10):
if link_up():
break
time.sleep(1)
else:
print("Chestnut not ready, skipping big model build")
return
if ret := env.Execute(command):
return ret
chunk_file(pkl, chunks)
def do_chunk(target, source, env, pkl=target_pkl_path, chunks=chunk_targets):
chunk_file(pkl, chunks)
actions = Action(do_compile, " [USBGPU] $TARGET") if usbgpu else [cmd, Action(do_chunk, " [CHUNK] $TARGET")]
node = lenv.Command(
chunk_targets,
tinygrad_files + compile_modeld_script + driving_onnx_deps + [Value(chunk_targets), chunker_file],
actions,
)
if usbgpu:
lenv.SideEffect(usbgpu_lock, node)
if not os.getenv('SKIP_TINYGRAD_COMPILE'):
for chestnut in [False, True] if CHESTNUT else [False]:
target_pkl_path = File(modeld_pkl_path(chestnut)).abspath
file_prefix, cmd_flags = ('big_', chestnut_tg_flags) if chestnut else ('', tg_flags)
driving_onnx_deps = get_existing_chunks(File(f"models/{file_prefix}driving_supercombo.onnx").abspath)
camera_res_args = ' '.join(f'{cw}x{ch}' for cw, ch in camera_configs)
# CPU 7 is isolated with isolcpus on AGNOS, so explicitly pin the compiler to it.
taskset = 'taskset -c 7 ' if arch == 'comma_arm64' else ''
cmd = (f'{cmd_flags} {mac_brew_string} {taskset}python3 {modeld_dir}/compile_modeld.py '
f'--model-size {model_w}x{model_h} '
f'--camera-resolutions {camera_res_args} '
f'--onnx {File(f"models/{file_prefix}driving_supercombo.onnx").abspath} '
f'--output {target_pkl_path} --frame-skip {frame_skip}')
onnx_sizes_sum = sum(os.path.getsize(f) for f in driving_onnx_deps)
chunk_targets = get_chunk_targets(target_pkl_path, estimate_pickle_max_size(onnx_sizes_sum))
def do_compile(target, source, env, command=cmd, pkl=target_pkl_path, chunks=chunk_targets):
from openpilot.system.hardware.chestnut.flash import link_up
# chestnut can enumerate before its PCIe link is up due to varying 12V power behavior across cars
for _ in range(10):
if link_up():
break
time.sleep(1)
else:
print("Chestnut not ready, skipping big model build")
return
if ret := env.Execute(command):
return ret
chunk_file(pkl, chunks)
def do_chunk(target, source, env, pkl=target_pkl_path, chunks=chunk_targets):
chunk_file(pkl, chunks)
actions = Action(do_compile, " [CHESTNUT] $TARGET") if chestnut else [cmd, Action(do_chunk, " [CHUNK] $TARGET")]
node = lenv.Command(
chunk_targets,
tinygrad_files + compile_modeld_script + driving_onnx_deps + [Value(camera_res_args), Value(chunk_targets), chunker_file],
actions,
)
if chestnut:
lenv.SideEffect(chestnut_lock, node)
# get model metadata
fn = File(f"models/dmonitoring_model").abspath
@@ -120,7 +117,7 @@ lenv.Command(fn + "_metadata.pkl", [fn + ".onnx"] + tinygrad_files + script_file
dm_w, dm_h = DM_INPUT_SIZE
compile_dm_warp_script = [File(f"{modeld_dir}/compile_dm_warp.py")]
for cam_w, cam_h in CAMERA_CONFIGS:
for cam_w, cam_h in camera_configs:
dm_pkl_path = File(f"models/dm_warp_{cam_w}x{cam_h}_tinygrad.pkl").abspath
cmd = (f'{tg_flags} {mac_brew_string} python3 {modeld_dir}/compile_dm_warp.py '
f'--camera-resolution {cam_w}x{cam_h} --warp-to {dm_w}x{dm_h} '
+86 -73
View File
@@ -37,17 +37,12 @@ from tinygrad.engine.jit import TinyJit
NV12Frame = namedtuple("NV12Frame", ['width', 'height', 'stride', 'y_height', 'uv_height', 'size'])
WARP_INPUTS = ['tfm', 'big_tfm']
POLICY_INPUTS = ['img_q', 'big_img_q', 'feat_q', 'desire_q', 'packed_npy_inputs']
UV_SCALE_MATRIX = np.array([[0.5, 0, 0], [0, 0.5, 0], [0, 0, 1]], dtype=np.float32)
UV_SCALE_MATRIX_INV = np.linalg.inv(UV_SCALE_MATRIX)
WARP_DEV = os.getenv('WARP_DEV')
MODELD_INPUTS = ['img_q', 'big_img_q', 'feat_q', 'desire_q', 'packed_npy_inputs']
def make_random_images(keys, shape, device=None):
return {k: Tensor.randint(shape, low=0, high=256, dtype='uint8', device=device).realize() for k in keys}
def nv12_copy_size(stride: int, y_height: int, uv_height: int) -> int:
# Retain the padded Y and UV plane storage, but skip the trailing kernel/guard allocation.
return stride * (y_height + uv_height)
def warp_perspective_tinygrad(src_flat, M_inv, dst_shape, src_shape, stride_pad, border_fill_val=None):
@@ -99,7 +94,7 @@ def make_frame_prepare(nv12: NV12Frame, model_w, model_h):
def frame_prepare_tinygrad(input_frame, M_inv):
# UV_SCALE @ M_inv @ UV_SCALE_INV simplifies to elementwise scaling
M_inv_uv = M_inv * Tensor([[1.0, 1.0, 0.5], [1.0, 1.0, 0.5], [2.0, 2.0, 1.0]], device=WARP_DEV)
M_inv_uv = M_inv * Tensor([[1.0, 1.0, 0.5], [1.0, 1.0, 0.5], [2.0, 2.0, 1.0]], device=Device.DEFAULT)
# deinterleave NV12 UV plane (UVUV... -> separate U, V)
uv = input_frame[uv_offset:uv_offset + uv_height * stride].reshape(uv_height, stride)
with Context(SPLIT_REDUCEOP=0):
@@ -118,49 +113,43 @@ def make_frame_prepare(nv12: NV12Frame, model_w, model_h):
return frame_prepare_tinygrad
def make_warp_input_queues(vision_input_shapes, frame_skip, device):
img = vision_input_shapes['img'] # (1, 12, 128, 256)
n_frames = img[1] // 6
img_buf_shape = (frame_skip * (n_frames - 1) + 1, 6, img[2], img[3])
npy = {
'tfm': np.zeros((3, 3), dtype=np.float32),
'big_tfm': np.zeros((3, 3), dtype=np.float32),
}
input_queues = {
'img_q': Tensor(np.zeros(img_buf_shape, dtype=np.uint8), device=device).contiguous().realize(),
'big_img_q': Tensor(np.zeros(img_buf_shape, dtype=np.uint8), device=device).contiguous().realize(),
**{k: Tensor(v, device='NPY').realize() for k, v in npy.items()},
}
return input_queues, npy
def get_policy_npy_shapes(input_shapes):
dp = input_shapes['desire_pulse'] # (1, 25, 8)
tc = input_shapes['traffic_convention'] # (1, 2)
at = input_shapes['action_t'] # (1, 2)
fb = input_shapes['features_buffer'] # (1, 24, 512)
fb = input_shapes['features_buffer'] # (1, T-1, ...) e.g. (1, 24, 32, 512) with spatial features
feat_dim = math.prod(fb[2:])
# TODO prev_feat shouldn't exist and be handled inside the JIT, but corrupt on QCOM for now
shapes = {'desire': (dp[2],), 'traffic_convention': tuple(tc), 'action_t': tuple(at), 'prev_feat': (fb[0], fb[2])}
shapes = {'desire': (dp[2],), 'traffic_convention': tuple(tc), 'action_t': tuple(at), 'prev_feat': (fb[0], feat_dim)}
return shapes, [math.prod(s) for s in shapes.values()]
def make_input_queues(input_shapes, frame_skip, device):
input_queues, npy = make_warp_input_queues(input_shapes, frame_skip, device)
fb = input_shapes['features_buffer'] # (1, 24, 512), past features only; the model appends the current frame's feature
def make_input_queues(input_shapes, frame_skip, device, frame_copy_size):
img = input_shapes['img'] # (1, 12, 128, 256)
fb = input_shapes['features_buffer'] # (1, T-1, ...), past features only; the model appends the current frame's feature
feat_dim = math.prod(fb[2:])
dp = input_shapes['desire_pulse'] # (1, 25, 8)
n_frames = img[1] // 6
img_buf_shape = (frame_skip * (n_frames - 1) + 1, 6, img[2], img[3])
shapes, sizes = get_policy_npy_shapes(input_shapes)
packed_npy_inputs = np.zeros(sum(sizes), dtype=np.float32)
policy_shapes, _ = get_policy_npy_shapes(input_shapes)
shapes = {'tfm': (3, 3), 'big_tfm': (3, 3)} | policy_shapes
sizes = [math.prod(s) for s in shapes.values()]
packed_npy_size = sum(sizes) * np.dtype(np.float32).itemsize
packed_input = np.zeros(packed_npy_size + 2 * frame_copy_size, dtype=np.uint8)
packed_npy_inputs = packed_input[:packed_npy_size].view(np.float32)
frames = packed_input[packed_npy_size:]
frame_views = {'img': frames[:frame_copy_size], 'big_img': frames[frame_copy_size:]}
# views into the packed inputs, to be refilled at runtime
npy.update({k: v.reshape(s) for (k, s), v in zip(shapes.items(), np.split(packed_npy_inputs, np.cumsum(sizes[:-1])), strict=True)})
input_queues.update({
'feat_q': Tensor(np.zeros((frame_skip * fb[1], fb[0], fb[2]), dtype=np.float32), device=device).contiguous().realize(),
npy = {k: v.reshape(s) for (k, s), v in zip(shapes.items(), np.split(packed_npy_inputs, np.cumsum(sizes[:-1])), strict=True)}
input_queues = {
'img_q': Tensor(np.zeros(img_buf_shape, dtype=np.uint8), device=device).contiguous().realize(),
'big_img_q': Tensor(np.zeros(img_buf_shape, dtype=np.uint8), device=device).contiguous().realize(),
'feat_q': Tensor(np.zeros((frame_skip * fb[1], fb[0], feat_dim), dtype=np.float32), device=device).contiguous().realize(),
'desire_q': Tensor(np.zeros((frame_skip * dp[1], dp[0], dp[2]), dtype=np.float32), device=device).contiguous().realize(),
'packed_npy_inputs': Tensor(packed_npy_inputs, device='NPY').realize(),
})
return input_queues, npy
'packed_npy_inputs': Tensor(packed_input, device='NPY').realize(),
}
return input_queues, npy, frame_views
def shift_and_sample(buf, new_val, sample_fn):
@@ -176,13 +165,15 @@ def sample_desire(buf, frame_skip):
return buf.reshape(-1, frame_skip, *buf.shape[1:]).max(1).flatten(0, 1).unsqueeze(0)
def make_warp(nv12, model_w, model_h, frame_skip):
def make_warp(nv12, model_w, model_h):
frame_prepare = make_frame_prepare(nv12, model_w, model_h)
def warp(tfm, big_tfm, frame, big_frame):
tfm = tfm.to(WARP_DEV)
big_tfm = big_tfm.to(WARP_DEV)
Tensor.realize(tfm, big_tfm)
tfm = tfm.to(Device.DEFAULT)
big_tfm = big_tfm.to(Device.DEFAULT)
frame = frame.to(Device.DEFAULT)
big_frame = big_frame.to(Device.DEFAULT)
Tensor.realize(tfm, big_tfm, frame, big_frame)
warped_frame = frame_prepare(frame, tfm).unsqueeze(0)
warped_big_frame = frame_prepare(big_frame, big_tfm).unsqueeze(0)
@@ -195,10 +186,10 @@ def make_run_policy(model_runner, model_metadata, frame_skip):
sample_desire_fn = partial(sample_desire, frame_skip=frame_skip)
sample_skip_fn = partial(sample_skip, frame_skip=frame_skip)
npy_shapes, npy_sizes = get_policy_npy_shapes(model_metadata['input_shapes'])
model_input_dtypes = {name: spec.dtype for name, spec in model_runner.graph_inputs.items()}
def run_policy(warped, img_q, big_img_q, feat_q, desire_q, packed_npy_inputs):
packed_npy_inputs = packed_npy_inputs.to(Device.DEFAULT)
warped = warped.to(Device.DEFAULT)
Tensor.realize(packed_npy_inputs, warped)
img = shift_and_sample(img_q, warped[0:1], sample_skip_fn)
@@ -211,33 +202,50 @@ def make_run_policy(model_runner, model_metadata, frame_skip):
inputs = {
'img': img,
'big_img': big_img,
'features_buffer': feat_buf,
'features_buffer': feat_buf.reshape(model_metadata['input_shapes']['features_buffer']),
'desire_pulse': desire_buf,
'traffic_convention': traffic_convention,
'action_t': action_t,
}
inputs = {name: value.cast(model_input_dtypes[name]) for name, value in inputs.items()}
out = next(iter(model_runner(inputs).values())).cast('float32')
return out,
return run_policy
def compile_jit(jit, make_random_inputs, input_keys, make_queues):
SEED = 42
def random_inputs_run(fn, seed, test_val=None, test_buffers=None, expect_match=True):
input_queues, npy = make_queues(Device.DEFAULT)
rng = np.random.default_rng(seed)
Tensor.manual_seed(seed)
def make_run_model(warp, run_policy, model_metadata, frame_copy_size):
_, policy_sizes = get_policy_npy_shapes(model_metadata['input_shapes'])
packed_npy_size = (18 + sum(policy_sizes)) * np.dtype(np.float32).itemsize
testing = test_val is not None or test_buffers is not None
n_runs = 1 if testing else 3
def run_model(img_q, big_img_q, feat_q, desire_q, packed_npy_inputs):
packed_input = packed_npy_inputs.to(Device.DEFAULT)
Tensor.realize(packed_input)
packed_npy_inputs = packed_input[:packed_npy_size].bitcast('float32')
frame = packed_input[packed_npy_size:packed_npy_size + frame_copy_size]
big_frame = packed_input[packed_npy_size + frame_copy_size:]
tfm, big_tfm, policy_inputs = packed_npy_inputs.split([9, 9, sum(policy_sizes)])
warped = warp(tfm.reshape(3, 3), big_tfm.reshape(3, 3), frame, big_frame)
return run_policy(warped, img_q, big_img_q, feat_q, desire_q, policy_inputs)
return run_model
def compile_jit(jit, input_keys, make_queues, benchmark_runs):
if benchmark_runs < 1:
raise ValueError("benchmark_runs must be at least 1")
SEED = 42
def random_inputs_run(fn, seed, n_runs, test_val=None, test_buffers=None, expect_match=True):
input_queues, npy, frame_views = make_queues(Device.DEFAULT)
rng = np.random.default_rng(seed)
for i in range(n_runs):
for v in npy.values():
v[:] = rng.standard_normal(v.shape).astype(v.dtype)
for v in frame_views.values():
v[:] = rng.integers(0, 256, size=v.shape, dtype=np.uint8)
Device.default.synchronize()
random_inputs = make_random_inputs()
st = time.perf_counter()
outs = fn(**{k: input_queues[k] for k in input_keys}, **random_inputs)
outs = fn(**{k: input_queues[k] for k in input_keys})
mt = time.perf_counter()
Device.default.synchronize()
et = time.perf_counter()
@@ -256,14 +264,15 @@ def compile_jit(jit, make_random_inputs, input_keys, make_queues):
return val, buffers
print('capture + replay')
test_val, test_buffers = random_inputs_run(jit, SEED)
print('pickle round trip')
test_val, test_buffers = random_inputs_run(jit, SEED, 3)
print(f'pickle round trip ({benchmark_runs} runs per seed)')
with tempfile.TemporaryFile(dir=".") as f:
dump_oob(jit, f)
f.seek(0)
jit = load_oob(f)
random_inputs_run(jit, SEED, test_val, test_buffers, expect_match=True)
random_inputs_run(jit, SEED+1, test_val, test_buffers, expect_match=False)
loaded_jit = load_oob(f)
random_inputs_run(loaded_jit, SEED, benchmark_runs, test_val, test_buffers, expect_match=True)
random_inputs_run(loaded_jit, SEED+1, benchmark_runs, test_val, test_buffers, expect_match=False)
# Keep the original so per-resolution JITs share model weight buffers in the final pickle.
return jit
@@ -292,27 +301,31 @@ if __name__ == "__main__":
p.add_argument('--onnx', required=True)
p.add_argument('--output', required=True)
p.add_argument('--frame-skip', type=int, required=True)
p.add_argument('--benchmark-runs', type=int, default=1,
help='timed loaded-JIT runs for each correctness seed')
args = p.parse_args()
model_path = read_file_chunked_to_disk(args.onnx)
model_w, model_h = args.model_size
model_runner = OnnxRunner(model_path)
out = {'metadata': make_metadata_dict(model_path)}
out = {
'metadata': make_metadata_dict(model_path),
'input_devices': {'model': Device.DEFAULT},
'run_model': {},
}
run_policy_jit = TinyJit(make_run_policy(model_runner, out['metadata'], args.frame_skip), prune=True)
make_policy_queues = partial(make_input_queues, out['metadata']['input_shapes'], args.frame_skip)
make_random_model_inputs = partial(make_random_images, keys=['warped'], shape=(2, 6, *out['metadata']['input_shapes']['img'][2:]), device=WARP_DEV)
out['run_policy'] = compile_jit(run_policy_jit, make_random_model_inputs, POLICY_INPUTS,
make_policy_queues)
run_policy = make_run_policy(model_runner, out['metadata'], args.frame_skip)
for cam_w, cam_h in args.camera_resolutions:
nv12 = NV12Frame(cam_w, cam_h, *get_nv12_info(cam_w, cam_h))
make_random_warp_inputs = partial(make_random_images, keys=['frame', 'big_frame'], shape=nv12.size, device=WARP_DEV)
warp = TinyJit(make_warp(nv12, model_w, model_h, args.frame_skip), prune=True)
make_warp_queues = partial(make_warp_input_queues, out['metadata']['input_shapes'], args.frame_skip)
out[(cam_w,cam_h)] = compile_jit(warp, make_random_warp_inputs, WARP_INPUTS, make_warp_queues)
frame_copy_size = nv12_copy_size(nv12.stride, nv12.y_height, nv12.uv_height)
make_model_queues = partial(make_input_queues, out['metadata']['input_shapes'], args.frame_skip,
frame_copy_size=frame_copy_size)
warp = make_warp(nv12, model_w, model_h)
run_model_jit = TinyJit(make_run_model(warp, run_policy, out['metadata'], frame_copy_size), prune=True)
out['run_model'][(cam_w,cam_h)] = compile_jit(run_model_jit, MODELD_INPUTS, make_model_queues,
args.benchmark_runs)
with open(args.output, "wb") as f:
dump_oob(out, f)
@@ -29,7 +29,7 @@ class ModelState:
output: np.ndarray
def __init__(self, cam_w: int, cam_h: int):
self.DEV = get_tg_input_devices(PROCESS_NAME, usbgpu=False)['DEV']
self.DEV = get_tg_input_devices(PROCESS_NAME, chestnut=False)['DEV']
with open(METADATA_PATH, 'rb') as f:
model_metadata = pickle.load(f)
self.input_shapes = model_metadata['input_shapes']
@@ -64,6 +64,7 @@ def fill_driving_model_data(msg: capnp._DynamicStructBuilder, modelv2_send: capn
driving_model_data.frameIdExtra = modelV2.frameIdExtra
driving_model_data.frameDropPerc = modelV2.frameDropPerc
driving_model_data.modelExecutionTime = modelV2.modelExecutionTime
driving_model_data.big = modelV2.big
driving_model_data.action = modelV2.action
driving_model_data.meta.laneChangeState = modelV2.meta.laneChangeState
driving_model_data.meta.laneChangeDirection = modelV2.meta.laneChangeDirection
+15 -9
View File
@@ -7,18 +7,20 @@ import tempfile
from pathlib import Path
from openpilot.common.file_chunker import get_manifest_path
from openpilot.common.hardware.usb import CHESTNUT_FW_VERSION, CHESTNUT_USB_IDS, USB_DEVICES_PATH
from openpilot.common.hardware.usb import CHESTNUT_USB_PRODUCT, USB_DEVICES_PATH, is_chestnut_usb_id
MODELS_DIR = Path(__file__).resolve().parent / 'models'
TG_INPUT_DEVICES_PATH = MODELS_DIR / 'tg_input_devices.json'
CHESTNUT_POWERED_VOLTAGE = 5000
CHESTNUT_PCIE_READY = 0x78
def get_tg_input_devices(process_name: str, usbgpu: bool):
def get_tg_input_devices(process_name: str, chestnut: bool):
with open(TG_INPUT_DEVICES_PATH) as f:
return json.load(f)[process_name]['default' if not usbgpu else 'usbgpu']
return json.load(f)[process_name]['default' if not chestnut else 'chestnut']
def modeld_pkl_path(usbgpu: bool):
prefix = 'big_' if usbgpu else ''
def modeld_pkl_path(chestnut: bool):
prefix = 'big_' if chestnut else ''
return MODELS_DIR / f'{prefix}driving_tinygrad.pkl'
def dump_oob(obj, f):
@@ -45,16 +47,20 @@ def load_oob(f):
yield pb
return pickle.load(io.BytesIO(opcodes), buffers=buffers())
def usbgpu_present() -> bool:
def chestnut_present() -> bool:
for d in USB_DEVICES_PATH.glob("*"):
try:
usb_id = (int((d / "idVendor").read_text(), 16), int((d / "idProduct").read_text(), 16))
product = (d / "product").read_text().strip()
if usb_id in CHESTNUT_USB_IDS and product == f"custom {CHESTNUT_FW_VERSION}-CLEAN":
if is_chestnut_usb_id(*usb_id) and product == CHESTNUT_USB_PRODUCT:
return True
except Exception:
pass
return False
def usbgpu_compiled() -> bool:
return Path(get_manifest_path(modeld_pkl_path(usbgpu=True))).is_file()
def chestnut_compiled() -> bool:
return Path(get_manifest_path(modeld_pkl_path(chestnut=True))).is_file()
def chestnut_ready(state) -> bool:
return state.supplyVoltage >= CHESTNUT_POWERED_VOLTAGE and not state.supplyFault and state.pcieLtssm == CHESTNUT_PCIE_READY
+105 -66
View File
@@ -1,9 +1,11 @@
#!/usr/bin/env python3
from collections.abc import Callable
import ctypes
from functools import cached_property
import os
os.environ['GMMU'] = '0' # for usbgpu fast loading, noop for qcom
from tinygrad.tensor import Tensor
os.environ['GMMU'] = '0' # for chestnut fast loading, noop for qcom
from tinygrad.device import Device
import usb1
import struct
import threading
import time
@@ -26,17 +28,17 @@ from openpilot.common.transformations.model import get_warp_matrix
from openpilot.selfdrive.controls.lib.desire_helper import DesireHelper
from openpilot.selfdrive.controls.lib.drive_helpers import get_accel_from_plan, should_stop, smooth_value, get_curvature_from_plan
from openpilot.selfdrive.modeld.parse_model_outputs import Parser
from openpilot.selfdrive.modeld.compile_modeld import make_input_queues, WARP_INPUTS, POLICY_INPUTS
from openpilot.selfdrive.modeld.compile_modeld import make_input_queues, nv12_copy_size, MODELD_INPUTS
from openpilot.selfdrive.modeld.fill_model_msg import fill_model_msg, fill_driving_model_data, fill_pose_msg, PublishState
from openpilot.common.file_chunker import open_file_chunked
from openpilot.common.hardware.usb import CHESTNUT_USB_IDS
from openpilot.selfdrive.modeld.constants import ModelConstants, Plan
from openpilot.selfdrive.modeld.helpers import usbgpu_present, usbgpu_compiled, modeld_pkl_path, get_tg_input_devices, load_oob
from openpilot.selfdrive.modeld.helpers import chestnut_present, chestnut_compiled, chestnut_ready, modeld_pkl_path, load_oob
from openpilot.sunnypilot.livedelay.helpers import get_lat_delay
from openpilot.sunnypilot.modeld_v2.modeld_base import ModelStateBase
from openpilot.sunnypilot.selfdrive.controls.lib.relc import RoadEdgeLaneChangeController
PROCESS_NAME = "openpilot.selfdrive.modeld.modeld"
SEND_RAW_PRED = os.getenv('SEND_RAW_PRED')
LAT_SMOOTH_SECONDS = 0.0
@@ -81,6 +83,37 @@ class ChestnutState:
self.valid = True
self.sends = 0
self.metrics = {}
self._asm_usb = None
def _close_asm_usb(self) -> None:
if self._asm_usb is not None:
self._asm_usb.close()
self._asm_usb = None
def _open_asm_usb(self):
context = usb1.USBContext()
for vendor_id, product_id in CHESTNUT_USB_IDS:
if (handle := context.openByVendorIDAndProductID(vendor_id, product_id, skip_on_error=True)) is not None:
return handle
context.close()
def _read_ina(self) -> tuple[int, int, bool]:
if "AMD" in Device._opened_devices and self._asm_usb is None:
try:
raw = Device["AMD"].iface.pci_dev.usb.usb.control_read(0xC0, 5)
return struct.unpack('<Hh?', bytes(raw))
except Exception:
pass
if self._asm_usb is None:
self._asm_usb = self._open_asm_usb()
if self._asm_usb is None:
raise usb1.USBErrorNoDevice
try:
raw = self._asm_usb.controlRead(0xC0, 0xC0, 0, 0, 5, timeout=100)
except usb1.USBError:
self._close_asm_usb()
raise
return struct.unpack('<Hh?', bytes(raw))
@cached_property
def power_limit(self) -> int:
@@ -94,8 +127,10 @@ class ChestnutState:
if self.big and "AMD" in Device._opened_devices and self.sends % 100 == 1:
try:
smu = Device["AMD"].iface.dev_impl.smu
metrics_t = smu.smu_mod.SmuMetricsExternal_t
smu._send_msg(smu.smu_mod.PPSMC_MSG_TransferTableSmu2Dram, smu.smu_mod.TABLE_SMU_METRICS, timeout=100)
metrics = smu.read_table(smu.smu_mod.SmuMetricsExternal_t, smu.smu_mod.TABLE_SMU_METRICS).SmuMetrics
metrics_buf = bytearray(smu.adev.vram.view(smu.driver_table_paddr, ctypes.sizeof(metrics_t))[:])
metrics = metrics_t.from_buffer(metrics_buf).SmuMetrics
self.metrics = {'tempC': metrics.AvgTemperature[smu.smu_mod.TEMP_HOTSPOT],
'memoryTempC': metrics.AvgTemperature[smu.smu_mod.TEMP_MEM],
'powerDrawW': metrics.AverageSocketPower,
@@ -114,13 +149,15 @@ class ChestnutState:
setattr(state, k, v)
asm_valid = False
try:
# ASM runs on USB-C power, these still read without a gpu
state.supplyVoltage, state.supplyCurrent, state.supplyFault = self._read_ina()
asm_valid = True
except Exception:
pass
if "AMD" in Device._opened_devices:
try:
# ASM runs on USB-C power, these still read without a gpu
asm = Device["AMD"].iface.pci_dev.usb
state.pcieLtssm = asm.read(0xB450, 1)[0]
state.supplyVoltage, state.supplyCurrent = struct.unpack('<Hh', bytes(asm.usb.control_read(0xC0, 5))[:4])
asm_valid = True
state.pcieLtssm = Device["AMD"].iface.pci_dev.usb.read(0xB450, 1)[0]
except Exception:
pass
@@ -141,42 +178,34 @@ class FrameMeta:
class ModelState(ModelStateBase):
prev_desire: np.ndarray # for tracking the rising edge of the pulse
def __init__(self, cam_w: int, cam_h: int, usbgpu: bool):
def __init__(self, cam_w: int, cam_h: int, chestnut: bool):
ModelStateBase.__init__(self)
input_devices = get_tg_input_devices(PROCESS_NAME, usbgpu)
self.WARP_DEV, self.QUEUE_DEV = input_devices['WARP_DEV'], input_devices['QUEUE_DEV']
jits = load_oob(open_file_chunked(modeld_pkl_path(usbgpu)))
jits = load_oob(open_file_chunked(modeld_pkl_path(chestnut)))
input_devices = jits['input_devices']
self.model_device = input_devices['model']
metadata = jits['metadata']
self.input_shapes = metadata['input_shapes']
self.vision_input_names = [k for k in self.input_shapes if 'img' in k]
self.output_slices = metadata['output_slices']
self.prev_desire = np.zeros(ModelConstants.DESIRE_LEN, dtype=np.float32)
self.usbgpu = usbgpu
self.chestnut = chestnut
self.frame_skip = ModelConstants.MODEL_RUN_FREQ // ModelConstants.MODEL_CONTEXT_FREQ
self.input_queues, self.npy = make_input_queues(self.input_shapes, self.frame_skip, device=self.QUEUE_DEV)
self.full_frames: dict[str, Tensor] = {}
self._blob_cache: dict[tuple[str, int], Tensor] = {}
self.frame_copy_size = nv12_copy_size(*get_nv12_info(cam_w, cam_h)[:3])
self.input_queues, self.npy, self.frame_views = make_input_queues(
self.input_shapes, self.frame_skip, device=self.model_device, frame_copy_size=self.frame_copy_size)
self.parser = Parser()
self.frame_buf_params = {k: get_nv12_info(cam_w, cam_h) for k in ('img', 'big_img')}
self.run_policy = jits['run_policy']
self.warp = jits[(cam_w,cam_h)]
self.run_model = jits['run_model'][(cam_w,cam_h)]
def slice_outputs(self, model_outputs: np.ndarray, output_slices: dict[str, slice]) -> dict[str, np.ndarray]:
parsed_model_outputs = {k: model_outputs[np.newaxis, v] for k,v in output_slices.items()}
return parsed_model_outputs
def run(self, bufs: dict[str, VisionBuf], transforms: dict[str, np.ndarray],
inputs: dict[str, np.ndarray]) -> dict[str, np.ndarray] | None:
for key in bufs.keys():
ptr = np.frombuffer(bufs[key].data, dtype=np.uint8).ctypes.data
yuv_size = self.frame_buf_params[key][3]
# There is a ringbuffer of imgs, just cache tensors pointing to all of them
cache_key = (key, ptr)
if cache_key not in self._blob_cache:
self._blob_cache[cache_key] = Tensor.from_blob(ptr, (yuv_size,), dtype='uint8', device=self.WARP_DEV)
self.full_frames[key] = self._blob_cache[cache_key]
inputs: dict[str, np.ndarray], after_enqueue: Callable[[], None] | None = None) -> dict[str, np.ndarray]:
for key, buf in bufs.items():
np.copyto(self.frame_views[key], np.frombuffer(buf.data, dtype=np.uint8, count=self.frame_copy_size))
# Model decides when action is completed, so desire input is just a pulse triggered on rising edge
inputs['desire_pulse'][0] = 0
@@ -187,16 +216,12 @@ class ModelState(ModelStateBase):
self.npy['tfm'][:,:] = transforms['img'][:,:]
self.npy['big_tfm'][:,:] = transforms['big_img'][:,:]
warped = self.warp(**{k: self.input_queues[k] for k in WARP_INPUTS}, frame=self.full_frames['img'], big_frame=self.full_frames['big_img'])
outs, = self.run_policy(
**{k: self.input_queues[k] for k in POLICY_INPUTS if k in self.input_queues}, warped=warped
)
outs, = self.run_model(**{k: self.input_queues[k] for k in MODELD_INPUTS})
if after_enqueue is not None:
after_enqueue()
model_output = outs.numpy()[0]
if self.usbgpu and not np.all(np.isfinite(model_output)):
# TODO remove with prev_feat
cloudlog.error("model output not finite, dropping frame")
return None
if self.chestnut and not np.all(np.isfinite(model_output)):
raise RuntimeError("model output not finite")
outputs_dict = self.parser.parse_outputs(self.slice_outputs(model_output, self.output_slices))
self.npy['prev_feat'][:] = model_output[self.output_slices['hidden_state']]
@@ -205,25 +230,37 @@ class ModelState(ModelStateBase):
return outputs_dict
def warmup(self) -> None:
dummy_frames = {k: np.zeros(self.frame_buf_params[k][3], dtype=np.uint8) for k in self.vision_input_names}
dummy_frames = {k: np.zeros(self.frame_copy_size, dtype=np.uint8) for k in self.vision_input_names}
eye = np.eye(3, dtype=np.float32)
dims = {'desire_pulse': ModelConstants.DESIRE_LEN, 'traffic_convention': 2, 'action_t': 2}
self.run(dummy_frames, dict.fromkeys(self.vision_input_names, eye), {k: np.zeros(v, dtype=np.float32) for k, v in dims.items()})
self.input_queues, self.npy = make_input_queues(self.input_shapes, self.frame_skip, device=self.QUEUE_DEV)
self.input_queues, self.npy, self.frame_views = make_input_queues(
self.input_shapes, self.frame_skip, device=self.model_device, frame_copy_size=self.frame_copy_size)
self.prev_desire[:] = 0
self.full_frames.clear()
self._blob_cache.clear()
def main(demo=False):
cloudlog.warning("modeld init")
USBGPU = usbgpu_present() and usbgpu_compiled()
if USBGPU:
chestnut_available = chestnut_present() and chestnut_compiled()
CHESTNUT = False
if chestnut_available:
poller = messaging.Poller()
sock = messaging.sub_sock("chestnutState", poller=poller, conflate=True)
deadline = time.monotonic() + 4. / SERVICE_LIST['deviceState'].frequency
while not CHESTNUT and (remaining := deadline - time.monotonic()) > 0.:
if not poller.poll(round(remaining * 1000)):
break
msg = messaging.recv_one_or_none(sock)
CHESTNUT = msg is not None and msg.valid and chestnut_ready(msg.chestnutState)
if CHESTNUT:
os.environ['HCQDEV_WAIT_TIMEOUT_MS'] = '3000'
params = Params()
params.put_bool("UsbGpuLoading", USBGPU)
params.remove("UsbGpuActive")
params.put_bool("ChestnutLoading", CHESTNUT)
if chestnut_available and not CHESTNUT:
params.put_bool("ChestnutActive", False)
else:
params.remove("ChestnutActive")
config_realtime_process(7, 54)
@@ -253,7 +290,7 @@ def main(demo=False):
st = time.monotonic()
cloudlog.warning("loading model")
model = None
if USBGPU:
if CHESTNUT:
big_model = None
def load_big():
nonlocal big_model
@@ -267,23 +304,27 @@ def main(demo=False):
loader.start()
loader.join(BIG_MODEL_TIMEOUT)
model = big_model
params.put_bool("UsbGpuActive", model is not None)
if model is None:
params.put_bool("ChestnutModelError", True)
params.put_bool("ChestnutActive", model is not None)
if model is not None:
params.remove("ChestnutModelError")
small_model = ModelState(vipc_client_main.width, vipc_client_main.height, False) if model is None or USBGPU else None
small_model = ModelState(vipc_client_main.width, vipc_client_main.height, False) if model is None or CHESTNUT else None
if model is None:
model = small_model
params.put_bool("UsbGpuLoading", False)
params.put_bool("ChestnutLoading", False)
assert model is not None
cloudlog.warning(f"models loaded in {time.monotonic() - st:.1f}s, modeld starting")
# messaging
pub_socks = ["modelV2", "drivingModelData", "cameraOdometry", "modelDataV2SP"] + (["chestnutState"] if USBGPU else [])
pub_socks = ["modelV2", "drivingModelData", "cameraOdometry", "modelDataV2SP"] + (["chestnutState"] if CHESTNUT else [])
pm = PubMaster(pub_socks)
sm = SubMaster(["deviceState", "carState", "narrowRoadCameraState", "extrinsicsCalibration", "driverMonitoringState", "carControl", "lateralDelay"])
publish_state = PublishState()
params = Params()
chestnut_state = ChestnutState(pm, model.usbgpu) if USBGPU else None
chestnut_state = ChestnutState(pm, model.chestnut) if CHESTNUT else None
# setup filter to track dropped frames
frame_dropped_filter = FirstOrderFilter(0., 10., 1. / ModelConstants.MODEL_RUN_FREQ)
@@ -393,13 +434,16 @@ def main(demo=False):
mt1 = time.perf_counter()
try:
model_output = model.run(bufs, transforms, inputs)
send_chestnut = (chestnut_state is not None and
run_count % round(ModelConstants.MODEL_RUN_FREQ / SERVICE_LIST['chestnutState'].frequency) == 0)
model_output = model.run(bufs, transforms, inputs, chestnut_state.send if send_chestnut else None)
except Exception:
if not params.get_bool("UsbGpuActive"):
if not params.get_bool("ChestnutActive"):
raise
# fallback to small model
cloudlog.exception("big model failed, fall back to small")
params.put_bool("UsbGpuActive", False)
params.put_bool("ChestnutModelError", True)
params.put_bool("ChestnutActive", False)
assert small_model is not None
model = small_model
if chestnut_state is not None:
@@ -419,18 +463,17 @@ def main(demo=False):
fill_model_msg(modelv2_send, model_output, action,
publish_state, meta_main.frame_id, meta_extra.frame_id, frame_id,
frame_drop_ratio, meta_main.timestamp_eof, model_execution_time, extrinsics_calibration_seen)
modelv2_send.modelV2.big = model.usbgpu
modelv2_send.modelV2.big = model.chestnut
desire_state = modelv2_send.modelV2.meta.desireState
l_lane_change_prob = desire_state[log.Desire.laneChangeLeft]
r_lane_change_prob = desire_state[log.Desire.laneChangeRight]
lane_change_prob = l_lane_change_prob + r_lane_change_prob
DH.update(sm['carState'], sm['carControl'].latActive, lane_change_prob)
modelv2_send.modelV2.meta.laneChangeState = DH.lane_change_state
modelv2_send.modelV2.meta.laneChangeDirection = DH.lane_change_direction
mdv2sp_send = messaging.new_message('modelDataV2SP')
left_edge, right_edge = RELC.update_and_fill(modelv2_send.modelV2, mdv2sp_send.modelDataV2SP, v_ego)
DH.update(sm['carState'], sm['carControl'].latActive, lane_change_prob, left_edge, right_edge)
modelv2_send.modelV2.meta.laneChangeState = DH.lane_change_state
modelv2_send.modelV2.meta.laneChangeDirection = DH.lane_change_direction
mdv2sp_send.modelDataV2SP.laneTurnDirection = DH.lane_turn_direction
fill_driving_model_data(drivingdata_send, modelv2_send)
@@ -441,10 +484,6 @@ def main(demo=False):
pm.send('modelDataV2SP', mdv2sp_send)
last_vipc_frame_id = meta_main.frame_id
if chestnut_state is not None and run_count % round(ModelConstants.MODEL_RUN_FREQ / SERVICE_LIST['chestnutState'].frequency) == 0:
chestnut_state.send()
if __name__ == "__main__":
try:
import argparse
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a501760a9d1d5fef0eab2b8c5d122d06124fc26dc8e0782e0aa94b82a208f0ff
size 1757355221
oid sha256:1791d5940b2c048d0639813426dd2cf1d6f2a6727ed51e17c8bcea8bbe754123
size 765950064
+10 -10
View File
@@ -123,22 +123,22 @@ void fill_panda_state(cereal::PandaState::Builder &ps, cereal::PandaState::Panda
ps.setUptime(health.uptime_pkt);
ps.setSafetyTxBlocked(health.safety_tx_blocked_pkt);
ps.setSafetyRxInvalid(health.safety_rx_invalid_pkt);
ps.setIgnitionLine(health.ignition_line_pkt);
ps.setIgnitionCan(health.ignition_can_pkt);
ps.setControlsAllowed(health.controls_allowed_pkt);
ps.setIgnitionLine((health.flags_pkt & HEALTH_FLAG_IGNITION_LINE) != 0U);
ps.setIgnitionCan((health.flags_pkt & HEALTH_FLAG_IGNITION_CAN) != 0U);
ps.setControlsAllowed((health.flags_pkt & HEALTH_FLAG_CONTROLS_ALLOWED) != 0U);
ps.setTxBufferOverflow(health.tx_buffer_overflow_pkt);
ps.setRxBufferOverflow(health.rx_buffer_overflow_pkt);
ps.setPandaType(hw_type);
ps.setSafetyModel(cereal::CarParams::SafetyModel(health.safety_mode_pkt));
ps.setSafetyParam(health.safety_param_pkt);
ps.setFaultStatus(cereal::PandaState::FaultStatus(health.fault_status_pkt));
ps.setPowerSaveEnabled((bool)(health.power_save_enabled_pkt));
ps.setHeartbeatLost((bool)(health.heartbeat_lost_pkt));
ps.setPowerSaveEnabled((health.flags_pkt & HEALTH_FLAG_POWER_SAVE_ENABLED) != 0U);
ps.setHeartbeatLost((health.flags_pkt & HEALTH_FLAG_HEARTBEAT_LOST) != 0U);
ps.setAlternativeExperience(health.alternative_experience_pkt);
ps.setHarnessStatus(cereal::PandaState::HarnessStatus(health.car_harness_status_pkt));
ps.setInterruptLoad(health.interrupt_load_pkt);
ps.setInterruptLoad(health.interrupt_load_pkt / 255.0f);
ps.setFanPower(health.fan_power);
ps.setSafetyRxChecksInvalid((bool)(health.safety_rx_checks_invalid_pkt));
ps.setSafetyRxChecksInvalid((health.flags_pkt & HEALTH_FLAG_SAFETY_RX_CHECKS_INVALID) != 0U);
ps.setSpiErrorCount(health.spi_error_count_pkt);
ps.setSbu1Voltage(health.sbu1_voltage_mV / 1000.0f);
ps.setSbu2Voltage(health.sbu2_voltage_mV / 1000.0f);
@@ -198,10 +198,10 @@ std::optional<bool> send_panda_states(PubMaster *pm, Panda *panda, bool is_onroa
}
if (spoofing_started) {
health.ignition_line_pkt = 1;
health.flags_pkt |= HEALTH_FLAG_IGNITION_LINE;
}
bool ignition_local = ((health.ignition_line_pkt != 0) || (health.ignition_can_pkt != 0)) && !always_offroad;
bool ignition_local = ((health.flags_pkt & (HEALTH_FLAG_IGNITION_LINE | HEALTH_FLAG_IGNITION_CAN)) != 0U) && !always_offroad;
// Make sure CAN buses are live: safety_setter_thread does not work if Panda CAN are silent and there is only one other CAN node
if (health.safety_mode_pkt == (uint8_t)(cereal::CarParams::SafetyModel::SILENT)) {
@@ -209,7 +209,7 @@ std::optional<bool> send_panda_states(PubMaster *pm, Panda *panda, bool is_onroa
}
bool power_save_desired = !ignition_local;
if (health.power_save_enabled_pkt != power_save_desired) {
if (((health.flags_pkt & HEALTH_FLAG_POWER_SAVE_ENABLED) != 0U) != power_save_desired) {
panda->set_power_saving(power_save_desired);
}
@@ -18,7 +18,31 @@
"_comment": "Set extra field to the failed reason."
},
"Offroad_ChestnutBranch": {
"text": "Chestnut detected! Switch to the release-chestnut branch to use chestnut-class models.",
"text": "Chestnut detected! Switch to the %1 branch to use chestnut-class models.",
"severity": -1
},
"Offroad_ChestnutNotDetected": {
"text": "Chestnut not detected. Check USB and 12V connections.",
"severity": 0
},
"Offroad_ChestnutOverheated": {
"text": "Chestnut overheated. Ensure good airflow. Current GPU temperature is %1.",
"severity": 0
},
"Offroad_ChestnutPcieUnavailable": {
"text": "%1",
"severity": 0
},
"Offroad_ChestnutUncompiled": {
"text": "Chestnut model not compiled. Keep ignition on and reboot the comma.",
"severity": 0
},
"Offroad_ChestnutUpdateFailed": {
"text": "Chestnut update failed. Check the USB cable.",
"severity": 0
},
"Offroad_ChestnutUsbSlow": {
"text": "Chestnut USB link is slow. Check the USB cable. The current speed is %1.",
"severity": 0
},
"Offroad_UnregisteredHardware": {
+5 -4
View File
@@ -195,17 +195,18 @@ class SelfdriveD(CruiseHelper):
self.events.add(EventName.joystickDebug)
self.startup_event = None
loading = self.params.get_bool("UsbGpuLoading")
loading = self.params.get_bool("ChestnutLoading")
if self.big_model_loading and not loading:
self.big_model_ready_t = time.monotonic()
self.events_sp.add(custom.OnroadEventSP.EventName.bigModelReady)
self.big_model_loading = loading
if self.big_model_loading:
self.events.add(EventName.bigModelLoading)
big_active = self.params.get("UsbGpuActive")
usbgpu_present = self.sm['deviceState'].chestnutPresent
big_active = self.params.get("ChestnutActive")
chestnut_present = self.sm['deviceState'].chestnutPresent
model_unavailable = big_active is True and self.sm.seen['modelV2'] and not self.sm.alive['modelV2']
big_failed = big_active is False or model_unavailable or (self.big_model_active and not usbgpu_present)
big_failed = big_active is False or model_unavailable or (self.big_model_active and not chestnut_present)
if big_failed and not self.big_model_failed:
self.events.add(EventName.bigModelFailed)
self.big_model_failed = big_failed
@@ -152,7 +152,7 @@ def migrate_drivingModelData(msgs):
add_ops = []
for _, msg in msgs:
dmd = messaging.new_message('drivingModelData', valid=msg.valid, logMonoTime=msg.logMonoTime)
for field in ["frameId", "frameIdExtra", "frameDropPerc", "modelExecutionTime", "action"]:
for field in ["frameId", "frameIdExtra", "frameDropPerc", "modelExecutionTime", "big", "action"]:
setattr(dmd.drivingModelData, field, getattr(msg.modelV2, field))
for meta_field in ["laneChangeState", "laneChangeState"]:
setattr(dmd.drivingModelData.meta, meta_field, getattr(msg.modelV2.meta, meta_field))
@@ -33,9 +33,9 @@ MODEL_REPLAY_BUCKET="model_replay_master"
GITHUB = GithubUtils(API_TOKEN, DATA_TOKEN)
EXEC_TIMINGS = [
# model, instant max, average max
("modelV2", 0.05, 0.028),
("driverStateV2", 0.05, 0.018),
# model, instant max, average max, chestnut average max
("modelV2", 0.05, 0.03, 0.05),
("driverStateV2", 0.05, 0.018, 0.018),
]
def get_log_fn(test_route, ref="master"):
@@ -169,11 +169,13 @@ def model_replay(lr, frs):
dmonitoringmodeld_msgs = replay_process(dmonitoringmodeld, dmodeld_logs, frs)
msgs = modeld_msgs + dmonitoringmodeld_msgs
chestnut = any(m.modelV2.big for m in modeld_msgs if m.which() == "modelV2")
header = ['model', 'max instant', 'max instant allowed', 'average', 'max average allowed', 'test result']
rows = []
timings_ok = True
for (s, instant_max, avg_max) in EXEC_TIMINGS:
for (s, instant_max, avg_max, chestnut_avg_max) in EXEC_TIMINGS:
avg_max = chestnut_avg_max if chestnut else avg_max
ts = [getattr(m, s).modelExecutionTime for m in msgs if m.which() == s]
# TODO some init can happen in first iteration
ts = ts[1:]
@@ -1,7 +1,7 @@
import time
import pyray as rl
from openpilot.system.ui.lib.application import gui_app, FontWeight
from openpilot.system.ui.lib.application import gui_app, FontWeight, TextAlignment, TextAlignmentVertical
from openpilot.system.ui.widgets import Widget
from openpilot.system.ui.widgets.label import UnifiedLabel
from openpilot.selfdrive.ui.ui_state import ui_state
@@ -26,8 +26,8 @@ class BodyLayout(Widget):
self._last_input_time = time.monotonic()
self._was_active = False
self._offroad_label = UnifiedLabel("turn on ignition to use", 95 if gui_app.big_ui() else 45, FontWeight.DISPLAY,
alignment=rl.GuiTextAlignment.TEXT_ALIGN_CENTER,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_MIDDLE)
alignment=TextAlignment.CENTER,
alignment_vertical=TextAlignmentVertical.MIDDLE)
def draw_dot_grid(self, rect: rl.Rectangle, dots: list[tuple[int, int]], color: rl.Color):
spacing = min(rect.height / GRID_ROWS, rect.width / GRID_COLS)
+2 -2
View File
@@ -8,7 +8,7 @@ from openpilot.selfdrive.ui.widgets.exp_mode_button import ExperimentalModeButto
from openpilot.selfdrive.ui.widgets.prime import PrimeWidget
from openpilot.selfdrive.ui.widgets.setup import SetupWidget
from openpilot.system.ui.lib.text_measure import measure_text_cached
from openpilot.system.ui.lib.application import gui_app, FontWeight, MousePos
from openpilot.system.ui.lib.application import gui_app, FontWeight, MousePos, TextAlignment
from openpilot.system.ui.lib.multilang import tr, trn
from openpilot.system.ui.widgets.label import gui_label
from openpilot.system.ui.widgets import Widget
@@ -178,7 +178,7 @@ class HomeLayout(Widget):
version_rect = rl.Rectangle(self.header_rect.x + self.header_rect.width - version_text_width, self.header_rect.y,
version_text_width, self.header_rect.height)
gui_label(version_rect, self._version_text, 48, rl.WHITE, alignment=rl.GuiTextAlignment.TEXT_ALIGN_RIGHT)
gui_label(version_rect, self._version_text, 48, rl.WHITE, alignment=TextAlignment.RIGHT)
def _render_home_content(self):
self._render_left_column()
+4 -4
View File
@@ -5,7 +5,7 @@ from enum import IntEnum
import pyray as rl
from openpilot.common.basedir import BASEDIR
from openpilot.system.ui.lib.application import FontWeight, gui_app
from openpilot.system.ui.lib.application import FontWeight, TextAlignment, gui_app
from openpilot.system.ui.lib.multilang import tr
from openpilot.system.ui.widgets import Widget
from openpilot.system.ui.widgets.button import Button, ButtonStyle
@@ -115,9 +115,9 @@ class TermsPage(Widget):
self._on_accept = on_accept
self._on_decline = on_decline
self._title = Label(tr("Welcome to sunnypilot"), font_size=90, font_weight=FontWeight.BOLD, text_alignment=rl.GuiTextAlignment.TEXT_ALIGN_LEFT)
self._title = Label(tr("Welcome to sunnypilot"), font_size=90, font_weight=FontWeight.BOLD, text_alignment=TextAlignment.LEFT)
self._desc = Label(tr("You must accept the Terms of Service to use sunnypilot. Read the latest terms at https://sunnypilot.ai/terms before continuing."),
font_size=90, font_weight=FontWeight.MEDIUM, text_alignment=rl.GuiTextAlignment.TEXT_ALIGN_LEFT)
font_size=90, font_weight=FontWeight.MEDIUM, text_alignment=TextAlignment.LEFT)
self._decline_btn = Button(tr("Decline"), click_callback=on_decline)
self._accept_btn = Button(tr("Agree"), button_style=ButtonStyle.PRIMARY, click_callback=on_accept)
@@ -150,7 +150,7 @@ class DeclinePage(Widget):
def __init__(self, back_callback=None):
super().__init__()
self._text = Label(tr("You must accept the Terms of Service in order to use sunnypilot."),
font_size=90, font_weight=FontWeight.MEDIUM, text_alignment=rl.GuiTextAlignment.TEXT_ALIGN_LEFT)
font_size=90, font_weight=FontWeight.MEDIUM, text_alignment=TextAlignment.LEFT)
self._back_btn = Button(tr("Back"), click_callback=back_callback)
self._uninstall_btn = Button(tr("Decline, uninstall sunnypilot"), button_style=ButtonStyle.DANGER,
click_callback=self._on_uninstall_clicked)
@@ -199,6 +199,9 @@ class SoftwareLayout(Widget):
selection = self._branch_dialog.selection
ui_state.params.put("UpdaterTargetBranch", selection, block=True)
self._branch_btn.action_item.set_value(selection)
self._download_btn.action_item.set_enabled(False)
self._waiting_for_updater = True
self._waiting_start_ts = time.monotonic()
subprocess.run("pkill -SIGUSR1 -f openpilot.system.updated.updated", shell=True)
self._branch_dialog = None
+8 -1
View File
@@ -168,9 +168,16 @@ class Sidebar(Widget, SidebarSP):
# Home/Flag button
flag_pressed = mouse_down and rl.check_collision_point_rec(mouse_pos, HOME_BTN)
button_img = self._flag_img if ui_state.started else self._home_img
button_pos = rl.Vector2(HOME_BTN.x, HOME_BTN.y)
icon_opacity = 1.0
if gui_app.sunnypilot_ui():
button_img, button_pos, icon_opacity = SidebarSP._get_home_icon(self, button_img)
tint = Colors.BUTTON_PRESSED if (ui_state.started and flag_pressed) else Colors.BUTTON_NORMAL
rl.draw_texture_ex(button_img, rl.Vector2(HOME_BTN.x, HOME_BTN.y), 0.0, 1.0, tint)
if icon_opacity < 1.0:
tint = rl.Color(tint[0], tint[1], tint[2], int(255 * icon_opacity))
rl.draw_texture_ex(button_img, button_pos, 0.0, 1.0, tint)
# Microphone button
if self._recording_audio:
+27 -11
View File
@@ -1,4 +1,5 @@
import datetime
import math
import time
from openpilot.cereal import log
@@ -8,8 +9,8 @@ from openpilot.system.ui.widgets import Widget
from openpilot.system.ui.widgets.layouts import HBoxLayout
from openpilot.system.ui.widgets.icon_widget import IconWidget
from openpilot.system.ui.widgets.label import UnifiedLabel, gui_label
from openpilot.system.ui.lib.application import gui_app, FontWeight, MousePos
from openpilot.selfdrive.ui.ui_state import ui_state
from openpilot.system.ui.lib.application import gui_app, FontWeight, MousePos, TextAlignment, TextAlignmentVertical
from openpilot.selfdrive.ui.ui_state import ui_state, ChestnutState
from openpilot.common.version import RELEASE_BRANCHES
HEAD_BUTTON_FONT_SIZE = 40
@@ -69,8 +70,8 @@ class AlertsPill(Widget):
count_rect = rl.Rectangle(self.rect.x + self.COUNT_OFFSET, self.rect.y, pill_w - self.COUNT_OFFSET, pill_h)
gui_label(count_rect, str(alert_count), font_size=36,
alignment=rl.GuiTextAlignment.TEXT_ALIGN_CENTER,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_MIDDLE)
alignment=TextAlignment.CENTER,
alignment_vertical=TextAlignmentVertical.MIDDLE)
class NetworkIcon(Widget):
@@ -139,8 +140,10 @@ class MiciHomeLayout(Widget):
self._version_text = self._get_version_text()
self._experimental_icon = IconWidget("icons_mici/experimental_mode.png", (48, 48))
self._egpu_icon = IconWidget("icons_mici/egpu_green.png", (50, 37))
self._egpu_icon_gray = IconWidget("icons_mici/egpu_gray.png", (50, 37))
self._usb_icon = IconWidget("icons_mici/usb.png", (62, 40))
self._chestnut_icon = IconWidget("icons_mici/chestnut_green.png", (68, 40))
self._chestnut_loading_icon = IconWidget("icons_mici/chestnut.png", (68, 40))
self._chestnut_failed_icon = IconWidget("icons_mici/chestnut_orange.png", (68, 40))
self._mic_icon = IconWidget("icons_mici/microphone.png", (32, 46))
self._body_icon = IconWidget("icons_mici/body.png", (54, 37))
@@ -150,13 +153,15 @@ class MiciHomeLayout(Widget):
IconWidget("icons_mici/settings.png", (48, 48), opacity=0.9),
NetworkIcon(),
self._experimental_icon,
self._egpu_icon,
self._egpu_icon_gray,
self._usb_icon,
self._chestnut_icon,
self._chestnut_loading_icon,
self._chestnut_failed_icon,
self._body_icon,
self._mic_icon,
], spacing=18)
self._openpilot_label = UnifiedLabel("sunnypilot", font_size=96, font_weight=FontWeight.DISPLAY, max_width=480, wrap_text=False)
self._openpilot_label = UnifiedLabel("openpilot", font_size=96, font_weight=FontWeight.DISPLAY, max_width=480, wrap_text=False)
self._version_label = UnifiedLabel("", font_size=36, font_weight=FontWeight.ROMAN, max_width=480, wrap_text=False)
self._large_version_label = UnifiedLabel("", font_size=64, text_color=rl.GRAY, font_weight=FontWeight.ROMAN, max_width=480, wrap_text=False)
self._date_label = UnifiedLabel("", font_size=36, text_color=rl.GRAY, font_weight=FontWeight.ROMAN, max_width=480, wrap_text=False)
@@ -247,9 +252,20 @@ class MiciHomeLayout(Widget):
self._version_commit_label.render()
# ***** Center-aligned bottom section icons *****
usb_connected = ui_state.usb_connected
usb_unknown = ui_state.usb_unknown
chestnut_state = ui_state.chestnut_state
self._experimental_icon.set_visible(ui_state.experimental_mode)
self._egpu_icon.set_visible(ui_state.sm["deviceState"].chestnutPresent and ui_state.usbgpu_compiled)
self._egpu_icon_gray.set_visible(ui_state.sm["deviceState"].chestnutPresent and not ui_state.usbgpu_compiled)
if gui_app.sunnypilot_ui():
self._set_chestnut_visibility()
else:
self._usb_icon.set_visible(usb_connected and usb_unknown)
self._chestnut_icon.set_visible(not usb_unknown and chestnut_state not in
(ChestnutState.LOADING, ChestnutState.UNCOMPILED, ChestnutState.FAILED) and
(usb_connected or chestnut_state in (ChestnutState.READY, ChestnutState.ACTIVE)))
self._chestnut_loading_icon.set_visible(not usb_unknown and chestnut_state == ChestnutState.LOADING)
self._chestnut_loading_icon.set_opacity(0.35 + 0.65 * (0.5 - 0.5 * math.cos(rl.get_time() * 6.0)))
self._chestnut_failed_icon.set_visible(not usb_unknown and chestnut_state in (ChestnutState.UNCOMPILED, ChestnutState.FAILED))
self._mic_icon.set_visible(ui_state.recording_audio)
self._body_icon.set_visible(bool(ui_state.is_body))
@@ -11,7 +11,7 @@ from openpilot.common.hardware import HARDWARE
from openpilot.system.ui.widgets import Widget
from openpilot.system.ui.widgets.label import UnifiedLabel
from openpilot.system.ui.widgets.scroller import Scroller
from openpilot.system.ui.lib.application import gui_app, FontWeight
from openpilot.system.ui.lib.application import gui_app, FontWeight, TextAlignment, TextAlignmentVertical
from openpilot.system.ui.lib.multilang import tr
REFRESH_INTERVAL = 5.0 # seconds
@@ -62,12 +62,12 @@ class AlertItem(Widget):
self._icon_green = gui_app.texture("icons_mici/offroad_alerts/green_wheel.png", self.ICON_SIZE, self.ICON_SIZE)
self._title_label = UnifiedLabel(text="", font_size=32, font_weight=FontWeight.SEMI_BOLD, text_color=self.TEXT_COLOR,
alignment=rl.GuiTextAlignment.TEXT_ALIGN_LEFT,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_TOP, line_height=0.95)
alignment=TextAlignment.LEFT,
alignment_vertical=TextAlignmentVertical.TOP, line_height=0.95)
self._body_label = UnifiedLabel(text="", font_size=28, font_weight=FontWeight.ROMAN, text_color=self.TEXT_COLOR,
alignment=rl.GuiTextAlignment.TEXT_ALIGN_LEFT,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_BOTTOM, line_height=0.95)
alignment=TextAlignment.LEFT,
alignment_vertical=TextAlignmentVertical.BOTTOM, line_height=0.95)
self._title_text = ""
self._body_text = ""
@@ -200,8 +200,8 @@ class MiciOffroadAlerts(Scroller):
# Create empty state label
self._empty_label = UnifiedLabel(tr("no alerts"), 65, FontWeight.DISPLAY, rl.WHITE,
alignment=rl.GuiTextAlignment.TEXT_ALIGN_CENTER,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_MIDDLE)
alignment=TextAlignment.CENTER,
alignment_vertical=TextAlignmentVertical.MIDDLE)
# Build initial alert list
self._build_alerts()
@@ -4,7 +4,7 @@ import pyray as rl
from collections.abc import Callable
from openpilot.common.filter_simple import FirstOrderFilter
from openpilot.common.qrcode import make_texture
from openpilot.system.ui.lib.application import FontWeight, gui_app
from openpilot.system.ui.lib.application import FontWeight, gui_app, TextAlignment
from openpilot.system.ui.widgets import Widget
from openpilot.system.ui.widgets.button import SmallCircleIconButton
from openpilot.system.ui.widgets.scroller import NavScroller, Scroller
@@ -35,7 +35,7 @@ class DriverCameraSetupDialog(BaseCabinCameraDialog):
if not self._camera_view.frame:
gui_label(rect, tr("camera starting"), font_size=64, font_weight=FontWeight.BOLD,
alignment=rl.GuiTextAlignment.TEXT_ALIGN_CENTER)
alignment=TextAlignment.CENTER)
rl.end_scissor_mode()
return
@@ -74,6 +74,10 @@ class SoftwareInfoLayoutMici(Widget):
class CheckUpdateButton(BigButton):
UPDATER_PROC = "openpilot.system.updated.updated"
CHECK_FOR_UPDATE = "SIGUSR1"
DOWNLOAD_UPDATE = "SIGHUP"
def __init__(self):
self._txt_update_icon = gui_app.texture("icons_mici/settings/device/update.png", 64, 75)
self._txt_up_to_date_icon = gui_app.texture("icons_mici/settings/device/up_to_date.png", 64, 64)
@@ -97,15 +101,20 @@ class CheckUpdateButton(BigButton):
gui_app.push_widget(dlg)
return
self._signal_updater(self.DOWNLOAD_UPDATE if self.get_value() == "download update" else self.CHECK_FOR_UPDATE)
def check_for_update(self):
self._signal_updater(self.CHECK_FOR_UPDATE)
def _signal_updater(self, sig: str):
self.set_enabled(False)
self._state = UpdaterState.WAITING_FOR_UPDATER
self._hide_value_t = None
self.set_value("")
self.set_icon(self._txt_update_icon)
def run():
if self.get_value() == "download update":
subprocess.run("pkill -SIGHUP -f openpilot.system.updated.updated", shell=True)
else:
subprocess.run("pkill -SIGUSR1 -f openpilot.system.updated.updated", shell=True)
subprocess.run(f"pkill -{sig} -f {self.UPDATER_PROC}", shell=True)
threading.Thread(target=run, daemon=True).start()
@@ -184,7 +193,7 @@ class CheckUpdateButton(BigButton):
class InstallUpdateButton(BigButton):
def __init__(self):
super().__init__("install update", "", gui_app.texture("icons_mici/settings/device/reboot.png", 64, 70))
super().__init__("install now", "", gui_app.texture("icons_mici/settings/device/reboot.png", 64, 70))
self.set_visible(lambda: ui_state.is_offroad() and ui_state.params.get_bool("UpdateAvailable"))
def _update_state(self):
@@ -232,8 +241,9 @@ class BranchSelectPage(NavScroller):
class TargetBranchButton(BigButton):
def __init__(self):
def __init__(self, check_update_btn: CheckUpdateButton):
super().__init__("target branch", ui_state.params.get("UpdaterTargetBranch") or "")
self._check_update_btn = check_update_btn
self.set_click_callback(self._on_click)
self.set_visible(not ui_state.params.get_bool("IsTestedBranch"))
self.set_enabled(lambda: ui_state.is_offroad())
@@ -246,12 +256,15 @@ class TargetBranchButton(BigButton):
self.set_value(target)
def _on_click(self):
if not ui_state.params.get("UpdaterAvailableBranches"):
gui_app.push_widget(BigDialog("", tr("Failed to get available branches. Ensure you're connected to the internet and try again.")))
return
gui_app.push_widget(BranchSelectPage(self._on_select))
def _on_select(self, branch: str):
ui_state.params.put("UpdaterTargetBranch", branch, block=True)
self.set_value(branch)
subprocess.run("pkill -SIGUSR1 -f openpilot.system.updated.updated", shell=True)
self._check_update_btn.check_for_update()
class SoftwareLayoutMici(NavScroller):
@@ -265,10 +278,11 @@ class SoftwareLayoutMici(NavScroller):
gui_app.texture("icons_mici/settings/device/uninstall.png", 64, 64),
uninstall_openpilot_callback, exit_on_confirm=False)
check_update_btn = CheckUpdateButton()
self._scroller.add_widgets([
SoftwareInfoLayoutMici(),
CheckUpdateButton(),
check_update_btn,
InstallUpdateButton(),
TargetBranchButton(),
TargetBranchButton(check_update_btn),
uninstall_openpilot_btn,
])
@@ -10,7 +10,7 @@ from opendbc.car.structs import car
from openpilot.selfdrive.ui.ui_state import ui_state
from openpilot.common.filter_simple import BounceFilter, FirstOrderFilter
from openpilot.common.hardware import COMMA_HARDWARE
from openpilot.system.ui.lib.application import gui_app, FontWeight
from openpilot.system.ui.lib.application import gui_app, FontWeight, TextAlignment
from openpilot.system.ui.widgets import Widget
from openpilot.system.ui.widgets.label import UnifiedLabel
@@ -333,7 +333,7 @@ class AlertRenderer(Widget, SpeedLimitAlertRenderer):
self._alert_text1_label.set_text(alert_text1)
self._alert_text1_label.set_text_color(color)
self._alert_text1_label.set_font_size(font_size)
self._alert_text1_label.set_alignment(rl.GuiTextAlignment.TEXT_ALIGN_LEFT if icon_side != 'left' else rl.GuiTextAlignment.TEXT_ALIGN_RIGHT)
self._alert_text1_label.set_alignment(TextAlignment.LEFT if icon_side != 'left' else TextAlignment.RIGHT)
self._alert_text1_label.render(text_rect1)
alert_text2 = alert.text2.lower()
@@ -365,5 +365,5 @@ class AlertRenderer(Widget, SpeedLimitAlertRenderer):
self._alert_text2_label.set_text(alert_text2)
self._alert_text2_label.set_text_color(color)
self._alert_text2_label.set_font_size(small_font_size)
self._alert_text2_label.set_alignment(rl.GuiTextAlignment.TEXT_ALIGN_LEFT if icon_side != 'left' else rl.GuiTextAlignment.TEXT_ALIGN_RIGHT)
self._alert_text2_label.set_alignment(TextAlignment.LEFT if icon_side != 'left' else TextAlignment.RIGHT)
self._alert_text2_label.render(text_rect2)
@@ -11,7 +11,7 @@ from openpilot.selfdrive.ui.mici.onroad.hud_renderer import HudRenderer
from openpilot.selfdrive.ui.mici.onroad.model_renderer import ModelRenderer
from openpilot.selfdrive.ui.mici.onroad.confidence_ball import ConfidenceBall
from openpilot.selfdrive.ui.mici.onroad.cameraview import CameraView
from openpilot.system.ui.lib.application import FontWeight, gui_app, MousePos, MouseEvent
from openpilot.system.ui.lib.application import FontWeight, gui_app, MousePos, MouseEvent, TextAlignment, TextAlignmentVertical
from openpilot.system.ui.widgets.label import UnifiedLabel
from openpilot.system.ui.widgets import Widget
from openpilot.common.filter_simple import BounceFilter
@@ -158,8 +158,8 @@ class AugmentedRoadView(CameraView):
self._confidence_ball = ConfidenceBall()
self._offroad_label = UnifiedLabel("start the car to\nuse sunnypilot", 54, FontWeight.DISPLAY,
text_color=rl.Color(255, 255, 255, int(255 * 0.9)),
alignment=rl.GuiTextAlignment.TEXT_ALIGN_CENTER,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_MIDDLE)
alignment=TextAlignment.CENTER,
alignment_vertical=TextAlignmentVertical.MIDDLE)
self._fade_texture = gui_app.texture("icons_mici/onroad/onroad_fade.png")
@@ -4,7 +4,7 @@ from openpilot.cereal.visionipc import VisionStreamType
from openpilot.selfdrive.ui.mici.onroad.cameraview import CameraView
from openpilot.selfdrive.ui.mici.onroad.driver_state import DriverStateRenderer
from openpilot.selfdrive.ui.ui_state import ui_state, device
from openpilot.system.ui.lib.application import gui_app, FontWeight
from openpilot.system.ui.lib.application import gui_app, FontWeight, TextAlignment, TextAlignmentVertical
from openpilot.system.ui.lib.multilang import tr
from openpilot.system.ui.widgets import Widget
from openpilot.system.ui.widgets.nav_widget import NavWidget
@@ -76,7 +76,7 @@ class BaseCabinCameraDialog(Widget):
if not self._camera_view.frame:
gui_label(rect, tr("camera starting"), font_size=54, font_weight=FontWeight.BOLD,
alignment=rl.GuiTextAlignment.TEXT_ALIGN_CENTER)
alignment=TextAlignment.CENTER)
rl.end_scissor_mode()
self._publish_alert_sound(None)
return
@@ -124,12 +124,12 @@ class BaseCabinCameraDialog(Widget):
awareness_pct = dm_state.visionPolicyState.awarenessPercent if is_vision else dm_state.wheeltouchPolicyState.awarenessPercent
gui_label(rl.Rectangle(rect.x + 2, rect.y + 2, rect.width, rect.height),
f"Awareness: {awareness_pct:.0f}%", font_size=44, font_weight=FontWeight.MEDIUM,
alignment=rl.GuiTextAlignment.TEXT_ALIGN_RIGHT,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_TOP,
alignment=TextAlignment.RIGHT,
alignment_vertical=TextAlignmentVertical.TOP,
color=rl.Color(0, 0, 0, 180))
gui_label(rect, f"Awareness: {awareness_pct:.0f}%", font_size=44, font_weight=FontWeight.MEDIUM,
alignment=rl.GuiTextAlignment.TEXT_ALIGN_RIGHT,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_TOP,
alignment=TextAlignment.RIGHT,
alignment_vertical=TextAlignmentVertical.TOP,
color=rl.Color(255, 255, 255, int(255 * 0.9)))
if dm_state.alertLevel == log.DriverMonitoringState.AlertLevel.none:
@@ -137,16 +137,16 @@ class BaseCabinCameraDialog(Widget):
# Show alert level
alert_level_str = f"{'Pay Attention' if is_vision else 'Touch Wheel'} - level {dm_state.alertLevel}"
alignment = rl.GuiTextAlignment.TEXT_ALIGN_RIGHT if self.driver_state_renderer.is_rhd else rl.GuiTextAlignment.TEXT_ALIGN_LEFT
alignment = TextAlignment.RIGHT if self.driver_state_renderer.is_rhd else TextAlignment.LEFT
shadow_rect = rl.Rectangle(rect.x + 2, rect.y + 2, rect.width, rect.height)
gui_label(shadow_rect, alert_level_str, font_size=40, font_weight=FontWeight.BOLD,
alignment=alignment,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_BOTTOM,
alignment_vertical=TextAlignmentVertical.BOTTOM,
color=rl.Color(0, 0, 0, 180))
gui_label(rect, alert_level_str, font_size=40, font_weight=FontWeight.BOLD,
alignment=alignment,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_BOTTOM,
alignment_vertical=TextAlignmentVertical.BOTTOM,
color=rl.Color(255, 255, 255, int(255 * 0.9)))
def _load_eye_textures(self):
@@ -3,7 +3,7 @@ import pyray as rl
from dataclasses import dataclass
from openpilot.common.constants import CV
from openpilot.selfdrive.ui.mici.onroad.torque_bar import TorqueBar
from openpilot.selfdrive.ui.ui_state import ui_state, UIStatus
from openpilot.selfdrive.ui.ui_state import ui_state, UIStatus, ChestnutState
from openpilot.system.ui.lib.application import gui_app, FontWeight
from openpilot.system.ui.lib.multilang import tr
from openpilot.system.ui.lib.text_measure import measure_text_cached
@@ -107,8 +107,7 @@ class HudRenderer(Widget):
self.speed: float = 0.0
self.v_ego_cluster_seen: bool = False
self._engaged: bool = False
self._small_model_engaged: bool = False
self._egpu_fade_time: float = 0
self._chestnut_fade_time: float = 0
self._can_draw_top_icons = True
self._show_wheel_critical = False
@@ -124,17 +123,15 @@ class HudRenderer(Widget):
self._txt_wheel: rl.Texture = gui_app.texture('icons_mici/wheel.png', 50, 50)
self._txt_wheel_critical: rl.Texture = gui_app.texture('icons_mici/wheel_critical.png', 50, 50)
self._txt_exclamation_point: rl.Texture = gui_app.texture('icons_mici/exclamation_point.png', 9, 44)
self._txt_egpu: rl.Texture = gui_app.texture('icons_mici/egpu.png', 60, 44)
self._txt_egpu_green: rl.Texture = gui_app.texture('icons_mici/egpu_green.png', 60, 44)
self._txt_egpu_orange: rl.Texture = gui_app.texture('icons_mici/egpu_orange.png', 60, 44)
self._txt_egpu_crossed: rl.Texture = gui_app.texture('icons_mici/egpu_crossed.png', 60, 52)
self._egpu_icon: rl.Texture | None = None
self._txt_chestnut: rl.Texture = gui_app.texture('icons_mici/chestnut.png', 60, 44)
self._txt_chestnut_green: rl.Texture = gui_app.texture('icons_mici/chestnut_green.png', 60, 44)
self._txt_chestnut_orange: rl.Texture = gui_app.texture('icons_mici/chestnut_orange.png', 75, 44)
self._chestnut_icon: rl.Texture | None = None
self._wheel_alpha_filter = FirstOrderFilter(0, 0.05, 1 / gui_app.target_fps)
self._wheel_y_filter = FirstOrderFilter(0, 0.1, 1 / gui_app.target_fps)
self._set_speed_alpha_filter = FirstOrderFilter(0.0, 0.1, 1 / gui_app.target_fps)
self._egpu_alpha_filter = FirstOrderFilter(0.0, 0.1, 1 / gui_app.target_fps)
self._chestnut_alpha_filter = FirstOrderFilter(0.0, 0.1, 1 / gui_app.target_fps)
def set_wheel_critical_icon(self, critical: bool):
"""Set the wheel icon to critical or normal state."""
@@ -165,13 +162,10 @@ class HudRenderer(Widget):
controls_state.deprecated.vCruise if v_cruise_cluster == 0.0 else v_cruise_cluster
)
engaged = sm['selfdriveState'].enabled
if (engaged and not self._engaged and not ui_state.usbgpu_loading and ui_state.usbgpu_active is not True and
ui_state.sm.recv_frame['modelV2'] > ui_state.started_frame):
self._small_model_engaged = True
if engaged != self._engaged:
self._egpu_fade_time = rl.get_time() if engaged else 0
if (set_speed != self.set_speed and engaged) or (engaged and not self._engaged):
self._set_speed_changed_time = rl.get_time()
if engaged != self._engaged:
self._chestnut_fade_time = rl.get_time() if engaged else 0
self._engaged = engaged
self.set_speed = set_speed
self.is_cruise_set = 0 < self.set_speed < SET_SPEED_NA
@@ -191,8 +185,7 @@ class HudRenderer(Widget):
if self.is_cruise_set:
self._draw_set_speed(rect)
if ui_state.usbgpu and ui_state.usbgpu_compiled:
self._draw_model_source(rect)
self._draw_model_source(rect)
self._draw_steering_wheel(rect)
@@ -200,30 +193,24 @@ class HudRenderer(Widget):
if ui_state.sm.recv_frame['selfdriveState'] < ui_state.started_frame:
return
big_failed = (ui_state.usbgpu_active is False or not ui_state.sm['deviceState'].chestnutPresent or
(ui_state.usbgpu_active is True and ui_state.sm.recv_frame['modelV2'] > ui_state.started_frame and
not ui_state.sm.alive['modelV2']) or
(ui_state.usbgpu_active is None and ui_state.sm.recv_frame['modelV2'] > ui_state.started_frame))
self._small_model_engaged &= big_failed
loading = ui_state.usbgpu_loading or (ui_state.usbgpu_active is None and not big_failed)
loading = ui_state.chestnut_state == ChestnutState.LOADING
if loading:
pulse = 0.5 - 0.5 * math.cos(rl.get_time() * 6.0)
icon = self._txt_egpu
opacity = 0.35 + 0.65 * pulse
elif self._small_model_engaged:
icon = self._txt_egpu_crossed
opacity = 0.65
elif big_failed:
icon = self._txt_egpu_orange
icon = self._txt_chestnut
opacity = 0.35 + 0.65 * (0.5 - 0.5 * math.cos(rl.get_time() * 6.0))
elif ui_state.chestnut_state in (ChestnutState.UNCOMPILED, ChestnutState.FAILED):
icon = self._txt_chestnut_orange
opacity = 1.0
elif ui_state.chestnut_state == ChestnutState.ACTIVE:
icon = self._txt_chestnut_green
opacity = 1.0
else:
icon = self._txt_egpu_green
opacity = 1.0
return
if icon is not self._egpu_icon:
self._egpu_fade_time = rl.get_time()
self._egpu_icon = icon
alpha = self._egpu_alpha_filter.update(loading or 0 < rl.get_time() - self._egpu_fade_time < SET_SPEED_PERSISTENCE)
if icon is not self._chestnut_icon:
self._chestnut_fade_time = rl.get_time()
self._chestnut_icon = icon
visible = loading or rl.get_time() - self._chestnut_fade_time < SET_SPEED_PERSISTENCE
alpha = self._chestnut_alpha_filter.update(visible)
if alpha < 1e-2:
return
+16 -18
View File
@@ -6,7 +6,7 @@ from collections.abc import Callable
from openpilot.system.ui.widgets import Widget
from openpilot.system.ui.widgets.label import UnifiedLabel
from openpilot.system.ui.widgets.scroller import DO_ZOOM
from openpilot.system.ui.lib.application import gui_app, FontWeight, MousePos
from openpilot.system.ui.lib.application import gui_app, FontWeight, MousePos, TextAlignmentVertical
from openpilot.common.filter_simple import BounceFilter
if TYPE_CHECKING:
@@ -125,10 +125,10 @@ class BigButton(Widget):
self._rotate_icon_t: float | None = None
self._label = UnifiedLabel(text, font_size=self._get_label_font_size(), font_weight=FontWeight.BOLD,
text_color=LABEL_COLOR, alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_BOTTOM, scroll=scroll,
text_color=LABEL_COLOR, alignment_vertical=TextAlignmentVertical.BOTTOM, scroll=scroll,
line_height=0.9)
self._sub_label = UnifiedLabel(value, font_size=COMPLICATION_SIZE, font_weight=FontWeight.ROMAN,
text_color=COMPLICATION_GREY, alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_BOTTOM)
text_color=COMPLICATION_GREY, alignment_vertical=TextAlignmentVertical.BOTTOM)
self._update_label_layout()
self._load_images()
@@ -149,11 +149,15 @@ class BigButton(Widget):
def set_touch_valid_callback(self, touch_callback: Callable[[], bool]) -> None:
super().set_touch_valid_callback(lambda: touch_callback() and self._grow_animation_until is None)
def _width_hint(self) -> int:
# A value moves the title to the top, where it shares space with the icon.
def _title_width_hint(self) -> int:
# A value moves the title to the top, where it shares space with the icon
icon_size = self._txt_icon.width if self._txt_icon and self.value else 0
return int(self._rect.width - self.LABEL_HORIZONTAL_PADDING * 2 - icon_size)
def _subtitle_width_hint(self) -> int:
# Bottom aligned, so it sits below the icon
return int(self._rect.width - self.LABEL_HORIZONTAL_PADDING * 2)
def _get_label_font_size(self):
if len(self.text) <= 18:
return 48
@@ -163,9 +167,9 @@ class BigButton(Widget):
def _update_label_layout(self):
self._label.set_font_size(self._get_label_font_size())
if self.value:
self._label.set_alignment_vertical(rl.GuiTextAlignmentVertical.TEXT_ALIGN_TOP)
self._label.set_alignment_vertical(TextAlignmentVertical.TOP)
else:
self._label.set_alignment_vertical(rl.GuiTextAlignmentVertical.TEXT_ALIGN_BOTTOM)
self._label.set_alignment_vertical(TextAlignmentVertical.BOTTOM)
def set_text(self, text: str):
self.text = text
@@ -228,14 +232,14 @@ class BigButton(Widget):
label_color = LABEL_COLOR if self.enabled else rl.Color(255, 255, 255, int(255 * 0.35))
self._label.set_color(label_color)
label_rect = rl.Rectangle(label_x, btn_y + self.LABEL_VERTICAL_PADDING, self._width_hint(),
label_rect = rl.Rectangle(label_x, btn_y + self.LABEL_VERTICAL_PADDING, self._title_width_hint(),
self._rect.height - self.LABEL_VERTICAL_PADDING * 2)
self._label.render(label_rect)
if self.value:
label_y = btn_y + self.LABEL_VERTICAL_PADDING + self._label.get_content_height(self._width_hint())
label_y = label_rect.y + self._label.get_content_height(int(label_rect.width))
sub_label_height = btn_y + self._rect.height - self.LABEL_VERTICAL_PADDING - label_y
sub_label_rect = rl.Rectangle(label_x, label_y, self._width_hint(), sub_label_height)
sub_label_rect = rl.Rectangle(label_x, label_y, self._subtitle_width_hint(), sub_label_height)
self._sub_label.render(sub_label_rect)
# ICON -------------------------------------------------------------------
@@ -312,9 +316,6 @@ class BigMultiToggle(BigToggle):
self.set_value(self._options[0])
def _width_hint(self) -> int:
return int(self._rect.width - self.LABEL_HORIZONTAL_PADDING * 2 - self._txt_enabled_toggle.width)
def _handle_mouse_release(self, mouse_pos: MousePos):
super()._handle_mouse_release(mouse_pos)
cur_idx = self._options.index(self.value)
@@ -355,17 +356,14 @@ class GreyBigButton(BigButton):
self._sub_label.set_font_size(36)
self._sub_label.set_text_color(rl.Color(255, 255, 255, int(255 * 0.9)))
self._sub_label.set_font_weight(FontWeight.DISPLAY_REGULAR)
self._sub_label.set_alignment_vertical(rl.GuiTextAlignmentVertical.TEXT_ALIGN_MIDDLE if not self._label.text else
rl.GuiTextAlignmentVertical.TEXT_ALIGN_BOTTOM)
self._sub_label.set_alignment_vertical(TextAlignmentVertical.MIDDLE if not self._label.text else
TextAlignmentVertical.BOTTOM)
self._sub_label.set_line_height(0.95)
@property
def LABEL_VERTICAL_PADDING(self):
return BigButton.LABEL_VERTICAL_PADDING if self._label.text else 18
def _width_hint(self) -> int:
return int(self._rect.width - self.LABEL_HORIZONTAL_PADDING * 2)
def _get_label_font_size(self):
return 36
@@ -4,7 +4,7 @@ from dataclasses import dataclass
from openpilot.cereal import messaging, log
from openpilot.selfdrive.ui.ui_state import ui_state
from openpilot.common.hardware import COMMA_HARDWARE
from openpilot.system.ui.lib.application import gui_app, FontWeight
from openpilot.system.ui.lib.application import gui_app, FontWeight, TextAlignment, TextAlignmentVertical
from openpilot.system.ui.lib.multilang import tr
from openpilot.system.ui.lib.text_measure import measure_text_cached
from openpilot.system.ui.widgets import Widget
@@ -76,10 +76,10 @@ class AlertRenderer(Widget):
self.font_bold: rl.Font = gui_app.font(FontWeight.BOLD)
# font size is set dynamically
self._full_text1_label = Label("", font_size=0, font_weight=FontWeight.BOLD, text_alignment=rl.GuiTextAlignment.TEXT_ALIGN_CENTER,
text_alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_TOP)
self._full_text2_label = Label("", font_size=ALERT_FONT_BIG, text_alignment=rl.GuiTextAlignment.TEXT_ALIGN_CENTER,
text_alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_TOP)
self._full_text1_label = Label("", font_size=0, font_weight=FontWeight.BOLD, text_alignment=TextAlignment.CENTER,
text_alignment_vertical=TextAlignmentVertical.TOP)
self._full_text2_label = Label("", font_size=ALERT_FONT_BIG, text_alignment=TextAlignment.CENTER,
text_alignment_vertical=TextAlignmentVertical.TOP)
def get_alert(self, sm: messaging.SubMaster) -> Alert | None:
"""Generate the current alert based on selfdrive state."""
@@ -4,7 +4,7 @@ from openpilot.cereal.visionipc import VisionStreamType
from openpilot.selfdrive.ui.onroad.cameraview import CameraView
from openpilot.selfdrive.ui.onroad.driver_state import DriverStateRenderer
from openpilot.selfdrive.ui.ui_state import ui_state, device
from openpilot.system.ui.lib.application import gui_app, FontWeight
from openpilot.system.ui.lib.application import gui_app, FontWeight, TextAlignment
from openpilot.system.ui.lib.multilang import tr
from openpilot.system.ui.widgets.label import gui_label
@@ -38,7 +38,7 @@ class CabinCameraDialog(CameraView):
tr("camera starting"),
font_size=100,
font_weight=FontWeight.BOLD,
alignment=rl.GuiTextAlignment.TEXT_ALIGN_CENTER,
alignment=TextAlignment.CENTER,
)
return -1
@@ -192,7 +192,7 @@ class ModelRenderer(Widget, ChevronMetrics, ModelRendererSP):
max_idx = self._get_path_length_idx(path_x_array, max_distance)
self._path.projected_points = self._map_line_to_polygon(
self._path.raw_points, 0.9, self._path_offset_z, max_idx, max_distance, allow_invert=False
self._path.raw_points, self._get_path_half_width(), self._path_offset_z, max_idx, max_distance, allow_invert=False
)
self._update_experimental_gradient()
@@ -292,7 +292,7 @@ class ModelRenderer(Widget, ChevronMetrics, ModelRendererSP):
allow_throttle = sm['longitudinalPlan'].allowThrottle or not self._longitudinal_control
self._blend_filter.update(int(allow_throttle))
if ui_state.rainbow_path:
if ui_state.rainbow_path and self._lateral_active:
self.rainbow_path.draw_rainbow_path(self._rect, self._path)
return
@@ -6,7 +6,7 @@ See the LICENSE.md file in the root directory for more details.
"""
import pyray as rl
from openpilot.selfdrive.ui.layouts.home import HomeLayout, HomeLayoutState, HEAD_BUTTON_FONT_SIZE, SPACING
from openpilot.system.ui.lib.application import gui_app, FontWeight
from openpilot.system.ui.lib.application import gui_app, FontWeight, TextAlignment
from openpilot.system.ui.lib.text_measure import measure_text_cached
from openpilot.system.ui.lib.multilang import tr, trn
from openpilot.system.ui.widgets.label import gui_label
@@ -59,7 +59,7 @@ class HomeLayoutSP(HomeLayout):
desc_size = measure_text_cached(gui_app.font(FontWeight.NORMAL), description, BRAND_FONT_SIZE)
desc_width = desc_size.x
desc_rect = rl.Rectangle(version_right - desc_width, self.header_rect.y, desc_width, self.header_rect.height)
gui_label(desc_rect, description, BRAND_FONT_SIZE, rl.WHITE, alignment=rl.GuiTextAlignment.TEXT_ALIGN_RIGHT)
gui_label(desc_rect, description, BRAND_FONT_SIZE, rl.WHITE, alignment=TextAlignment.RIGHT)
brand_size = measure_text_cached(gui_app.font(FontWeight.AUDIOWIDE), brand, BRAND_FONT_SIZE)
spacing = BRAND_DESC_SPACING if description else 0
@@ -6,7 +6,7 @@ See the LICENSE.md file in the root directory for more details.
"""
import pyray as rl
from openpilot.selfdrive.ui.ui_state import ui_state
from openpilot.system.ui.lib.application import FontWeight
from openpilot.system.ui.lib.application import FontWeight, TextAlignment
from openpilot.system.ui.lib.multilang import tr
from openpilot.system.ui.widgets import Widget
from openpilot.system.ui.widgets.button import Button, ButtonStyle
@@ -20,7 +20,7 @@ class SunnylinkConsentPage(Widget):
self._done_callback = done_callback
self._step = 0
self._title = self._child(Label(tr("sunnylink"), font_size=90, font_weight=FontWeight.AUDIOWIDE, text_alignment=rl.GuiTextAlignment.TEXT_ALIGN_LEFT))
self._title = self._child(Label(tr("sunnylink"), font_size=90, font_weight=FontWeight.AUDIOWIDE, text_alignment=TextAlignment.LEFT))
self._content = [
{
@@ -43,7 +43,7 @@ class SunnylinkConsentPage(Widget):
self._primary_btn = self._child(Button("", button_style=ButtonStyle.PRIMARY, click_callback=lambda: self._handle_choice("enable")))
self._secondary_btn = self._child(Button("", button_style=ButtonStyle.NORMAL, click_callback=lambda: self._handle_choice("secondary")))
self._danger_btn = self._child(Button("", button_style=ButtonStyle.DANGER, click_callback=lambda: self._handle_choice("disable")))
self._desc = self._child(Label("", font_size=90, font_weight=FontWeight.MEDIUM, text_alignment=rl.GuiTextAlignment.TEXT_ALIGN_LEFT))
self._desc = self._child(Label("", font_size=90, font_weight=FontWeight.MEDIUM, text_alignment=TextAlignment.LEFT))
def _handle_choice(self, choice):
if choice == "enable":
@@ -10,9 +10,10 @@ import time
import pyray as rl
from openpilot.cereal import custom
from openpilot.sunnypilot.models.default_model import DEFAULT_MODEL
from openpilot.sunnypilot.models.helpers import ACTIVE_BUNDLE_KEYS, get_selected_bundle, resolve_bundle_by_ref
from openpilot.common.constants import CV
from openpilot.selfdrive.ui.ui_state import device, ui_state
from openpilot.selfdrive.ui.sunnypilot.model_info import big_model_state, bundles_for_source, carrying_model, default_model_name, queued_name
from openpilot.system.ui.lib.multilang import tr
from openpilot.system.ui.lib.application import gui_app
from openpilot.system.ui.widgets import DialogResult, Widget
@@ -36,7 +37,10 @@ class ModelsLayout(Widget):
super().__init__()
self.model_manager = None
self.model_dialog = None
self._selection_source = None
self._downloading = False
self._verifying = False
self._last_note = None
self.last_cache_calc_time = 0
self._initialize_items()
@@ -48,17 +52,24 @@ class ModelsLayout(Widget):
self._scroller = Scroller(self.items, line_separator=True, spacing=0)
def _initialize_items(self):
self.current_model_item = ListItemSP(
title=tr("Current Model"),
self.small_model_item = ListItemSP(
title=tr("Small Model"),
description="",
action_item=ScrollingButtonAction(tr("SELECT")),
callback=self._handle_current_model_clicked
callback=lambda: self._open_source_dialog("qcom")
)
self.big_model_item = ListItemSP(
title=tr("Big Model"),
action_item=ScrollingButtonAction(tr("SELECT")),
callback=lambda: self._open_source_dialog("chestnut")
)
self.download_item = download_status_item(lambda: tr("Download") if self._downloading else tr("Model Status"))
self.refresh_item = button_item(tr("Refresh Model List"), tr("REFRESH"), "",
lambda: (ui_state.params.put("ModelManager_LastSyncTime", 0),
ui_state.params.put("ModelManager_LastSyncTime_Chestnut", 0),
gui_app.push_widget(alert_dialog(tr("Fetching Latest Models")))))
self.clear_cache_item = ListItemSP(
@@ -68,7 +79,9 @@ class ModelsLayout(Widget):
callback=self._clear_cache
)
self.cancel_download_item = button_item(tr("Cancel Download"), tr("Cancel"), "", lambda: ui_state.params.remove("ModelManager_DownloadIndex"))
self.cancel_download_item = button_item(lambda: tr("Cancel Verification") if self._verifying else tr("Cancel Download"),
tr("Cancel"), "",
lambda: ui_state.params.remove("ModelManager_DownloadRef"))
self.lane_turn_value_control = option_item_sp(tr("Adjust Lane Turn Speed"), "LaneTurnValue", 500, 2000,
tr("Set the maximum speed for lane turn desires. Default is 19 mph."),
@@ -93,7 +106,7 @@ class ModelsLayout(Widget):
1, None, True, "", style.BUTTON_ACTION_WIDTH, None, True,
lambda v: f"{v / 100:.2f} m")
self.items = [self.current_model_item, self.cancel_download_item, self.download_item, self.refresh_item, self.clear_cache_item,
self.items = [self.small_model_item, self.big_model_item, self.cancel_download_item, self.download_item, self.refresh_item, self.clear_cache_item,
self.lane_turn_desire_toggle, self.lane_turn_value_control, self.lagd_toggle, self.delay_control, self.camera_offset]
def _update_lagd_description(self, lagd_toggle: bool):
@@ -107,16 +120,16 @@ class ModelsLayout(Widget):
desc += f"<br>{tr('Actuator Delay:')} {cp:.2f} s + {tr('Software Delay:')} {sw:.2f} s = {tr('Total Delay:')} {cp + sw:.2f} s"
self.lagd_toggle.set_description(desc)
def _is_downloading(self):
return (self.model_manager and self.model_manager.selectedBundle and
self.model_manager.selectedBundle.status == custom.ModelManagerSP.DownloadStatus.downloading)
@staticmethod
def calculate_cache_size():
cache_size = 0.0
if os.path.exists(CUSTOM_MODEL_PATH):
cache_size = sum(os.path.getsize(os.path.join(CUSTOM_MODEL_PATH, file)) for file in os.listdir(CUSTOM_MODEL_PATH)) / (1024**2)
return cache_size
for file in os.listdir(CUSTOM_MODEL_PATH):
try:
cache_size += os.path.getsize(os.path.join(CUSTOM_MODEL_PATH, file))
except OSError:
continue
return cache_size / (1024**2)
def _clear_cache(self):
def _callback(response):
@@ -129,36 +142,90 @@ class ModelsLayout(Widget):
gui_app.push_widget(dialog)
def _handle_bundle_download_progress(self):
self.download_item.set_visible(False)
self.cancel_download_item.set_visible(False)
self._downloading = False
if not self.model_manager or (not self.model_manager.selectedBundle and not self.model_manager.activeBundle):
return
bundle = self.model_manager.selectedBundle if self._is_downloading() or (
self.model_manager.selectedBundle and self.model_manager.selectedBundle.status == custom.ModelManagerSP.DownloadStatus.failed
) else self.model_manager.activeBundle
if not bundle:
return
self.cancel_download_item.set_visible(bool(self.model_manager.selectedBundle) and ui_state.params.get("ModelManager_DownloadIndex") is not None)
self._verifying = False
self.download_item.set_visible(True)
if (current_time := time.monotonic()) - self.last_cache_calc_time > 0.5:
self.last_cache_calc_time = current_time
self.clear_cache_item.action_item.set_value(f"{self.calculate_cache_size():.2f} MB")
bundle = self.model_manager.selectedBundle if self.model_manager else None
progresses = [model.artifact.downloadProgress for model in bundle.models if model.artifact.fileName] if bundle else []
if not progresses or bundle.status not in (custom.ModelManagerSP.DownloadStatus.downloading,
custom.ModelManagerSP.DownloadStatus.failed):
self.download_item.action_item.update(name="", segments=self._slot_segments())
return
self.cancel_download_item.set_visible(ui_state.params.get("ModelManager_DownloadRef") is not None)
if bundle.status == custom.ModelManagerSP.DownloadStatus.downloading:
device._reset_interactive_timeout()
# every bundle is a single chunked artifact now
progresses = [model.artifact.downloadProgress for model in bundle.models if model.artifact.fileName]
if not progresses:
return
self.download_item.set_visible(True)
self.download_item.action_item.update(**self._download_row_state(progresses, bundle.internalName))
state = self._download_row_state(progresses, bundle.internalName)
if queued := queued_name(bundle.ref):
state["name"] += f" | {queued} {tr('queued')}"
self.download_item.action_item.update(**state)
self._downloading = self.download_item.action_item.downloading
ds = custom.ModelManagerSP.DownloadStatus
self._verifying = any(getattr(p.status, 'raw', p.status) == ds.verifying for p in progresses)
def _slot_segments(self):
"""small and big slots side by side; green marks the slot whose pick is actually
driving (runner-matched, so a failed Default big greens neither slot), an empty
slot shows its default."""
big_state = big_model_state()
carry_source, carry_internal, _ = carrying_model()
segments = []
for source, label in (("qcom", tr("small")), ("chestnut", tr("big"))):
if segments:
segments.append(("|", rl.GRAY, None, None))
bundle = get_selected_bundle(ui_state.params, source)
name = bundle.internalName if bundle else default_model_name(source)
color = ON_COLOR if (source == carry_source and name == carry_internal) else rl.LIGHTGRAY
name = "" + name
if source == "chestnut":
if big_state == 'failed':
color = rl.RED
elif big_state == 'loading':
color = rl.GOLD
segments.append((label, rl.GRAY, None, None))
segments.append((name, color, None, None))
return segments
@staticmethod
def _set_item_note(item, text):
# a description renders only while shown; hide before clearing or the
# empty description keeps its visible state
if text:
item.set_description(text)
item.show_description(True)
else:
item.show_description(False)
item.set_description("")
def _status_note(self) -> str:
"""The failover story for the Model Status row. One-way big -> small, and the
fallback is runner-matched: a Default big can only fall back to the Default
small (stock modeld), a custom big has no automatic fallback yet."""
if not ui_state.chestnut_present:
return ""
big_bundle = get_selected_bundle(ui_state.params, "chestnut")
big_name = big_bundle.internalName if big_bundle else default_model_name("chestnut")
big_is_default = big_bundle is None
fallback_name = default_model_name("qcom")
state = big_model_state()
if state == 'failed':
if big_is_default:
return tr("Big model unavailable, {} is driving until the next drive.").format(fallback_name)
return tr("Big model unavailable until the next drive.")
if state == 'loading':
if big_is_default:
return tr("{} drives until the big model is ready.").format(fallback_name)
return tr("Getting the big model ready.")
if big_is_default:
return tr("{} will drive. If it fails during a drive, {} takes over until the next drive.").format(big_name, fallback_name)
return tr("{} will drive when the chestnut is ready.").format(big_name)
@staticmethod
def _download_row_state(progresses, name: str) -> dict:
@@ -171,6 +238,8 @@ class ModelsLayout(Widget):
if ds.failed in statuses:
# close.png is authored black and a tint cannot lift it, hence close2
return {"name": name, "status_text": tr("download failed"), "text_color": rl.RED, "icon": "icons/close2.png"}
if ds.verifying in statuses:
return {"name": name, "downloading": True, "progress": progress, "status_text": tr("verifying")}
if ds.downloading in statuses:
return {"name": name, "downloading": True, "progress": progress}
if statuses <= {ds.downloaded, ds.cached}:
@@ -178,64 +247,73 @@ class ModelsLayout(Widget):
# circled_slash is authored grey; tinting it again only darkens it
return {"name": name, "text_color": rl.GRAY, "icon": "icons/circled_slash.png", "icon_color": rl.WHITE}
@staticmethod
def _show_reset_params_dialog():
def _callback(response):
if response == DialogResult.CONFIRM:
ui_state.params.remove("CalibrationParams")
ui_state.params.remove("LiveTorqueParameters")
msg = tr("Model download has started in the background. We suggest resetting calibration. Would you like to do that now?")
dialog = ConfirmDialog(msg, tr("Reset Calibration"), callback=_callback)
gui_app.push_widget(dialog)
def _on_model_selected(self, result):
if result != DialogResult.CONFIRM:
self.model_dialog = None
return
selected_ref = self.model_dialog.selection_ref
if selected_ref == "Default":
ui_state.params.remove("ModelManager_ActiveBundle")
self._show_reset_params_dialog()
elif selected_bundle := next((bundle for bundle in self.model_manager.availableBundles if bundle.ref == selected_ref), None):
ui_state.params.put("ModelManager_DownloadIndex", selected_bundle.index)
if self.model_manager.activeBundle and selected_bundle.generation != self.model_manager.activeBundle.generation:
self._show_reset_params_dialog()
self.model_dialog = None
if selected_ref == "Default":
if self._selection_source in ACTIVE_BUNDLE_KEYS:
ui_state.params.remove(ACTIVE_BUNDLE_KEYS[self._selection_source])
return
if selected_bundle := self._resolve_selected_bundle(selected_ref):
ui_state.params.put("ModelManager_DownloadRef", selected_bundle.ref)
def _resolve_selected_bundle(self, ref):
source_bundles = {source: bundles_for_source(source) for source in ("qcom", "chestnut")}
resolved = resolve_bundle_by_ref(ref, source_bundles)
return resolved[0] if resolved else None
@staticmethod
def _bundle_to_node(bundle):
return TreeNode(bundle.ref, {'display_name': bundle.displayName, 'short_name': bundle.internalName})
def _get_folders(self, favorites):
bundles = self.model_manager.availableBundles
def _get_folders(self, favorites, bundles):
folders = {}
for bundle in bundles:
folders.setdefault(next((ov_ride.value for ov_ride in bundle.overrides if ov_ride.key == "folder"), ""), []).append(bundle)
folders_list = [TreeFolder("", [TreeNode("Default", {'display_name': f"{DEFAULT_MODEL} (Default)", 'short_name': "Default"})])]
folders_list = []
for folder, folder_bundles in sorted(folders.items(), key=lambda x: max((bundle.index for bundle in x[1]), default=-1), reverse=True):
folder_bundles.sort(key=lambda bundle: bundle.index, reverse=True)
name = folder + (f" - (Updated: {m.group(1)})" if folder_bundles and (m := re.search(r'\(([^)]*)\)[^(]*$', folder_bundles[0].displayName)) else "")
folders_list.append(TreeFolder(name, [self._bundle_to_node(bundle) for bundle in folder_bundles]))
if favorites and (fav_bundles := [bundle for bundle in bundles if bundle.ref in favorites]):
folders_list.insert(1, TreeFolder("Favorites", [self._bundle_to_node(bundle) for bundle in fav_bundles]))
folders_list.insert(0, TreeFolder("Favorites", [self._bundle_to_node(bundle) for bundle in fav_bundles]))
return folders_list
def _handle_current_model_clicked(self):
def _open_source_dialog(self, source):
self._selection_source = source
favs = ui_state.params.get("ModelManager_Favs")
favorites = set(favs.split(';')) if favs else set()
folders_list = self._get_folders(favorites)
active_ref = self.model_manager.activeBundle.ref if self.model_manager.activeBundle else "Default"
self.model_dialog = TreeOptionDialog(tr("Select a Model"), folders_list, active_ref, "ModelManager_Favs",
get_folders_fn=self._get_folders, on_exit=self._on_model_selected)
folders_list = self._source_folders(favorites, source)
if not folders_list:
gui_app.push_widget(alert_dialog(tr("No models are available for this hardware yet. Connect to the internet and refresh the model list.")))
return
self.model_dialog = TreeOptionDialog(tr("Select a Model"), folders_list, self._slot_active_ref(source), "ModelManager_Favs",
get_folders_fn=lambda favs: self._source_folders(favs, source), on_exit=self._on_model_selected)
gui_app.push_widget(self.model_dialog)
def _source_folders(self, favorites, source):
bundles = bundles_for_source(source)
if not bundles:
return []
folders_list = [TreeFolder("", [TreeNode("Default", {'display_name': default_model_name(source)})])]
folders_list.extend(self._get_folders(favorites, bundles))
return folders_list
@staticmethod
def _slot_active_ref(source: str) -> str:
bundle = get_selected_bundle(ui_state.params, source)
return bundle.ref if bundle else "Default"
def _update_state(self):
advanced_controls: bool = ui_state.params.get_bool("ShowAdvancedControls")
turn_desire: bool = ui_state.params.get_bool("LaneTurnDesire")
live_delay: bool = ui_state.params.get_bool("LagdToggle")
camera_offset: bool = ui_state.params.get("ModelManager_ActiveBundle") is not None
camera_offset: bool = ui_state.active_bundle is not None
self.lane_turn_desire_toggle.action_item.set_state(turn_desire)
self.lane_turn_value_control.set_visible(turn_desire and advanced_controls)
@@ -249,18 +327,27 @@ class ModelsLayout(Widget):
self._update_lagd_description(live_delay)
self.model_manager = ui_state.sm["modelManagerSP"]
self._handle_bundle_download_progress()
active_name = self.model_manager.activeBundle.displayName if self.model_manager and self.model_manager.activeBundle.ref else f"{DEFAULT_MODEL} (Default)"
self.current_model_item.action_item.set_value(active_name)
if not ui_state.is_offroad():
self.current_model_item.action_item.set_enabled(False)
self.current_model_item.set_description(tr("Only available when vehicle is off, or always offroad mode is on"))
else:
self.current_model_item.action_item.set_enabled(True)
self.current_model_item.set_description("")
carry_source, _, carry_display = carrying_model()
for item, item_source in ((self.small_model_item, "qcom"), (self.big_model_item, "chestnut")):
bundle = get_selected_bundle(ui_state.params, item_source)
name = bundle.displayName if bundle else default_model_name(item_source)
color = ON_COLOR if (item_source == carry_source and name == carry_display) else style.ITEM_TEXT_VALUE_COLOR
item.action_item.set_value(name, color)
note = self._status_note()
if note != self._last_note:
self._last_note = note
self._set_item_note(self.download_item, note)
offroad = ui_state.is_offroad()
self.small_model_item.action_item.set_enabled(offroad)
self.big_model_item.action_item.set_enabled(offroad)
self.small_model_item.set_description("" if offroad else tr("Only available when vehicle is off, or always offroad mode is on"))
def _render(self, rect):
self._scroller.render(rect)
def show_event(self):
self._scroller.show_event()
self._last_note = None # re-expand the failover note every time the page opens
@@ -8,7 +8,6 @@ import datetime
import os
import platform
import requests
import shutil
import threading
from pathlib import Path
from time import monotonic
@@ -75,22 +74,12 @@ class OSMLayout(Widget):
def _update_map_size(self):
threading.Thread(target=self.calculate_size, daemon=True).start()
def _do_delete_maps(self):
if MAP_PATH.exists():
shutil.rmtree(MAP_PATH)
for param in ("OsmDownloadedDate", "OsmLocal", "OsmLocationName", "OsmLocationTitle", "OsmStateName", "OsmStateTitle"):
ui_state.params.remove(param)
def _on_confirm_delete_maps(self):
ui_state.params.put_bool("Mapd_ClearCache", True)
self._delete_maps_btn.action_item.set_enabled(True)
self._delete_maps_btn.action_item.set_text(tr("DELETE"))
self._update_map_size()
def _on_confirm_delete_maps(self):
self._delete_maps_btn.action_item.set_enabled(False)
self._delete_maps_btn.action_item.set_text("DELETING...")
threading.Thread(target=self._do_delete_maps).start()
def _delete_maps(self):
self._show_confirm(tr("This will delete ALL downloaded maps\n\nAre you sure you want to delete all maps?"),
tr("Yes, delete all maps"), self._on_confirm_delete_maps)
@@ -9,7 +9,7 @@ from openpilot.cereal import custom
from openpilot.selfdrive.ui.sunnypilot.layouts.onboarding import SunnylinkConsentPage
from openpilot.selfdrive.ui.ui_state import ui_state
from openpilot.sunnypilot.sunnylink.api import UNREGISTERED_SUNNYLINK_DONGLE_ID
from openpilot.system.ui.lib.application import gui_app, FontWeight
from openpilot.system.ui.lib.application import gui_app, FontWeight, TextAlignment, TextAlignmentVertical
from openpilot.system.ui.lib.multilang import tr
from openpilot.system.ui.sunnypilot.widgets.list_view import button_item_sp
from openpilot.system.ui.sunnypilot.widgets.list_view import toggle_item_sp
@@ -32,8 +32,8 @@ class SunnylinkHeader(Widget):
font_size=90,
font_weight=FontWeight.AUDIOWIDE,
text_color=rl.WHITE,
alignment=rl.GuiTextAlignment.TEXT_ALIGN_CENTER,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_TOP,
alignment=TextAlignment.CENTER,
alignment_vertical=TextAlignmentVertical.TOP,
wrap_text=False,
elide=False
)
@@ -43,8 +43,8 @@ class SunnylinkHeader(Widget):
font_size=40,
font_weight=FontWeight.NORMAL,
text_color=rl.Color(0, 255, 0, 255), # Green
alignment=rl.GuiTextAlignment.TEXT_ALIGN_CENTER,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_TOP,
alignment=TextAlignment.CENTER,
alignment_vertical=TextAlignmentVertical.TOP,
wrap_text=True,
elide=False
)
@@ -55,8 +55,8 @@ class SunnylinkHeader(Widget):
font_size=35,
font_weight=FontWeight.NORMAL,
text_color=rl.Color(255, 165, 0, 255), # Orange
alignment=rl.GuiTextAlignment.TEXT_ALIGN_CENTER,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_TOP,
alignment=TextAlignment.CENTER,
alignment_vertical=TextAlignmentVertical.TOP,
wrap_text=True,
elide=False
)
@@ -109,8 +109,8 @@ class SunnylinkDescriptionItem(Widget):
font_size=40,
font_weight=FontWeight.NORMAL,
text_color=rl.WHITE,
alignment=rl.GuiTextAlignment.TEXT_ALIGN_LEFT,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_TOP,
alignment=TextAlignment.LEFT,
alignment_vertical=TextAlignmentVertical.TOP,
wrap_text=True,
elide=False,
)
@@ -4,11 +4,14 @@ Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
import math
import pyray as rl
import time
from dataclasses import dataclass
from openpilot.selfdrive.ui.ui_state import ui_state
from openpilot.selfdrive.ui.ui_state import ui_state, ChestnutState
from openpilot.sunnypilot.sunnylink.api import UNREGISTERED_SUNNYLINK_DONGLE_ID
from openpilot.system.ui.lib.application import gui_app
from openpilot.system.ui.lib.multilang import tr_noop
@@ -18,6 +21,9 @@ METRIC_MARGIN = 30
METRIC_START_Y = 300
HOME_BTN = rl.Rectangle(60, 860, 180, 180)
CHESTNUT_ICON_WIDTH = 180
CHESTNUT_ICON_HEIGHT = 133
# Color scheme
class Colors:
@@ -53,6 +59,9 @@ class MetricData:
class SidebarSP:
def __init__(self):
self._sunnylink_status = MetricData(tr_noop("SUNNYLINK"), tr_noop("OFFLINE"), Colors.WARNING)
self._chestnut_green_img = gui_app.texture("icons_mici/chestnut_green.png", CHESTNUT_ICON_WIDTH, CHESTNUT_ICON_HEIGHT)
self._chestnut_default_img = gui_app.texture("icons_mici/chestnut.png", CHESTNUT_ICON_WIDTH, CHESTNUT_ICON_HEIGHT)
self._chestnut_orange_img = gui_app.texture("icons_mici/chestnut_orange.png", CHESTNUT_ICON_WIDTH, CHESTNUT_ICON_HEIGHT)
def _update_sunnylink_status(self):
if not ui_state.params.get_bool("SunnylinkEnabled"):
@@ -78,6 +87,24 @@ class SidebarSP:
self._sunnylink_status.update(tr_noop("SUNNYLINK"), status, color)
def _get_home_icon(self, default_img: rl.Texture) -> tuple[rl.Texture, rl.Vector2, float]:
default_pos = rl.Vector2(HOME_BTN.x, HOME_BTN.y)
state = ui_state.chestnut_state
if state == ChestnutState.DISCONNECTED:
return default_img, default_pos, 1.0
if state == ChestnutState.LOADING:
icon = self._chestnut_default_img
opacity = 0.35 + 0.65 * (0.5 - 0.5 * math.cos(rl.get_time() * 6.0))
elif state in (ChestnutState.UNCOMPILED, ChestnutState.FAILED):
icon, opacity = self._chestnut_orange_img, 1.0
else:
icon, opacity = self._chestnut_green_img, 1.0
x = HOME_BTN.x + (HOME_BTN.width - icon.width) / 2
y = HOME_BTN.y + (HOME_BTN.height - icon.height) / 2
return icon, rl.Vector2(x, y), opacity
def _draw_metrics_w_sunnylink(self, rect: rl.Rectangle, _temp, _panda, _connect):
metrics = [_temp, _panda, _connect, self._sunnylink_status]
start_y = int(rect.y) + METRIC_START_Y
@@ -4,7 +4,12 @@ Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
import math
import pyray as rl
from openpilot.selfdrive.ui.mici.layouts.home import MiciHomeLayout
from openpilot.selfdrive.ui.ui_state import ui_state, ChestnutState
from openpilot.system.ui.lib.application import FontWeight
from openpilot.system.ui.widgets.label import UnifiedLabel
@@ -13,3 +18,16 @@ class MiciHomeLayoutSP(MiciHomeLayout):
def __init__(self):
super().__init__()
self._openpilot_label = UnifiedLabel("sunnypilot", font_size=88, font_weight=FontWeight.AUDIOWIDE, max_width=480, wrap_text=False)
def _set_chestnut_visibility(self):
usb_connected = ui_state.usb_connected
usb_unknown = ui_state.usb_unknown
chestnut_state = ui_state.chestnut_state
loading = chestnut_state == ChestnutState.LOADING
self._usb_icon.set_visible(usb_connected and usb_unknown)
self._chestnut_loading_icon.set_opacity(0.35 + 0.65 * (0.5 - 0.5 * math.cos(rl.get_time() * 6.0)))
self._chestnut_loading_icon.set_visible(not usb_unknown and loading)
self._chestnut_icon.set_visible(not usb_unknown and not loading and
chestnut_state in (ChestnutState.READY, ChestnutState.ACTIVE))
self._chestnut_failed_icon.set_visible(not usb_unknown and chestnut_state in (ChestnutState.UNCOMPILED, ChestnutState.FAILED))
@@ -7,16 +7,37 @@ See the LICENSE.md file in the root directory for more details.
import pyray as rl
from openpilot.cereal import custom
from openpilot.sunnypilot.models.default_model import DEFAULT_MODEL
from openpilot.selfdrive.ui.mici.widgets.dialog import BigDialog
from openpilot.sunnypilot.models.helpers import ACTIVE_BUNDLE_KEYS, get_selected_bundle
from openpilot.selfdrive.ui.mici.widgets.button import BigButton
from openpilot.selfdrive.ui.sunnypilot.layouts.settings.models import ModelsLayout
from openpilot.selfdrive.ui.ui_state import ui_state, device
from openpilot.selfdrive.ui.sunnypilot.model_info import (active_source, big_model_state, bundles_for_source, carrying_model,
default_model_name, model_info, queued_name)
from openpilot.system.ui.lib.application import FontWeight, gui_app
from openpilot.system.ui.lib.multilang import tr
from openpilot.system.ui.widgets import Widget
from openpilot.system.ui.widgets.label import UnifiedLabel
from openpilot.system.ui.widgets.scroller import NavScroller
def _model_info() -> tuple[str, str, str]:
"""(active model, info header, info text) for the panel. Runner-matched: the
active line names what actually drives, and a notable big-model state takes
the info pair."""
source, active_name, other_name = model_info()
state = big_model_state()
_, _, carry_display = carrying_model()
if carry_display is None:
big = get_selected_bundle(ui_state.params, "chestnut")
carry_display = big.displayName if big else default_model_name("chestnut")
active_text = (carry_display or active_name).lower()
if state == 'failed':
return active_text, tr("big model"), tr("unavailable")
if state == 'loading':
return active_text, tr("big model"), tr("getting ready")
header = tr("small model") if source == "chestnut" else tr("big model")
return active_text, header, other_name.lower()
class CurrentModelInfo(Widget):
def __init__(self):
super().__init__()
@@ -26,12 +47,12 @@ class CurrentModelInfo(Widget):
header_color = rl.Color(255, 255, 255, int(255 * 0.9))
subheader_color = rl.Color(255, 255, 255, int(255 * 0.9 * 0.65))
max_width = int(self._rect.width - 20)
active_text, info_header, info_text = _model_info()
self.current_model_header = UnifiedLabel(tr("active model"), 48, max_width=max_width, text_color=header_color, font_weight=FontWeight.DISPLAY)
default_text = f"{DEFAULT_MODEL} (Default)".lower()
self.current_model_text = UnifiedLabel(default_text, 32, max_width=max_width, text_color=subheader_color, font_weight=FontWeight.ROMAN, scroll=True)
self.current_model_text = UnifiedLabel(active_text, 32, max_width=max_width, text_color=subheader_color, font_weight=FontWeight.ROMAN, scroll=True)
self.info_header = UnifiedLabel("cache size", 48, max_width=max_width, text_color=header_color, font_weight=FontWeight.DISPLAY)
self.info_text = UnifiedLabel("0 mb", 32, max_width=max_width, text_color=subheader_color, font_weight=FontWeight.ROMAN)
self.info_header = UnifiedLabel(info_header, 48, max_width=max_width, text_color=header_color, font_weight=FontWeight.DISPLAY)
self.info_text = UnifiedLabel(info_text, 32, max_width=max_width, text_color=subheader_color, font_weight=FontWeight.ROMAN, scroll=True)
def _render(self, _):
self.current_model_header.set_position(self._rect.x + 20, self._rect.y - 10)
@@ -55,12 +76,13 @@ class ModelsLayoutMici(NavScroller):
self._download_progress = "."
self._download_frame = 0
self._was_downloading = False
self._selection_source: str | None = None
self.select_model_btn = BigButton(tr("select model"))
self.select_model_btn.set_click_callback(self._show_folders)
self.cancel_download_btn = BigButton(tr("cancel download"))
self.cancel_download_btn.set_click_callback(lambda: ui_state.params.remove("ModelManager_DownloadIndex"))
self.cancel_download_btn.set_click_callback(lambda: ui_state.params.remove("ModelManager_DownloadRef"))
self.main_items = [self.current_model_info, self.select_model_btn, self.cancel_download_btn]
self._scroller.add_widgets(self.main_items)
@@ -69,8 +91,7 @@ class ModelsLayoutMici(NavScroller):
def model_manager(self):
return ui_state.sm["modelManagerSP"]
def _get_grouped_bundles(self, favorites = None):
bundles = self.model_manager.availableBundles
def _get_grouped_bundles(self, bundles, favorites = None):
folders = {}
for bundle in bundles:
folder = next((override.value for override in bundle.overrides if override.key == "folder"), "")
@@ -90,47 +111,70 @@ class ModelsLayoutMici(NavScroller):
def _show_folders(self):
self.focused_widget = self.select_model_btn
hardware_btns = []
active = active_source()
for source, label in (("qcom", tr("small models")), ("chestnut", tr("big models"))):
bundle = get_selected_bundle(ui_state.params, source)
value = (bundle.internalName if bundle else default_model_name(source)).lower()
if source == active:
value += f" ({tr('active')})"
btn = BigButton(label.lower(), value=value)
btn.set_click_callback(lambda s=source: self._select_hardware(s))
hardware_btns.append(btn)
self._push_selection_view(hardware_btns)
def _select_hardware(self, source):
self._selection_source = source
favs = ui_state.params.get("ModelManager_Favs")
favorites = set(favs.split(';')) if favs else set()
folders = self._get_grouped_bundles(favorites)
bundles = bundles_for_source(source)
if not bundles:
gui_app.push_widget(BigDialog(title=tr("No models available"),
description=tr("No models are available for this hardware yet. Connect to the internet and refresh the model list.")))
return
folders = self._get_grouped_bundles(bundles, favorites)
folder_buttons = []
default_btn = BigButton(f"{DEFAULT_MODEL} (Default)".lower())
default_btn.set_click_callback(self._select_default)
default_btn = BigButton(default_model_name(source).lower())
default_btn.set_click_callback(lambda s=source: self._select_default(s))
folder_buttons.append(default_btn)
for folder in sorted(folders.keys(), key=lambda f: max((bundle.index for bundle in folders[f]), default=-1), reverse=True):
if folder.lower() in ["release models", "master models", "favorites"]:
btn = BigButton(folder.lower())
btn.set_click_callback(lambda f=folder: self._select_folder(f))
if folder.lower() == "favorites":
folder_buttons.insert(0, btn)
else:
folder_buttons.append(btn)
btn = BigButton(folder.lower())
btn.set_click_callback(lambda f=folder: self._select_folder(f))
if folder.lower() == "favorites":
folder_buttons.insert(0, btn)
else:
folder_buttons.append(btn)
self._push_selection_view(folder_buttons)
def _pop_to_main(self):
gui_app.pop_widgets_to(self)
self._scroller.scroll_panel.set_offset(0.0)
def _select_model(self, bundle):
ui_state.params.put("ModelManager_DownloadIndex", bundle.index)
ui_state.params.put("ModelManager_DownloadRef", bundle.ref)
self._pop_to_main()
def _select_default(self):
ui_state.params.remove("ModelManager_ActiveBundle")
def _select_default(self, source):
ui_state.params.remove(ACTIVE_BUNDLE_KEYS[source])
self._pop_to_main()
def _select_folder(self, folder_name):
source = self._selection_source
if source is None: # folders are only reachable after picking a hardware
return
favs = ui_state.params.get("ModelManager_Favs")
favorites = set(favs.split(';')) if favs else set()
folders = self._get_grouped_bundles(favorites)
folders = self._get_grouped_bundles(bundles_for_source(source), favorites)
bundles = sorted(folders.get(folder_name, []), key=lambda b: b.index, reverse=True)
btns = []
for bundle in bundles:
txt = bundle.displayName.lower()
btn = BigButton(txt)
btn = BigButton(bundle.displayName.lower())
btn.set_click_callback(lambda b=bundle: self._select_model(b))
btns.append(btn)
self._push_selection_view(btns)
@@ -162,10 +206,10 @@ class ModelsLayoutMici(NavScroller):
self._was_downloading = is_downloading
self.current_model_info.current_model_header.set_text(tr("active model"))
model_text = manager.activeBundle.displayName.lower() if manager.activeBundle.ref else f"{DEFAULT_MODEL} (Default)".lower()
self.current_model_info.current_model_text.set_text(model_text)
self.current_model_info.info_header.set_text(tr("cache size"))
self.current_model_info.info_text.set_text(f"{ModelsLayout.calculate_cache_size():.2f} MB")
active_text, info_header, info_text = _model_info()
self.current_model_info.current_model_text.set_text(active_text)
self.current_model_info.info_header.set_text(info_header)
self.current_model_info.info_text.set_text(info_text)
if manager.selectedBundle and manager.selectedBundle.status == custom.ModelManagerSP.DownloadStatus.failed:
self.current_model_info.info_header.set_text(tr("error") + self._download_progress)
@@ -176,19 +220,29 @@ class ModelsLayoutMici(NavScroller):
device.set_override_interactive_timeout(5)
progress = 0.0
count = 0
verifying = False
for model in manager.selectedBundle.models:
count += 1
p = model.artifact.downloadProgress
if p.status == custom.ModelManagerSP.DownloadStatus.downloading:
if p.status in (custom.ModelManagerSP.DownloadStatus.downloading,
custom.ModelManagerSP.DownloadStatus.verifying):
progress += p.progress
verifying = verifying or p.status == custom.ModelManagerSP.DownloadStatus.verifying
elif p.status in (custom.ModelManagerSP.DownloadStatus.downloaded,
custom.ModelManagerSP.DownloadStatus.cached):
progress += 100.0
self.current_model_info.current_model_header.set_text(tr("downloading"))
self.current_model_info.current_model_header.set_text(tr("verifying") if verifying else tr("downloading"))
self.cancel_download_btn.set_text(tr("cancel verification") if verifying else tr("cancel download"))
self.current_model_info.current_model_header._shimmer = True
self.current_model_info.current_model_text.set_text(f"{manager.selectedBundle.internalName.lower()}")
name_text = manager.selectedBundle.internalName.lower()
if queued := queued_name(manager.selectedBundle.ref):
name_text += f" | {queued.lower()} {tr('queued')}"
self.current_model_info.current_model_text.set_text(name_text)
self.current_model_info.info_header.set_text(tr("progress") + self._download_progress)
self.current_model_info.info_header._shimmer = True
self.current_model_info.info_text.set_text(f"{progress/count:.2f}%")
elif manager.selectedBundle and manager.selectedBundle.status == custom.ModelManagerSP.DownloadStatus.downloaded:
self.current_model_info.info_header.set_text(tr("downloaded"))
self.current_model_info.info_text.set_text(tr("downloaded"))
@@ -12,13 +12,23 @@ from openpilot.selfdrive.ui.mici.widgets.dialog import BigConfirmationDialog, Bi
from openpilot.selfdrive.ui.sunnypilot.mici.layouts.sunnylink import SunnylinkLayoutMici
from openpilot.selfdrive.ui.sunnypilot.mici.layouts.models import ModelsLayoutMici
from openpilot.selfdrive.ui.ui_state import ui_state
from openpilot.system.ui.lib.application import gui_app
from openpilot.system.ui.lib.application import gui_app, FontWeight
from openpilot.system.ui.lib.multilang import tr
ICON_SIZE = 70
BIG_ICON_SIZE = 110
class SunnylinkBigButton(SettingsBigButton):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._label.set_font_weight(FontWeight.AUDIOWIDE)
def _get_label_font_size(self):
# Audiowide runs wider than Inter: "sunnylink" wraps to two lines at 64
return 56
class SettingsLayoutSP(OP.SettingsLayout):
def __init__(self):
OP.SettingsLayout.__init__(self)
@@ -33,7 +43,7 @@ class SettingsLayoutSP(OP.SettingsLayout):
self.icon_offroad_slider = gui_app.texture("icons_mici/settings/device/lkas.png", BIG_ICON_SIZE, BIG_ICON_SIZE)
sunnylink_panel = SunnylinkLayoutMici()
sunnylink_btn = SettingsBigButton(tr("sunnylink"), "", gui_app.texture("icons_mici/settings/developer/ssh.png", 55, 55))
sunnylink_btn = SunnylinkBigButton(tr("sunnylink"), "", gui_app.texture("../../sunnypilot/selfdrive/assets/icons_mici/sunnylink.png", 76, 44))
sunnylink_btn.set_click_callback(lambda: gui_app.push_widget(sunnylink_panel))
models_panel = ModelsLayoutMici()
@@ -56,8 +66,8 @@ class SettingsLayoutSP(OP.SettingsLayout):
items = self._scroller._items.copy()
items.insert(1, sunnylink_btn)
items.insert(2, models_btn)
items.insert(1, models_btn)
items.insert(5, sunnylink_btn)
# front slots (only one ever visible at a time): exit-always-offroad, then enable-onroad
items.insert(0, self._enable_offroad_btn_onroad)
@@ -0,0 +1,85 @@
"""
Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
from openpilot.selfdrive.ui.ui_state import ui_state, ChestnutState
from openpilot.sunnypilot.models.fetcher import get_cached_bundles
from openpilot.sunnypilot.models.helpers import get_active_source, get_selected_bundle, resolve_bundle_by_ref
from openpilot.sunnypilot.models.model_name import DEFAULT_BIG_MODEL, DEFAULT_MODEL
def active_source() -> str:
return get_active_source(chestnut=ui_state.chestnut_present,
chestnut_active=ui_state.chestnut_active, chestnut_loading=ui_state.chestnut_loading,
offroad=ui_state.is_offroad())
def bundles_for_source(source: str):
if source == active_source():
return ui_state.sm["modelManagerSP"].availableBundles
return get_cached_bundles(ui_state.params, source)
def default_model(source: str) -> str:
return DEFAULT_BIG_MODEL if source == 'chestnut' else DEFAULT_MODEL
def default_model_name(source: str) -> str:
return f"{default_model(source)} (Default)"
def big_model_state() -> str | None:
"""'failed' | 'loading' | None, from the same state the icons render."""
return {ChestnutState.UNCOMPILED: 'failed',
ChestnutState.FAILED: 'failed',
ChestnutState.LOADING: 'loading'}.get(ui_state.chestnut_state)
def carrying_model() -> tuple[str | None, str | None, str | None]:
"""(source, internal name, display name) of what actually drives. Runner-matched:
when a Default big cannot carry, stock modeld runs the Default small, never the
small slot's pick; a custom big has no automatic fallback yet -> (None, None, None)."""
source = active_source()
if source == "chestnut":
bundle = get_selected_bundle(ui_state.params, "chestnut")
if bundle:
return "chestnut", bundle.internalName, bundle.displayName
name = default_model_name("chestnut")
return "chestnut", name, name
if ui_state.chestnut_present:
if get_selected_bundle(ui_state.params, "chestnut") is None:
name = default_model_name("qcom")
return "qcom", name, name
return None, None, None
bundle = get_selected_bundle(ui_state.params, "qcom")
if bundle:
return "qcom", bundle.internalName, bundle.displayName
name = default_model_name("qcom")
return "qcom", name, name
def queued_name(current_ref) -> str | None:
ref = ui_state.params.get("ModelManager_DownloadRef")
if ref and ref != current_ref:
source_bundles = {source: bundles_for_source(source) for source in ("qcom", "chestnut")}
if resolved := resolve_bundle_by_ref(ref, source_bundles):
return resolved[0].internalName
return None
def model_info() -> tuple[str, str, str]:
"""returns (active source, active model name, other model name)
Names come from the params slots, never modelManagerSP.activeBundle the
manager republishes a tick after a chestnut change, so the stale bundle
would flash the wrong model."""
source = active_source()
other = "qcom" if source == "chestnut" else "chestnut"
active_bundle = get_selected_bundle(ui_state.params, source)
other_bundle = get_selected_bundle(ui_state.params, other)
active_name = active_bundle.displayName if active_bundle else default_model_name(source)
other_name = other_bundle.displayName if other_bundle else default_model_name(other)
return source, active_name, other_name
@@ -4,11 +4,29 @@ Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
from openpilot.common.filter_simple import FirstOrderFilter
from openpilot.selfdrive.ui.ui_state import ui_state, UIStatus
from openpilot.selfdrive.ui.sunnypilot.onroad.chevron_metrics import ChevronMetrics
from openpilot.selfdrive.ui.sunnypilot.onroad.rainbow_path import RainbowPath
from openpilot.selfdrive.ui.sunnypilot.ui_state import MADSState
from openpilot.system.ui.lib.application import gui_app
class ModelRendererSP:
def __init__(self):
self.rainbow_path = RainbowPath()
self.chevron_metrics = ChevronMetrics()
self._width_filter = FirstOrderFilter(0.9, 0.1, 1 / gui_app.target_fps)
@property
def _lateral_active(self) -> bool:
sm = ui_state.sm
if sm.valid["selfdriveStateSP"]:
mads = sm["selfdriveStateSP"].mads
if mads.available:
return mads.enabled and mads.state != MADSState.paused
return ui_state.status in (UIStatus.ENGAGED, UIStatus.LAT_ONLY)
def _get_path_half_width(self) -> float:
target = 0.9 if self._lateral_active else 0.40
return self._width_filter.update(target)
@@ -10,6 +10,7 @@ from openpilot.cereal import messaging, log, custom
from opendbc.car.structs import car
from openpilot.common.params import Params
from openpilot.selfdrive.ui.sunnypilot.layouts.settings.display import OnroadBrightness
from openpilot.sunnypilot.models.helpers import ACTIVE_BUNDLE_KEYS, get_active_source
from openpilot.sunnypilot.sunnylink.sunnylink_state import SunnylinkState
from openpilot.system.ui.lib.application import gui_app
from openpilot.system.ui.sunnypilot.widgets.screen_saver import ScreenSaverSP
@@ -43,6 +44,7 @@ class UIStateSP:
self.screensaver_enabled: bool = False
self.active_bundle = None
self.model_runner_tinygrad: bool = False
self.blindspot: bool = False
self.chevron_metrics = None
self.custom_interactive_timeout: int = 0
@@ -150,7 +152,13 @@ class UIStateSP:
self.has_icbm = self.CP_SP.intelligentCruiseButtonManagementAvailable and self.params.get_bool("IntelligentCruiseButtonManagement")
self._enforce_constraints()
self.active_bundle = self.params.get("ModelManager_ActiveBundle")
source = get_active_source(chestnut=self.chestnut_present, chestnut_active=self.chestnut_active,
chestnut_loading=self.chestnut_loading, offroad=self.is_offroad())
self.active_bundle = self.params.get(ACTIVE_BUNDLE_KEYS[source])
self.model_runner_tinygrad = self.active_bundle is not None and self.active_bundle.get("runner") == "tinygrad"
# stock only counts the default big model's compiled pkl. a downloaded big bundle runs on the
# chestnut just the same, so ChestnutState has to see it as available too.
self.chestnut_compiled = self.chestnut_compiled or self.model_runner_tinygrad
self.blindspot = self.params.get_bool("BlindSpot")
self.chevron_metrics = self.params.get("ChevronInfo")
self.custom_interactive_timeout = self.params.get("InteractivityTimeout", return_default=True)
+65 -11
View File
@@ -12,7 +12,8 @@ from openpilot.common.swaglog import cloudlog
from openpilot.selfdrive.ui.lib.prime_state import PrimeState
from openpilot.system.ui.lib.application import gui_app
from openpilot.common.hardware import HARDWARE, PC
from openpilot.selfdrive.modeld.helpers import usbgpu_compiled
from openpilot.common.hardware.usb import TYPEC_CC_ORIENTATION_PATH, get_usb_state, is_chestnut_usb_id, read_int
from openpilot.selfdrive.modeld.helpers import chestnut_compiled
from openpilot.selfdrive.ui.sunnypilot.ui_state import UIStateSP, DeviceSP
@@ -28,6 +29,15 @@ class UIStatus(Enum):
LONG_ONLY = "long_only"
class ChestnutState(Enum):
DISCONNECTED = "disconnected"
UNCOMPILED = "uncompiled"
READY = "ready"
LOADING = "loading"
ACTIVE = "active"
FAILED = "failed"
class UIState(UIStateSP):
_instance: 'UIState | None' = None
@@ -82,10 +92,15 @@ class UIState(UIStateSP):
self.always_on_dm: bool = self.params.get_bool("AlwaysOnDM")
self.experimental_mode: bool = self.params.get_bool("ExperimentalMode")
self.experimental_mode_confirmed: bool = self.params.get_bool("ExperimentalModeConfirmed")
self.usbgpu: bool = False
self.usbgpu_compiled: bool = usbgpu_compiled()
self.usbgpu_active: bool | None = self.params.get("UsbGpuActive")
self.usbgpu_loading: bool = self.params.get_bool("UsbGpuLoading")
self.chestnut_present: bool = False
self.chestnut_compiled: bool = chestnut_compiled()
self.chestnut_active: bool | None = None
self.chestnut_loading: bool = False
self.usb_connected: bool = False
self.usb_connected_ts: float | None = None
self.usb_disconnected_ts: float | None = None
self.usb_unknown: bool = False
self.chestnut_state = ChestnutState.DISCONNECTED
self.started: bool = False
self.ignition: bool = False
self.recording_audio: bool = False
@@ -131,6 +146,7 @@ class UIState(UIStateSP):
self.sm.update(0)
self._update_state()
self._update_status()
self._update_chestnut_state()
device.update()
UIStateSP.update(self)
@@ -194,12 +210,35 @@ class UIState(UIStateSP):
self.status = UIStatus.DISENGAGED
self.started_frame = self.sm.frame
self.started_time = time.monotonic()
self.chestnut_present = self.sm["deviceState"].chestnutPresent
for callback in self._offroad_transition_callbacks:
callback()
self._started_prev = self.started
def _update_chestnut_state(self) -> None:
detected = self.sm["deviceState"].chestnutPresent
if not self.started:
self.chestnut_present = detected
self.chestnut_state = (ChestnutState.READY if detected and self.chestnut_compiled else
ChestnutState.UNCOMPILED if detected else ChestnutState.DISCONNECTED)
return
model_seen = self.sm.recv_frame["modelV2"] > self.started_frame
if not self.chestnut_present:
self.chestnut_state = ChestnutState.DISCONNECTED
elif not self.chestnut_compiled:
self.chestnut_state = ChestnutState.UNCOMPILED
elif self.chestnut_state == ChestnutState.FAILED or not detected or (model_seen and (not self.sm.alive["modelV2"] or not self.sm["modelV2"].big)):
self.chestnut_state = ChestnutState.FAILED
elif self.chestnut_loading or not model_seen:
self.chestnut_state = ChestnutState.LOADING
elif self.chestnut_active is False:
self.chestnut_state = ChestnutState.FAILED
else:
self.chestnut_state = ChestnutState.ACTIVE
def update_params(self) -> None:
# For slower operations
# Update longitudinal control state
@@ -216,12 +255,27 @@ class UIState(UIStateSP):
self.always_on_dm = self.params.get_bool("AlwaysOnDM")
self.experimental_mode = self.params.get_bool("ExperimentalMode")
self.experimental_mode_confirmed = self.params.get_bool("ExperimentalModeConfirmed")
# keep usbgpu UI active until offroad transition when gpu disappears
self.usbgpu = self.sm["deviceState"].chestnutPresent or (self.usbgpu and self.started)
if not self.usbgpu_compiled:
self.usbgpu_compiled = usbgpu_compiled()
self.usbgpu_active = self.params.get("UsbGpuActive")
self.usbgpu_loading = self.params.get_bool("UsbGpuLoading")
if not self.chestnut_compiled:
self.chestnut_compiled = chestnut_compiled()
self.chestnut_active = self.params.get("ChestnutActive")
self.chestnut_loading = self.params.get_bool("ChestnutLoading")
now = time.monotonic()
if read_int(TYPEC_CC_ORIENTATION_PATH) != 0:
self.usb_disconnected_ts = None
if not self.usb_connected:
self.usb_connected = True
self.usb_connected_ts = now
self.usb_unknown = False
elif self.usb_connected_ts is not None and now - self.usb_connected_ts > 10.:
self.usb_unknown = not any(is_chestnut_usb_id(d["vendorId"], d["productId"], True) for d in get_usb_state())
self.usb_connected_ts = None
elif self.usb_connected:
if self.usb_disconnected_ts is None:
self.usb_disconnected_ts = now
elif now - self.usb_disconnected_ts > PARAM_UPDATE_TIME:
self.usb_connected = False
self.usb_connected_ts = None
self.usb_unknown = False
UIStateSP.update_params(self)
+17
View File
@@ -55,6 +55,19 @@ def cleanup_old_osm_data(files_to_remove: list[str]) -> None:
shutil.rmtree(file, ignore_errors=False)
def clear_downloaded_maps() -> None:
"""Deletes downloaded OSM map data and resets params."""
path = f"{Paths.mapd_root()}/offline"
if os.path.exists(path):
shutil.rmtree(path, ignore_errors=True)
for param in ("OsmDownloadedDate", "OsmLocal", "OsmLocationName", "OsmLocationTitle",
"OsmStateName", "OsmStateTitle"):
params.remove(param)
cloudlog.info("mapd: downloaded maps cleared")
def request_refresh_osm_location_data(nations: list[str], states: list[str] | None = None) -> None:
params.put("OsmDownloadedDate", str(datetime.now().timestamp()), block=True)
params.put_bool("OsmDbUpdatesCheck", False, block=True)
@@ -131,6 +144,10 @@ def main_thread():
show_alert = bool(get_files_for_cleanup() and params.get_bool("OsmLocal"))
set_offroad_alert("Offroad_OSMUpdateRequired", show_alert, "This alert will be cleared when new maps are downloaded.")
if params.get("Mapd_ClearCache"):
clear_downloaded_maps()
params.remove("Mapd_ClearCache")
update_osm_db()
live_map_sp.tick()
rk.keep_time()
@@ -7,6 +7,7 @@ See the LICENSE.md file in the root directory for more details.
"""
import argparse
import math
import os
import tempfile
import time
@@ -66,14 +67,15 @@ def get_policy_npy_shapes(input_shapes: dict, is_supercombo: bool = False) -> tu
if desire_key:
shapes['desire'] = (input_shapes[desire_key][2],)
if is_supercombo and 'features_buffer' in input_shapes:
fb = input_shapes['features_buffer']
shapes['prev_feat'] = (fb[0], fb[2])
for key, shape in input_shapes.items():
if key not in (desire_key, 'features_buffer') and 'img' not in key:
shapes[key] = tuple(shape)
if is_supercombo and 'features_buffer' in input_shapes:
fb = input_shapes['features_buffer']
feat_dim = math.prod(fb[2:])
shapes['prev_feat'] = (fb[0], feat_dim)
sizes = [int(np.prod(size)) for size in shapes.values()]
return shapes, sizes
@@ -117,8 +119,9 @@ def generate_queues_and_npy(input_shapes: dict, frame_skip: int, device: str = D
}
if features_buffer:
feat_dim = math.prod(features_buffer[2:])
feat_q_len = frame_skip * features_buffer[1] if is_supercombo else frame_skip * (features_buffer[1] - 1) + 1
queues['feat_q'] = Tensor(np.zeros((feat_q_len, features_buffer[0], features_buffer[2]),
queues['feat_q'] = Tensor(np.zeros((feat_q_len, features_buffer[0], feat_dim),
dtype=np.float32), device=device).contiguous().realize()
queues.update({key: Tensor(value, device='NPY').realize() for key, value in npy_arrays.items() if key in ('tfm', 'big_tfm')})
@@ -183,14 +186,14 @@ def make_run_policy(vision_runner, policy_runners: list, features_slice: slice,
warped_dev = warped.to(Device.DEFAULT)
Tensor.realize(packed_npy_inputs_dev, warped_dev)
img = shift_and_sample(img_q, warped_dev[0:1], sample_skip_fn).realize()
big_img = shift_and_sample(big_img_q, warped_dev[1:2], sample_skip_fn).realize()
img = shift_and_sample(img_q, warped_dev[0:1], sample_skip_fn)
big_img = shift_and_sample(big_img_q, warped_dev[1:2], sample_skip_fn)
unpacked_tensors = [tensor.reshape(shape) for tensor, shape in zip(packed_npy_inputs_dev.split(npy_sizes), npy_shapes.values(), strict=True)]
unpacked_dict = dict(zip(npy_shapes.keys(), unpacked_tensors, strict=True))
desire_dev = unpacked_dict['desire']
desire_buf = shift_and_sample(desire_q, desire_dev.reshape(1, 1, -1), sample_desire_fn).realize()
desire_buf = shift_and_sample(desire_q, desire_dev.reshape(1, 1, -1), sample_desire_fn)
inputs = {desire_key: desire_buf}
for key, tensor_val in unpacked_dict.items():
@@ -199,7 +202,7 @@ def make_run_policy(vision_runner, policy_runners: list, features_slice: slice,
if 'prev_feat' in unpacked_dict:
prev_feat_dev = unpacked_dict['prev_feat']
inputs['features_buffer'] = shift_and_sample(feat_q, prev_feat_dev.reshape(1, 1, -1), sample_skip_fn).realize()
inputs['features_buffer'] = shift_and_sample(feat_q, prev_feat_dev.reshape(1, 1, -1), sample_skip_fn).reshape(input_shapes['features_buffer'])
if vision_runner:
vision_out_cast = next(iter(vision_runner({road_key: img, wide_key: big_img}).values())).cast('float32').realize()
@@ -211,7 +214,7 @@ def make_run_policy(vision_runner, policy_runners: list, features_slice: slice,
inputs.update({road_key: img, wide_key: big_img})
if 'features_buffer' not in inputs:
inputs['features_buffer'] = sample_skip_fn(feat_q)
inputs['features_buffer'] = sample_skip_fn(feat_q).reshape(input_shapes['features_buffer'])
policy_out = next(iter(policy_runners[0](inputs).values())).cast('float32').realize()
if 'features_buffer' not in inputs and features_slice is not None:
@@ -272,18 +275,17 @@ def _parse_size(size_str: str) -> tuple[int, int]:
return int(width), int(height)
def read_file_chunked_to_shm(path):
def read_file_chunked_to_disk(path):
if not path:
return None
import atexit
import shutil
from openpilot.common.file_chunker import open_file_chunked
from openpilot.common.hardware.hw import Paths
shm_path = os.path.join(Paths.shm_path(), os.path.basename(path))
atexit.register(lambda: os.path.exists(shm_path) and os.remove(shm_path))
with open(shm_path, 'wb') as dst, open_file_chunked(path) as src:
shutil.copyfileobj(src, dst)
return shm_path
tmp_path = f'{path}.unchunked'
with open(tmp_path, 'wb') as f, open_file_chunked(path) as src:
shutil.copyfileobj(src, f)
atexit.register(lambda: os.path.exists(tmp_path) and os.remove(tmp_path))
return tmp_path
def _load_policy_runners(args: argparse.Namespace) -> tuple[list, list]:
@@ -296,7 +298,7 @@ def _load_policy_runners(args: argparse.Namespace) -> tuple[list, list]:
if __name__ == "__main__":
if 'USB' in os.getenv('DEV', '') or os.getenv('USBGPU'):
if 'USB' in os.getenv('DEV', '') or os.getenv('CHESTNUT'):
from openpilot.system.hardware.chestnut.flash import link_up
for _ in range(10):
if link_up():
@@ -327,11 +329,11 @@ if __name__ == "__main__":
model_w, model_h = args.model_size
output_data = {}
args.vision_onnx = read_file_chunked_to_shm(args.vision_onnx)
args.policy_onnx = read_file_chunked_to_shm(args.policy_onnx)
args.off_policy_onnx = read_file_chunked_to_shm(args.off_policy_onnx)
args.on_policy_onnx = read_file_chunked_to_shm(args.on_policy_onnx)
args.supercombo_onnx = read_file_chunked_to_shm(args.supercombo_onnx)
args.vision_onnx = read_file_chunked_to_disk(args.vision_onnx)
args.policy_onnx = read_file_chunked_to_disk(args.policy_onnx)
args.off_policy_onnx = read_file_chunked_to_disk(args.off_policy_onnx)
args.on_policy_onnx = read_file_chunked_to_disk(args.on_policy_onnx)
args.supercombo_onnx = read_file_chunked_to_disk(args.supercombo_onnx)
vision_runner = OnnxRunner(args.vision_onnx) if args.vision_onnx else None
+56 -42
View File
@@ -8,22 +8,22 @@ See the LICENSE.md file in the root directory for more details.
import os
os.environ['GMMU'] = '0'
from openpilot.common.hardware import COMMA_HARDWARE
from openpilot.selfdrive.modeld.helpers import usbgpu_present, load_oob
import time
import numpy as np
import threading
import time
from setproctitle import setproctitle
from tinygrad.tensor import Tensor
import openpilot.cereal.messaging as messaging
from openpilot.common.hardware import COMMA_HARDWARE
from openpilot.selfdrive.modeld.helpers import chestnut_present, load_oob
from openpilot.cereal import log
from opendbc.car.structs import car
from openpilot.cereal.services import SERVICE_LIST
from setproctitle import setproctitle
from openpilot.cereal.messaging import PubMaster, SubMaster
from openpilot.cereal.visionipc import VisionStreamType
from msgq.visionipc import VisionIpcClient, VisionBuf
from opendbc.car.car_helpers import get_demo_car_params
from tinygrad.tensor import Tensor
from openpilot.common.file_chunker import open_file_chunked
from openpilot.common.swaglog import cloudlog
from openpilot.common.params import Params
@@ -42,13 +42,13 @@ from openpilot.sunnypilot.modeld_v2.constants import Plan
from openpilot.sunnypilot.modeld_v2.meta_helper import load_meta_constants
from openpilot.sunnypilot.modeld_v2.camera_offset_helper import CameraOffsetHelper
from openpilot.sunnypilot.modeld_v2.compile_modeld import derive_frame_skip, make_split_input_queues, make_supercombo_input_queues, WARP_INPUTS, POLICY_INPUTS
from openpilot.sunnypilot.livedelay.helpers import get_lat_delay
from openpilot.sunnypilot.modeld_v2.modeld_base import ModelStateBase
from openpilot.sunnypilot.models.helpers import get_active_bundle
from openpilot.sunnypilot.selfdrive.controls.lib.relc import RoadEdgeLaneChangeController
PROCESS_NAME = "openpilot.selfdrive.modeld.modeld_tinygrad"
BIG_MODEL_TIMEOUT = 60
def _pkl_exists(path):
@@ -68,6 +68,7 @@ def _find_driving_pkl(bundle):
pkl_path = os.path.join(model_root, pkl_name)
if _pkl_exists(pkl_path):
return pkl_path
return None
class FrameMeta:
@@ -84,14 +85,14 @@ class ModelState(ModelStateBase):
inputs: dict[str, np.ndarray]
prev_desire: np.ndarray
def __init__(self, cam_w: int, cam_h: int, usbgpu: bool = False):
def __init__(self, cam_w: int, cam_h: int, chestnut: bool = False):
ModelStateBase.__init__(self)
env_pkl = os.environ.get('COMBINED_MODEL_PKL')
if env_pkl and os.path.exists(env_pkl):
model_bundle = None
else:
model_bundle = get_active_bundle()
model_bundle = get_active_bundle(chestnut=chestnut)
self.generation = model_bundle.generation if model_bundle is not None else None
overrides = {override.key: override.value for override in model_bundle.overrides} if model_bundle else {}
@@ -99,10 +100,10 @@ class ModelState(ModelStateBase):
self.LONG_SMOOTH_SECONDS = float(overrides.get('long', ".0"))
self.MIN_LAT_CONTROL_SPEED = 0.3
self.PLANPLUS_CONTROL: float = 1.0
self.usbgpu = usbgpu
self.chestnut = chestnut
pkl_path = _find_driving_pkl(model_bundle)
assert pkl_path is not None, "No driving pkl found — all models must be compiled with compile_modeld.py"
assert pkl_path is not None, f"No driving pkl found for {'chestnut' if chestnut else 'small model'} — all models must be compiled with compile_modeld.py"
self._init_combined(pkl_path, cam_w, cam_h, model_bundle)
def _init_combined(self, pkl_path, cam_w, cam_h, bundle):
@@ -110,7 +111,7 @@ class ModelState(ModelStateBase):
jits = load_oob(open_file_chunked(pkl_path))
self.WARP_DEV = 'QCOM' if COMMA_HARDWARE else 'CPU'
self.DEV = 'AMD' if self.usbgpu else self.WARP_DEV
self.DEV = 'AMD' if self.chestnut else self.WARP_DEV
self.QUEUE_DEV = self.DEV
metadata = jits['metadata']
@@ -185,9 +186,6 @@ class ModelState(ModelStateBase):
else:
self.warp(**{k: self.input_queues[k] for k in WARP_INPUTS}, frame=frame_tensor, big_frame=big_frame_tensor)
if self.usbgpu:
self.warmup()
def warmup(self) -> None:
dummy_frames = {k: np.zeros(self.frame_buf_params[k][3], dtype=np.uint8) for k in self._vision_input_names}
transforms = {k: np.eye(3, dtype=np.float32) for k in [self._road_key, self._wide_key] if k}
@@ -287,9 +285,8 @@ class ModelState(ModelStateBase):
buf[0, :-1] = buf[0, 1:]
buf[0, -1, :] = outputs['desired_curvature'][0, :] if not self.mlsim else 0
if self.usbgpu and not np.all(np.isfinite(outputs.get('plan', np.array([0.])))):
cloudlog.error("model output not finite, dropping frame")
return None
if self.chestnut and not np.all(np.isfinite(outputs.get('plan', np.array([0.])))):
raise RuntimeError("model output not finite")
return outputs
@@ -327,13 +324,13 @@ def main(demo=False):
setproctitle(PROCESS_NAME)
config_realtime_process(7, 54)
USBGPU = usbgpu_present()
if USBGPU:
CHESTNUT = chestnut_present()
if CHESTNUT:
os.environ['HCQDEV_WAIT_TIMEOUT_MS'] = '3000'
params = Params()
params.put_bool("UsbGpuLoading", USBGPU)
params.remove("UsbGpuActive")
params.put_bool("ChestnutLoading", CHESTNUT)
params.remove("ChestnutActive")
# visionipc clients
while True:
@@ -362,31 +359,36 @@ def main(demo=False):
st = time.monotonic()
model = None
if USBGPU:
import threading
def load():
nonlocal model
model = ModelState(cam_w=vipc_client_main.width, cam_h=vipc_client_main.height, usbgpu=True)
t = threading.Thread(target=load, daemon=True)
t.start()
t.join(60)
if model is None:
params.put_bool("UsbGpuActive", False)
raise RuntimeError("eGPU model load failed or timed out (60s)")
params.put_bool("UsbGpuActive", True)
else:
model = ModelState(cam_w=vipc_client_main.width, cam_h=vipc_client_main.height, usbgpu=False)
if CHESTNUT:
big_model = None
def load_big():
nonlocal big_model
try:
m = ModelState(cam_w=vipc_client_main.width, cam_h=vipc_client_main.height, chestnut=True)
m.warmup()
big_model = m
except Exception:
cloudlog.exception("chestnut load failed")
loader = threading.Thread(target=load_big, daemon=True)
loader.start()
loader.join(BIG_MODEL_TIMEOUT)
model = big_model
params.put_bool("ChestnutActive", model is not None)
params.put_bool("UsbGpuLoading", False)
small_model = ModelState(cam_w=vipc_client_main.width, cam_h=vipc_client_main.height, chestnut=False) if model is None or CHESTNUT else None
if model is None:
model = small_model
params.put_bool("ChestnutLoading", False)
assert model is not None
cloudlog.warning(f"models loaded in {time.monotonic() - st:.1f}s, modeld starting")
# messaging
pub_socks = ["modelV2", "drivingModelData", "cameraOdometry", "modelDataV2SP"] + (["chestnutState"] if USBGPU else [])
pub_socks = ["modelV2", "drivingModelData", "cameraOdometry", "modelDataV2SP"] + (["chestnutState"] if CHESTNUT else [])
pm = PubMaster(pub_socks)
sm = SubMaster(["deviceState", "carState", "narrowRoadCameraState", "extrinsicsCalibration", "driverMonitoringState", "carControl", "lateralDelay"])
publish_state = PublishState()
chestnut_state = ChestnutState(pm, USBGPU) if USBGPU else None
chestnut_state = ChestnutState(pm, model.chestnut) if CHESTNUT else None
# setup filter to track dropped frames
frame_dropped_filter = FirstOrderFilter(0., 10., 1. / model.constants.MODEL_FREQ)
@@ -509,7 +511,19 @@ def main(demo=False):
inputs['action_t'] = np.array([lat_action_t, long_action_t], dtype=np.float32)
mt1 = time.perf_counter()
model_output = model.run(bufs, transforms, inputs, prepare_only)
try:
model_output = model.run(bufs, transforms, inputs, prepare_only)
except Exception:
if not params.get_bool("ChestnutActive"):
raise
cloudlog.exception("chestnut failed, falling back to small")
params.put_bool("ChestnutActive", False)
assert small_model is not None
model = small_model
if chestnut_state is not None:
chestnut_state.big = False
run_count = 0
model_output = None
mt2 = time.perf_counter()
model_execution_time = mt2 - mt1
@@ -524,7 +538,7 @@ def main(demo=False):
fill_model_msg(drivingdata_send, modelv2_send, model_output, action,
publish_state, meta_main.frame_id, meta_extra.frame_id, frame_id,
frame_drop_ratio, meta_main.timestamp_eof, model_execution_time, live_calib_seen, meta_constants)
modelv2_send.modelV2.big = model.usbgpu
modelv2_send.modelV2.big = model.chestnut
desire_state = modelv2_send.modelV2.meta.desireState
l_lane_change_prob = desire_state[log.Desire.laneChangeLeft]
@@ -190,8 +190,8 @@ def tmp_path():
def patch_modeld(monkeypatch):
def _patch(bundle):
monkeypatch.setattr(helpers, 'get_active_bundle', lambda params=None: bundle)
monkeypatch.setattr(modeld_module, 'get_active_bundle', lambda params=None: bundle)
monkeypatch.setattr(helpers, 'get_active_bundle', lambda params=None, *, chestnut=None: bundle)
monkeypatch.setattr(modeld_module, 'get_active_bundle', lambda params=None, *, chestnut=None: bundle)
return _patch
@@ -59,8 +59,8 @@ class TestFindDrivingPkl(OpenpilotTestCase):
class TestModelStateCombinedInit(OpenpilotTestCase):
def test_asserts_when_no_pkl(self, monkeypatch):
bundle = DummyBundle(models=[], is_20hz=True)
monkeypatch.setattr(helpers, 'get_active_bundle', lambda params=None: bundle)
monkeypatch.setattr(modeld_module, 'get_active_bundle', lambda params=None: bundle)
monkeypatch.setattr(helpers, 'get_active_bundle', lambda params=None, *, chestnut=None: bundle)
monkeypatch.setattr(modeld_module, 'get_active_bundle', lambda params=None, *, chestnut=None: bundle)
with self.assertRaisesRegex(AssertionError, "No driving pkl found"):
ModelState(cam_w=CAM_W, cam_h=CAM_H)
@@ -75,11 +75,11 @@ class TestStockEquivalence(OpenpilotTestCase):
frame_skip = derive_frame_skip(SPLIT_VISION_INPUT_SHAPES, SPLIT_POLICY_INPUT_SHAPES)
stock_shapes = {**SPLIT_VISION_INPUT_SHAPES, **SPLIT_POLICY_INPUT_SHAPES, 'action_t': (1, 2)}
stock_queues, stock_npy = make_input_queues(stock_shapes, frame_skip, device='NPY')
stock_queues, stock_npy, _frame_views = make_input_queues(stock_shapes, frame_skip, device='NPY', frame_copy_size=49152)
assert set(state.input_queues.keys()) == set(stock_queues.keys())
# sunnypilot split pipeline has tfm/big_tfm as queues (stock has them in npy only)
assert set(stock_queues.keys()) <= set(state.input_queues.keys())
assert {'desire', 'traffic_convention'} <= set(state.numpy_inputs.keys())
assert set(state.numpy_inputs.keys()) == set(stock_npy.keys()) - {'action_t', 'prev_feat'}
def test_split_queue_keys_work_with_desire_key(self, model_state_factory):
from openpilot.sunnypilot.modeld_v2.compile_modeld import derive_frame_skip, make_split_input_queues
@@ -5,10 +5,16 @@ This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
import os
import tempfile
import unittest
from pathlib import Path
import numpy as np
from openpilot.common.parameterized import parameterized
from openpilot.sunnypilot.modeld_v2.compile_modeld import derive_frame_skip, _detect_desire_key
from openpilot.common.file_chunker import chunk_file, get_chunk_targets
from openpilot.sunnypilot.modeld_v2.compile_modeld import derive_frame_skip, _detect_desire_key, read_file_chunked_to_disk
from openpilot.common.test import OpenpilotTestCase
@@ -160,3 +166,119 @@ class TestOutputSlicePreservation(OpenpilotTestCase):
policy_slices = {'plan': slice(0, 495), 'meta': slice(495, 550)}
assert set(vision_slices.keys()) & set(policy_slices.keys()) == set(), \
"vision and policy slices should not overlap in keys"
class TestReadFileChunkedToDisk(OpenpilotTestCase):
def test_none_passthrough(self):
assert read_file_chunked_to_disk(None) is None
def test_unchunked_source_staged_on_disk(self):
with tempfile.TemporaryDirectory() as d:
src = Path(d) / "driving_supercombo.onnx"
payload = os.urandom(1024)
src.write_bytes(payload)
out = Path(read_file_chunked_to_disk(str(src)))
assert out.parent == Path(d)
assert out.name == "driving_supercombo.onnx.unchunked"
assert out.read_bytes() == payload
def test_chunked_source_reassembled_on_disk(self):
with tempfile.TemporaryDirectory() as d:
src = Path(d) / "driving_supercombo.onnx"
payload = os.urandom(4096)
src.write_bytes(payload)
chunk_file(str(src), get_chunk_targets(str(src), len(payload)))
assert not src.exists()
out = Path(read_file_chunked_to_disk(str(src)))
assert out.parent == Path(d)
assert out.read_bytes() == payload
class Test4DFeaturesBuffer(OpenpilotTestCase):
def test_get_policy_npy_shapes_4d(self):
from openpilot.sunnypilot.modeld_v2.compile_modeld import get_policy_npy_shapes
input_shapes = {
'desire_pulse': (1, 25, 8),
'features_buffer': (1, 24, 32, 512), # compare 4d to 3d for regression
'traffic_convention': (1, 2),
'action_t': (1, 2)
}
shapes, sizes = get_policy_npy_shapes(input_shapes, is_supercombo=True)
assert shapes['prev_feat'] == (1, 16384)
assert sizes == [8, 2, 2, 16384]
def test_get_policy_npy_shapes_3d(self):
from openpilot.sunnypilot.modeld_v2.compile_modeld import get_policy_npy_shapes
input_shapes = {
'desire_pulse': (1, 25, 8),
'features_buffer': (1, 24, 512),
'traffic_convention': (1, 2),
'action_t': (1, 2)
}
shapes, sizes = get_policy_npy_shapes(input_shapes, is_supercombo=True)
assert shapes['prev_feat'] == (1, 512)
assert sizes == [8, 2, 2, 512]
class TestStockCompileModeldEquivalence(OpenpilotTestCase):
def test_get_policy_npy_shapes_matches_stock(self):
from openpilot.selfdrive.modeld.compile_modeld import get_policy_npy_shapes as stock_get_policy_npy_shapes
from openpilot.sunnypilot.modeld_v2.compile_modeld import get_policy_npy_shapes as sunny_get_policy_npy_shapes
stock_input_shapes = {
'desire_pulse': (1, 25, 8),
'features_buffer': (1, 24, 512), # see below comment
'traffic_convention': (1, 2),
'action_t': (1, 2),
}
stock_shapes, stock_sizes = stock_get_policy_npy_shapes(stock_input_shapes)
sunny_shapes, sunny_sizes = sunny_get_policy_npy_shapes(stock_input_shapes, is_supercombo=True)
assert sunny_shapes == stock_shapes
assert sunny_sizes == stock_sizes
assert sunny_shapes['prev_feat'] == (1, 512)
def test_make_input_queues_full_stock_equivalence(self):
from openpilot.selfdrive.modeld.compile_modeld import make_input_queues as stock_make_input_queues
from openpilot.sunnypilot.modeld_v2.compile_modeld import make_supercombo_input_queues as sunny_make_supercombo_input_queues
input_shapes = {
'img': (1, 12, 128, 256),
'desire_pulse': (1, 25, 8),
'features_buffer': (1, 24, 32, 512),
'traffic_convention': (1, 2),
'action_t': (1, 2),
}
frame_skip = 4
stock_queues, stock_npy, _frame_views = stock_make_input_queues(input_shapes, frame_skip, device='NPY', frame_copy_size=49152)
sunny_queues, sunny_npy = sunny_make_supercombo_input_queues(input_shapes, frame_skip, device='NPY')
# sunnypilot split pipeline has tfm/big_tfm as queues; packed_npy_inputs size differs (different frame packing)
assert set(stock_queues.keys()) <= set(sunny_queues.keys())
for key in stock_queues:
if key == 'packed_npy_inputs':
continue
assert sunny_queues[key].shape == stock_queues[key].shape, \
f"Queue shape mismatch for {key}: sunny {sunny_queues[key].shape} != stock {stock_queues[key].shape}"
assert set(stock_npy.keys()) <= set(sunny_npy.keys())
for key in stock_npy:
assert sunny_npy[key].shape == stock_npy[key].shape, \
f"Numpy array shape mismatch for {key}: sunny {sunny_npy[key].shape} != stock {stock_npy[key].shape}"
@unittest.skip("upstream removed make_warp_input_queues — warp merged into run_model")
def test_make_warp_queues_stock_equivalence(self):
from openpilot.selfdrive.modeld.compile_modeld import make_warp_input_queues as stock_make_warp_queues
from openpilot.sunnypilot.modeld_v2.compile_modeld import make_warp_queues as sunny_make_warp_queues
stock_vision_shapes = {'img': (1, 12, 128, 256)} # for now?
stock_queues, stock_npy = stock_make_warp_queues(stock_vision_shapes, frame_skip=4, device='NPY')
sunny_queues, sunny_npy = sunny_make_warp_queues(device='NPY')
assert set(sunny_npy.keys()) == set(stock_npy.keys()) == {'tfm', 'big_tfm'}
for key in sunny_npy:
assert sunny_npy[key].shape == stock_npy[key].shape == (3, 3)
@@ -0,0 +1,62 @@
"""
Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
import io
import requests
from openpilot.common.file_chunker import get_chunk_name
from openpilot.common.hardware import hw
from openpilot.common.test import OpenpilotTestCase
from openpilot.selfdrive.modeld.helpers import dump_oob
import openpilot.sunnypilot.modeld_v2.modeld as modeld_module
from openpilot.sunnypilot.modeld_v2.tests import helpers as tests_helpers
from openpilot.sunnypilot.modeld_v2.tests.helpers import DummyModel, DummyBundle, CAM_W, CAM_H
from openpilot.sunnypilot.models.fetcher import ModelParser, ModelFetcher
tmp_path = tests_helpers.tmp_path
class TestFallback(OpenpilotTestCase):
def test_find_dual_model_in_bundle(self, tmp_path, monkeypatch):
lebowski_file = 'driving_lebowski.pkl'
tsfdo_file = 'driving_tsfdo.pkl'
(tmp_path / lebowski_file).write_bytes(b'fkasdjfkljf')
(tmp_path / tsfdo_file).write_bytes(b'dskfajklsdjlsfka')
monkeypatch.setattr(hw.Paths, 'model_root', staticmethod(lambda: str(tmp_path)))
big_bundle = DummyBundle(models=[DummyModel('supercombo', lebowski_file)])
small_bundle = DummyBundle(models=[DummyModel('supercombo', tsfdo_file)])
big_pkl = modeld_module._find_driving_pkl(big_bundle)
small_pkl = modeld_module._find_driving_pkl(small_bundle)
assert big_pkl is not None and lebowski_file in big_pkl
assert small_pkl is not None and tsfdo_file in small_pkl
def test_download_models_and_init_modelstate_fallback(self, tmp_path, monkeypatch):
monkeypatch.setattr(hw.Paths, 'model_root', staticmethod(lambda: str(tmp_path)))
big_json = requests.get(ModelFetcher.MODEL_URL_CHESTNUT).json()
big_bundle = ModelParser.parse_models(big_json)[-1]
small_json = requests.get(ModelFetcher.MODEL_URL).json()
small_bundle = ModelParser.parse_models(small_json)[-1]
buf = io.BytesIO()
dump_oob(tests_helpers.make_pkl_data(tests_helpers.ARCHETYPES['supercombo_non20hz']), buf)
oob_bytes = buf.getvalue()
for bundle in (big_bundle, small_bundle):
artifact = bundle.models[0].artifact
for i in range(len(artifact.chunks)):
(tmp_path / get_chunk_name(artifact.fileName, i, len(artifact.chunks))).write_bytes(oob_bytes if i == 0 else b"")
monkeypatch.setattr(modeld_module, 'get_active_bundle', lambda params=None, *, chestnut=None: small_bundle)
assert modeld_module.ModelState(CAM_W, CAM_H, chestnut=False).chestnut is False
monkeypatch.setattr(modeld_module, 'get_active_bundle', lambda params=None, *, chestnut=None: big_bundle)
try:
assert modeld_module.ModelState(CAM_W, CAM_H, chestnut=True).chestnut is True
except Exception as e:
assert "AMD" in str(e) or "device" in str(e).lower()
+70 -28
View File
@@ -1,61 +1,103 @@
import argparse
import os
import hashlib
import requests
import re
from openpilot.common.basedir import BASEDIR
from openpilot.selfdrive.ui.ui_state import ui_state
from openpilot.sunnypilot import get_file_hash
from openpilot.sunnypilot.models.model_name import DEFAULT_MODEL
from openpilot.sunnypilot.models.model_name import DEFAULT_MODEL, DEFAULT_BIG_MODEL
from openpilot.sunnypilot.models.fetcher import ModelFetcher
def get_default_model() -> str:
show_big_model = (ui_state.chestnut_present
and (ui_state.chestnut_active or ui_state.chestnut_loading or ui_state.is_offroad()))
return DEFAULT_BIG_MODEL if show_big_model else DEFAULT_MODEL
DEFAULT_MODEL_NAME_PATH = os.path.join(BASEDIR, "openpilot", "sunnypilot", "models", "model_name.py")
MODEL_HASH_PATH = os.path.join(BASEDIR, "openpilot", "sunnypilot", "models", "tests", "model_hash")
BIG_MODEL_HASH_PATH = os.path.join(BASEDIR, "openpilot", "sunnypilot", "models", "tests", "big_model_hash")
SUPERCOMBO_ONNX_PATH = os.path.join(BASEDIR, "openpilot", "selfdrive", "modeld", "models", "driving_supercombo.onnx")
BIG_SUPERCOMBO_ONNX_PATH = os.path.join(BASEDIR, "openpilot", "selfdrive", "modeld", "models", "big_driving_supercombo.onnx")
def _read_model_name_fields():
with open(DEFAULT_MODEL_NAME_PATH) as f:
content = f.read()
fields = {}
for line in content.splitlines():
if "=" in line:
key, val = line.split("=", 1)
fields[key.strip()] = val.strip().strip('"')
return fields
def update_model_hash():
fields = _read_model_name_fields()
supercombo_hash = get_file_hash(SUPERCOMBO_ONNX_PATH)
combined_hash = hashlib.sha256(supercombo_hash.encode()).hexdigest()
fingerprint = f"{supercombo_hash}:{fields.get('DEFAULT_MODEL', '')}:{fields.get('DEFAULT_MODEL_REF', '')}"
combined_hash = hashlib.sha256(fingerprint.encode()).hexdigest()
with open(MODEL_HASH_PATH, "w") as f:
f.write(combined_hash)
print(f"Generated and updated new combined model hash to {MODEL_HASH_PATH}")
if os.path.exists(BIG_SUPERCOMBO_ONNX_PATH):
import subprocess
rel = os.path.relpath(BIG_SUPERCOMBO_ONNX_PATH, os.getcwd())
pointer = subprocess.check_output(["git", "show", f"HEAD:{rel}"], text=True)
oid = next(l.split(":", 1)[1] for l in pointer.splitlines() if l.startswith("oid sha256:"))
big_fingerprint = f"{oid}:{fields.get('DEFAULT_BIG_MODEL', '')}:{fields.get('DEFAULT_BIG_MODEL_REF', '')}"
big_combined_hash = hashlib.sha256(big_fingerprint.encode()).hexdigest()
def get_current_default_model_name():
print("[GET DEFAULT MODEL NAME]")
name = DEFAULT_MODEL
print(f'Current default model name: "{name}"')
with open(BIG_MODEL_HASH_PATH, "w") as f:
f.write(big_combined_hash)
return name
print(f"Generated and updated new big model hash to {BIG_MODEL_HASH_PATH}")
def update_default_model_name(name: str):
print("[CHANGE DEFAULT MODEL NAME]")
def get_ref_for_name(url: str, name: str) -> str:
response = requests.get(url, timeout=10)
if response.status_code == 200:
bundles = response.json()["bundles"]
matching = [b for b in bundles if re.search(name, f"{b['short_name']} {b['display_name']}", re.IGNORECASE)]
if matching:
return max(matching, key=lambda b: int(b["index"]))["ref"]
return ""
def update_default_model_names(default_model_name: str, default_big_model_name: str):
print("[CHANGE DEFAULT MODEL NAMES]")
small_ref = get_ref_for_name(ModelFetcher.MODEL_URL, default_model_name)
big_ref = get_ref_for_name(ModelFetcher.MODEL_URL_CHESTNUT, default_big_model_name)
with open(DEFAULT_MODEL_NAME_PATH, "w") as f:
f.write(f'DEFAULT_MODEL = "{name}"\n')
print(f'New default model name: "{name}"')
f.write(f'DEFAULT_MODEL = "{default_model_name}"\n')
f.write(f'DEFAULT_MODEL_REF = "{small_ref}"\n')
f.write(f'DEFAULT_BIG_MODEL = "{default_big_model_name}"\n')
f.write(f'DEFAULT_BIG_MODEL_REF = "{big_ref}"\n')
print(f'New default small model name: "{default_model_name}" (ref: {small_ref})')
print(f'New default big model name: "{default_big_model_name}" (ref: {big_ref})')
print("[DONE]")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Update default model name and hash")
parser.add_argument("--new_name", type=str, help="New default model name")
parser = argparse.ArgumentParser(description="Update default model names and hash")
parser.add_argument("--new_small_model_name", type=str, help="New default small model name")
parser.add_argument("--new_big_model_name", type=str, help="New default big model name")
args = parser.parse_args()
if not args.new_name:
print("Warning: No new default model name provided. Use --new_name to specify")
print("Default model name and hash will not be updated! (aborted)")
exit(0)
if args.new_small_model_name is None and args.new_big_model_name is None:
new_name = input(f'Enter new default small model name (current: "{DEFAULT_MODEL}", leave empty to keep): ').strip()
new_big_model_name = input(f'Enter new default big model name (current: "{DEFAULT_BIG_MODEL}", leave empty to keep): ').strip()
else:
new_name, new_big_model_name = args.new_small_model_name, args.new_big_model_name
current_name = get_current_default_model_name()
new_name = args.new_name
if current_name == new_name:
print(f'Proposed default model name: "{new_name}"')
confirm = input("Proposed default model name is the same as the current default model name. Confirm? (y/n): ").upper().strip()
if confirm != "Y":
print("Default model name and hash will not be updated! (aborted)")
exit(0)
update_default_model_name(new_name)
update_default_model_names(new_name or DEFAULT_MODEL, new_big_model_name or DEFAULT_BIG_MODEL)
update_model_hash()
+83 -32
View File
@@ -13,8 +13,6 @@ from openpilot.common.params import Params
from openpilot.common.swaglog import cloudlog
from openpilot.common.hardware.hw import Paths
from openpilot.sunnypilot.models.helpers import is_bundle_version_compatible
from openpilot.selfdrive.modeld.helpers import usbgpu_present
from openpilot.cereal import custom
@@ -140,45 +138,53 @@ class ModelCache:
class ModelFetcher:
"""Handles fetching and caching of model data from remote source"""
MODEL_URL = "https://raw.githubusercontent.com/sunnypilot/sunnypilot-models/refs/heads/gh-pages/docs/driving_models_v20.json"
MODEL_URL_USBGPU = "https://raw.githubusercontent.com/sunnypilot/sunnypilot-models/refs/heads/gh-pages/docs/driving_models_usbgpu_v20.json"
MODEL_URL = "https://raw.githubusercontent.com/sunnypilot/sunnypilot-models/refs/heads/gh-pages/docs/driving_models_v22.json"
MODEL_URL_CHESTNUT = "https://raw.githubusercontent.com/sunnypilot/sunnypilot-models/refs/heads/gh-pages/docs/driving_models_chestnut_v23.json"
MODEL_SOURCES = {
"qcom": (MODEL_URL, ""),
"chestnut": (MODEL_URL_CHESTNUT, "_Chestnut"),
}
def __init__(self, params: Params):
self.params = params
self.model_parser = ModelParser()
self._is_usbgpu: bool | None = None
self.model_cache = ModelCache(params)
self.model_url = self.MODEL_URL
self._update_model_source()
self.model_caches = {
source: ModelCache(params, suffix=suffix)
for source, (_, suffix) in self.MODEL_SOURCES.items()
}
self._refetched: set[str] = set()
self.params.put("ModelManager_ActiveJson", {
"qcom": self.MODEL_URL,
"chestnut": self.MODEL_URL_CHESTNUT,
}, block=True)
def _update_model_source(self) -> None:
"""Updates what json to use based on usbgpu availability"""
is_usbgpu = usbgpu_present()
if is_usbgpu != self._is_usbgpu:
self._is_usbgpu = is_usbgpu
self.model_cache = ModelCache(self.params, suffix="_USBGPU" if is_usbgpu else "")
self.model_url = self.MODEL_URL_USBGPU if is_usbgpu else self.MODEL_URL
self.params.put("ModelManager_ActiveJson", self.model_url, block=True)
@staticmethod
def active_source(chestnut_present: bool) -> str:
return "chestnut" if chestnut_present else "qcom"
def _fetch_and_cache_models(self) -> list[custom.ModelManagerSP.ModelBundle] | None:
def _fetch_and_cache_models(self, source: str) -> list[custom.ModelManagerSP.ModelBundle] | None:
"""Fetches fresh model data from remote and updates cache.
Returns None on transport errors. Raises on 404 and other fatal HTTP errors.
"""
model_url, _ = self.MODEL_SOURCES[source]
try:
response = requests.get(self.model_url, timeout=10)
response = requests.get(model_url, timeout=10)
# Explicitly handle 404 differently
if response.status_code == 404:
cloudlog.error(f"Models URL returned 404 Not Found: {self.model_url}")
raise HTTPError(f"404 Not Found: {self.model_url}", response=response)
cloudlog.error(f"Models URL returned 404 Not Found: {model_url}")
raise HTTPError(f"404 Not Found: {model_url}", response=response)
# Raise for any other 4xx/5xx
response.raise_for_status()
json_data = response.json()
self.model_cache.set(json_data)
cloudlog.debug("Successfully updated models cache")
return self.model_parser.parse_models(json_data)
parsed = self.model_parser.parse_models(json_data)
if parsed:
self.model_caches[source].set(json_data)
cloudlog.debug(f"Successfully updated models cache for {source}")
return parsed
except ConnectionError as e:
cloudlog.warning(f"DNS/connection error while fetching models: {e}")
@@ -191,16 +197,40 @@ class ModelFetcher:
return None
def get_available_bundles(self) -> list[custom.ModelManagerSP.ModelBundle]:
"""Gets the list of available models, with smart cache handling"""
self._update_model_source()
cached_data, is_expired = self.model_cache.get()
@staticmethod
def _cache_matches_source(source: str, cached_data: dict) -> bool:
bundles = cached_data.get("bundles", [])
if source == "chestnut":
return any(bundle.get("is_big") is True for bundle in bundles)
return not any(bundle.get("is_big") is True for bundle in bundles)
def get_bundles_for_source(self, source: str) -> list[custom.ModelManagerSP.ModelBundle]:
if source not in self.MODEL_SOURCES:
cloudlog.warning(f"Unknown model source: {source}")
return []
cached_data, is_expired = self.model_caches[source].get()
if cached_data and not is_expired:
cloudlog.debug("Using valid cached models data")
return self.model_parser.parse_models(cached_data)
# a source is refetched over a mismatch at most once per process: if the fresh
# manifest still mismatches, the URL is authoritative and the cache is trusted
if self._cache_matches_source(source, cached_data) or source in self._refetched:
try:
parsed = self.model_parser.parse_models(cached_data)
except Exception:
cloudlog.warning(f"Failed to parse cached models for {source}; refetching", exc_info=True)
else:
if parsed:
cloudlog.debug(f"Using valid cached models data for source {source}")
return parsed
# a source-matching cache that yields no valid bundles is stale (e.g. an old
# manifest version) - do not trust it, refetch so the source is repopulated
cloudlog.warning(f"Cached models for {source} have no valid bundles; refetching")
else:
self._refetched.add(source)
cloudlog.warning(f"Cached models for {source} not valid; refetching once")
fetched_bundles = self._fetch_and_cache_models()
fetched_bundles = self._fetch_and_cache_models(source)
if fetched_bundles is not None:
return fetched_bundles
@@ -208,12 +238,33 @@ class ModelFetcher:
cloudlog.warning("Failed to fetch fresh data and no cache available")
cloudlog.warning("Failed to fetch fresh data. Using expired cache as fallback")
return self.model_parser.parse_models(cached_data)
try:
return self.model_parser.parse_models(cached_data)
except Exception:
return []
def get_cached_bundles(params: Params, source: str) -> list[custom.ModelManagerSP.ModelBundle]:
if source not in ModelFetcher.MODEL_SOURCES:
cloudlog.warning(f"Unknown model source: {source}")
return []
_, suffix = ModelFetcher.MODEL_SOURCES[source]
cached_data = params.get(f"ModelManager_ModelsCache{suffix}")
if not cached_data:
return []
try:
return ModelParser.parse_models(cached_data)
except Exception as e:
cloudlog.warning(f"Failed to parse cached models for source {source}: {e}")
return []
if __name__ == "__main__":
from openpilot.selfdrive.modeld.helpers import chestnut_present
params = Params()
model_fetcher = ModelFetcher(params)
bundles = model_fetcher.get_available_bundles()
bundles = model_fetcher.get_bundles_for_source(ModelFetcher.active_source(chestnut_present()))
for bundle in bundles:
for model in bundle.models:
model_overrides = {override.key: override.value for override in bundle.overrides}
+73 -35
View File
@@ -16,14 +16,20 @@ from openpilot.common.params import Params
from openpilot.common.swaglog import cloudlog
from openpilot.sunnypilot.models.constants import Meta, MetaSimPose, MetaTombRaider
from openpilot.common.hardware.hw import Paths
from openpilot.selfdrive.modeld.helpers import chestnut_present
# SET ME TO THE EXACT JSON VERSION WE SET IN SUNNYPILOT_MODELS REPO
REQUIRED_JSON_VERSION = 17
REQUIRED_JSON_VERSION = 19
CUSTOM_MODEL_PATH = Paths.model_root()
METADATA_PATH = Path(__file__).parent / '../models/supercombo_metadata.pkl'
ModelManager = custom.ModelManagerSP
_LAST_VALIDATED_RAW = None
ACTIVE_BUNDLE_KEYS = {
"qcom": "ModelManager_ActiveBundle",
"chestnut": "ModelManager_ActiveBundleChestnut",
}
_LAST_VALIDATED_RAW: dict[str, dict | None] = {}
def _compute_hash(file_path: str) -> str | None:
@@ -86,11 +92,11 @@ def _bundle_needs_reset(active_bundle: custom.ModelManagerSP.ModelBundle, availa
if available_bundles is not None:
matching_bundle = None
for bundle in available_bundles:
if getattr(active_bundle, 'ref', None) and getattr(bundle, 'ref', None):
if active_bundle.ref and bundle.ref:
if active_bundle.ref == bundle.ref:
matching_bundle = bundle
break
elif getattr(active_bundle, 'internalName', None) == getattr(bundle, 'internalName', None):
elif active_bundle.internalName == bundle.internalName:
matching_bundle = bundle
break
@@ -98,49 +104,81 @@ def _bundle_needs_reset(active_bundle: custom.ModelManagerSP.ModelBundle, availa
return True
if active_bundle.minimumSelectorVersion != matching_bundle.minimumSelectorVersion:
return True
active_runner = getattr(active_bundle, 'runner', None)
matching_runner = getattr(matching_bundle, 'runner', None)
if active_runner is not None and matching_runner is not None:
if getattr(active_runner, 'raw', active_runner) != getattr(matching_runner, 'raw', matching_runner):
return True
if active_bundle.runner != matching_bundle.runner:
return True
if set(_bundle_artifacts(active_bundle)) != set(_bundle_artifacts(matching_bundle)):
return True
return not _bundle_is_valid_locally(active_bundle)
def validate_active_bundle(params: Params, available_bundles: list[custom.ModelManagerSP.ModelBundle] | None = None) -> None:
global _LAST_VALIDATED_RAW
raw_bundle = params.get("ModelManager_ActiveBundle")
if not raw_bundle:
return
if raw_bundle == _LAST_VALIDATED_RAW:
return
active_bundle = get_active_bundle(params, raw_bundle_dict=raw_bundle)
if active_bundle is None or _bundle_needs_reset(active_bundle, available_bundles):
cloudlog.warning("Active model bundle invalid; resetting to default")
params.remove("ModelManager_ActiveBundle")
params.put("ModelRunnerTypeCache", int(custom.ModelManagerSP.Runner.stock), block=True)
_LAST_VALIDATED_RAW = None
else:
_LAST_VALIDATED_RAW = raw_bundle
def get_active_bundle(params: Params | None = None, raw_bundle_dict: dict | bytes | None = None) -> "custom.ModelManagerSP.ModelBundle | None":
params = params or Params()
def _parse_active_bundle(raw_bundle) -> "custom.ModelManagerSP.ModelBundle | None":
try:
active_bundle_dict = raw_bundle_dict if raw_bundle_dict is not None else (params.get("ModelManager_ActiveBundle") or {})
if isinstance(active_bundle_dict, dict) and active_bundle_dict and is_bundle_version_compatible(active_bundle_dict):
return custom.ModelManagerSP.ModelBundle(**active_bundle_dict)
if isinstance(raw_bundle, dict) and raw_bundle and is_bundle_version_compatible(raw_bundle):
return custom.ModelManagerSP.ModelBundle(**raw_bundle)
except Exception:
pass
return None
def get_selected_bundle(params: Params | None = None, source: str = "qcom") -> "custom.ModelManagerSP.ModelBundle | None":
params = params or Params()
return _parse_active_bundle(params.get(ACTIVE_BUNDLE_KEYS[source]))
def get_active_source(chestnut: bool | None = None, chestnut_active: bool | None = None,
chestnut_loading: bool | None = None, offroad: bool | None = None) -> str:
if chestnut is None:
chestnut = chestnut_present()
state_valid = chestnut_active is not None or chestnut_loading is not None or offroad is not None
big_active = chestnut and (not state_valid or chestnut_active or chestnut_loading or offroad)
return "chestnut" if big_active else "qcom"
def get_active_bundle(params: Params | None = None, *, chestnut: bool | None = None) -> "custom.ModelManagerSP.ModelBundle | None":
# no cross-slot fallback: an empty active slot means the hardware default, which
# only stock modeld can run - modeld_v2 requires a real bundle
params = params or Params()
return get_selected_bundle(params, get_active_source(chestnut=chestnut))
def resolve_bundle_by_ref(
ref: str, source_bundles: dict[str, list[custom.ModelManagerSP.ModelBundle]],
) -> "tuple[custom.ModelManagerSP.ModelBundle, str] | None":
for source, bundles in source_bundles.items():
for bundle in bundles:
if bundle.ref == ref:
return bundle, source
return None
def _validate_active_bundle(params: Params, source: str, available_bundles: list[custom.ModelManagerSP.ModelBundle] | None = None) -> None:
global _LAST_VALIDATED_RAW
key = ACTIVE_BUNDLE_KEYS[source]
raw_bundle = params.get(key)
if not raw_bundle:
return
if _LAST_VALIDATED_RAW.get(key) == raw_bundle:
return
active_bundle = _parse_active_bundle(raw_bundle)
if active_bundle is None or _bundle_needs_reset(active_bundle, available_bundles):
cloudlog.warning(f"Active model bundle invalid for {source}; resetting to default")
params.remove(key)
_LAST_VALIDATED_RAW[key] = None
else:
_LAST_VALIDATED_RAW[key] = raw_bundle
def validate_active_bundles(params: Params, source_bundles: dict[str, list[custom.ModelManagerSP.ModelBundle]]) -> None:
# an empty list means the fetch failed, not that the catalog dropped the bundle
for source, bundles in source_bundles.items():
_validate_active_bundle(params, source, bundles or None)
get_active_model_runner(params, force_check=True)
def get_active_model_runner(params: Params | None = None, force_check: bool = False) -> int:
params = params or Params()
cached_runner_type = params.get("ModelRunnerTypeCache")
+110 -41
View File
@@ -17,12 +17,17 @@ from openpilot.common.hardware.hw import Paths
from openpilot.cereal import messaging, custom
from openpilot.sunnypilot.models.fetcher import ModelFetcher
from openpilot.sunnypilot.models.helpers import get_active_bundle, validate_active_bundle, verify_file
from openpilot.sunnypilot.models.helpers import (ACTIVE_BUNDLE_KEYS, get_active_bundle, get_selected_bundle,
resolve_bundle_by_ref, validate_active_bundles, verify_file)
# (connect, read) seconds. read is per-request inactivity, not a total cap
DOWNLOAD_TIMEOUT = (30, 30)
class DownloadCancelled(Exception):
pass
class ModelManagerSP:
"""Manages model downloads and status reporting"""
@@ -30,11 +35,25 @@ class ModelManagerSP:
self.params = Params()
self.model_fetcher = ModelFetcher(self.params)
self.pm = messaging.PubMaster(["modelManagerSP"])
self.sm = messaging.SubMaster(["deviceState"])
self.chestnut_present = False
self.available_models: list[custom.ModelManagerSP.ModelBundle] = []
self.source_models: dict[str, list[custom.ModelManagerSP.ModelBundle]] = {}
self.selected_bundle: custom.ModelManagerSP.ModelBundle = None
self.active_bundle: custom.ModelManagerSP.ModelBundle = get_active_bundle(self.params)
self.active_bundle: custom.ModelManagerSP.ModelBundle = get_active_bundle(self.params, chestnut=self.chestnut_present)
self._chunk_size = 128 * 1000 # 128 KB chunks
self._download_start_times: dict[str, float] = {} # Track start time per model
self._download_ref: bytes | str | None = None
def _download_interrupted(self) -> bool:
# only removal cancels: a different ref is a queued selection that
# _release_download_ref leaves in place for the next tick
return self.params.get("ModelManager_DownloadRef") is None
def _release_download_ref(self) -> None:
if self.params.get("ModelManager_DownloadRef") == self._download_ref:
self.params.remove("ModelManager_DownloadRef")
self._download_ref = None
def _sync_artifact_progress(self, source_artifact) -> None:
"""Mirror download progress to all artifacts sharing the same filename in the selected bundle."""
@@ -76,8 +95,8 @@ class ModelManagerSP:
f.write(chunk)
bytes_downloaded += len(chunk)
if self.params.get("ModelManager_DownloadIndex") is None:
raise Exception("Download cancelled")
if self._download_interrupted():
raise DownloadCancelled("Download cancelled")
if total_size > 0:
progress = (bytes_downloaded / total_size) * 100
@@ -90,7 +109,7 @@ class ModelManagerSP:
# Clean up start time after download completes
del self._download_start_times[model.fileName]
async def _download_chunked(self, base_url: str, base_path: str, artifact) -> None:
async def _download_chunked(self, base_url: str, base_path: str, artifact, skip: frozenset[int] | set[int] = frozenset()) -> None:
from openpilot.common.file_chunker import get_chunk_name, get_manifest_path
num_chunks = len(artifact.chunks)
@@ -102,8 +121,11 @@ class ModelManagerSP:
# Shared connection saves a TCP+TLS handshake per chunk.
# Keep sequential: the link saturates on one stream and Session is not thread-safe.
completed = len(skip)
with requests.Session() as session:
for i, _ in enumerate(artifact.chunks):
if i in skip:
continue
chunk_url = get_chunk_name(base_url, i, num_chunks)
chunk_path = get_chunk_name(base_path, i, num_chunks)
chunk_downloaded = 0
@@ -114,15 +136,16 @@ class ModelManagerSP:
for data in response.iter_content(chunk_size=self._chunk_size):
f.write(data)
chunk_downloaded += len(data)
if self.params.get("ModelManager_DownloadIndex") is None:
raise Exception("Download cancelled")
if self._download_interrupted():
raise DownloadCancelled("Download cancelled")
intra = chunk_downloaded / max(chunk_size, 1)
progress = min(99.0, ((i + intra) / num_chunks) * 100)
progress = min(99.0, ((completed + intra) / num_chunks) * 100)
artifact.downloadProgress.status = custom.ModelManagerSP.DownloadStatus.downloading
artifact.downloadProgress.progress = progress
artifact.downloadProgress.eta = self._calculate_eta(artifact.fileName, progress)
self._sync_artifact_progress(artifact)
self._report_status()
completed += 1
with open(manifest_path, 'w') as f: # noqa: ASYNC230
f.write(str(num_chunks))
@@ -133,6 +156,8 @@ class ModelManagerSP:
async def _process_artifact(self, artifact, destination_path: str) -> None:
if not artifact.downloadUri.uri:
return None
if self._download_interrupted():
raise DownloadCancelled("Download cancelled")
url = artifact.downloadUri.uri
expected_hash = artifact.downloadUri.sha256
@@ -140,17 +165,23 @@ class ModelManagerSP:
full_path = os.path.join(destination_path, filename)
try:
# progress counts only valid chunks so a resumed download continues the
# bar from where verification left it, instead of falling back to zero
is_cached = False
valid_chunks: set[int] = set()
if len(artifact.chunks) > 0:
from openpilot.common.file_chunker import get_chunk_name
chunks_valid = True
num_chunks = len(artifact.chunks)
for i, chunk in enumerate(artifact.chunks):
chunk_path = get_chunk_name(full_path, i, len(artifact.chunks))
if not await verify_file(chunk_path, chunk.sha256):
chunks_valid = False
break
if chunks_valid and len(artifact.chunks) > 0:
is_cached = True
if self._download_interrupted():
raise DownloadCancelled("Download cancelled")
if await verify_file(get_chunk_name(full_path, i, num_chunks), chunk.sha256):
valid_chunks.add(i)
artifact.downloadProgress.status = custom.ModelManagerSP.DownloadStatus.verifying
artifact.downloadProgress.progress = (len(valid_chunks) / num_chunks) * 100
self._sync_artifact_progress(artifact)
self._report_status()
is_cached = len(valid_chunks) == num_chunks
else:
if await verify_file(full_path, expected_hash):
is_cached = True
@@ -164,7 +195,7 @@ class ModelManagerSP:
return
if len(artifact.chunks) > 0:
await self._download_chunked(url, full_path, artifact)
await self._download_chunked(url, full_path, artifact, skip=valid_chunks)
from openpilot.common.file_chunker import get_chunk_name
for i, chunk in enumerate(artifact.chunks):
chunk_path = get_chunk_name(full_path, i, len(artifact.chunks))
@@ -181,6 +212,17 @@ class ModelManagerSP:
self._sync_artifact_progress(artifact)
self._report_status()
except DownloadCancelled:
# a cancel keeps whatever is on disk: complete chunks resume the next attempt
self._download_start_times.pop(artifact.fileName, None)
artifact.downloadProgress.status = custom.ModelManagerSP.DownloadStatus.failed
artifact.downloadProgress.eta = 0
self._sync_artifact_progress(artifact)
if self.selected_bundle:
self.selected_bundle.status = custom.ModelManagerSP.DownloadStatus.failed
self._report_status()
raise
except Exception as e:
cloudlog.error(f"Error downloading {filename}: {str(e)}")
for f in [full_path] + [p for p in (os.path.join(destination_path, f) for f in os.listdir(destination_path)) if filename in p]:
@@ -212,10 +254,12 @@ class ModelManagerSP:
model_manager_state.availableBundles = self.available_models
self.pm.send('modelManagerSP', msg)
async def _download_bundle(self, model_bundle: custom.ModelManagerSP.ModelBundle, destination_path: str) -> None:
"""Downloads all models in a bundle"""
async def _download_bundle(self, model_bundle: custom.ModelManagerSP.ModelBundle, destination_path: str, source: str) -> None:
self.selected_bundle = model_bundle
self.selected_bundle.status = custom.ModelManagerSP.DownloadStatus.downloading
for model in self.selected_bundle.models:
model.artifact.downloadProgress.status = custom.ModelManagerSP.DownloadStatus.downloading
self._report_status()
os.makedirs(destination_path, exist_ok=True)
try:
@@ -232,10 +276,11 @@ class ModelManagerSP:
seen_artifacts.add(artifact.fileName)
await self._process_artifact(artifact, destination_path)
self.active_bundle = self.selected_bundle
self.active_bundle.status = custom.ModelManagerSP.DownloadStatus.downloaded
self.params.put("ModelManager_ActiveBundle", self.active_bundle.to_dict(), block=True)
self.selected_bundle = None
if self._download_interrupted():
raise DownloadCancelled("Download cancelled")
self.selected_bundle.status = custom.ModelManagerSP.DownloadStatus.downloaded
self.params.put(ACTIVE_BUNDLE_KEYS[source], model_bundle.to_dict(), block=True)
self.active_bundle = get_active_bundle(self.params, chestnut=self.chestnut_present)
except Exception:
if self.selected_bundle is not None:
@@ -245,9 +290,30 @@ class ModelManagerSP:
finally:
self._report_status()
def download(self, model_bundle: custom.ModelManagerSP.ModelBundle, destination_path: str) -> None:
def download(self, model_bundle: custom.ModelManagerSP.ModelBundle, destination_path: str, source: str) -> None:
"""Main entry point for downloading a model bundle"""
asyncio.run(self._download_bundle(model_bundle, destination_path))
asyncio.run(self._download_bundle(model_bundle, destination_path, source))
def _process_download_requests(self) -> None:
# loops so a ref queued during a download starts in the same tick, without
# the bar dropping to idle for a tick between the two transfers
last_ref = None
while (ref_to_download := self.params.get("ModelManager_DownloadRef")) is not None:
if ref_to_download == last_ref: # a repeating ref falls back to the next tick instead of spinning
return
last_ref = ref_to_download
resolved = resolve_bundle_by_ref(ref_to_download, self.source_models)
if not resolved:
return
model_to_download, source = resolved
self._download_ref = ref_to_download
try:
self.download(model_to_download, Paths.model_root(), source)
except Exception as e:
cloudlog.exception(e)
finally:
self._release_download_ref()
self.selected_bundle = None
def main_thread(self) -> None:
"""Main thread for model management"""
@@ -255,19 +321,20 @@ class ModelManagerSP:
while True:
try:
self.available_models = self.model_fetcher.get_available_bundles()
validate_active_bundle(self.params, self.available_models)
self.active_bundle = get_active_bundle(self.params)
self.sm.update(0)
self.chestnut_present = self.sm['deviceState'].chestnutPresent
self.source_models = {source: self.model_fetcher.get_bundles_for_source(source) for source in ModelFetcher.MODEL_SOURCES}
self.available_models = self.source_models[ModelFetcher.active_source(self.chestnut_present)]
validate_active_bundles(self.params, self.source_models)
self.active_bundle = get_active_bundle(self.params, chestnut=self.chestnut_present)
if (index_to_download := self.params.get("ModelManager_DownloadIndex")) is not None:
if model_to_download := next((model for model in self.available_models if model.index == index_to_download), None):
try:
self.download(model_to_download, Paths.model_root())
except Exception as e:
cloudlog.exception(e)
finally:
self.params.remove("ModelManager_DownloadIndex")
self.selected_bundle = None
if get_selected_bundle(self.params, "chestnut") is not None and get_selected_bundle(self.params, "qcom") is None:
if self.params.get("ModelManager_DownloadRef") is None:
from openpilot.sunnypilot.models.model_name import DEFAULT_MODEL_REF
if DEFAULT_MODEL_REF:
self.params.put("ModelManager_DownloadRef", DEFAULT_MODEL_REF)
self._process_download_requests()
if self.params.get("ModelManager_ClearCache"):
self.clear_model_cache()
@@ -285,12 +352,14 @@ class ModelManagerSP:
Clears the model cache directory of all files except those in the active model bundle.
"""
# Get list of files used by active model bundle
# Get list of files used by both slots' selected bundles (either may become
# the truly active bundle depending on hardware availability)
active_files = []
if self.active_bundle is not None: # When the default model is active
for model in self.active_bundle.models:
if hasattr(model, 'artifact') and model.artifact.fileName:
active_files.append(model.artifact.fileName)
for source in ACTIVE_BUNDLE_KEYS:
if selected_bundle := get_selected_bundle(self.params, source):
for model in selected_bundle.models:
if model.artifact.fileName:
active_files.append(model.artifact.fileName)
# Remove all files except active ones (including their chunk files)
model_dir = Paths.model_root()
@@ -1 +1,4 @@
DEFAULT_MODEL = "CD210"
DEFAULT_MODEL_REF = "5b6436a90cf6902b8aaa71c2b6f3d7164d8ae391"
DEFAULT_BIG_MODEL = "BMRLNAP Model v4"
DEFAULT_BIG_MODEL_REF = "f877d7a0ccc3cce943c76e285214c020cd65c899"
@@ -0,0 +1 @@
2c814f08a2c51323b87839fbf8d2c2a9853a2b5536271b3d67f7b7a2de7f9374
+1 -1
View File
@@ -1 +1 @@
49133798d9cd9cacf47085c7ef8122bfee88cd9c6192a8314c81bfb1b37f5809
c5be11d2fb1115be953c541f30c50f7c71a00bc4a0e128e19aa11b60689317fc
@@ -5,19 +5,47 @@ This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
import os
import subprocess
from openpilot.sunnypilot import get_file_hash
from openpilot.sunnypilot.models.default_model import MODEL_HASH_PATH, SUPERCOMBO_ONNX_PATH
from openpilot.sunnypilot.models.default_model import MODEL_HASH_PATH, SUPERCOMBO_ONNX_PATH, BIG_MODEL_HASH_PATH, \
BIG_SUPERCOMBO_ONNX_PATH, _read_model_name_fields
import hashlib
from openpilot.common.test import OpenpilotTestCase
def _get_lfs_oid(path: str) -> str:
"""Extract the LFS OID (SHA256 of actual content) from git, works whether the file is smudged or not."""
pointer = subprocess.check_output(["git", "show", f"HEAD:{path}"], text=True)
for line in pointer.splitlines():
if line.startswith("oid sha256:"):
return line.split(":", 1)[1]
raise ValueError(f"No LFS OID found for {path}")
class TestDefaultModel(OpenpilotTestCase):
def test_compare_onnx_hashes(self):
fields = _read_model_name_fields()
supercombo_hash = get_file_hash(SUPERCOMBO_ONNX_PATH)
combined_hash = hashlib.sha256(supercombo_hash.encode()).hexdigest()
fingerprint = f"{supercombo_hash}:{fields.get('DEFAULT_MODEL', '')}:{fields.get('DEFAULT_MODEL_REF', '')}"
combined_hash = hashlib.sha256(fingerprint.encode()).hexdigest()
with open(MODEL_HASH_PATH) as f:
current_hash = f.read().strip()
assert combined_hash == current_hash, "Run sunnypilot/models/default_model.py to update the default model name and hash"
assert combined_hash == current_hash, "Run openpilot/sunnypilot/models/default_model.py to update the default model name and hash"
def test_compare_big_onnx_hashes(self):
if not os.path.exists(BIG_SUPERCOMBO_ONNX_PATH):
self.skipTest("big_driving_supercombo.onnx not present")
fields = _read_model_name_fields()
oid = _get_lfs_oid(os.path.relpath(BIG_SUPERCOMBO_ONNX_PATH, os.getcwd()))
big_fingerprint = f"{oid}:{fields.get('DEFAULT_BIG_MODEL', '')}:{fields.get('DEFAULT_BIG_MODEL_REF', '')}"
combined_hash = hashlib.sha256(big_fingerprint.encode()).hexdigest()
with open(BIG_MODEL_HASH_PATH) as f:
current_hash = f.read().strip()
assert combined_hash == current_hash, "Run openpilot/sunnypilot/models/default_model.py to update the default model name and hash"
@@ -11,6 +11,7 @@ import http.server
import os
import tempfile
import threading
import time
import unittest
from typing import Any
from unittest import mock
@@ -23,6 +24,10 @@ from openpilot.common.test import OpenpilotTestCase
from openpilot.common.file_chunker import get_chunk_name, get_manifest_path
from openpilot.selfdrive.test.helpers import http_server_context
from openpilot.sunnypilot.models import manager as manager_module
from openpilot.sunnypilot.models.fetcher import ModelFetcher, get_cached_bundles
from openpilot.sunnypilot.models import helpers
from openpilot.sunnypilot.models.helpers import (get_active_bundle, get_active_source, get_selected_bundle,
resolve_bundle_by_ref, validate_active_bundles)
from openpilot.sunnypilot.models.manager import ModelManagerSP
CHUNK_BODIES = [b'A' * 5000, b'B' * 5000, b'C' * 3000]
@@ -98,11 +103,13 @@ class ManagerDownloadTestBase(OpenpilotTestCase):
self.manager = ModelManagerSP.__new__(ModelManagerSP)
self.manager.params = mock.MagicMock()
self.manager.params.get.return_value = b'0' # not cancelled
self.manager._download_ref = b'0'
self.manager.pm = mock.MagicMock()
self.manager.pm.send.side_effect = self._record_progress
self.manager.selected_bundle = None
self.manager.active_bundle = None
self.manager.available_models = []
self.manager.chestnut_present = False
self.manager._chunk_size = 1024
self.manager._download_start_times = {}
@@ -249,6 +256,166 @@ class TestManagerDownload(ManagerDownloadTestBase):
assert self.manager._download_start_times == {}
self.run_with_server(body)
def test_download_ref_present_keeps_download_alive(self):
"""A pending download request (DownloadRef set) must not be cancelled mid-transfer."""
def body():
artifact = self.make_artifact(chunked=True)
base_path = os.path.join(self.dest, artifact.fileName)
self.manager.params.get.side_effect = lambda key: b"ref" if key == "ModelManager_DownloadRef" else None
self.manager._download_ref = b"ref"
asyncio.run(self.manager._download_chunked(artifact.downloadUri.uri, base_path, artifact))
assert os.path.isfile(get_manifest_path(base_path))
self.run_with_server(body)
def test_cancellation_via_download_ref(self):
"""Removing DownloadRef mid-transfer cancels the download."""
def body():
artifact = self.make_artifact(chunked=True)
base_path = os.path.join(self.dest, artifact.fileName)
checks = {"n": 0}
def get(key):
if key == "ModelManager_DownloadRef":
checks["n"] += 1
return b"ref" if checks["n"] <= 2 else None
return b"0"
self.manager.params.get.side_effect = get
self.manager._download_ref = b"ref"
with self.assertRaises(Exception) as ctx:
asyncio.run(self.manager._download_chunked(artifact.downloadUri.uri, base_path, artifact))
assert 'cancelled' in str(ctx.exception).lower()
assert not os.path.isfile(get_manifest_path(base_path))
self.run_with_server(body)
def test_replaced_download_ref_queues_instead_of_cancelling(self):
"""Selecting another model mid-transfer lets the running download finish."""
def body():
artifact = self.make_artifact(chunked=True)
base_path = os.path.join(self.dest, artifact.fileName)
self.manager.params.get.side_effect = lambda key: b"other-ref" if key == "ModelManager_DownloadRef" else None
self.manager._download_ref = b"ref"
asyncio.run(self.manager._download_chunked(artifact.downloadUri.uri, base_path, artifact))
assert os.path.isfile(get_manifest_path(base_path))
self.run_with_server(body)
def test_replaced_download_ref_is_kept(self):
"""A selection made during a download must survive that download's cleanup."""
self.manager.params.get.return_value = b"new-ref"
self.manager._download_ref = b"old-ref"
self.manager._release_download_ref()
self.manager.params.remove.assert_not_called()
def test_own_download_ref_is_released(self):
self.manager.params.get.return_value = b"ref"
self.manager._download_ref = b"ref"
self.manager._release_download_ref()
self.manager.params.remove.assert_called_once_with("ModelManager_DownloadRef")
def test_cached_bundle_cancel_skips_slot_write(self):
"""A cancel must stop an already-on-disk bundle before it is applied to the slot."""
def body():
artifact = self.make_artifact(chunked=True)
base_path = os.path.join(self.dest, artifact.fileName)
for i, data in enumerate(CHUNK_BODIES):
with open(get_chunk_name(base_path, i, len(CHUNK_BODIES)), 'wb') as f:
f.write(data)
self._bundle.ref = "test-ref"
params, store = self._make_params_with_store()
store["ModelManager_DownloadRef"] = None # removed -> cancelled
self.manager.params = params
self.manager._download_ref = b"ref"
with self.assertRaises(Exception) as ctx:
asyncio.run(self.manager._download_bundle(self._bundle, self.dest, "qcom"))
assert 'cancelled' in str(ctx.exception).lower()
assert "ModelManager_ActiveBundle" not in store
assert all(os.path.isfile(p) for p in self.chunk_paths(base_path)), "cancel must not delete cached chunks"
self.run_with_server(body)
def test_resume_skips_valid_chunks(self):
"""A chunk already on disk is kept and not re-downloaded; progress starts above its share."""
def body():
artifact = self.make_artifact(chunked=True)
base_path = os.path.join(self.dest, artifact.fileName)
with open(get_chunk_name(base_path, 0, len(CHUNK_BODIES)), 'wb') as f:
f.write(CHUNK_BODIES[0])
asyncio.run(self.manager._process_artifact(artifact, self.dest))
chunk0_suffix = get_chunk_name('', 0, len(CHUNK_BODIES))
assert not any(p.endswith(chunk0_suffix) for p in DownloadHandler.request_paths), "valid chunk was re-downloaded"
for i, expected in enumerate(CHUNK_BODIES):
with open(get_chunk_name(base_path, i, len(CHUNK_BODIES)), 'rb') as f:
assert f.read() == expected
assert os.path.isfile(get_manifest_path(base_path))
assert min(self.reported) >= (1 / len(CHUNK_BODIES)) * 100 - 1, "progress must not restart below the resumed share"
self.run_with_server(body)
def test_verify_reports_valid_fraction_then_cached(self):
"""A fully cached bundle publishes climbing verify progress and ends cached."""
def body():
artifact = self.make_artifact(chunked=True)
base_path = os.path.join(self.dest, artifact.fileName)
for i, data in enumerate(CHUNK_BODIES):
with open(get_chunk_name(base_path, i, len(CHUNK_BODIES)), 'wb') as f:
f.write(data)
asyncio.run(self.manager._process_artifact(artifact, self.dest))
assert DownloadHandler.request_paths == [], "cached bundle must not hit the network"
assert [round(p) for p in self.reported[:3]] == [33, 67, 100]
assert artifact.downloadProgress.status == custom.ModelManagerSP.DownloadStatus.cached
self.run_with_server(body)
def _make_params_with_store(self):
params = mock.MagicMock()
store = {}
def get(key, *args, **kwargs):
return store.get(key, b"0") # b"0" -> download not cancelled
def put(key, value, *args, **kwargs):
store[key] = value
params.get.side_effect = get
params.put.side_effect = put
return params, store
def test_download_writes_qcom_slot(self):
"""A download resolved to the qcom source writes the qcom active bundle slot only."""
def body():
artifact = self.make_artifact(chunked=True)
self._bundle.ref = "test-ref"
self._bundle.minimumSelectorVersion = helpers.REQUIRED_JSON_VERSION
params, store = self._make_params_with_store()
self.manager.params = params
asyncio.run(self.manager._download_bundle(self._bundle, self.dest, "qcom"))
assert "ModelManager_ActiveBundle" in store, "qcom download must write the qcom slot"
assert "ModelManager_ActiveBundleChestnut" not in store, "qcom download must not touch the chestnut slot"
assert self.manager.selected_bundle.status == custom.ModelManagerSP.DownloadStatus.downloaded
assert self.manager.active_bundle is not None and self.manager.active_bundle.ref == "test-ref"
assert self.manager.active_bundle.status == custom.ModelManagerSP.DownloadStatus.downloaded
chunk_names = [get_chunk_name(artifact.fileName, i, len(artifact.chunks)) for i in range(len(artifact.chunks))]
missing = [c for c in chunk_names if not os.path.isfile(os.path.join(self.dest, c))]
assert missing == [], f"chunks missing from the cache: {missing}"
self.run_with_server(body)
def test_download_writes_chestnut_slot(self):
"""A download resolved to the chestnut source writes the chestnut active bundle slot only."""
def body():
self.make_artifact(chunked=True)
self._bundle.ref = "big-ref"
self._bundle.minimumSelectorVersion = helpers.REQUIRED_JSON_VERSION
params, store = self._make_params_with_store()
self.manager.params = params
asyncio.run(self.manager._download_bundle(self._bundle, self.dest, "chestnut"))
assert "ModelManager_ActiveBundleChestnut" in store, "chestnut download must write the chestnut slot"
assert "ModelManager_ActiveBundle" not in store, "chestnut download must not touch the qcom slot"
assert self.manager.selected_bundle.status == custom.ModelManagerSP.DownloadStatus.downloaded
self.run_with_server(body)
class TestManagerImports(OpenpilotTestCase):
"""Catches undeclared dependencies. aiohttp lived only in the AGNOS venv; 19.6 dropped
@@ -267,6 +434,352 @@ class TestManagerImports(OpenpilotTestCase):
assert connect > 0 and read > 0, "requests defaults to no timeout; downloads would hang forever"
class TestResolveBundleByRef(OpenpilotTestCase):
"""A ref resolves to (bundle, source) across both hardware manifests. Refs are
unique per manifest and never overlap across sources, so a ref maps to exactly
one slot. Shared by the manager's download flow and the settings UI."""
@staticmethod
def _bundle(ref: str):
bundle = custom.ModelManagerSP.ModelBundle.new_message()
bundle.ref = ref
return bundle
def test_qcom_ref_resolves_to_qcom_slot(self):
small = self._bundle("small")
assert resolve_bundle_by_ref("small", {"qcom": [small], "chestnut": []}) == (small, "qcom")
def test_chestnut_ref_resolves_to_chestnut_slot(self):
big = self._bundle("big")
assert resolve_bundle_by_ref("big", {"qcom": [], "chestnut": [big]}) == (big, "chestnut")
def test_unknown_ref_returns_none(self):
source_bundles = {"qcom": [self._bundle("small")], "chestnut": []}
assert resolve_bundle_by_ref("nope", source_bundles) is None
def manifest_bundle(short_name: str, ref: str, index: int = 0, is_big: bool = False) -> dict:
"""Minimal manifest bundle dict, version-compatible (no chunks to avoid disk side effects).
Big (chestnut) bundles carry `is_big: true` in the manifest JSON."""
return {
"index": index,
"short_name": short_name,
"display_name": short_name.upper(),
"generation": 1,
"environment": "release",
"runner": "tinygrad",
"is_big": is_big,
"minimum_selector_version": str(helpers.REQUIRED_JSON_VERSION),
"ref": ref,
"models": [{
"type": "supercombo",
"artifact": {
"file_name": f"{short_name}.pkl",
"download_uri": {"url": f"https://example.com/{short_name}.pkl", "sha256": "s"},
},
}],
}
def fresh_sync_time() -> int:
return int(time.monotonic() * 1e9)
class TestModelFetcherSources(OpenpilotTestCase):
"""Both manifests are always maintained: get_bundles_for_source exposes either
source by name, and active_source picks which one matches the attached hardware."""
def _make_params(self, qcom_manifest, chestnut_manifest):
params = mock.MagicMock()
def get(key):
if key == "ModelManager_ModelsCache":
return qcom_manifest
if key == "ModelManager_ModelsCache_Chestnut":
return chestnut_manifest
if key in ("ModelManager_LastSyncTime", "ModelManager_LastSyncTime_Chestnut"):
return fresh_sync_time()
return None
params.get.side_effect = get
return params
def test_active_source_follows_chestnut_presence(self):
assert ModelFetcher.active_source(False) == "qcom"
assert ModelFetcher.active_source(True) == "chestnut"
def test_get_bundles_for_source_returns_each_source(self):
params = self._make_params({"bundles": [manifest_bundle("small", "aaa")]},
{"bundles": [manifest_bundle("big", "bbb", is_big=True)]})
fetcher = ModelFetcher(params)
assert [bundle.ref for bundle in fetcher.get_bundles_for_source("qcom")] == ["aaa"]
assert [bundle.ref for bundle in fetcher.get_bundles_for_source("chestnut")] == ["bbb"]
def test_get_bundles_for_source_unknown(self):
assert ModelFetcher(mock.MagicMock()).get_bundles_for_source("bogus") == []
def test_get_cached_bundles_parses_source(self):
params = self._make_params({"bundles": [manifest_bundle("small", "aaa")]},
{"bundles": [manifest_bundle("big", "bbb", is_big=True)]})
qcom_bundles = get_cached_bundles(params, "qcom")
chestnut_bundles = get_cached_bundles(params, "chestnut")
assert [b.ref for b in qcom_bundles] == ["aaa"]
assert [b.ref for b in chestnut_bundles] == ["bbb"]
assert qcom_bundles[0].displayName == "SMALL"
def test_get_cached_bundles_empty_when_missing(self):
params = mock.MagicMock()
params.get.return_value = None
assert get_cached_bundles(params, "qcom") == []
assert get_cached_bundles(params, "chestnut") == []
def test_get_cached_bundles_unknown_source(self):
assert get_cached_bundles(mock.MagicMock(), "bogus") == []
def test_active_json_has_both_urls(self):
params = mock.MagicMock()
ModelFetcher(params)
active_json_calls = [call for call in params.put.call_args_list if call.args[0] == "ModelManager_ActiveJson"]
assert active_json_calls, "expected ModelManager_ActiveJson to be written"
assert active_json_calls[-1].args[1] == {
"qcom": ModelFetcher.MODEL_URL,
"chestnut": ModelFetcher.MODEL_URL_CHESTNUT,
}
class TestSourceCacheIntegrity(OpenpilotTestCase):
"""Each source's cached manifest must contain only that source's models; the
`is_big` flag in the JSON marks the big (chestnut) models. A mismatched cache is
legacy data from before the per-source split (the active manifest was cached
under the unsuffixed key regardless of hardware) and is refetched. This
replaces the old one-time bundle migration."""
def _make_params(self, qcom_manifest, chestnut_manifest):
params = mock.MagicMock()
def get(key):
if key == "ModelManager_ModelsCache":
return qcom_manifest
if key == "ModelManager_ModelsCache_Chestnut":
return chestnut_manifest
if key in ("ModelManager_LastSyncTime", "ModelManager_LastSyncTime_Chestnut"):
return fresh_sync_time()
return None
params.get.side_effect = get
return params
def _fetched(self, *bundles):
return ModelFetcher(mock.MagicMock()).model_parser.parse_models({"bundles": list(bundles)})
def test_qcom_cache_with_big_models_is_refetched(self):
"""Legacy: the unsuffixed cache holds the big manifest. is_big confirms it is
the wrong set for qcom, so a fresh fetch replaces it."""
params = self._make_params({"bundles": [manifest_bundle("big", "bbb", is_big=True)]},
{"bundles": [manifest_bundle("big2", "ccc", is_big=True)]})
fetcher = ModelFetcher(params)
fetched = self._fetched(manifest_bundle("small", "aaa"))
with mock.patch.object(fetcher, "_fetch_and_cache_models", return_value=fetched):
bundles = fetcher.get_bundles_for_source("qcom")
assert [bundle.ref for bundle in bundles] == ["aaa"]
def test_chestnut_cache_without_big_models_is_refetched(self):
params = self._make_params({"bundles": [manifest_bundle("small", "aaa")]},
{"bundles": [manifest_bundle("big2", "ccc")]})
fetcher = ModelFetcher(params)
fetched = self._fetched(manifest_bundle("big", "bbb", is_big=True))
with mock.patch.object(fetcher, "_fetch_and_cache_models", return_value=fetched):
bundles = fetcher.get_bundles_for_source("chestnut")
assert [bundle.ref for bundle in bundles] == ["bbb"]
def test_matching_caches_are_used_without_fetch(self):
params = self._make_params({"bundles": [manifest_bundle("small", "aaa")]},
{"bundles": [manifest_bundle("big", "bbb", is_big=True)]})
fetcher = ModelFetcher(params)
with mock.patch.object(fetcher, "_fetch_and_cache_models", side_effect=AssertionError("cache should be used")):
assert [bundle.ref for bundle in fetcher.get_bundles_for_source("qcom")] == ["aaa"]
assert [bundle.ref for bundle in fetcher.get_bundles_for_source("chestnut")] == ["bbb"]
def test_stale_version_cache_is_refetched(self):
"""A source-matching cache whose bundles are all filtered by the selector version
check parses to zero valid bundles; it is stale (e.g. an old manifest) and must be
refetched instead of silently returning an empty list forever."""
stale = manifest_bundle("small", "aaa")
stale["minimum_selector_version"] = "16"
params = self._make_params({"bundles": [stale]},
{"bundles": [manifest_bundle("big", "bbb", is_big=True)]})
fetcher = ModelFetcher(params)
fetched = self._fetched(manifest_bundle("small2", "ddd"))
with mock.patch.object(fetcher, "_fetch_and_cache_models", return_value=fetched) as fetch:
bundles = fetcher.get_bundles_for_source("qcom")
fetch.assert_called_once_with("qcom")
assert [bundle.ref for bundle in bundles] == ["ddd"]
def test_mismatched_refetch_happens_once(self):
"""If the fresh manifest still fails the source check, the URL is authoritative:
trust it instead of refetching at 1 Hz forever."""
params = self._make_params({"bundles": [manifest_bundle("big", "bbb", is_big=True)]},
{"bundles": [manifest_bundle("big2", "ccc", is_big=True)]})
fetcher = ModelFetcher(params)
fetched = self._fetched(manifest_bundle("big", "bbb", is_big=True))
with mock.patch.object(fetcher, "_fetch_and_cache_models", return_value=fetched) as fetch:
first = fetcher.get_bundles_for_source("qcom")
second = fetcher.get_bundles_for_source("qcom")
fetch.assert_called_once_with("qcom")
assert [bundle.ref for bundle in first] == ["bbb"]
assert [bundle.ref for bundle in second] == ["bbb"]
def test_corrupt_cache_is_refetched(self):
"""A cache that fails to parse (e.g. truncated/foreign JSON) must trigger a
refetch instead of raising every loop and never recovering."""
corrupt = {"bundles": [{"short_name": "broken"}]} # missing required fields
params = self._make_params(corrupt, {"bundles": [manifest_bundle("big", "bbb", is_big=True)]})
fetcher = ModelFetcher(params)
fetched = self._fetched(manifest_bundle("small", "aaa"))
with mock.patch.object(fetcher, "_fetch_and_cache_models", return_value=fetched) as fetch:
bundles = fetcher.get_bundles_for_source("qcom")
fetch.assert_called_once_with("qcom")
assert [bundle.ref for bundle in bundles] == ["aaa"]
class TestActiveBundleValidation(OpenpilotTestCase):
"""Validation is per-slot: a failed fetch (empty bundle list) must not reset a slot,
and resetting one slot must not stomp the runner cache derived from the other."""
def setUp(self):
super().setUp()
helpers._LAST_VALIDATED_RAW.clear()
@staticmethod
def _raw_bundle(ref: str, runner: int | None = None) -> dict:
bundle = custom.ModelManagerSP.ModelBundle.new_message()
bundle.ref = ref
bundle.minimumSelectorVersion = helpers.REQUIRED_JSON_VERSION
if runner is not None:
bundle.runner = runner
return bundle.to_dict()
def _params(self, qcom=None, chestnut=None):
params = mock.MagicMock()
def get(key, *args, **kwargs):
return {"ModelManager_ActiveBundle": qcom, "ModelManager_ActiveBundleChestnut": chestnut}.get(key)
params.get.side_effect = get
return params
def test_empty_catalog_does_not_reset_slot(self):
params = self._params(qcom=self._raw_bundle("small"))
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=False):
validate_active_bundles(params, {"qcom": [], "chestnut": []})
params.remove.assert_not_called()
def test_reset_recomputes_runner_from_surviving_slot(self):
tinygrad = int(custom.ModelManagerSP.Runner.tinygrad)
big_raw = self._raw_bundle("big", runner=tinygrad)
params = self._params(qcom=self._raw_bundle("gone"), chestnut=big_raw)
catalog = {"qcom": [custom.ModelManagerSP.ModelBundle(**self._raw_bundle("other"))],
"chestnut": [custom.ModelManagerSP.ModelBundle(**big_raw)]}
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=True):
validate_active_bundles(params, catalog)
params.remove.assert_called_once_with("ModelManager_ActiveBundle")
runner_puts = [call for call in params.put.call_args_list if call.args[0] == "ModelRunnerTypeCache"]
assert [call.args[1] for call in runner_puts] == [tinygrad]
class TestActiveBundleSelection(OpenpilotTestCase):
"""The effective active bundle is the active source's slot: chestnut when a GPU is
present, qcom otherwise. An empty active slot means the hardware default (stock
runner), never the other slot's pick - modeld_v2 requires a real bundle."""
@staticmethod
def _raw_bundle(ref: str) -> dict:
bundle = custom.ModelManagerSP.ModelBundle.new_message()
bundle.ref = ref
bundle.minimumSelectorVersion = helpers.REQUIRED_JSON_VERSION
return bundle.to_dict()
def _params(self, qcom=None, chestnut=None):
params = mock.MagicMock()
def get(key, *args, **kwargs):
if key == "ModelManager_ActiveBundle":
return qcom
if key == "ModelManager_ActiveBundleChestnut":
return chestnut
return None
params.get.side_effect = get
return params
def test_selected_bundle_is_per_slot(self):
params = self._params(qcom=self._raw_bundle("small"), chestnut=self._raw_bundle("big"))
assert get_selected_bundle(params, "qcom").ref == "small"
assert get_selected_bundle(params, "chestnut").ref == "big"
def test_no_gpu_uses_qcom_slot(self):
params = self._params(qcom=self._raw_bundle("small"), chestnut=self._raw_bundle("big"))
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=False):
assert get_active_bundle(params).ref == "small"
def test_gpu_uses_chestnut_slot(self):
params = self._params(qcom=self._raw_bundle("small"), chestnut=self._raw_bundle("big"))
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=True):
assert get_active_bundle(params).ref == "big"
def test_gpu_without_big_selection_is_hardware_default(self):
params = self._params(qcom=self._raw_bundle("small"), chestnut=None)
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=True):
assert get_active_bundle(params) is None
class TestEffectiveSource(OpenpilotTestCase):
"""One gate decides the active source. With no flags it is runtime truth (GPU
attached); display callers (mici) pass the ui_state flags, which additionally
require the big model to be loading, active, or the device offroad. The active
bundle is simply the selected bundle of that source."""
@staticmethod
def _raw_bundle(ref: str) -> dict:
bundle = custom.ModelManagerSP.ModelBundle.new_message()
bundle.ref = ref
bundle.minimumSelectorVersion = helpers.REQUIRED_JSON_VERSION
return bundle.to_dict()
def test_runtime_no_gpu(self):
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=False):
assert get_active_source() == "qcom"
def test_runtime_gpu_present(self):
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=True):
assert get_active_source() == "chestnut"
def test_display_offroad_gpu_present_shows_big(self):
assert get_active_source(chestnut=True, chestnut_active=False, chestnut_loading=False, offroad=True) == "chestnut"
def test_display_onroad_gpu_loading_shows_big(self):
assert get_active_source(chestnut=True, chestnut_active=False, chestnut_loading=True, offroad=False) == "chestnut"
def test_display_onroad_gpu_active_shows_big(self):
assert get_active_source(chestnut=True, chestnut_active=True, chestnut_loading=False, offroad=False) == "chestnut"
def test_display_onroad_gpu_idle_shows_small(self):
assert get_active_source(chestnut=True, chestnut_active=False, chestnut_loading=False, offroad=False) == "qcom"
def test_display_active_none_is_idle(self):
assert get_active_source(chestnut=True, chestnut_active=None, chestnut_loading=False, offroad=False) == "qcom"
def test_active_bundle_follows_source(self):
params = mock.MagicMock()
params.get.side_effect = lambda key: {"ModelManager_ActiveBundle": self._raw_bundle("small"),
"ModelManager_ActiveBundleChestnut": self._raw_bundle("big")}.get(key)
with mock.patch("openpilot.sunnypilot.models.helpers.chestnut_present", return_value=False):
assert get_active_bundle(params).ref == "small"
assert get_selected_bundle(params, get_active_source(chestnut=True, chestnut_active=False,
chestnut_loading=False, offroad=True)).ref == "big"
@unittest.skipUnless(os.environ.get('RUN_INTEGRATION_TESTS'), 'requires external network')
class TestLiveModelManifest(OpenpilotTestCase):
"""Every artifact and chunk URL in the published manifest must resolve."""
@@ -1,13 +1,11 @@
import requests
from openpilot.common.params import Params
from openpilot.sunnypilot.models.tinygrad_ref import get_tinygrad_ref
from openpilot.sunnypilot.models.fetcher import ModelFetcher
from openpilot.common.test import OpenpilotTestCase
def fetch_tinygrad_ref():
fetcher = ModelFetcher(Params())
response = requests.get(fetcher.model_url, timeout=10)
response = requests.get(ModelFetcher.MODEL_URL, timeout=10)
response.raise_for_status()
json_data = response.json()
return json_data.get("tinygrad_ref")
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:447099e93e303b29e7b3eac237bb0f27f8c5e12786991139aee2432532a75f58
size 12310
@@ -83,9 +83,14 @@ class TestLocationdProc(OpenpilotTestCase):
self.pm.send(msg.which(), msg)
if msg.which() == "cameraOdometry":
self.pm.wait_for_readers_to_update(msg.which(), timeout=1, dt=0.005)
time.sleep(1) # wait for async params write
for _ in range(50):
val = self.params.get('LastGPSPositionLLK')
if val is not None:
break
time.sleep(0.1)
lastGPS = json.loads(self.params.get('LastGPSPositionLLK'))
self.assertIsNotNone(val, "LastGPSPositionLLK not written within 5s")
lastGPS = json.loads(val)
self.assertAlmostEqual(lastGPS['latitude'], self.lat, delta=0.001)
self.assertAlmostEqual(lastGPS['longitude'], self.lon, delta=0.001)
self.assertAlmostEqual(lastGPS['altitude'], self.alt, delta=0.001)
@@ -252,4 +252,12 @@ EVENTS_SP: dict[int, dict[str, Alert | AlertCallbackType]] = {
AlertStatus.userPrompt, AlertSize.small,
Priority.LOW, VisualAlert.none, AudibleAlert.prompt, 0.1),
},
EventNameSP.bigModelReady: {
ET.PERMANENT: Alert(
"Big Model Ready",
"",
AlertStatus.normal, AlertSize.small,
Priority.LOW, VisualAlert.none, AudibleAlert.prompt, 2.),
},
}
@@ -28,7 +28,7 @@ from websocket import (ABNF, WebSocket, WebSocketException, WebSocketTimeoutExce
create_connection, WebSocketConnectionClosedException)
import openpilot.cereal.messaging as messaging
from openpilot.sunnypilot.models.default_model import DEFAULT_MODEL
from openpilot.sunnypilot.models.model_name import DEFAULT_MODEL, DEFAULT_BIG_MODEL
from openpilot.sunnypilot.selfdrive.car.sync_sunnylink_params import update_car_list_param
from openpilot.sunnypilot.sunnylink.api import SunnylinkApi
from openpilot.sunnypilot.sunnylink.utils import sunnylink_need_register, sunnylink_ready, get_param_as_byte, save_param_from_base64_encoded_string
@@ -182,6 +182,8 @@ def getParamsMetadata() -> str:
schema["capabilities"] = generate_capabilities()
schema["capability_labels"] = CAPABILITY_LABELS
schema["default_model"] = DEFAULT_MODEL
schema["default_big_model"] = DEFAULT_BIG_MODEL
schema["chestnut_active"] = params.get_bool("ChestnutActive")
raw = json.dumps(schema, separators=(",", ":")).encode("utf-8")
return base64.b64encode(gzip.compress(raw)).decode("utf-8")
except Exception:
+1
View File
@@ -65,6 +65,7 @@ def sp_stats(end_event):
'MadsSteeringMode',
'MadsUnifiedEngagementMode',
'ModelManager_ActiveBundle',
'ModelManager_ActiveBundleChestnut',
'ModelManager_Favs',
'EnableSunnylinkUploader',
'SunnylinkEnabled',
@@ -84,6 +84,25 @@ def _migrate_tesla_mads_screen_button(_params):
cloudlog.exception(f"Error migrating TeslaMadsScreenButton: {e}")
def _migrate_model_bundle_slots(_params):
# Pre-split, a chestnut user's big-model selection lived in the single
# ActiveBundle. Seed both slots; validation drops whichever does not match
# its own manifest.
try:
if _params.get("ModelManager_ActiveBundleChestnut") is not None:
return
if (chestnut_bundle := _params.get("ModelManager_ActiveBundleUSBGPU")) is not None:
_params.put("ModelManager_ActiveBundleChestnut", chestnut_bundle, block=True)
cloudlog.info("params_migration: seeded ModelManager_ActiveBundleChestnut from ModelManager_ActiveBundleUSBGPU")
return
if (bundle := _params.get("ModelManager_ActiveBundle")) is None:
return
_params.put("ModelManager_ActiveBundleChestnut", bundle, block=True)
cloudlog.info("params_migration: seeded ModelManager_ActiveBundleChestnut from ModelManager_ActiveBundle")
except Exception as e:
cloudlog.exception(f"Error migrating model bundle slots: {e}")
def run_migration(_params):
# migrate OnroadScreenOffBrightness
if _params.get("OnroadScreenOffBrightnessMigrated") != ONROAD_BRIGHTNESS_MIGRATION_VERSION:
@@ -120,3 +139,6 @@ def run_migration(_params):
# seed TeslaMadsScreenButton for existing Tesla installs
_migrate_tesla_mads_screen_button(_params)
# seed the chestnut model slot from the pre-split single slot
_migrate_model_bundle_slots(_params)
@@ -0,0 +1,36 @@
"""
Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
from openpilot.common.params import Params
from openpilot.common.test import OpenpilotTestCase
from openpilot.sunnypilot.system.params_migration import _migrate_model_bundle_slots
class TestModelBundleSlotMigration(OpenpilotTestCase):
"""Pre-split, a chestnut user's big-model selection lived in the single ActiveBundle.
The migration seeds both slots; per-source validation later drops whichever does not
match its own manifest."""
def test_seeds_chestnut_slot_from_active_bundle(self):
params = Params()
bundle = {"ref": "big", "minimumSelectorVersion": 18}
params.put("ModelManager_ActiveBundle", bundle, block=True)
_migrate_model_bundle_slots(params)
assert params.get("ModelManager_ActiveBundleChestnut") == bundle
assert params.get("ModelManager_ActiveBundle") == bundle
def test_noop_when_chestnut_slot_already_set(self):
params = Params()
params.put("ModelManager_ActiveBundle", {"ref": "small"}, block=True)
params.put("ModelManager_ActiveBundleChestnut", {"ref": "big"}, block=True)
_migrate_model_bundle_slots(params)
assert params.get("ModelManager_ActiveBundleChestnut") == {"ref": "big"}
def test_noop_when_no_selection(self):
params = Params()
_migrate_model_bundle_slots(params)
assert params.get("ModelManager_ActiveBundleChestnut") is None
+6 -4
View File
@@ -828,20 +828,22 @@ def startStream(sdp: str, enabled: bool) -> dict:
bridge_services_in = []
# stale car params case taken care of by webrtcd being shut off on ignition
cp_bytes = Params().get("CarParamsPersistent")
cp_bytes = params.get("CarParamsPersistent")
if cp_bytes is not None:
with car.CarParams.from_bytes(cp_bytes) as CP:
if CP.notCar:
bridge_services_in.append("testJoystick")
else:
raise Exception("failed to get CarParamsPersistent")
if params.get_bool("IsOffroad"):
# manager owns camerad/stream_encoderd/webrtcd; flip the param and let it bring them up.
# webrtcd clears IsLiveStreaming when the session ends
params.put_bool("IsLiveStreaming", True)
# wait for webrtcd end points to wake up
wait_for_webrtcd()
try:
wait_for_webrtcd()
except TimeoutError:
cloudlog.event("athena.startStream.webrtcd_offroad_start_timeout", error=True)
raise
return post_stream_request(StreamRequestBody(sdp, ["wideRoad"], enabled, bridge_services_in, ["carState", "deviceState"]))
@@ -0,0 +1,101 @@
import time
from openpilot.common.hardware.usb import CHESTNUT_USB_PRODUCT, is_chestnut_usb_id
from openpilot.common.version import get_build_metadata, CHESTNUT_BRANCHES
from openpilot.selfdrive.modeld.helpers import MODELS_DIR, chestnut_compiled
CHESTNUT_RELEASE_BRANCHES = ("release-chestnut", "release-chestnut-staging")
CHESTNUT_POWERED_VOLTAGE = 5000
GPU_TEMP_LIMIT = 100.
MEMORY_TEMP_LIMIT = 95.
TEMP_HYSTERESIS = 5.
class ChestnutStatus:
def __init__(self):
self.started = time.monotonic()
self.offroad = True
self.pcie_failed = False
self.power_seen = False
self.power_unavailable = False
self.power_lost = False
self.power_restored = False
self.link_failures = 0
self.model_loading_seen = False
self.model_attempted = False
self.overheated = False
self.usb_seen = False
self.usb_failed = False
def update(self, offroad: bool, branch: str, usb_state: list[dict], firmware_failed: bool,
model_loading: bool, model_active: bool | None, state, set_alert) -> None:
detected = [d for d in usb_state if is_chestnut_usb_id(d["vendorId"], d["productId"], include_bootloader=True)]
devices = [d for d in detected if is_chestnut_usb_id(d["vendorId"], d["productId"])]
firmware_ok = len(devices) == 1 and devices[0]["product"] == CHESTNUT_USB_PRODUCT
if self.offroad and not offroad:
self.pcie_failed = False
self.power_seen = False
self.power_unavailable = False
self.power_lost = False
self.power_restored = False
self.link_failures = 0
self.model_loading_seen = False
self.model_attempted = False
self.usb_seen = firmware_ok
self.usb_failed = False
self.model_loading_seen |= model_loading
self.model_attempted |= self.model_loading_seen and not model_loading and model_active is not None
if not offroad and self.usb_seen and not firmware_ok:
self.usb_failed = True
if not offroad and state is not None:
powered = state.supplyVoltage >= CHESTNUT_POWERED_VOLTAGE
power_lost = state.supplyFault or not powered
if self.model_attempted and power_lost and not self.power_lost:
self.power_unavailable = not self.power_seen
self.power_seen |= powered
if not offroad and self.model_attempted and state is not None:
self.link_failures = self.link_failures + 1 if state.pcieLtssm != 0x78 else 0
self.pcie_failed |= self.link_failures >= 2 or power_lost
self.power_lost |= power_lost
if self.pcie_failed and self.power_lost and state is not None:
self.power_restored |= not state.supplyFault and state.supplyVoltage >= CHESTNUT_POWERED_VOLTAGE
if self.usb_failed:
self.pcie_failed = False
self.power_seen = False
self.power_unavailable = False
self.power_lost = False
self.power_restored = False
if state is not None:
gpu_limit = GPU_TEMP_LIMIT - (TEMP_HYSTERESIS if self.overheated else 0.)
memory_limit = MEMORY_TEMP_LIMIT - (TEMP_HYSTERESIS if self.overheated else 0.)
self.overheated = state.tempC >= gpu_limit or state.memoryTempC >= memory_limit
release = branch in CHESTNUT_RELEASE_BRANCHES
missing = self.usb_failed or (offroad and release and time.monotonic() - self.started > 10. and len(detected) != 1)
slow_usb = offroad and len(devices) == 1 and devices[0]["speedMbps"] < 5000
big_model_available = (MODELS_DIR / 'big_driving_supercombo.onnx').is_file() or chestnut_compiled()
current_channel = get_build_metadata().channel
chestnut_target = CHESTNUT_BRANCHES.get(current_channel)
chestnut_needs_switch = len(devices) == 1 and not big_model_available and chestnut_target is not None
set_alert("Offroad_ChestnutBranch", chestnut_needs_switch, chestnut_target if chestnut_needs_switch else None)
set_alert("Offroad_ChestnutNotDetected", missing)
set_alert("Offroad_ChestnutOverheated", self.overheated, f"{state.tempC:.0f} °C" if state is not None else None)
set_alert("Offroad_ChestnutUsbSlow", slow_usb, f"{devices[0]['speedMbps']} Mbps" if slow_usb else None)
if self.power_lost:
pcie_alert = ("Chestnut power restored. 12V is stable again, cycle ignition." if self.power_restored else
"Chestnut power disconnected. Check 12V connection, then cycle ignition." if self.power_unavailable else
"Chestnut power lost. Possibly caused by an engine-crank voltage drop. Check 12V connection, then cycle ignition.")
else:
pcie_alert = "Chestnut GPU unavailable. PCIe link is not up. Check the GPU is securely seated."
set_alert("Offroad_ChestnutPcieUnavailable", self.pcie_failed, pcie_alert)
set_alert("Offroad_ChestnutUncompiled", offroad and firmware_ok and not chestnut_compiled())
set_alert("Offroad_ChestnutUpdateFailed", offroad and firmware_failed)
self.offroad = offroad
+19 -10
View File
@@ -16,20 +16,20 @@ from openpilot.common.utils import strip_deprecated_keys
from openpilot.common.filter_simple import FirstOrderFilter
from openpilot.common.params import Params
from openpilot.common.realtime import DT_HW
from openpilot.selfdrive.modeld.helpers import MODELS_DIR, usbgpu_compiled
from openpilot.selfdrive.selfdrived.alertmanager import set_offroad_alert
from openpilot.common.hardware import HARDWARE, COMMA_HARDWARE
from openpilot.common.basedir import BASEDIR
from openpilot.common.hardware.usb import CHESTNUT_FW_VERSION, CHESTNUT_ROM_USB_IDS, CHESTNUT_USB_IDS, get_usb_state, get_usb_topology, set_usb_state
from openpilot.common.git import get_short_branch
from openpilot.common.hardware.usb import CHESTNUT_FW_VERSION, CHESTNUT_USB_PRODUCT, get_usb_state, get_usb_topology, is_chestnut_usb_id, set_usb_state
from openpilot.common.linux import LinuxSystemStats
from openpilot.system.loggerd.config import get_available_percent
from openpilot.common.swaglog import cloudlog
from openpilot.sunnypilot.system.statsd import statlog
from openpilot.system.hardware.power_monitoring import PowerMonitoring
from openpilot.system.hardware.fan_controller import FanController
from openpilot.system.hardware.chestnut.status import ChestnutStatus
from openpilot.common.version import terms_version, training_version, get_build_metadata, terms_version_sp
ThermalStatus = log.DeviceState.ThermalStatus
NetworkType = log.DeviceState.NetworkType
NetworkStrength = log.DeviceState.NetworkStrength
@@ -49,6 +49,11 @@ class Chestnut:
self.attempts = 0
self.last_attempt = 0.
self.flashed = False
self.mismatch = False
@property
def failed(self) -> bool:
return self.mismatch and self.attempts >= self.MAX_ATTEMPTS and self.thread is not None and not self.thread.is_alive() and not self.flashed
def flash(self) -> None:
ret = subprocess.run(["sudo", sys.executable, os.path.join(BASEDIR, "openpilot/system/hardware/chestnut/flash.py"), CHESTNUT_FW_VERSION],
@@ -57,9 +62,9 @@ class Chestnut:
self.flashed = ret.returncode == 0
def update(self, offroad: bool, usb_state: list[dict]) -> None:
mismatch = any((d["vendorId"], d["productId"]) in CHESTNUT_USB_IDS + CHESTNUT_ROM_USB_IDS and
d["product"] != f"custom {CHESTNUT_FW_VERSION}-CLEAN" for d in usb_state)
if not mismatch:
self.mismatch = any(is_chestnut_usb_id(d["vendorId"], d["productId"], include_bootloader=True) and
d["product"] != CHESTNUT_USB_PRODUCT for d in usb_state)
if not self.mismatch:
self.flashed = False
return
@@ -191,7 +196,7 @@ def hw_state_thread(end_event, hw_queue):
def hardware_thread(end_event, hw_queue) -> None:
system_stats = LinuxSystemStats()
pm = messaging.PubMaster(['deviceState'])
sm = messaging.SubMaster(["peripheralState", "gpsLocationExternal", "selfdriveState", "pandaStates"], poll="pandaStates")
sm = messaging.SubMaster(["peripheralState", "gpsLocationExternal", "selfdriveState", "pandaStates", "chestnutState"], poll="pandaStates")
count = 0
@@ -239,7 +244,8 @@ def hardware_thread(end_event, hw_queue) -> None:
fan_controller = FanController(int(1./DT_HW))
chestnut = Chestnut()
big_model_available = (MODELS_DIR / 'big_driving_supercombo.onnx').is_file() or usbgpu_compiled()
chestnut_status = ChestnutStatus()
branch = get_short_branch()
while not end_event.is_set():
sm.update(PANDA_STATES_TIMEOUT)
@@ -301,8 +307,11 @@ def hardware_thread(end_event, hw_queue) -> None:
set_usb_state(msg.deviceState, last_hw_state.usb_state)
chestnut.update(started_ts is None, last_hw_state.usb_state)
set_offroad_alert_if_changed("Offroad_ChestnutBranch", msg.deviceState.chestnutPresent and not big_model_available)
chestnut_state = sm["chestnutState"]
chestnut_valid = sm.alive["chestnutState"] and sm.valid["chestnutState"]
chestnut_status.update(started_ts is None, branch, last_hw_state.usb_state, chestnut.failed,
params.get_bool("ChestnutLoading"), params.get("ChestnutActive"),
chestnut_state if chestnut_valid else None, set_offroad_alert_if_changed)
# this subset is only used for offroad
temp_sources = [
msg.deviceState.memoryTempC,
-40
View File
@@ -1,40 +0,0 @@
#!/usr/bin/env bash
# Define the service name
SERVICE_NAME="actions.runner.sunnypilot.$(uname -n)"
# Function to control the service
control_service() {
local action=$1 # Store the function argument in a local variable
sudo systemctl $action ${SERVICE_NAME}
}
service_exists_and_is_loaded() {
sudo systemctl status ${SERVICE_NAME} &>/dev/null
if [[ $? -ne 4 ]]; then
return 0 # Service is known to systemd (i.e., loaded)
else
return 1 # Service is unknown to systemd (i.e., not loaded)
fi
}
# Check for required argument
if [[ -z $1 ]] || { [[ $1 != "start" ]] && [[ $1 != "stop" ]]; }; then
echo "Usage: $0 {start|stop}"
exit 1
fi
# Store the script argument in a descriptive variable
ACTION=$1
# Trap EXIT signal (Ctrl+C) and stop the service
trap 'control_service stop ; exit' SIGINT SIGKILL EXIT
# Enter the main loop
while true; do
# Check if the service is actually present on the system
if service_exists_and_is_loaded; then
control_service $ACTION # Call the function with the specified action
fi
sleep 1 # Pause before the next iteration
done
+2 -13
View File
@@ -68,10 +68,6 @@ def only_offroad(started: bool, params: Params, CP: car.CarParams) -> bool:
def livestream(started: bool, params: Params, CP: car.CarParams) -> bool:
return params.get_bool("IsLiveStreaming")
def use_github_runner(started, params, CP: car.CarParams) -> bool:
return not PC and params.get_bool("EnableGithubRunner") and (
not params.get_bool("NetworkMetered") and not params.get_bool("GithubRunnerSufficientVoltage"))
def use_copyparty(started, params, CP: car.CarParams) -> bool:
return bool(params.get_bool("EnableCopyparty"))
@@ -110,15 +106,12 @@ def or_(*fns):
def and_(*fns):
return lambda *args: operator.and_(*(fn(*args) for fn in fns))
def not_(*fns):
return lambda *args: operator.not_(*(fn(*args) for fn in fns))
procs = [
DaemonProcess("manage_athenad", "openpilot.system.athena.manage_athenad", "AthenadPid"),
NativeProcess("loggerd", "openpilot/system/loggerd", ["./loggerd"], logging),
NativeProcess("encoderd", "openpilot/system/loggerd", ["./encoderd"], only_onroad),
NativeProcess("stream_encoderd", "openpilot/system/loggerd", ["./encoderd", "--stream"], or_(and_(livestream, not_(iscar)), notcar)),
NativeProcess("stream_encoderd", "openpilot/system/loggerd", ["./encoderd", "--stream"], or_(livestream, notcar)),
PythonProcess("logmessaged", "openpilot.system.logmessaged", always_run),
NativeProcess("camerad", "openpilot/system/camerad", ["./camerad"], or_(driverview, livestream), enabled=not WEBCAM),
@@ -163,7 +156,7 @@ procs = [
# debug procs
NativeProcess("bridge", "openpilot/cereal/messaging", ["./bridge"], notcar),
PythonProcess("webrtcd", "openpilot.system.webrtc.webrtcd", or_(and_(livestream, not_(iscar)), notcar)),
PythonProcess("webrtcd", "openpilot.system.webrtc.webrtcd", or_(livestream, notcar)),
PythonProcess("joystick", "openpilot.tools.joystick.joystick_control", and_(joystick, iscar)),
# sunnylink <3
@@ -189,10 +182,6 @@ procs += [
NativeProcess("locationd_llk", "openpilot/sunnypilot/selfdrive/locationd", ["./locationd"], only_onroad),
]
if os.path.exists("./github_runner.sh"):
procs += [NativeProcess("github_runner_start", "openpilot/system/manager",
["./github_runner.sh", "start"], and_(only_offroad, use_github_runner), sigkill=False)]
if os.path.exists("../../sunnypilot/sunnylink/uploader.py"):
procs += [PythonProcess("sunnylink_uploader", "openpilot.sunnypilot.sunnylink.uploader", use_sunnylink_uploader_shim)]
+13 -10
View File
@@ -13,7 +13,7 @@ import subprocess
from contextlib import contextmanager
from collections.abc import Callable
from collections import deque
from enum import StrEnum
from enum import IntEnum, StrEnum
from pathlib import Path
from typing import NamedTuple
from importlib.resources import as_file, files
@@ -118,6 +118,18 @@ class FontWeight(StrEnum):
DISPLAY = "Inter-Bold.ttf"
class TextAlignment(IntEnum):
LEFT = 0
CENTER = 1
RIGHT = 2
class TextAlignmentVertical(IntEnum):
TOP = 0
MIDDLE = 1
BOTTOM = 2
def font_fallback(font: rl.Font) -> rl.Font:
"""Use a Noto fallback for languages not covered by Inter."""
if multilang.requires_font_fallback():
@@ -335,7 +347,6 @@ class GuiApplication(GuiApplicationExt):
rl.set_target_fps(0 if OFFSCREEN or vblank_control else fps)
self._target_fps = fps
self._set_styles()
self._load_fonts()
self._patch_text_functions()
self._patch_scissor_mode()
@@ -739,14 +750,6 @@ class GuiApplication(GuiApplicationExt):
self._fonts[font_weight_file] = font
if multilang.requires_font_fallback():
self.fallback_font()
rl.gui_set_font(self._fonts[FontWeight.NORMAL])
def _set_styles(self):
rl.gui_set_style(rl.GuiControl.DEFAULT, rl.GuiControlProperty.BORDER_WIDTH, 0)
rl.gui_set_style(rl.GuiControl.DEFAULT, rl.GuiDefaultProperty.TEXT_SIZE, DEFAULT_TEXT_SIZE)
rl.gui_set_style(rl.GuiControl.DEFAULT, rl.GuiDefaultProperty.BACKGROUND_COLOR, rl.color_to_int(rl.BLACK))
rl.gui_set_style(rl.GuiControl.DEFAULT, rl.GuiControlProperty.TEXT_COLOR_NORMAL, rl.color_to_int(DEFAULT_TEXT_COLOR))
rl.gui_set_style(rl.GuiControl.DEFAULT, rl.GuiControlProperty.BASE_COLOR_NORMAL, rl.color_to_int(rl.Color(50, 50, 50, 255)))
def _patch_text_functions(self):
# Wrap pyray text APIs to apply a global text size scale so our px sizes match Qt
-19
View File
@@ -1,19 +0,0 @@
import pyray as rl
from collections.abc import Sequence
class GuiStyleContext:
def __init__(self, styles: Sequence[tuple[int, int, int]]):
"""styles is a list of tuples (control, prop, new_value)"""
self.styles = styles
self.prev_styles: list[tuple[int, int, int]] = []
def __enter__(self):
for control, prop, new_value in self.styles:
prev_value = rl.gui_get_style(control, prop)
self.prev_styles.append((control, prop, prev_value))
rl.gui_set_style(control, prop, new_value)
def __exit__(self, exc_type, exc_value, traceback):
for control, prop, prev_value in self.prev_styles:
rl.gui_set_style(control, prop, prev_value)
+6 -6
View File
@@ -18,7 +18,7 @@ from openpilot.common.realtime import config_realtime_process, set_core_affinity
from openpilot.common.swaglog import cloudlog
from openpilot.common.time_helpers import system_time_valid
from openpilot.common.utils import run_cmd
from openpilot.system.ui.lib.application import gui_app, FontWeight
from openpilot.system.ui.lib.application import gui_app, FontWeight, TextAlignment, TextAlignmentVertical
from openpilot.system.ui.lib.wifi_manager import WifiManager, ConnectStatus
from openpilot.system.ui.widgets import Widget
from openpilot.system.ui.widgets.nav_widget import NavWidget
@@ -105,8 +105,8 @@ class StartPage(Widget):
super().__init__()
self._title = UnifiedLabel("start", 64, text_color=rl.Color(255, 255, 255, int(255 * 0.9)),
font_weight=FontWeight.DISPLAY, alignment=rl.GuiTextAlignment.TEXT_ALIGN_CENTER,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_MIDDLE)
font_weight=FontWeight.DISPLAY, alignment=TextAlignment.CENTER,
alignment_vertical=TextAlignmentVertical.MIDDLE)
self._start_bg_txt = gui_app.texture("icons_mici/setup/start_button.png", 500, 224, keep_aspect_ratio=False)
self._start_bg_pressed_txt = gui_app.texture("icons_mici/setup/start_button_pressed.png", 500, 224, keep_aspect_ratio=False)
@@ -197,7 +197,7 @@ class DownloadingPage(NavWidget):
self._title_label = UnifiedLabel("downloading...", 64, text_color=rl.Color(255, 255, 255, int(255 * 0.9)),
font_weight=FontWeight.DISPLAY)
self._progress_label = UnifiedLabel("", 132, text_color=rl.Color(255, 255, 255, int(255 * 0.9 * 0.65)),
font_weight=FontWeight.ROMAN, alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_BOTTOM)
font_weight=FontWeight.ROMAN, alignment_vertical=TextAlignmentVertical.BOTTOM)
self._progress = 0
def _back_enabled(self) -> bool:
@@ -261,8 +261,8 @@ class BigPillButton(BigButton):
super().__init__(*args, **kwargs)
self._label.set_font_size(48)
self._label.set_alignment(rl.GuiTextAlignment.TEXT_ALIGN_CENTER)
self._label.set_alignment_vertical(rl.GuiTextAlignmentVertical.TEXT_ALIGN_MIDDLE)
self._label.set_alignment(TextAlignment.CENTER)
self._label.set_alignment_vertical(TextAlignmentVertical.MIDDLE)
def _load_images(self):
if self._green:
+2 -2
View File
@@ -7,7 +7,7 @@ import pyray as rl
from openpilot.common.realtime import config_realtime_process, set_core_affinity
from openpilot.common.hardware import HARDWARE, COMMA_HARDWARE
from openpilot.common.swaglog import cloudlog
from openpilot.system.ui.lib.application import gui_app, FontWeight
from openpilot.system.ui.lib.application import gui_app, FontWeight, TextAlignmentVertical
from openpilot.system.ui.widgets.nav_widget import NavWidget
from openpilot.system.ui.widgets.scroller import Scroller
from openpilot.system.ui.widgets.label import UnifiedLabel
@@ -30,7 +30,7 @@ class ProgressPage(NavWidget):
font_weight=FontWeight.DISPLAY, line_height=0.8)
self._progress_percent_label = UnifiedLabel("", 132, text_color=rl.Color(255, 255, 255, int(255 * 0.9 * 0.65)),
font_weight=FontWeight.ROMAN,
alignment_vertical=rl.GuiTextAlignmentVertical.TEXT_ALIGN_BOTTOM)
alignment_vertical=TextAlignmentVertical.BOTTOM)
def _back_enabled(self) -> bool:
return False

Some files were not shown because too many files have changed in this diff Show More