Compare commits

..

704 Commits

Author SHA1 Message Date
Dom a31cc9bb5d Verify existing ARM loader guard ABI during native qualification 2026-10-11 19:48:00 -06:00
Dom c7f14368b9 Add pinned Python 3.12 build dependency for ARM qualification 2026-10-11 19:37:40 -06:00
Dom c32912dbd6 Correct ARM qualification workflow context and ancestry checks 2026-10-11 19:30:21 -06:00
Dom 402a55b49d Qualify temporary Card receive timing on native ARM
Build-only diagnostic branch. Add bounded receive/GIL and Card phase
observations plus tests, and an isolated source-pinned ARM artifact job.
No control-policy change, device deployment, receipt refresh or main
branch update. Artifact adoption requires separate evidence review.
2026-10-11 19:23:45 -06:00
Dom e8389b0b75 Restore executable mode on selfdrived entry point
Restore the existing 100755 mode lost during the preceding publication.
The selfdrived.py source blob is unchanged.
2026-10-11 16:23:10 -06:00
Dom 13ca03f11f Respect active axes when bounding control input waits
An unchanged pause on an inactive, unrequested axis must not veto the
other axis's existing bounded input wait. Preserve new-pause vetoes,
requested/active-axis pauses, and every existing unpause admission.

The original absolute output deadline and all source/native/fault gates
remain unchanged. Add full-owner tests for LONG-only continuity, late
arrival and processing, missing/invalid input, faults, pause transitions,
and the unresolved source-cutoff-after-final-refresh boundary.

Validated 31 targeted regressions, affected production suites, lint and
independent review. Device-level dropout resolution remains unverified.
2026-10-11 16:19:32 -06:00
Dom 46dd0df949 Reuse exact Ioniq native structural admission per observation
Retain all live freshness, session, permission and original lease checks.
Reuse only successful structural validation for exact payload, timestamp
and complete CP inputs within each owner. Unsupported profiles fall back
to full validation. Add 50 invalidation and fallback regressions.

Validated affected AOL, SelfdriveD and Controls suites, unchanged Card
fixture failures, full lint/type checks and deterministic owner outputs.
Vehicle-level dropout resolution remains unverified.
2026-10-11 15:48:58 -06:00
Dom 805051be92 Reduce redundant Ioniq native admission and inventory work
Dispatch the exact Ioniq 6 fingerprint directly to its existing full qualifier instead of evaluating unrelated Hyundai profile owners first. Revalidate the Ioniq contract on every call; do not cache qualification or live authority.

When transport health obtains current_native itself, avoid immediately repeating the same structural inventory validation on that result. Explicitly supplied native observations still receive the structural check. Preserve all source/session/deadline, latest Panda configuration/health, controlsAllowed and vehicle-health vetoes.

Historical full-module comparisons identified added per-cycle feedback work after df38093f and further expansion in 1279c092. Later changes removed part of that expansion; this patch targets redundant work remaining on the current path. A matched 200-cycle Card fixture drops duplicate inventory validations from 424 to 222 with unchanged native reads and control outputs. No claim that a single introducing commit or all on-road dropouts has been causally resolved.

Validation: 694 affected AOL/Hyundai cases, 43 SelfdriveD and 47 Controls cases passed. Card had 85 passes, six skips and four separately established baseline LKAS fixture failures. Final test-only lint correction passed its 51-case suite; all 11 lint/type checks passed. Independent profile-equivalence and live-fault/deadline review completed. Broader compatibility and hosted CI are separate follow-up gates; no device or road qualification.

Only two Python runtime files and two regression test files change. No diagnostic instrumentation, arming/rearming, alert suppression, deadline changes, native artifacts, models or dependency changes. Build receipt remains unchanged; use the normal update path rather than manual Fast Update.
2026-10-11 15:25:18 -06:00
Dom d2b25f1c39 Reduce idle wheel audits and reconcile late Ioniq inputs
Skip repeated neutral media settings audits only while every gesture tracker is idle. Preserve edge, held-input, epoch, periodic and command-time qualification.

For qualified Ioniq 6 output continuity, reconcile sources arriving during car-event work before selected-source decisions. Retain earlier exact intent restrictions and queued cancellation/fault events, process samples once within the existing bound, and preserve the original absolute output deadline.

Add production-module regression coverage for late arrivals, source restrictions, GM ordering and wheel remaps. No arming/rearming, alert, freshness-threshold, native, model or dependency changes.

Validation: 101 focused cases passed; affected authority, controller, conditional, SelfdriveD/Controls and cross-brand functional checks completed. Baseline fixture/order and timing failures were separately reproduced; full aggregate, interrupted compatibility runs and native-process qualification are not claimed green. No device or road qualification. This is a scoped partial timing repair, not proof that genuine Card producer stalls are eliminated.

Build receipt and native artifacts remain unchanged. Manual Fast Update rejects the stale protected-source receipt; the normal updater path is required.
2026-10-11 13:38:02 -06:00
firestar5683 3ab4674ac1 Fix control input timing and temporary steering recovery 2026-10-11 11:51:01 -06:00
firestar5683 f858e68898 Fix AOL recovery and control input timing 2026-10-11 10:33:18 -06:00
firestar5683 568de54a9f Fix invalid vehicle message batch expectation 2026-10-11 09:31:47 -06:00
firestar5683 6788419227 Avoid rebuilding vehicle flags in the control loop 2026-10-11 09:11:19 -06:00
firestar5683 c0455630b7 Keep communication warnings within their control episode 2026-10-11 09:04:43 -06:00
firestar5683 8afc09d499 Fix control input test substitution 2026-10-11 08:55:47 -06:00
firestar5683 2da663894f Fix control input catch-up and planner scheduling 2026-10-11 08:47:17 -06:00
firestar5683 3ba9503103 Fix Honda AOL engagement and authority loss 2026-10-11 08:20:31 -06:00
firestarsdog 7fc969bf14 Add button pressed state color 2026-10-11 03:52:16 -04:00
firestar5683 74139a60ef Serialize historical wire fixtures once 2026-10-11 01:20:33 -06:00
firestar5683 0830771789 Preserve historical steering learning schema 2026-10-11 01:03:39 -06:00
firestar5683 c204f0b165 Add bounded automatic steering learning 2026-10-11 00:25:52 -06:00
Prabhaav Pillai df91c501fa build 2026-10-11 02:08:22 -04:00
firestarsdog 85341237ae Pagination Edge Gradient Cleanup 2026-10-11 02:00:38 -04:00
Prabhaav Pillai db52c131c2 Software update status handling 2026-10-11 01:51:34 -04:00
Prabhaav Pillai b099ccf37d Add support for OS update requirements and validation in updater 2026-10-11 01:51:34 -04:00
firestar5683 ec0e5fe694 Fix build job arguments 2026-10-10 23:27:00 -06:00
firestar5683 44fba6ef27 Fix AOL transport state 2026-10-10 23:11:56 -06:00
firestar5683 a8f53df7ca Restore control authority timing 2026-10-10 22:41:07 -06:00
firestarsdog 05621af29e Pagination QoL : Allow swipe to start outside panel if ends in panel 2026-10-11 00:28:37 -04:00
Prabhaav Pillai e140dc5078 Fix updater status messages and enhance update process 2026-10-11 00:22:31 -04:00
firestar5683 1a4da8bc44 Refresh Galaxy update status 2026-10-10 22:00:17 -06:00
firestarsdog 3fa8e99c02 Pagination swiping persistence across panel thresholds 2026-10-10 23:44:22 -04:00
firestar5683 48af4915f5 Share radar processing across control cores 2026-10-10 21:35:22 -06:00
firestar5683 48c9652f77 Preserve AOL intent through availability recovery 2026-10-10 21:29:56 -06:00
firestarsdog 4429499362 Move software install button to first page when visible 2026-10-10 23:10:38 -04:00
firestar5683 28d04bffb8 Preserve validated transport pauses 2026-10-10 20:57:00 -06:00
firestar5683 c058bb8bc2 Restore shared control availability 2026-10-10 20:23:43 -06:00
firestar5683 5d7c249c68 Initialize AOL test callers 2026-10-10 19:01:58 -06:00
firestar5683 cb0ade2c1a Preserve pending AOL intent 2026-10-10 18:36:56 -06:00
firestar5683 ff766ad175 Ioniq authority timing 2026-10-10 17:46:09 -06:00
firestar5683 89f4cbd1be Fresh Ioniq control state 2026-10-10 17:27:51 -06:00
firestar5683 2ac07aef16 Build runtime 2026-10-10 16:54:45 -06:00
firestar5683 086e7451af AOL communication recovery 2026-10-10 16:49:56 -06:00
firestarsdog 0669069d8d Fix C3 Panda detection by skipping unused SPI 2026-10-10 17:41:53 -04:00
firestar5683 0c1b0070f5 Build 2026-10-10 13:47:13 -06:00
firestar5683 1e988f0bdc Ford serialization fixture 2026-10-10 13:45:02 -06:00
firestar5683 4efa732259 AOL transport validation 2026-10-10 13:45:02 -06:00
firestar5683 adf0b7158b Ford control publication checks 2026-10-10 12:30:04 -06:00
firestar5683 6dd504cb51 GM test callback binding 2026-10-10 12:14:18 -06:00
firestar5683 e8aeccb03a Vehicle interface fixtures 2026-10-10 12:00:50 -06:00
firestar5683 d6b2cfe067 Ford curvature admission 2026-10-10 12:00:50 -06:00
firestar5683 4284d7e4b3 EV6 replay initial state 2026-10-10 11:55:11 -06:00
firestar5683 ae8d0f0eb7 Hyundai two-Panda startup ignition 2026-10-10 11:53:49 -06:00
firestar5683 fe9a037417 GM control publication checks 2026-10-10 11:53:45 -06:00
firestar5683 1448c89e8f Ford safety validation 2026-10-10 11:53:42 -06:00
firestar5683 fff1f5baa7 Build 2026-10-10 10:25:17 -06:00
firestar5683 75b947322f Ford longitudinal admission 2026-10-10 09:32:27 -06:00
firestar5683 1279c092b1 Ford angle steering and Panda routing 2026-10-10 07:10:07 -06:00
firestar5683 71b02a7975 Font atlas validation 2026-10-10 01:05:46 -06:00
firestar5683 a2d019c12d Build 2026-10-10 00:58:33 -06:00
firestar5683 28ed48e35e Car loop scheduling 2026-10-10 00:55:57 -06:00
firestar5683 8a0cea2bbf GM control profiles 2026-10-10 00:46:10 -06:00
firestarsdog cc6a9330d4 Speed limit indicator 2026-10-10 00:36:39 -06:00
firestar5683 eb19d343a6 CI validation fixes 2026-10-10 00:36:38 -06:00
prabhaavp 1ad418ca35 Merge pull request #211 from dirwin31:Stardust-PR-5
Improve Android Auto rendering, Galaxy hotspot access, and Allow soft Exit
2026-10-10 01:39:15 -04:00
Prabhaav Pillai 678a90f070 Unused Uniden protocol package and its tests; no runtime code referenced it.
Unused font arguments and preview state.
A headless cache flag that nothing enabled.
2026-10-10 01:37:47 -04:00
Prabhaav Pillai 8df85fa083 Updated Android Auto page to clarify Wi-Fi usage and connection requirements.
Renamed button text in app.js for better user understanding.
Enhanced Galaxy page layout and messaging for remote access and installation instructions.
Improved hotspot settings interface with clearer instructions and error handling.
Streamlined local access messaging for better user guidance.
Adjusted navigation status handling to reflect accurate network states.
Updated tests to align with changes in network status reporting and command execution.
Refined system snapshot collection to ensure proper output handling and error reporting.
2026-10-10 01:29:42 -04:00
firestar5683 683b4e51a7 Build 2026-10-09 22:45:10 -06:00
firestar5683 29a4fe5d30 Adjacent lane width parity 2026-10-09 22:45:00 -06:00
firestar5683 9ee201ea28 Build 2026-10-09 22:20:44 -06:00
firestar5683 b7435a43bc AOL clock recovery 2026-10-09 22:20:28 -06:00
firestar5683 cc181947b6 Build 2026-10-09 19:48:02 -05:00
firestar5683 8984fb8d26 Publication test fixtures 2026-10-09 19:47:47 -05:00
firestar5683 1fd07f1cd5 Intent projection recovery 2026-10-09 19:47:38 -05:00
firestar5683 34625050a9 Forced offroad display timeout 2026-10-09 18:49:02 -05:00
firestar5683 7267af7411 Build 2026-10-09 18:35:38 -05:00
firestar5683 df38093ff1 AOL transport recovery 2026-10-09 18:34:26 -05:00
firestar5683 5c337e297a Build 2026-10-09 16:41:24 -05:00
firestar5683 c0c1c0353d GM lateral recovery 2026-10-09 16:41:11 -05:00
firestar5683 ac1968a384 Build 2026-10-09 16:13:24 -05:00
firestar5683 89aa8a294e Volt steering recovery 2026-10-09 16:11:28 -05:00
firestar5683 26a3ec8fcd Build 2026-10-09 14:35:01 -05:00
firestar5683 969ce02810 AOL message pairing 2026-10-09 14:34:54 -05:00
dirwin31 71f121a68b Smaller, no arrow 2026-10-09 12:14:39 -07:00
dirwin31 0c1b81741c Improve Android Auto, Galaxy connectivity, and navigation 2026-10-09 12:02:26 -07:00
firestar5683 9ff53d39c2 Build 2026-10-09 12:43:43 -05:00
firestar5683 65a5b48c30 CEM green release 2026-10-09 12:42:52 -05:00
firestar5683 0e76d0c6f6 Build 2026-10-09 12:35:58 -05:00
firestar5683 2764593fff Replay schema contract 2026-10-09 12:35:58 -05:00
firestar5683 3fc56ef7e4 Replay speed limit display 2026-10-09 12:35:57 -05:00
firestar5683 6dbfce2fc5 Build 2026-10-09 12:19:56 -05:00
firestar5683 0e30087e9a Volt steering authority 2026-10-09 12:13:42 -05:00
firestar5683 41bc6d6704 CAN FD replay reference 2026-10-09 12:13:42 -05:00
firestar5683 6086120108 Vehicle caller checks 2026-10-09 12:13:42 -05:00
firestarsdog 77b44257a0 Old Route -> New Replay shim
added retirement instructions
2026-10-09 12:58:07 -04:00
firestar5683 139a5302e9 I6 Domi Trial 2026-10-09 11:21:56 -05:00
firestarsdog 6ddeb91d16 Large UI Flicker Pass 2 2026-10-09 11:54:06 -04:00
firestar5683 931620ff00 I6 Domi Trial 2026-10-09 10:30:13 -05:00
firestarsdog b41f3df1fb Large UI Replay Flicker Be Gone 2026-10-09 11:19:26 -04:00
firestar5683 9d46558823 Build 2026-10-09 09:08:49 -05:00
firestar5683 5be29a9ad0 CAN FD steering 2026-10-09 09:05:37 -05:00
firestar5683 fc3290c1f1 Build 2026-10-09 08:42:06 -05:00
firestar5683 45de4e2929 Lateral authority 2026-10-09 08:36:42 -05:00
firestar5683 86e3002cb2 Build 2026-10-09 06:19:42 -05:00
firestar5683 c8ec1ae5ec GM recovery checks 2026-10-09 06:19:18 -05:00
firestar5683 531daa1f22 Build 2026-10-09 05:15:26 -05:00
firestar5683 24dc7b9112 Safety checks 2026-10-09 05:12:53 -05:00
firestar5683 ae7b168a86 Build 2026-10-09 03:35:05 -05:00
firestar5683 eb46cdf31a Galaxy capability tests 2026-10-09 03:07:24 -05:00
firestar5683 63fbaa6b24 Hyundai AOL button handling 2026-10-09 03:07:24 -05:00
firestarsdog c34b0f97de Revert "Fix replay jitter"
This reverts commit ef42c19924.
2026-10-09 03:44:36 -04:00
firestarsdog ef42c19924 Fix replay jitter 2026-10-09 03:12:10 -04:00
Prabhaav Pillai c6bc07a9a0 (Another sixth) round of Galaxy UI bug fixes
Improved busy state handling across sound packs, backup settings, and vehicle controls.
Prevented duplicate requests and user interactions during ongoing processes.
Added haptic feedback for smoother user interactions.
Introduced automated settings state tests to improve reliability.
2026-10-09 02:32:17 -04:00
firestar5683 d2ae98eb39 Build 2026-10-08 22:25:02 -05:00
firestar5683 670927368c Bolt cruise cancellation 2026-10-08 22:19:14 -05:00
firestar5683 71c7bd8411 Build 2026-10-08 21:49:19 -05:00
firestar5683 deb1702cc2 Automatic lane change activation 2026-10-08 21:45:44 -05:00
firestar5683 980f9fb081 AOL pause settings 2026-10-08 21:45:44 -05:00
firestar5683 65586189f7 Bolt regen command cadence 2026-10-08 21:45:43 -05:00
Prabhaav Pillai 21806fa504 (Another fifth) round of Galaxy UI bug fixes
Updated headers and descriptions across various pages to enhance user understanding.
Added navigation and interaction improvements in the logs, longitudinal curves, and model laboratory sections.
Enhanced error handling and user notifications in map operations and vehicle controls.
Improved accessibility features, including ARIA labels and roles for better screen reader support.
Consolidated and clarified messages related to software updates and branch selections.
Streamlined the layout and organization of settings and configuration pages for better usability.
Smoother Transitions
2026-10-08 21:57:58 -04:00
firestar5683 9443422926 Build 2026-10-08 19:19:46 -05:00
firestar5683 015c810b17 Camera CPU recovery 2026-10-08 19:17:11 -05:00
firestar5683 cfc1243a2f Build 2026-10-08 18:52:10 -05:00
firestar5683 a1f1e53e87 Personality lead colors 2026-10-08 18:48:48 -05:00
firestar5683 e18c328081 Build 2026-10-08 18:25:06 -05:00
firestar5683 c7088d98d6 Volt optional gas limits 2026-10-08 18:23:11 -05:00
firestar5683 31abbcfe2c Build 2026-10-08 17:57:03 -05:00
firestar5683 1e0cb24fd2 Controller learning help 2026-10-08 17:54:06 -05:00
firestar5683 831413dd9e Volt low-speed controls 2026-10-08 17:54:06 -05:00
firestar5683 e05f7e9925 Volt AOL recovery 2026-10-08 17:54:05 -05:00
firestar5683 8c79866e22 Build 2026-10-08 17:00:51 -05:00
firestar5683 02043d1276 Sensor process recovery 2026-10-08 16:58:33 -05:00
firestar5683 4022b9304c Build 2026-10-08 16:39:17 -05:00
firestar5683 6f7fe4068c Volt safety checks 2026-10-08 16:31:31 -05:00
firestar5683 d0bf0728b0 RAV4 torque tuning 2026-10-08 16:31:30 -05:00
firestar5683 873089ef7a Sonata Hybrid wheel defaults 2026-10-08 16:31:30 -05:00
firestar5683 41d11d08af Mach-E control tuning 2026-10-08 16:31:30 -05:00
firestar5683 f499290a35 Build 2026-10-08 15:24:26 -05:00
firestar5683 89f60950ed Audio underrun diagnostics 2026-10-08 15:21:28 -05:00
firestar5683 35186e67fd Replay clock sampling 2026-10-08 15:21:27 -05:00
firestar5683 99ef4b9646 Model catalog checks 2026-10-08 15:21:27 -05:00
firestar5683 efbee4155e Vehicle selector guards 2026-10-08 15:21:27 -05:00
firestar5683 fa7c0e155f Volt SDGM acceleration envelope 2026-10-08 15:21:27 -05:00
firestar5683 f71f19cf7a Build 2026-10-08 14:25:05 -05:00
firestar5683 136d2f168c Onroad lead icon and lane-change widgets 2026-10-08 14:22:47 -05:00
whoisdomi 0709a36b5f Update Manifest 2026-10-08 13:20:35 -05:00
firestar5683 9507f5f07d Build 2026-10-08 12:34:48 -05:00
firestar5683 befa15cdfd GM configuration lookup performance 2026-10-08 12:31:45 -05:00
firestar5683 7ea69c1c0b Volt startup handoff 2026-10-08 12:31:45 -05:00
firestar5683 c054e427c4 Bugsink error reporting 2026-10-08 12:31:44 -05:00
firestar5683 2a3e0420dd Conditional mode test typing 2026-10-08 09:40:10 -05:00
firestar5683 3ce0894783 Build 2026-10-08 09:33:34 -05:00
firestar5683 5fe62337aa Toyota manual brake hold 2026-10-08 09:31:10 -05:00
firestar5683 2d8e40060e Personality controls 2026-10-08 09:31:09 -05:00
firestar5683 6bb0f8b0c2 Experimental driving mode 2026-10-08 09:31:09 -05:00
firestar5683 0632ab9f28 Safety mutation equivalence 2026-10-08 09:31:09 -05:00
firestar5683 1fb7aa116b Honda transmission detection 2026-10-08 09:31:09 -05:00
firestar5683 a74a6f2e45 Volt hold heartbeat 2026-10-08 09:31:08 -05:00
firestar5683 bbad2d9e4b GM control loop performance 2026-10-08 09:31:08 -05:00
firestar5683 43efb4ce66 Build 2026-10-08 07:04:51 -05:00
firestar5683 3c0301bce2 CI mutation progress 2026-10-08 07:03:45 -05:00
firestar5683 5b6dcdd5e8 Hyundai longitudinal parity 2026-10-08 06:42:59 -05:00
firestar5683 532a3604d0 Build 2026-10-08 03:46:39 -05:00
firestar5683 1830f41a09 GM test serialization 2026-10-08 03:44:49 -05:00
firestar5683 0d34efe7c3 CI build guards 2026-10-08 03:44:48 -05:00
firestar5683 be5fb88423 Build 2026-10-08 03:06:47 -05:00
firestar5683 e29f60b188 Migration and UI fixtures 2026-10-08 03:02:19 -05:00
firestar5683 65a0ef7679 Galaxy HTTP responses 2026-10-08 03:02:19 -05:00
firestar5683 17d3f08641 Runtime type contracts 2026-10-08 03:02:19 -05:00
firestar5683 c2a47219cf Vehicle regression fixtures 2026-10-08 03:02:19 -05:00
firestar5683 38b86cb991 Safety regression coverage 2026-10-08 03:02:18 -05:00
firestar5683 b59074b296 Native test coverage 2026-10-08 02:21:34 -05:00
firestar5683 51e08a5eda Galaxy test fixtures 2026-10-08 02:21:34 -05:00
firestar5683 602932a104 Safety qualification harness 2026-10-08 02:21:34 -05:00
firestar5683 2fde98dab6 Display regression expectations 2026-10-08 02:21:34 -05:00
firestar5683 a5ef6002ec Mac test prerequisites 2026-10-08 02:21:33 -05:00
firestar5683 65a2b29b34 Regression test fixtures 2026-10-08 02:21:33 -05:00
firestar5683 f7c0090274 Replay expectations 2026-10-08 02:21:33 -05:00
firestar5683 745f87e735 Runtime type contracts 2026-10-08 02:21:33 -05:00
firestar5683 be163b12da Recording metadata portability 2026-10-08 02:21:33 -05:00
firestar5683 dd7314e7a7 Galaxy HTTP server 2026-10-08 02:21:32 -05:00
firestar5683 e33c74d639 Replay clock overflow 2026-10-08 02:21:32 -05:00
firestar5683 32ea41e2e7 Hyundai wheel capabilities 2026-10-08 02:21:32 -05:00
firestar5683 51dbc31513 Schema policy 2026-10-08 02:21:32 -05:00
firestar5683 71daa79f66 CI workflow reporting 2026-10-08 02:21:32 -05:00
prabhaavp 144b48e65f Merge pull request #210 from dirwin31:Stardust-PR-4
Merged! This adds an on-road map with offline map downloads, improves navigation search and shortcuts, and expands layout customization. Awesome work and thanks for the contribution!
2026-10-08 03:11:39 -04:00
Prabhaav Pillai d7e4c021b6 Updated RouteRuntime to read routing settings instead of general settings.
Modified test cases to ensure proper handling of route changes and storage failures.
Enhanced tests for temporary destination handling and suggestions.
Improved scaling logic in Android Auto UI to account for letterbox offsets.
Added CSS for offline road maps in the Android Auto overlay.
Refined touch handling in projection favorites to prevent interaction issues.
Ensured proper state management in map overlays and tile reading.
2026-10-08 03:10:10 -04:00
dirwin31 d5f7852c0d Rebase Tweaks 2026-10-07 22:34:05 -07:00
dirwin31 6afa673a3b Buttins Be whack 2026-10-07 22:34:05 -07:00
dirwin31 e7a5040267 NBigger ETA, Easier Nav, Map Tail, Video Lag and Gaurd Rail Comments 2026-10-07 22:34:05 -07:00
Danny fca44e169c Make Favorites Easier, Condense Mobile Nav Page. 2026-10-07 22:34:05 -07:00
dirwin31 f2f32b76a7 Fix Screen Smoothing Again, AAwireless v1, Lane Changes 2026-10-07 22:34:05 -07:00
Danny 1b112b898d Pretty Nav Page 2026-10-07 22:34:05 -07:00
Danny 17463be653 Add Favorites Widgets 2026-10-07 22:34:05 -07:00
Danny 236c794eff Optimize Maps, No Frogs 2026-10-07 22:34:04 -07:00
dirwin31 ddd6eeb7f0 Better Drag and drop 2026-10-07 22:34:04 -07:00
dirwin31 014c8618ae Screen Setup Page ReOrg 2026-10-07 22:34:04 -07:00
dirwin31 ce83d86818 Allow for dropped params keys 2026-10-07 22:34:04 -07:00
Danny 6fc37f8aae Map Tweaks 2026-10-07 22:34:04 -07:00
Danny 2ce246f6a4 Mappsss 2026-10-07 22:34:04 -07:00
firestar5683 b005656ed7 Build 2026-10-07 23:58:07 -05:00
firestar5683 1af52d8a7b Mach-E curvature preview 2026-10-07 23:55:41 -05:00
firestar5683 b7d6827450 MPC terminal constraints 2026-10-07 23:55:41 -05:00
firestar5683 3c406e4374 Build 2026-10-07 23:30:59 -05:00
firestar5683 57967e709c Calibration reset 2026-10-07 23:29:12 -05:00
firestar5683 4a0e2eb0ba Speed limit defaults 2026-10-07 23:29:12 -05:00
firestar5683 8429b06436 Big UI torque bar 2026-10-07 23:29:11 -05:00
firestar5683 b078f9fe6a Build 2026-10-07 22:51:17 -05:00
firestar5683 b2896e7180 Test fixtures 2026-10-07 22:49:59 -05:00
firestar5683 fb63d4442d Build 2026-10-07 22:47:10 -05:00
firestar5683 3c0458e646 Volt resume test clocks 2026-10-07 22:47:10 -05:00
firestar5683 6800730446 Volt dashboard messages 2026-10-07 22:47:10 -05:00
firestar5683 b9df324f19 Build 2026-10-07 22:47:09 -05:00
firestar5683 14c1349a5d Control loop overhead 2026-10-07 22:47:09 -05:00
firestar5683 acaee3b624 Build 2026-10-07 22:47:09 -05:00
firestar5683 ce09aec43d Recorded replay state 2026-10-07 22:47:09 -05:00
firestar5683 3fa904765c Build 2026-10-07 22:47:09 -05:00
firestar5683 ef33866889 C4 personality indicator 2026-10-07 22:47:08 -05:00
firestar5683 67b213a5aa Driving personality switching 2026-10-07 22:47:08 -05:00
firestarsdog 30477f3cef Large UI Gamma Effect Trial 2026-10-07 21:17:47 -04:00
firestarsdog 2564796b78 Set speed favorite fix 2026-10-07 20:30:27 -04:00
Prabhaav Pillai 856320d333 (Another fourth) round of Galaxy UI bug fixes
Improved UI consistency and responsiveness
added Playwright tests
implemented toggle preference backup/restore with validation and unit tests
moved navigation styling and state management.
2026-10-07 20:28:47 -04:00
firestar5683 e41ff41107 Build 2026-10-07 18:22:53 -05:00
firestar5683 38bca18f8b Volt startup keepalives 2026-10-07 18:20:27 -05:00
firestar5683 fc60ef16c6 GM profile lookup 2026-10-07 17:52:03 -05:00
Daniel Koepping dd3241e9f9 chestnut: show switch branch alert only on release (#39049)
chestnut: only show branch alert on release branches

Original-upstream-commit: 3f06ed3bc7a4cc9c320be1f7d03dcd8a885a8e6f
Upstream: https://github.com/commaai/openpilot/commit/3f06ed3bc7a4cc9c320be1f7d03dcd8a885a8e6f
2026-10-07 15:13:02 -07:00
Harald Schäfer 3d7509c4ca Enable Experimental Mode by default without confirmation (#39043)
* Enable Experimental Mode by default without confirmation

* Update toggles.py

* Fix formatting in End-to-End Longitudinal Control description

Original-upstream-commit: f21bfc36be5f423f31b2c2a948d1e2eff91b02e8
Upstream: https://github.com/commaai/openpilot/commit/f21bfc36be5f423f31b2c2a948d1e2eff91b02e8
2026-10-07 07:59:43 -07:00
Adeeb Shihadeh 5d16c4c9af Fix Athena test port collision (#39048)
Original-upstream-commit: 657d3a4f7c248fd3dd37d6520bf4ea91eeba57d7
Upstream: https://github.com/commaai/openpilot/commit/657d3a4f7c248fd3dd37d6520bf4ea91eeba57d7
2026-10-06 23:43:16 -07:00
Daniel Koepping cf9fb44a5b modeld: drop chestnut on fallback (#39044)
Original-upstream-commit: a0d47bcd08d9c359ccc7bd73145ed90ade2255ab
Upstream: https://github.com/commaai/openpilot/commit/a0d47bcd08d9c359ccc7bd73145ed90ade2255ab
2026-10-06 18:43:50 -07:00
Daniel Koepping 120cf0adb6 ui: longitudinal policy indicator (#39040)
* ui: mici longitudinal icons

* ui: rm orange state

* ui: lead policy indicator

* fix

* show on op long

* make own widget

* sync long indicator with dmoji

* fast fade in with dmoji

Original-upstream-commit: 035a447d1f0f0ef5abf2ed9724e33d9265399019
Upstream: https://github.com/commaai/openpilot/commit/035a447d1f0f0ef5abf2ed9724e33d9265399019
2026-10-06 15:51:25 -07:00
stef 1e891dd9f2 ui: bookmark fill (#38631)
* bookmark fill

* fade and sticky click

* fade out animation

* no funny animation

* clean

---------

Co-authored-by: Daniel Koepping <elkoled@gmail.com>

Original-upstream-commit: 5b2a59d288f9d1eb3123f7af81ffab4159600b1f
Upstream: https://github.com/commaai/openpilot/commit/5b2a59d288f9d1eb3123f7af81ffab4159600b1f
2026-10-06 15:42:39 -07:00
firestar5683 72f094753e Galaxy vehicle categories 2026-10-07 17:41:39 -05:00
firestar5683 181b8059f1 Build 2026-10-07 16:56:09 -05:00
firestar5683 5c14b29fc1 Volt radar recovery alert 2026-10-07 16:48:50 -05:00
firestar5683 8efe5bac09 Build 2026-10-07 16:12:40 -05:00
firestar5683 217500af2a GM torque learning 2026-10-07 16:12:40 -05:00
firestar5683 65a1e2bef9 GM longitudinal preferences 2026-10-07 16:12:40 -05:00
firestar5683 0a3b849c5d GM runtime settings 2026-10-07 16:12:40 -05:00
firestar5683 4a13f306d8 Sound callback diagnostics 2026-10-07 16:12:40 -05:00
firestar5683 7ef391311e Cruise settings 2026-10-07 16:12:40 -05:00
firestar5683 68e655fc9e Build 2026-10-07 12:46:33 -05:00
firestar5683 8215eeb362 Bolt automatic identity 2026-10-07 12:43:24 -05:00
firestar5683 3d7c83eb5f Sound callback timing 2026-10-07 12:25:01 -05:00
firestar5683 3a60707936 Build 2026-10-07 12:10:27 -05:00
firestar5683 9b03a5b233 Vision workload priority 2026-10-07 12:06:03 -05:00
firestar5683 45ab728330 Bolt AOL startup and recovery 2026-10-07 11:42:39 -05:00
firestar5683 1380d84574 Build 2026-10-07 10:19:16 -05:00
firestar5683 959f0e837d Vision inference scheduling 2026-10-07 10:13:24 -05:00
firestar5683 3236168a74 Vision sign lifecycle 2026-10-07 10:13:24 -05:00
firestar5683 f3903a1051 AOL intent recovery 2026-10-07 10:00:53 -05:00
firestar5683 a7d0248b62 CEM stop release 2026-10-07 09:56:59 -05:00
firestar5683 c01f815736 Shared UI performance 2026-10-07 09:10:24 -05:00
firestar5683 e91fb76713 C4 UI performance 2026-10-07 08:50:10 -05:00
firestar5683 6f026efb13 Build 2026-10-07 07:27:18 -05:00
firestar5683 e4232643fa Restore EV acceleration presets 2026-10-07 07:23:34 -05:00
firestar5683 bfdad768b5 Restore Silverado truck tuning 2026-10-07 07:10:31 -05:00
firestar5683 0da456b93d Build 2026-10-07 06:37:36 -05:00
firestar5683 4238b61cdd Guard Hybrid cruise status 2026-10-07 06:34:06 -05:00
firestar5683 e80648bd20 Fix GM cruise transitions 2026-10-07 06:33:54 -05:00
firestar5683 01e80cf4ce Build 2026-10-07 05:34:32 -05:00
firestar5683 1bd1328531 Respect Volt cruise button timing 2026-10-07 05:33:37 -05:00
firestar5683 d798d7882a Respect GM cruise button state 2026-10-07 05:11:48 -05:00
firestar5683 eb78c0696c Restore GM geometry adjustments 2026-10-07 05:11:48 -05:00
firestar5683 53e9e4ec28 Build 2026-10-07 03:59:46 -05:00
firestar5683 a3c32e4633 Restore GM manual steering settings 2026-10-07 03:54:15 -05:00
firestar5683 3d79a80cce Build 2026-10-07 02:41:16 -05:00
firestar5683 4a59a56ae3 Match expanded paging controls 2026-10-07 02:40:06 -05:00
firestar5683 6affbf97eb Restore Volt cruise engagement 2026-10-07 02:38:46 -05:00
firestarsdog e58a905e07 Previous/Next expanded tap zones 2026-10-07 03:20:48 -04:00
firestar5683 06dfbec562 Build 2026-10-07 01:44:52 -05:00
firestar5683 f809a5610f Restore Bolt stock cruise cancellation 2026-10-07 01:42:57 -05:00
firestarsdog c67ade21d8 Extract Selector Dialog and add to System 2026-10-07 02:30:54 -04:00
firestar5683 50cae15248 Build 2026-10-07 00:58:15 -05:00
firestar5683 1b7219806d Restore GM cruise status 2026-10-07 00:56:40 -05:00
firestarsdog acc2b6191c Remove unneeded header tap dialog 2026-10-07 01:37:00 -04:00
firestarsdog e7e26ee074 Fix Sounds Panel Page 2 2026-10-07 01:31:18 -04:00
prabhaavp 0b81733e4d Merge pull request #209 from dirwin31:Stardust-PR-3
Live Edits to Galaxy Dev
2026-10-07 01:30:56 -04:00
Prabhaav Pillai be468d902b Unload guard for unsaved drafts and enhance live reload handling 2026-10-07 01:29:21 -04:00
firestar5683 7e11b01b0e Build 2026-10-07 00:27:41 -05:00
firestar5683 8864990859 Restore GM cruise cancellation 2026-10-07 00:25:31 -05:00
firestar5683 fd91811394 Fix validation fixtures and formatting 2026-10-07 00:25:31 -05:00
firestarsdog 10b27d6613 Large UI Sound Selection Update 2026-10-07 01:05:01 -04:00
prabhaavp 7ce888ed93 Merge pull request #208 from dirwin31:Stardust-PR-2
More Flowz
2026-10-07 00:56:51 -04:00
Prabhaav Pillai 104f88da3b Merge branch 'SecretGoodStarPilot' into pr/dirwin31/208 2026-10-07 00:51:17 -04:00
Prabhaav Pillai 2521eec0c0 streamline setup, pairing, and dialogs 2026-10-07 00:45:46 -04:00
firestar5683 244ed62cd6 Build 2026-10-06 23:35:30 -05:00
firestar5683 cdee63dd8e Restore Honda AOL engagement 2026-10-06 23:33:59 -05:00
Danny 69fa026fef Allow Live Edits to Galaxy 2026-10-06 21:03:32 -07:00
firestar5683 4085afd697 Build 2026-10-06 23:01:59 -05:00
firestar5683 e81b0c8e5e Restore Forte steering parity 2026-10-06 23:00:21 -05:00
firestar5683 33a9f104d2 Build 2026-10-06 22:50:37 -05:00
firestar5683 56761cbe15 Fix C4 audio toggle responsiveness 2026-10-06 22:49:00 -05:00
dirwin31 6b670e16c4 Prebuilt Lock 2026-10-06 20:45:43 -07:00
dirwin31 d0a5e1c43a Merge remote-tracking branch 'upstream/SecretGoodStarPilot' into Stardust-PR-2
# Conflicts:
#	openpilot/starpilot/galaxy/web/css/galaxy.css
2026-10-06 20:43:43 -07:00
firestar5683 2a9b98e8e6 Build 2026-10-06 22:37:07 -05:00
firestar5683 fa6a7c5b85 Fix GM AOL button capability 2026-10-06 22:35:25 -05:00
firestar5683 1d803c6506 Fix AOL startup 2026-10-06 22:35:25 -05:00
firestar5683 1acb1b1c8c Build 2026-10-06 22:15:10 -05:00
dirwin31 08d4634370 Car with built-in wireless Android Auto Wording 2026-10-06 20:13:13 -07:00
firestar5683 769dd1d0f3 Preserve Volkswagen GPS fix freshness 2026-10-06 22:12:33 -05:00
firestar5683 778269b0a5 Check vehicle GPS before updates 2026-10-06 22:09:45 -05:00
firestar5683 2ff07b2beb Restore vehicle GPS fallback 2026-10-06 22:08:02 -05:00
firestar5683 3cd4dce42d Update vehicle tuning and K4 coverage 2026-10-06 22:05:08 -05:00
firestar5683 5fd6b202f3 Build 2026-10-06 21:32:49 -05:00
firestar5683 987bdf9b9c Add driving mode descriptions 2026-10-06 21:29:28 -05:00
firestar5683 851fd50b2b Build 2026-10-06 21:18:51 -05:00
firestar5683 a913d65aa8 Restore low-speed steering advisory 2026-10-06 21:16:24 -05:00
firestar5683 9c0083820d Restore 2018-21 Bolt cruise cancellation 2026-10-06 21:16:24 -05:00
firestar5683 ca8146e63c Build 2026-10-06 20:48:07 -05:00
firestar5683 88f9a8eaf8 Retain audio underrun diagnostics 2026-10-06 20:45:40 -05:00
firestar5683 71ebd701eb panda 2026-10-06 20:15:24 -05:00
firestar5683 64a68b388a Build 2026-10-06 19:49:49 -05:00
firestar5683 92dc796811 Restore Bolt ACC pedal dashboard 2026-10-06 19:47:58 -05:00
firestar5683 587d30d455 Build 2026-10-06 19:05:32 -05:00
firestar5683 e1c48fa90f Verify shared steering pause inputs 2026-10-06 18:59:46 -05:00
firestar5683 4cded622b1 Cover map package validation in build dispatch 2026-10-06 18:53:44 -05:00
firestar5683 51ffef7025 Check map packaging before local tests 2026-10-06 18:51:14 -05:00
firestar5683 445a8c74f3 Preserve steering pause during invalid settings reads 2026-10-06 18:51:03 -05:00
firestar5683 9f0c443ffc Recover Galaxy after interrupted sessions 2026-10-06 18:50:03 -05:00
firestar5683 4c5fdfe100 Build 2026-10-06 18:46:51 -05:00
firestar5683 1ea9ad62a9 Improve navigation setup and map loading 2026-10-06 18:45:32 -05:00
firestar5683 caf2682c36 Ship the offline map provider 2026-10-06 18:43:51 -05:00
firestar5683 315c6689c2 Build 2026-10-06 18:40:56 -05:00
firestar5683 d51a78915e Restore steering pause 2026-10-06 18:39:29 -05:00
firestar5683 f339867727 Build 2026-10-06 18:25:39 -05:00
firestar5683 2a6de7f69a Restore Toyota gap buttons 2026-10-06 18:24:00 -05:00
firestar5683 00547795aa Keep zero speed visible 2026-10-06 18:17:31 -05:00
firestar5683 91a45b034e Build 2026-10-06 18:13:29 -05:00
firestar5683 ae28f2c70a Discover models from the remote catalog 2026-10-06 18:05:09 -05:00
firestar5683 50536f6f72 Preserve Jetlink artifacts across loader updates 2026-10-06 18:05:09 -05:00
firestar5683 25de3efff5 Clarify update progress 2026-10-06 17:51:04 -05:00
Danny 133d754b77 Dont Write if Not Needed, FAQ v2, Wonky Words.
(cherry picked from commit 423bbf6a36c0146a281c9366b4fc55155a7b334d)
2026-10-06 15:47:23 -07:00
Danny 7ad314f587 PiP Smoothing, Cer Expiring Warning, FAQ v1
(cherry picked from commit 135f7dd62aa3f6a2bc1e8ce10097a4c6f6a323be)
2026-10-06 15:47:23 -07:00
dirwin31 793b8857d8 Blind-spot side camera: wait for first cabin frame and show bubbles on Android Auto
(cherry picked from commit 60e39faa3e7c9c52df3cd314ac4c0645887aef2b)
2026-10-06 15:47:23 -07:00
dirwin31 354443893f Allow for dropped params keys
(cherry picked from commit 5ae8fc237c95c49f871b1e4377d25daef199177f)
2026-10-06 15:47:22 -07:00
firestar5683 222c6669ea Build 2026-10-06 17:40:08 -05:00
firestar5683 b6845519d2 Bind empty-arena loader runtime provenance 2026-10-06 17:30:07 -05:00
firestar5683 9f347b313c Optimize stateful model transfers 2026-10-06 17:28:19 -05:00
firestar5683 db5a559e73 Support stateful GPU model compilation 2026-10-06 17:17:39 -05:00
firestar5683 93fe9f1652 Protect longitudinal companion wire prefix 2026-10-06 17:15:16 -05:00
firestar5683 3b9f353ad4 Merge remote-tracking branch 'origin/SecretGoodStarPilot' into SecretGoodStarPilot 2026-10-06 16:58:22 -05:00
firestar5683 adc67c199c Adapt StarPilot hold and Tesla alerts to custom schema companions
Preserve official core schema and bind hold publication to the source plan.
2026-10-06 17:54:49 -05:00
Harald Schäfer f29fb8f724 Gas override boost (#39015)
* Add gas override boost, mici boost bar, and Rivian maximum set speed

* Refactor acceleration boost and gate accumulation on model limit

* Fade acceleration boost with model acceleration

* Limit boost buildup per gas override

* Latch boost eligibility before gas override

* Move model limiting check into AccelBoost

* Use DT_MDL directly in AccelBoost

* Clarify acceleration boost state names

* Scale acceleration boost by speed instead of decaying it

* Enable Experimental Mode by default

* Scale boost buildup with the application speed ramp

* Fix boost serialization and model-limit eligibility

* Remove boost UI and Experimental Mode default changes

* Combine boost update and application

Original-upstream-commit: ab8c84c3989bab34969d83e78c5fad3ee5b95112
Upstream: https://github.com/commaai/openpilot/commit/ab8c84c3989bab34969d83e78c5fad3ee5b95112
2026-10-06 17:54:49 -05:00
firestar5683 1364266b5a Prepare official longitudinal schema ordinal for upstream boost 2026-10-06 17:54:49 -05:00
Adeeb Shihadeh 017bf476f2 jp: fix file resolving
Original-upstream-commit: c933446486e6923c8468a979b6fab890b46f5cdb
Upstream: https://github.com/commaai/openpilot/commit/c933446486e6923c8468a979b6fab890b46f5cdb
2026-10-06 17:54:16 -05:00
Adeeb Shihadeh 448da4d39f jp: show text fields (#39042)
Original-upstream-commit: 2b7c34a5c32f78f4892a07598cbe3827be880438
Upstream: https://github.com/commaai/openpilot/commit/2b7c34a5c32f78f4892a07598cbe3827be880438
2026-10-06 17:54:16 -05:00
Adeeb Shihadeh 49c960a3f0 jotpluggler: fix subset time bounds (#39041)
Original-upstream-commit: fc616407826c52abd1338a34e85ce9d4fcbd20e1
Upstream: https://github.com/commaai/openpilot/commit/fc616407826c52abd1338a34e85ce9d4fcbd20e1
2026-10-06 17:54:15 -05:00
Adeeb Shihadeh 6c0074def3 jp: add initData tab (#39039)
Original-upstream-commit: 84f9bdbc10df0184976c14457bddab008b141208
Upstream: https://github.com/commaai/openpilot/commit/84f9bdbc10df0184976c14457bddab008b141208
2026-10-06 17:54:15 -05:00
Adeeb Shihadeh 7ec39898a8 deprecate unused initData fields
Original-upstream-commit: cce54f20d52c381499ae16119cb83c710c1f1083
Upstream: https://github.com/commaai/openpilot/commit/cce54f20d52c381499ae16119cb83c710c1f1083
2026-10-06 17:54:15 -05:00
Daniel Koepping 7073765e83 Revert #38464 (#39038)
* Revert #38464

Reverts aa0ac919ba.

Original-upstream-commit: 910c08e0828c54ce7919ae14606166779a52bf94
Upstream: https://github.com/commaai/openpilot/commit/910c08e0828c54ce7919ae14606166779a52bf94
2026-10-06 17:54:15 -05:00
Shane Smiskol d5748b0a61 ui: fix keyboard text overflow gradient
Original-upstream-commit: 01a670d6b2fc41c0fc79300b200906e4058f8d33
Upstream: https://github.com/commaai/openpilot/commit/01a670d6b2fc41c0fc79300b200906e4058f8d33
2026-10-06 17:54:15 -05:00
firestarsdog dcc47f2838 LaneLines 6->4 2026-10-06 17:22:31 -04:00
firestar5683 5ed05015a4 Build 2026-10-06 16:18:14 -05:00
firestar5683 80336a4eb7 Palisade Always On Lateral 2026-10-06 16:16:48 -05:00
firestar5683 bfc5a16280 Build 2026-10-06 15:48:42 -05:00
firestar5683 48c8a06b2f Palisade Longitudinal Startup 2026-10-06 15:46:21 -05:00
firestar5683 0a34c129f1 Build 2026-10-06 15:28:16 -05:00
firestar5683 c8de87e680 Malibu Hybrid Cruise Control 2026-10-06 15:26:49 -05:00
dirwin31 3600e3a461 Logs Page Weirdness and Step 4 2026-10-06 13:20:43 -07:00
dirwin31 229c9e7beb Make Uploads calm down 2026-10-06 13:07:02 -07:00
dirwin31 8a7770a4e0 Move Buttons 2026-10-06 13:07:02 -07:00
dirwin31 5b52ff4bc8 Android Auto config v4: move sync-readback configs back to async
Version 3 configs that turned gpu_nv12/async_readback off did so while a
failed async readback still crashed the car view. Now that it only drops the
frame, older files get today's defaults (both on); from v4 on a saved value
is a real choice again.
2026-10-06 13:06:51 -07:00
dirwin31 da82092bc9 Car view: drop a frame on a failed async readback instead of crashing
A failed GPU wait (GL_WAIT_FAILED as openpilot went onroad) raised out of the
renderer, three crashes in a row exhausted its restart budget and the car
showed 'display unavailable'. Now the frame is dropped, the GL error is
logged with it, and after three consecutive failures the session switches to
synchronous readback.
2026-10-06 13:06:51 -07:00
dirwin31 5c09df9b61 Fix Android Auto start screen logo and font paths
BASEDIR is the repo root, but assets live under openpilot/ in this layout,
so the logo and both fonts were never found and the car showed tiny
default-font text only.
2026-10-06 13:06:51 -07:00
dirwin31 889ff8506b Restart soundd and android_autod after a crash
Both can crash at boot before their hardware is ready: android_autod when
bluetoothd is not up yet (NoSuchUnit / adapter unavailable while restoring
a phone-role journal left by a reboot mid-projection), soundd when the audio
stream is not active. The manager never restarted them, so Android Auto was
dead and selfdrived refused to engage for the rest of the drive.
2026-10-06 13:06:51 -07:00
dirwin31 6030a79f8c Better Wording for Step 1 2026-10-06 13:06:51 -07:00
dirwin31 93a3cfe2bd VASM Camera and Helper Arrows 2026-10-06 13:06:51 -07:00
dirwin31 d1def0f266 Smooth Install Flow V1 2026-10-06 13:06:50 -07:00
Danny 5a6e17a0c8 Fix the Boot Logo Screen 2026-10-06 13:06:50 -07:00
Danny 738c7a2dbc Add bounded restart logic and output logging for crashed or stalled car UI renderers 2026-10-06 13:06:50 -07:00
Danny a636fa9d51 Allow Route Delete and Account for Route Counter Reset After Migration to Stop Auto Deleting Newest Routes 2026-10-06 13:06:50 -07:00
firestar5683 fb4a28774e Build 2026-10-06 14:00:42 -05:00
firestar5683 2d0cd49eea Malibu Cruise Control 2026-10-06 13:58:51 -05:00
firestar5683 baf1f55111 Build 2026-10-06 13:30:28 -05:00
firestar5683 1b89180b83 Turn Assist 2026-10-06 13:28:42 -05:00
firestar5683 13260d36c9 Lane Centering Strength 2026-10-06 13:14:32 -05:00
firestar5683 db4769cdde Build 2026-10-06 13:02:45 -05:00
firestar5683 3acfd0505c Settings Upgrade Recovery 2026-10-06 12:55:44 -05:00
firestar5683 9aa464f8e2 Build 2026-10-06 12:44:31 -05:00
firestar5683 0fcceb8294 Silverado Stock Lateral 2026-10-06 12:42:08 -05:00
firestar5683 8f71076e25 Build 2026-10-06 11:14:00 -05:00
firestar5683 bc27e48c5a Sienna Longitudinal Control 2026-10-06 11:12:12 -05:00
firestar5683 90050d8a95 Build 2026-10-06 10:56:11 -05:00
firestar5683 8269de1077 Corolla Longitudinal Control 2026-10-06 10:54:24 -05:00
firestar5683 b2d422d43a Build 2026-10-06 10:35:24 -05:00
firestar5683 84bb2a4110 Prius Always On Lateral 2026-10-06 10:31:18 -05:00
firestar5683 f7de968fcb Build 2026-10-06 10:02:55 -05:00
firestar5683 e91ca11e1c Toyota Radar AOL 2026-10-06 10:01:04 -05:00
firestar5683 caadb75088 Build 2026-10-06 09:39:22 -05:00
firestar5683 36553441f6 Startup Test Fixtures 2026-10-06 09:37:45 -05:00
firestar5683 83ee23c836 UI Test Fixtures 2026-10-06 09:29:31 -05:00
firestar5683 93d78f9b8c Build 2026-10-06 09:25:05 -05:00
firestar5683 75a505d851 Toyota Stopping and Steering 2026-10-06 09:25:05 -05:00
firestar5683 4daa209c06 Build 2026-10-06 09:25:04 -05:00
firestar5683 bcab96d071 Toyota Always On Lateral 2026-10-06 09:25:04 -05:00
firestar5683 7ae1a6a417 Build 2026-10-06 09:25:04 -05:00
firestar5683 d2a06a7a0e Toyota Hybrid Brake Hold 2026-10-06 09:25:04 -05:00
firestar5683 56d90ef0a6 Build 2026-10-06 09:25:04 -05:00
firestar5683 3db047a9af Mach-E Steering Unwind 2026-10-06 09:25:03 -05:00
firestar5683 249ab7cfc6 Build 2026-10-06 09:25:03 -05:00
firestar5683 81af2a0b32 Forte AOL Startup 2026-10-06 09:25:03 -05:00
firestar5683 aa5557acf1 Build 2026-10-06 09:25:03 -05:00
firestar5683 e05d3d3e99 G70 Steering Damping 2026-10-06 09:25:02 -05:00
firestar5683 f2e4f6084e Build 2026-10-06 09:25:02 -05:00
firestar5683 873f7059ea GM Pedal Control 2026-10-06 09:25:02 -05:00
firestar5683 2e3621ca8f Build 2026-10-06 09:25:02 -05:00
firestar5683 c219110392 GM Longitudinal Control 2026-10-06 09:25:01 -05:00
firestar5683 60868dd8bd Android Auto Report Permissions 2026-10-06 09:24:36 -05:00
firestar5683 0b9643b1e4 Build 2026-10-06 09:24:35 -05:00
firestar5683 dffeb7aa03 Honda Hybrid AOL
Co-authored-by: dirwin31 <83434411+dirwin31@users.noreply.github.com>
2026-10-06 09:24:35 -05:00
dirwin31 acffe8f112 Android Auto Connection and Rendering 2026-10-06 09:24:35 -05:00
firestar5683 875eb6bc8c Build 2026-10-06 09:24:06 -05:00
firestar5683 d18f4968fe Automatic Usage Statistics 2026-10-06 09:24:05 -05:00
Prabhaav Pillai 5ffc32f2a5 (Another third) round of Galaxy UI bug fixes
Centered loading states and unified page layouts, margins, and shared controls.
Added horizontal tab transitions and a moving glass navigation indicator.
Fixed mobile button overflow and compacted model summaries.
Removed unused code and styles.
2026-10-06 03:23:02 -04:00
firestarsdog b53e9201f3 Back button behavior 2026-10-06 02:37:43 -04:00
firestarsdog ccdfa0367c Header Styling 2026-10-06 02:23:09 -04:00
firestarsdog a47b1d4188 Pagination QoL #2 2026-10-06 01:44:26 -04:00
firestar5683 40e6b5747c Build 2026-10-05 23:51:16 -05:00
firestar5683 bbdaa61b28 Prius EPS Retrofit 2026-10-05 23:49:31 -05:00
Prabhaav Pillai 9bf387e709 (Another second) round of Galaxy UI bug fixes
Fixed logo/search overlap by aligning responsive breakpoints and preventing logo shrink.
Mobile search expands across the bar; closing it restores controls and keyboard focus.
Parked uses a small icon on phones, with an accessible label; desktop retains the text.
Verified no overlaps or page overflow at nine widths, from 320px to 1440px.
2026-10-06 00:23:54 -04:00
firestar5683 6f08e10686 Build 2026-10-05 23:21:01 -05:00
firestar5683 df540edd42 Galaxy Interface Contracts 2026-10-05 23:19:12 -05:00
firestar5683 5f08653c24 Shared Safety Headers 2026-10-05 23:10:40 -05:00
firestar5683 26f4468314 Steering Replay Checks 2026-10-05 23:10:40 -05:00
firestar5683 c310dacc2d CI Test Fixtures 2026-10-05 23:10:40 -05:00
firestar5683 8ade5916f4 Galaxy Control Settings 2026-10-05 23:10:39 -05:00
firestar5683 5cbe643f40 Bolt Pedal Control 2026-10-05 23:10:39 -05:00
firestarsdog 026ac0b1ce Pagination Swipe QoL 2026-10-05 23:42:46 -04:00
Prabhaav Pillai 7b84b4e621 (Another one) round of Galaxy UI bug fixes
VASM/PiP previews fixed with shared camera lifecycle, timeouts and automatic recovery.
Theme Maker side cameras now work independently.
VASM handles warning borders only; PiP handles cameras and saved layouts.
Refresh/Retry controls removed in favor of automatic state updates.
Disconnect/loading states now persist correctly and recover automatically.
Settings sync and shared-write propagation restored.
Toggle/editor and Vehicle Default flicker fixed.
Default actions simplified and numeric Default restored.
Sidebar and Tools now share one catalog.
Notices and form controls standardized.
Recordings UI improved with search, sorting and preservation filters.
Playback supports full-resolution cameras and automatic segment advancement.
Recording downloads and management restored.
Recording status now reflects actual scan state.
Map partial-recovery errors fixed.
Regression coverage expanded across browser, camera, polling, recording and placement flows.
2026-10-05 23:28:46 -04:00
firestar5683 9d02cc4f61 Build 2026-10-05 21:35:10 -05:00
firestar5683 0017b1992c CEM Standstill Hold 2026-10-05 21:33:23 -05:00
firestar5683 417f4e09d9 Cruise Resume Memory 2026-10-05 21:33:23 -05:00
firestar5683 85a6f7c992 Optional Android Auto Health 2026-10-05 21:33:10 -05:00
firestar5683 6c1977ba9f Build 2026-10-05 20:26:49 -05:00
firestar5683 0820a009cc Volt Cruise Pedal Control 2026-10-05 20:24:44 -05:00
firestar5683 1dba933c25 Build 2026-10-05 19:40:13 -05:00
firestar5683 94471f853e Lane Centering Strength 2026-10-05 19:38:11 -05:00
firestar5683 7a0c8f721c Ioniq 6 Steering Messages 2026-10-05 19:38:11 -05:00
firestar5683 79322cfb1e GPU Status 2026-10-05 19:38:11 -05:00
firestar5683 e01e290109 Build 2026-10-05 18:59:50 -05:00
firestar5683 66a706b4b6 Volt 2019 Pedal Control 2026-10-05 18:59:49 -05:00
firestar5683 af1c132e56 Build 2026-10-05 18:59:49 -05:00
firestar5683 bf7966efb9 Volt ASCM Pedal Control 2026-10-05 18:59:49 -05:00
firestar5683 58eef91c2a Build 2026-10-05 18:59:49 -05:00
firestar5683 4ea1b49b67 Volt Gateway Pedal Control 2026-10-05 18:59:48 -05:00
firestarsdog 74e08d7e38 Baseline Swipe Pagination 2026-10-05 19:01:08 -04:00
firestar5683 f36dcfa4f3 Build 2026-10-05 17:23:04 -05:00
firestar5683 f9e70f2ee9 Offroad Settings 2026-10-05 17:21:39 -05:00
firestar5683 a1bdaf855f Highlander Lateral Control 2026-10-05 17:21:39 -05:00
firestar5683 b4a49b74cf Volt Camera Pedal Control 2026-10-05 17:21:39 -05:00
firestar5683 c6e826b0e4 AOL Seatbelt Behavior 2026-10-05 16:02:20 -05:00
firestar5683 8aea0c1ac0 GM Camera Pedal Control 2026-10-05 16:00:59 -05:00
firestar5683 ae3cbf9340 UI and Update Checks 2026-10-05 15:53:20 -05:00
firestar5683 3e39b1531c AOL Calibration Recovery
Co-authored-by: Intelli <contact@intelli.software>
2026-10-05 15:53:20 -05:00
firestar5683 5debd4cd8e Recent Update Versions
Co-authored-by: dirwin31 <83434411+dirwin31@users.noreply.github.com>
2026-10-05 15:53:20 -05:00
firestar5683 82daa5bbe9 Updater Process Cleanup
Co-authored-by: dirwin31 <83434411+dirwin31@users.noreply.github.com>
2026-10-05 15:53:20 -05:00
firestar5683 63ea964a56 Training Texture Cleanup
Co-authored-by: dirwin31 <83434411+dirwin31@users.noreply.github.com>
2026-10-05 15:53:20 -05:00
firestar5683 cecba8590a Label Layout Cache
Co-authored-by: dirwin31 <83434411+dirwin31@users.noreply.github.com>
2026-10-05 15:53:20 -05:00
firestar5683 d29613b7c5 Bounded Text Caches
Co-authored-by: dirwin31 <83434411+dirwin31@users.noreply.github.com>
2026-10-05 15:53:20 -05:00
firestar5683 6e48383568 EV9 Steering Targets
Co-authored-by: Intelli <contact@intelli.software>
2026-10-05 15:53:20 -05:00
firestar5683 570fc2d71c Hyundai Angle Validation
Co-authored-by: Intelli <contact@intelli.software>
2026-10-05 15:53:20 -05:00
firestar5683 0b3997491d ID4 Vehicle Detection
Co-authored-by: Ian <ianpang126@gmail.com>
2026-10-05 15:53:20 -05:00
firestar5683 c8badeddf7 AOL Steering Feedback
Co-authored-by: Intelli <contact@intelli.software>
2026-10-05 15:53:20 -05:00
firestar5683 763f9171ee Steering Direction
Co-authored-by: pharmacomaniac <blittle65@gmail.com>
2026-10-05 15:53:20 -05:00
firestar5683 6465fa5e37 Niro Follow Tuning 2026-10-05 15:53:20 -05:00
firestar5683 54260529a5 Tesla Screen Controls 2026-10-05 15:53:20 -05:00
firestarsdog a62ef8e3ea Menu clamp 2026-10-05 15:18:32 -04:00
firestar5683 9999b0686a Fast Update Compatibility 2026-10-05 13:58:34 -05:00
firestar5683 5fbb935a2c Usage Statistics 2026-10-05 13:18:55 -05:00
firestar5683 c6b73f3afa Jetlink Support 2026-10-05 12:55:41 -05:00
firestar5683 6c8966d044 Jetlink Client 2026-10-05 12:21:26 -05:00
firestar5683 674ece7430 Developer Test Command 2026-10-05 11:41:45 -05:00
firestar5683 2f8a0096c2 Build 2026-10-05 11:02:36 -05:00
firestar5683 758942fdbf Large Settings Types 2026-10-05 11:00:54 -05:00
firestar5683 3046431700 Build 2026-10-05 10:57:54 -05:00
firestar5683 896994d17b Pedal Sensor Inputs 2026-10-05 10:57:54 -05:00
firestar5683 ed6c19f4d3 Build 2026-10-05 10:57:54 -05:00
firestar5683 525d9ab26d Paddle Modes 2026-10-05 10:57:53 -05:00
firestar5683 55b0c8e12c Build 2026-10-05 10:57:53 -05:00
firestar5683 d8cc751cef Volt One-Pedal 2026-10-05 10:57:53 -05:00
firestar5683 ead4c0cf73 Build 2026-10-05 10:57:53 -05:00
firestar5683 efa7781dfe GM Gateway Steering 2026-10-05 10:57:52 -05:00
firestar5683 2709f1c642 Build 2026-10-05 10:57:52 -05:00
firestar5683 de589222a7 GM Brake Hold 2026-10-05 10:57:52 -05:00
firestar5683 c1d07152c1 Build 2026-10-05 10:57:52 -05:00
firestar5683 fa9341ec6f Volt 2019 Brake Hold 2026-10-05 10:57:51 -05:00
firestar5683 00207c7b6c Build 2026-10-05 10:57:51 -05:00
firestar5683 09f7de29a5 Volt Camera Auto Hold 2026-10-05 10:57:51 -05:00
firestar5683 3ab13b3434 Build 2026-10-05 10:57:51 -05:00
firestar5683 6a11b09031 Volt Stop Controls 2026-10-05 10:57:50 -05:00
firestar5683 ca305dda2e Build 2026-10-05 10:57:50 -05:00
firestar5683 16fb0eee3b Replay Validation 2026-10-05 10:57:50 -05:00
firestar5683 6327ec50e0 Firmware Build Stamp 2026-10-05 10:57:50 -05:00
firestar5683 0986e61b4e UI Snapshot Reuse 2026-10-05 10:57:50 -05:00
firestar5683 3f0d494764 Ford Engagement 2026-10-05 10:57:50 -05:00
firestarsdog 06dc2cdfbf Add Stars to Header, very important 2026-10-05 02:21:33 -04:00
firestarsdog 972d918f39 Swap Large UI to Horizontal Pagination and StarPilot Styling 2026-10-05 02:05:28 -04:00
firestar5683 003a3bfded CI Workflow Validation 2026-10-04 23:57:53 -05:00
firestarsdog ed2ee4b772 Device menu : Galaxy line top, power options bottom 2026-10-05 00:57:00 -04:00
firestarsdog 3268cd014c Large UI Software Panel Fixes 2026-10-05 00:45:02 -04:00
firestar5683 195333a76a Build 2026-10-04 23:33:59 -05:00
firestar5683 7df3b2aef4 Mach-E Steering 2026-10-04 23:30:46 -05:00
firestar5683 cf9366e5e1 Toyota Auto Hold 2026-10-04 23:29:20 -05:00
firestar5683 1cdb868434 Build 2026-10-04 22:39:31 -05:00
firestar5683 41b2def9c4 GPIO Test Contracts 2026-10-04 22:31:08 -05:00
firestar5683 418b1c8e43 GPU Load Diagnostics 2026-10-04 22:31:07 -05:00
firestar5683 5c8b676c03 GPU Status 2026-10-04 22:31:05 -05:00
firestar5683 942a562a0a Bolt Gear State 2026-10-04 22:31:03 -05:00
firestarsdog a71528c485 Batch AGNOS GPIO Permission Rules 2026-10-04 22:56:08 -04:00
firestar5683 f6a01d2126 Build 2026-10-04 21:32:23 -05:00
firestar5683 262ec0b938 C3 Test Types 2026-10-04 21:32:03 -05:00
firestar5683 3c578f310e Galaxy Settings Defaults and Availability 2026-10-04 21:28:43 -05:00
firestar5683 5a6fbb0e92 Restore Steering Settings 2026-10-04 21:28:43 -05:00
firestar5683 93fcddaa9f Sidebar Telemetry Timing 2026-10-04 21:28:43 -05:00
firestar5683 a0950855e0 Build 2026-10-04 21:28:43 -05:00
firestar5683 f2ccd222cf Vehicle Preference Context 2026-10-04 21:28:43 -05:00
firestar5683 5b2371e6c8 Galaxy Connection Queue 2026-10-04 21:28:43 -05:00
firestar5683 530ce968e0 Manual Vehicle Settings 2026-10-04 21:28:42 -05:00
firestar5683 4e4ebd5866 Big UI Sidebars 2026-10-04 21:28:42 -05:00
firestar5683 360802641e Build 2026-10-04 21:28:42 -05:00
firestar5683 418b295dbe Release Build Checks 2026-10-04 21:28:42 -05:00
firestar5683 1bbcc22f1c Safety Test Frames 2026-10-04 21:28:42 -05:00
firestar5683 11a83e49f8 Bolt Control Checks 2026-10-04 21:28:42 -05:00
firestarsdog 348b35788c Fix C3 Panda Start Speed 2026-10-04 21:59:42 -04:00
firestarsdog ecff3fce24 C3 Panda Rules 2026-10-04 20:46:29 -04:00
firestar5683 a379d94725 Build 2026-10-04 19:31:40 -05:00
firestar5683 2e16b3bbac Validation Fixes 2026-10-04 19:29:11 -05:00
firestar5683 3a4e8f7625 Bolt Low Gear 2026-10-04 19:23:02 -05:00
firestar5683 da1820b7db Build 2026-10-04 19:23:02 -05:00
firestar5683 db1d677610 Vehicle Cache Migration 2026-10-04 19:23:02 -05:00
firestar5683 98c6c8b334 Build 2026-10-04 19:23:02 -05:00
firestar5683 e71c919fff Hyundai EV Longitudinal 2026-10-04 19:22:46 -05:00
firestar5683 00ce937ab9 Build 2026-10-04 19:22:45 -05:00
firestar5683 7d8baeb124 Ioniq Cluster Status 2026-10-04 19:22:45 -05:00
firestar5683 9d4414538e Build 2026-10-04 19:22:45 -05:00
firestar5683 724cbb0f00 Fleet Coverage 2026-10-04 19:22:30 -05:00
firestar5683 8d8f325a1b GM Control Tests 2026-10-04 19:22:30 -05:00
firestar5683 6f96d1b544 Android Auto Tests 2026-10-04 19:22:30 -05:00
firestar5683 25faf72be2 Map Package 2026-10-04 19:22:30 -05:00
firestar5683 0220ef9b1d Model Build Compatibility 2026-10-04 19:22:29 -05:00
firestarsdog 3296616331 C3 Panda + Build 2026-10-04 19:59:30 -04:00
Prabhaav Pillai 4b94a6a115 A round of Galaxy UI bug fixes:
Mobile "Search toggles": collapses to icon and stays in top bar.
Home page multiple Refresh buttons: consolidated to one.
Nav-bar blur lost on narrow widths: glass blur encases nav, search collapses to button.
Galaxy logo/home now looks clickable (border and hover).
Toggles page dual search: page search moved to an icon button near refresh.
Global search false "could not be searched" plus result flicker: only warns when nothing loaded, no per-batch republish.
Mobile native dropdown inconsistency: unified rendering.
Toggle change caused page re-render flicker: dimming no longer tied to save state.
Missing toggles reset button: restored.
Redundant and inconsistent toggle descriptions: moved to shared GxNotice, all pages covered.
Developer tab spacing: consistent card/section layout.
Force-drive Offroad/Onroad/Auto buttons: consistent tool styling.
Galaxy Tunnel install text: accurate PWA install and benefits copy.
Colors and Layout banner/margins plus misplaced "Park" note: consistent savebar and warn banner.
Quick Select oversized intro/margins plus Reload text: icon button.
Blind Spot crop X/Y toggles hidden (kept writable via API/editor).
Crop editor link dead-end: now routes to Blind Spot Camera, renamed "and Preview".
Sound pack "Park the vehicle" full-box description: inlined into header hint.
Tools/Logs/Plots inconsistent submenu tiles (icon centering, hover/press): shared MenuTile.
Navigation bottom button hidden behind nav bar: clearance fix.
Navigation page nav bar hidden plus blurred Exit button restored.
Redundant Onroad Appearance: removed tool, Colors and Layout promoted to tool.
Plots and Analysis submenu panel inconsistent: uses MenuTile.
Theme Maker and Onroad Appearance consolidation.
2026-10-04 19:12:49 -04:00
firestarsdog 7880a4e3be Possible C3 Panda Fix 2026-10-04 18:43:17 -04:00
firestar5683 b5d02a8fff Build 2026-10-04 17:26:41 -05:00
firestar5683 dca64b67ec Preserve Model Runtime 2026-10-04 17:25:04 -05:00
firestar5683 2c8e8b5879 CI Contracts 2026-10-04 17:23:26 -05:00
firestar5683 0858e178e3 Runtime Compatibility 2026-10-04 17:23:26 -05:00
firestar5683 da4bceb5ac Map IPC Compatibility 2026-10-04 17:23:25 -05:00
firestar5683 3b305b12b4 Fast Update 2026-10-04 17:23:25 -05:00
firestar5683 a0a40f9e83 Supported Driving Defaults 2026-10-04 17:23:25 -05:00
firestar5683 dad0143c9a Vehicle Engagement and Pedal Detection 2026-10-04 17:23:24 -05:00
firestar5683 f586da6eac Tesla Stock Steering 2026-10-04 16:54:56 -05:00
firestar5683 83c42cacda Mazda Steering Control 2026-10-04 16:54:56 -05:00
firestarsdog efa2c71553 Large UI Vehicle panel bugfixes 2026-10-04 17:47:28 -04:00
Prabhaav Pillai ae720a05b7 build command to include 'galaxy' in the target list 2026-10-04 16:54:01 -04:00
firestarsdog d861b2a3a5 Vehicle panel + 2026-10-04 14:30:40 -04:00
firestar5683 e68ed75ee6 Driving Controls and Wheel Actions 2026-10-04 12:24:27 -05:00
firestar5683 6aec1f863a Vehicle Test Discovery 2026-10-04 10:58:29 -05:00
firestar5683 11a4ef2662 Honda Stock Cruise 2026-10-04 10:41:53 -05:00
firestar5683 9737753948 Honda Steering and Pedal Control 2026-10-04 10:22:18 -05:00
firestar5683 09cd3317bc Build Validation 2026-10-04 10:02:54 -05:00
firestar5683 68ad7fdfc2 Honda Radar Tracking 2026-10-04 09:40:48 -05:00
firestar5683 f40bc5b8cb Honda Clarity Identification 2026-10-04 09:40:48 -05:00
firestar5683 42e43d7043 Honda City Identification 2026-10-04 09:40:48 -05:00
firestar5683 c399146acb Honda Cruise Signal Sources 2026-10-04 09:40:48 -05:00
firestar5683 c541313f2a Honda Modified EPS 2026-10-04 09:40:47 -05:00
firestar5683 0856cb868f Honda Longitudinal Control 2026-10-04 09:40:47 -05:00
firestar5683 98826823cb Honda Transmission Detection 2026-10-04 09:40:47 -05:00
firestar5683 a79ce16bf9 Honda Controller Settings 2026-10-04 09:40:47 -05:00
firestar5683 d4f67e6b59 build 2026-10-04 09:40:47 -05:00
firestar5683 8ac3813088 Ford Safety Storage 2026-10-04 09:40:46 -05:00
firestar5683 2a5ecb035b Ford Safety Memory 2026-10-04 09:40:46 -05:00
firestar5683 266c7367f6 Ford Always On Lateral 2026-10-04 09:40:46 -05:00
firestar5683 c92b3d597d Ford Cruise Selection Tests 2026-10-04 09:40:46 -05:00
firestar5683 661aa4d108 Ford Steering Settings 2026-10-04 09:40:46 -05:00
firestar5683 e244fd106e Mach-E Camera Leads 2026-10-04 09:40:46 -05:00
firestar5683 b9c4b00c60 Ford Startup Health 2026-10-04 09:40:45 -05:00
firestar5683 92de8bc96c Ford CAN FD Control 2026-10-04 09:40:45 -05:00
firestar5683 005f6dc682 build 2026-10-04 09:40:45 -05:00
firestar5683 845a476605 Hyundai Stock AOL 2026-10-04 09:40:45 -05:00
firestar5683 e13d16c476 build 2026-10-04 09:40:45 -05:00
firestar5683 b629ecd79c Panda Power State 2026-10-04 09:40:45 -05:00
firestar5683 025b6ca3ae Hyundai Safety Profiles 2026-10-04 09:40:45 -05:00
firestar5683 ea0198cf57 G90 Steering Startup 2026-10-04 09:40:44 -05:00
firestar5683 4440104f27 Hyundai AOL Profiles 2026-10-04 09:40:44 -05:00
firestar5683 1e4c7a3b31 Hyundai Stock Cruise Steering 2026-10-04 09:40:44 -05:00
firestar5683 9debca8d36 Hyundai Torque Options 2026-10-04 09:40:44 -05:00
firestar5683 15d3dc0a1d Hyundai Steering Policies 2026-10-04 09:40:44 -05:00
firestar5683 02eb9a0748 Hyundai Automatic Identification 2026-10-04 09:40:44 -05:00
firestar5683 ec8e99df71 Palisade Steering 2026-10-04 09:40:44 -05:00
firestar5683 dec33429fc Hyundai Startup Health 2026-10-04 09:40:43 -05:00
firestar5683 a3a4c8e7e4 GM Automatic Identification 2026-10-04 09:40:43 -05:00
firestar5683 660981b819 Bolt Camera Configuration 2026-10-04 09:40:43 -05:00
firestar5683 6dcab7c8f8 build 2026-10-04 09:40:43 -05:00
firestar5683 581392189a GM Startup Checks 2026-10-04 09:40:43 -05:00
firestar5683 67fb3b5ff7 Camera Session Cleanup 2026-10-04 09:40:43 -05:00
firestar5683 d21ccb6c11 Comma 3 Audio and Startup 2026-10-04 09:40:43 -05:00
firestarsdog 01a5cc4d91 Header border 2026-10-04 07:13:17 -04:00
firestarsdog f3e6ccda01 Header size correction 2026-10-04 07:04:43 -04:00
firestarsdog 22dae22085 Text Size Standardization and Fixes 2026-10-04 06:56:35 -04:00
firestarsdog c247948fc2 Dev tools fixes 2026-10-04 05:49:07 -04:00
firestarsdog cba196e25d Big UI Parity Pass Bugfixes 2026-10-04 05:33:15 -04:00
firestarsdog 42d3611932 StarPilot Big UI Styling 2026-10-04 04:36:14 -04:00
firestar5683 d31e11b20d build 2026-10-04 00:41:11 -05:00
firestar5683 54fbde0bf3 Comma 3 Panda Support 2026-10-04 00:39:39 -05:00
firestar5683 e87f51970b Panda Sidebar Status 2026-10-04 00:19:41 -05:00
firestar5683 6edbcde6e2 build 2026-10-04 00:19:41 -05:00
firestar5683 ccc0886604 RDFv4 Default Model 2026-10-04 00:19:40 -05:00
firestarsdog 3f0ce8f1b3 Start building up big ui bluetooth panel 2026-10-04 00:55:20 -04:00
firestar5683 67a9e60e23 Params Registry Checks 2026-10-03 23:34:03 -05:00
firestar5683 e7a92e4bee Standard Camera Default 2026-10-03 23:34:02 -05:00
firestar5683 90f40a6209 Bolt Controls Startup 2026-10-03 23:34:02 -05:00
firestar5683 ebd948937d Fast Update Fix 2026-10-03 23:34:02 -05:00
firestar5683 f7a041d575 Force Offroad Dialog 2026-10-03 23:34:01 -05:00
firestarsdog 49f7f18c46 Big UI Settings Fixes 2026-10-04 00:26:30 -04:00
firestar5683 de8336a84f Dom Compatibility 2026-10-03 23:06:07 -05:00
firestar5683 4fd849375e Soundd Fix 2026-10-03 22:52:51 -05:00
firestar5683 c6ab815cb3 Galaxy Pairing Migration 2026-10-03 22:52:46 -05:00
firestarsdog 5e54e17fa3 Ungate Network/Bluetooth panels 2026-10-03 23:46:23 -04:00
firestar5683 fe112f464d Bolt No-Acc Triage 2026-10-03 22:12:11 -05:00
firestar5683 39c1767f65 Fast Updater 2026-10-03 21:22:38 -05:00
firestar5683 a8915ca858 Force Drive State Icon 2026-10-03 20:54:12 -05:00
firestar5683 048094601f Show bundled model names on the home screen 2026-10-03 20:51:23 -05:00
firestar5683 105d2262e4 Restore Big UI Bluetooth and Developer menus 2026-10-03 20:51:23 -05:00
firestar5683 d84ab9d0ec Preserve taps within Big UI controls 2026-10-03 20:51:23 -05:00
firestar5683 14b2e8459f Restart the UI after unexpected process exits 2026-10-03 20:51:23 -05:00
firestar5683 2459ba878c Start new installations with fresh driving settings 2026-10-03 20:51:23 -05:00
firestar5683 89764d2d1b Handle Migration Cleanup 2026-10-03 19:46:08 -05:00
firestar5683 3a409748ac Match AGNOS version to system30 image 2026-10-03 18:32:20 -05:00
firestar5683 aff36eadc9 Fix AGNOS image download sources 2026-10-03 18:26:08 -05:00
firestar5683 b07d3dd5d4 build 2026-10-03 17:39:42 -05:00
firestar5683 9298db235b Galaxy HTTP and settings registration
Register Galaxy managed services, recording media and settings endpoints after their underlying owners exist. Cover authentication and cross-feature request behavior at the HTTP boundary.
2026-10-03 22:34:37 +00:00
firestar5683 523a73bbf9 Background drive analysis and statistics
Implement background drive analysis and statistics with explicit worker and persistence ownership.
2026-10-03 22:34:37 +00:00
firestar5683 2bf401019a Fleet learning presentation
Expose fleet-learning operations through Galaxy presentation endpoints.
2026-10-03 22:34:37 +00:00
firestar5683 6601a500e3 Fleet-learning analysis workers
Analyze closed recordings in bounded workers, validate static vehicle context, and retain actionable progress and failures. Honor finite and unlimited process resource limits.
2026-10-03 22:34:37 +00:00
firestar5683 6f30e17b83 Drive-history summaries and plots
Expose drive history summaries and plot data through Galaxy.
2026-10-03 22:34:37 +00:00
firestar5683 1bf86e10c3 Fleet-learning data contracts
Provide offline log decoding and torque-surface data contracts for fleet learning.
2026-10-03 22:34:37 +00:00
firestar5683 36a1f43816 UI application registration
Register shared settings and application pages in both UI layouts with capability-aware wheel assignments.
2026-10-03 22:34:37 +00:00
firestar5683 64abcb4094 Controller callbacks and joined feature coverage
Wire controller callbacks into the running process and add joined planner and mode-action regression coverage.
2026-10-03 22:34:37 +00:00
firestar5683 408ab1a9d7 Galaxy device and remote-access exports
Expose Galaxy device, crash and remote-access services with the bundled frpc license.
2026-10-03 22:34:37 +00:00
firestar5683 b011a1fdcd Vehicle and control process integration
Connect shared feature owners to vehicle and control processes. Migrate retired settings once before startup consumers while preserving effective preferences and vehicle-specific numeric adjustments.
2026-10-03 22:34:37 +00:00
firestar5683 de611f978c Compact Bluetooth settings
Add compact Bluetooth settings after the transport and shared UI owners are available.
2026-10-03 22:34:37 +00:00
firestar5683 ad88da0c8d Software presentation
Expose software operations and status through Galaxy.
2026-10-03 22:34:37 +00:00
firestar5683 c7d90d1715 Bluetooth and Android Auto presentation
Expose Android Auto setup and projection layout through Galaxy interfaces.
2026-10-03 22:34:37 +00:00
firestar5683 ba4d3b249b Display and annotation presentation
Expose onroad layout preferences through Galaxy.
2026-10-03 22:34:37 +00:00
firestar5683 99a8b1c07e Bluetooth and Android Auto
Manage named devices, car-first projection reconnect, and saved head-unit layouts. Keep runtime-safe Bluetooth operations available without changing controller ownership or restarting the radio unnecessarily.
2026-10-03 22:34:37 +00:00
firestar5683 aaac7a6733 Shared settings and onroad views
Present saved configuration and live driving state through native Small and Big UI layouts. Support per-field tune resets, movable widgets, camera availability, and a unified speed card while preserving decision-bound actions.
2026-10-03 22:34:37 +00:00
firestar5683 6e66450bd9 Vision assisted annotation
Implement vision annotation observations, inference and preferences through a dedicated runtime owner.
2026-10-03 22:34:37 +00:00
firestar5683 63ceaaafca Vendored runtime changes
Extend vendored runtime helpers and diagnostics while preserving the existing compiler and model ABI.
2026-10-03 22:34:37 +00:00
firestar5683 4e46324ad5 Protocol integration coverage
Add integrated wire and protocol fixtures that exercise the registered host and vehicle adapters together.
2026-10-03 22:34:37 +00:00
firestar5683 76523bb9e2 Native host protocol and adapter registration
Wire Panda host subscriptions and AOL adapters into the native daemon build and startup path.
2026-10-03 22:34:37 +00:00
firestar5683 942228400c Longitudinal registration and planner wiring
Wire longitudinal feature selection and inputs into planner hosts after their owners are available.
2026-10-03 22:34:37 +00:00
firestar5683 0051aa6047 Audio
Add selectable alert sounds, volume ownership and sound lifecycle handling, including dismissal and repeated-alert boundaries.
2026-10-03 22:34:37 +00:00
firestar5683 7db49d7f22 AOL registration and process wiring
Register vehicle AOL adapters and runtime dispatch after the vehicle factories and native contracts exist.
2026-10-03 22:34:37 +00:00
firestar5683 7bb0b5c6a0 Volvo C1 control port
Add the Volvo C1 port with matching Python registration and exact native parameter admission.
2026-10-03 22:34:37 +00:00
firestar5683 ae34cdca55 Volkswagen identification extensions
Update Volkswagen vehicle identification and parameter coverage.
2026-10-03 22:34:37 +00:00
firestar5683 1cac872d89 Vehicle preferences and startup registration
Resolve startup preferences and controller extensions through capability-aware vehicle owners.
2026-10-03 22:34:37 +00:00
firestar5683 ee1c3e65cc Toyota ADAS and control extensions
Update Toyota ADAS messages and control interfaces with platform-specific safety tests.
2026-10-03 22:34:37 +00:00
firestar5683 cde956b100 Tesla HW1 and namespace safety
Add Tesla HW1 and supported namespace handling with explicit native admission and unknown-bit rejection.
2026-10-03 22:34:37 +00:00
firestar5683 57323c820a Subaru control extensions
Update Subaru vehicle messages and controller/native safety integration.
2026-10-03 22:34:37 +00:00
firestar5683 14cb60b8e5 Longitudinal planner owners
Implement acceleration profiles, FollowJerk, force-stop and lead-departure owners. Keep selected acceleration limits and hazard arbitration explicit at their shared boundaries.
2026-10-03 22:34:37 +00:00
firestar5683 cf0f265ac7 Vehicle policy registration
Select longitudinal policy through the shared vehicle dispatch contract.
2026-10-03 22:34:37 +00:00
firestar5683 7388de875e Mazda interface extensions
Update Mazda vehicle interface behavior with focused stock-control tests.
2026-10-03 22:34:37 +00:00
firestar5683 a71d1f3e48 Torque and lateral controllers
Integrate torque feedback, lane-change smoothing and low-speed Turn Assist through dedicated lateral owners.
2026-10-03 22:34:37 +00:00
firestar5683 697c07e20f Navigation presentation
Expose map operations and status through Galaxy presentation endpoints.
2026-10-03 22:34:37 +00:00
firestar5683 6ba0dd8708 Curve speed
Resolve curve speed and smoothing through a dedicated learning and preference owner.
2026-10-03 22:34:37 +00:00
firestar5683 c74d05643d Controller feature owners
Route controller input and cruise actions through shared transport and mode-action owners.
2026-10-03 22:34:37 +00:00
firestar5683 1ab5058bc8 Conditional experimental mode
Implement conditional driving modes with explicit scene, button and traffic state ownership.
2026-10-03 22:34:37 +00:00
firestar5683 14241d72c8 Speed limits
Select qualified speed-limit sources, preserve acceptance and override decisions, and publish distinct control and display coordinates. Carry accepted and pending source identity with each displayed decision.
2026-10-03 22:34:37 +00:00
firestar5683 59e7e272ba Nostalgia
Add the optional nostalgia behavior as an isolated feature owner.
2026-10-03 22:34:37 +00:00
firestar5683 6369f355ff Hyundai/Kia/Genesis control extensions
Integrate vehicle-specific steering, longitudinal control, startup, and handoff policies, including Ioniq, EV9, GV70, and non-SCC configurations. Preserve the corresponding native safety and CAN routing contracts.
2026-10-03 22:34:37 +00:00
firestar5683 4b8be2b826 Honda steering and AOL extensions
Integrate Honda steering and AOL adapters with platform-specific native safety coverage.
2026-10-03 22:34:37 +00:00
firestar5683 3d8751b21d GM host AOL admission policy
Apply the GM host AOL policy after its control families and capabilities are available.
2026-10-03 22:34:37 +00:00
firestar5683 536fd0c8c6 Ford curvature and control extensions
Preserve vehicle-specific lateral strategies, stock cruise switch context, and native bus permissions for the supported Ford variants. Keep controller policy separate from shared steering integration.
2026-10-03 22:34:37 +00:00
firestar5683 bc8ae160fa Chrysler control extensions
Update Chrysler vehicle parameters and controller behavior with focused platform tests.
2026-10-03 22:34:37 +00:00
firestar5683 8cb60d82fe Developer and build tooling
Provide laptop and device build entry points, cache compatibility checks and developer tooling. Keep distribution assets and model packaging available before runtime consumers.
2026-10-03 22:34:37 +00:00
firestar5683 75327f5ade UI replay and startup helpers
Provide UI replay and startup helpers for inspecting onroad rendering without a live drive.
2026-10-03 22:34:37 +00:00
firestar5683 035fd3b547 Recorded vehicle qualification fixtures
Add compact recorded vehicle fixtures and a runner for reproducible source-level qualification.
2026-10-03 22:34:37 +00:00
firestar5683 e66380e8fa Software and AGNOS
Manage software updates and AGNOS retained state through explicit manifests and update preferences.
2026-10-03 22:34:37 +00:00
firestar5683 8c9dd4b00a Sentry presentation
Expose Sentry events and notification presentation through Galaxy.
2026-10-03 22:34:37 +00:00
firestar5683 2dd715a3d5 Sentry
Implement Sentry scanning, notification state and worker cancellation through explicit policies and preferences.
2026-10-03 22:34:37 +00:00
firestar5683 b282ced7fb Project documentation and attribution
Document supported behavior, contributor attribution and third-party licenses alongside the source distribution.
2026-10-03 22:34:37 +00:00
firestar5683 f71e88ce25 Power
Resolve offroad power preferences through a dedicated owner.
2026-10-03 22:34:37 +00:00
firestar5683 1ce9f73ae2 Navigation and maps
Add navigation and map matching providers with the vendored map service sources.
2026-10-03 22:34:37 +00:00
firestar5683 2818bb072e Model management
Own model discovery, compilation and registry selection through explicit model metadata and geometry contracts.
2026-10-03 22:34:37 +00:00
firestar5683 1ff98bacf6 Galaxy web presentation
Add Galaxy web assets and client settings interfaces with their bundled vendor notices.
2026-10-03 22:34:37 +00:00
firestar5683 51a0d50187 Galaxy access and remote service
Separate Galaxy authentication and local access policy from web presentation.
2026-10-03 22:34:37 +00:00
firestar5683 e38d39358e GM factory and native registration
Register GM controllers, car state, distance-button actions and feature capabilities against the shared platform groups.
2026-10-03 22:34:37 +00:00
firestar5683 6d477cb73e GM Bolt Pedal
Implement Bolt Pedal command generation and mode selection with dedicated native safety coverage. Keep pedal actuation separate from the factory cruise families.
2026-10-03 22:34:37 +00:00
firestar5683 deaa5061b2 GM Volt and factory ACC control owners
Add distinct Volt, Silverado, Suburban and factory ACC control owners rather than sharing pedal assumptions across platforms.
2026-10-03 22:34:37 +00:00
firestar5683 696372c0e8 GM Bolt Cruise Control
Implement Bolt cruise control with its own host and native admission gates.
2026-10-03 22:34:37 +00:00
firestar5683 0de0df06e0 GM shared longitudinal policies
Provide shared GM longitudinal policy helpers for the vehicle-specific controllers.
2026-10-03 22:34:37 +00:00
firestar5683 8235b19e77 GM shared CP and message contracts
Define GM platform groups, startup preferences and CAN messages before their control families are registered.
2026-10-03 22:34:37 +00:00
firestar5683 255f0306eb Drive lifecycle and evidence
Track drive identity and evidence lifecycle so consumers reset state at the same drive boundaries.
2026-10-03 22:34:37 +00:00
firestar5683 79a5cf4fcd Connect service
Own Connect credentials, registration and upload preferences through shared access providers.
2026-10-03 22:34:37 +00:00
firestar5683 7b1cc1f8e2 Camera presentation
Serve fresh camera snapshots with explicit image responses and expose live crop controls without placing file or camera startup work on the UI thread.
2026-10-03 22:34:37 +00:00
firestar5683 d6766098ff Camera lifecycle and source qualification
Coordinate finite camera leases with manager-owned processes and reject frames that predate a request. Preserve the native frame timestamp contract across camera variants.
2026-10-03 22:34:37 +00:00
firestar5683 d734bebcb2 CI and developer qualification
Add control trace and platform qualification tools that exercise explicit safety modes and vehicle coverage.
2026-10-03 22:34:37 +00:00
firestar5683 706e760db0 Vehicle shared contracts and registry
Centralize vehicle lookup, firmware identification and dashboard limits while retaining the stock vehicle registry.
2026-10-03 22:34:37 +00:00
firestar5683 a1af95a500 Common AOL protocol and intent
Define host-to-Panda AOL intent, policy and wire contracts with neutral defaults before vehicle adapters register capabilities.
2026-10-03 22:34:37 +00:00
firestar5683 7c665bf16f Cruise speed contracts
Resolve cruise speed increments through a shared interval owner for consistent button behavior.
2026-10-03 22:34:37 +00:00
firestar5683 ed0407d982 Native safety and AOL contracts
Extend native safety and health contracts with explicit AOL state and bounded message handling. Keep vehicle-specific admissions outside the generic layer.
2026-10-03 22:34:37 +00:00
firestar5683 e996787af0 Shared UI data contracts
Introduce shared feature settings and presentation state so both UI layouts can consume the same resolved values.
2026-10-03 22:34:37 +00:00
firestar5683 b06f37e2c7 Settings schema and storage contracts
Define persistent settings, typed schema fields and storage migrations before feature owners consume them. Keep calibration and parameter readers aligned with the shared data contracts.
2026-10-03 22:34:37 +00:00
2970 changed files with 29068 additions and 656452 deletions
-2
View File
@@ -23,7 +23,6 @@ panda/** !filter !diff !merge -text
rednose_repo/** !filter !diff !merge -text
teleoprtc_repo/** !filter !diff !merge -text
tinygrad_repo/** !filter !diff !merge -text
tinygrad_model_runtime/** !filter !diff !merge -text
# Compiled model parts must retain exact binary bytes.
*.pkl.chunk* -text
@@ -35,7 +34,6 @@ tinygrad_model_runtime/** !filter !diff !merge -text
*.so.* -text
openpilot/cereal/messaging/bridge -text
openpilot/selfdrive/pandad/pandad -text
openpilot/tools/replay/replay -text
openpilot/system/camerad/camerad -text
openpilot/system/loggerd/bootlog -text
openpilot/system/loggerd/encoderd -text
-6
View File
@@ -1,6 +0,0 @@
github: firestar5683
patreon: firestar4430
custom:
- "https://cash.app/$domkthompson"
- "https://account.venmo.com/u/firestar4430"
- "https://www.paypal.com/paypalme/firestar4430"
@@ -0,0 +1,92 @@
name: Card stall native artifact qualification
on:
push:
branches: [diagnostics/card-stall-build-20261012]
paths:
- .github/workflows/card_stall_native_artifact.yaml
- tools/ci/card_stall_native_artifact/**
- msgq_repo/**
- openpilot/selfdrive/car/card.py
- openpilot/starpilot/card_stall_timing.py
- openpilot/starpilot/aol/tests/test_card_stall_timing.py
permissions:
contents: read
concurrency:
group: card-stall-native-artifact-${{ github.ref }}
cancel-in-progress: false
jobs:
native_arm64:
if: github.repository == 'firestar5683/StarPilot' && github.ref == 'refs/heads/diagnostics/card-stall-build-20261012'
runs-on: ubuntu-24.04-arm
timeout-minutes: 25
env:
PYTHONNOUSERSITE: '1'
UV_NO_PROGRESS: '1'
steps:
- name: Fetch exact public source with no archive fallback
shell: bash
env:
GIT_TERMINAL_PROMPT: '0'
GIT_LFS_SKIP_SMUDGE: '1'
GIT_CONFIG_NOSYSTEM: '1'
GIT_CONFIG_GLOBAL: /dev/null
run: |
set -euo pipefail
printf 'ARTIFACT_DIR=%s/card-stall-native-artifact\n' "$RUNNER_TEMP" >> "$GITHUB_ENV"
test "$GITHUB_REPOSITORY" = firestar5683/StarPilot
test "$GITHUB_REF" = refs/heads/diagnostics/card-stall-build-20261012
[[ "$GITHUB_SHA" =~ ^[0-9a-f]{40}$ ]]
python3 -c 'import re, subprocess; v = subprocess.check_output(["git", "version"], text=True); m = re.search(r"(\d+)\.(\d+)\.(\d+)", v); assert m and tuple(map(int, m.groups())) >= (2, 28, 0), v; print(v.strip())'
test ! -e .git
git init
git remote add origin https://github.com/firestar5683/StarPilot.git
git config remote.origin.promisor true
git config remote.origin.partialclonefilter blob:none
git sparse-checkout init --no-cone
printf '%s\n' '/msgq_repo/' '/openpilot/selfdrive/car/card.py' '/openpilot/starpilot/card_stall_timing.py' '/openpilot/starpilot/aol/tests/test_card_stall_timing.py' '/tools/ci/card_stall_native_artifact/' '/uv.lock' '/pyproject.toml' '/.github/workflows/card_stall_native_artifact.yaml' | git sparse-checkout set --no-cone --stdin
git -c protocol.version=2 fetch --filter=blob:none --depth=5 origin "$GITHUB_SHA"
test "$(git rev-parse FETCH_HEAD)" = "$GITHUB_SHA"
test "$(git rev-parse FETCH_HEAD^)" = c7f14368b90ff43d75909200f01923312f420504
test "$(git rev-parse FETCH_HEAD^^)" = c32912dbd66aabb62b67afa793af936c547e4f94
test "$(git rev-parse FETCH_HEAD^^^)" = 402a55b49d15df5cff985d5034f0e2053a4c677a
test "$(git rev-parse FETCH_HEAD^^^^)" = e8389b0b7529bad402f1f4e5fbd672ff13d9d382
git checkout --detach "$GITHUB_SHA"
test ! -e openpilot/selfdrive/modeld/models
- name: Verify frozen source and baseline artifacts
run: python3 tools/ci/card_stall_native_artifact/verify_source.py "$ARTIFACT_DIR/provenance"
- name: Install bounded native build prerequisites
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends gcc-13 g++-13 binutils libzmq5
- name: Install pinned Python environment manager
uses: astral-sh/setup-uv@e92bafb6253dcd438e0484186d7669ea7a8ca1cc # v6.4.3
with:
version: '0.12.5'
enable-cache: false
- name: Qualify source, build and execute native ARM tests
shell: bash
env:
EXPECTED_BASE_COMMIT: e8389b0b7529bad402f1f4e5fbd672ff13d9d382
EXPECTED_SOURCE_COMMIT: ${{ github.sha }}
EXPECTED_GCC_VERSION: '13.3.0'
GCC: /usr/bin/gcc-13
GXX: /usr/bin/g++-13
run: |
uv python install 3.12.13
uv venv --python 3.12.13 "$RUNNER_TEMP/card-stall-native-venv"
export PYTHON="$RUNNER_TEMP/card-stall-native-venv/bin/python"
uv pip install --python "$PYTHON" --require-hashes -r tools/ci/card_stall_native_artifact/requirements.lock
bash tools/ci/card_stall_native_artifact/build_native.sh "$PWD" "$ARTIFACT_DIR/provenance/baseline-ipc_pyx.so" "$ARTIFACT_DIR/build"
- name: Retain artifact and complete qualification evidence
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: card-stall-native-arm64-${{ github.sha }}
path: ${{ env.ARTIFACT_DIR }}/
if-no-files-found: error
retention-days: 7
compression-level: 6
+18 -1
View File
@@ -17,7 +17,7 @@ jobs:
strategy:
fail-fast: false
matrix:
gate: [coverage, opendbc-misra, panda-misra, panda-host]
gate: [coverage, opendbc-misra, panda-misra, panda-host, mutation-list]
steps:
- uses: actions/checkout@v7
- run: ./tools/op.sh setup
@@ -30,3 +30,20 @@ jobs:
name: safety-${{ matrix.gate }}
path: ci-results/safety-${{ matrix.gate }}/
if-no-files-found: error
full_mutation:
name: full safety mutation
runs-on: ubuntu-24.04
timeout-minutes: 360
steps:
- uses: actions/checkout@v7
- run: ./tools/op.sh setup
- name: Run all discovered mutants
run: .venv/bin/python tools/ci/run_safety_qualification.py --gate mutation-full --output ci-results/safety-mutation-full
- name: Save full mutation evidence
if: always()
uses: actions/upload-artifact@v4
with:
name: safety-mutation-full
path: ci-results/safety-mutation-full/
if-no-files-found: error
-28
View File
@@ -1,28 +0,0 @@
name: simulator driving (manual)
on:
workflow_dispatch:
permissions:
contents: read
env:
CI: 1
PYTHONPATH: ${{ github.workspace }}
jobs:
simulator_driving:
name: simulator driving
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- uses: actions/checkout@v7
- run: ./tools/op.sh setup
- name: Build openpilot
run: scons
- name: Driving test
timeout-minutes: 2
env:
# MetaDrive renders offscreen through panda3d's EGL pipe on llvmpipe
EGL_PLATFORM: surfaceless
run: python openpilot/tools/sim/tests/test_metadrive_bridge.py
+22 -8
View File
@@ -186,11 +186,6 @@ jobs:
- name: Build openpilot
id: build
run: scons
- name: Provide H.264 encoder for Android Auto integration tests
if: ${{ !cancelled() && steps.build.outcome == 'success' }}
run: |
uv pip install --python .venv/bin/python --only-binary av av==19.0.1
.venv/bin/python -c "import av; assert 'libx264' in av.codecs_available"
- name: Run unit tests
if: ${{ !cancelled() && steps.build.outcome == 'success' }}
timeout-minutes: ${{ contains(runner.name, 'nsc') && 2 || 20 }}
@@ -199,7 +194,7 @@ jobs:
run: tools/op.sh test --json-output unit-test-results.json
- name: Test custom module-level pytest cases
if: ${{ !cancelled() && steps.build.outcome == 'success' }}
timeout-minutes: 15
timeout-minutes: 10
env:
RAYLIB_BACKEND: headless
run: python3 tools/ci/run_host_tests.py --pytest
@@ -214,8 +209,6 @@ jobs:
openpilot/selfdrive/pandad/tests/test_pandad_canprotocol
openpilot/selfdrive/pandad/tests/test_aol_protocol
openpilot/selfdrive/pandad/tests/test_aol_wire
openpilot/selfdrive/pandad/tests/test_producer_timing
openpilot/selfdrive/pandad/tests/test_producer_sink
- name: Test map provider to host messaging
if: ${{ !cancelled() && steps.build.outcome == 'success' }}
working-directory: mapd_repo
@@ -432,6 +425,27 @@ jobs:
max_attempts: 3
command: cd ${{ github.workspace }}/ci-artifacts && git push origin process-replay --force
simulator_driving:
name: simulator driving
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16"]')
|| fromJSON('["ubuntu-24.04"]') }}
if: false # FIXME: Started to timeout recently
steps:
- uses: actions/checkout@v7
- run: ./tools/op.sh setup
- name: Build openpilot
run: scons
- name: Driving test
timeout-minutes: 2
env:
# MetaDrive renders offscreen through panda3d's EGL pipe on llvmpipe
EGL_PLATFORM: surfaceless
run: python openpilot/tools/sim/tests/test_metadrive_bridge.py
create_ui_report:
name: Create UI Report
runs-on: ${{
-3
View File
@@ -43,7 +43,6 @@ bin/
*.os
*.os-*
*.so
!openpilot/tools/replay/libmedia_mux.so
*.a
st[0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z]
*.unchunked
@@ -136,5 +135,3 @@ openpilot/selfdrive/modeld/models/*.unchunked
# Jetlink editable-install metadata.
/jetlink_repo/jetlink.egg-info/
# Local Google Play resolver build output (source and lockfile are versioned).
+1 -5
View File
@@ -2,7 +2,7 @@
## Jetlink external inference
The comma-side Jetlink client and USB transport in `jetlink_repo` are by Zeph Leggett and the [Jetlink contributors](https://github.com/zoompilot/jetlink). They are vendored from revision `d24328d849b0e5154b55f149dc18c5dfe8bd79cc` under the MIT license, retained in [jetlink_repo/LICENSE](jetlink_repo/LICENSE). StarPilot's adapter (`openpilot/starpilot/models/jetlink_adapter.py`) builds on Jetlink's openpilot layer and owns its param names, Always On Lateral rule, model runner reset, settings and alerts. Companion applications and inference servers remain distributed by Jetlink.
The comma-side Jetlink client and USB transport in `jetlink_repo` are by Zeph Leggett and the [Jetlink contributors](https://github.com/zoompilot/jetlink). They are vendored from revision `c17cd5c52be0ee83b9a6265b7da69d24ad6c5d02` under the MIT license, retained in [jetlink_repo/LICENSE](jetlink_repo/LICENSE). StarPilot owns the adapter to its model runner, settings and process lifecycle. Companion applications and inference servers remain distributed by Jetlink.
## MRR35 Hyundai angle and radar
@@ -21,7 +21,3 @@ The vehicle strategy and CAN/native owners preserve that attribution while adapt
## Optional Ford angle controller
The vendored `opendbc_repo/opendbc/bluepilot_lateral` angle core, codecs and Ford-only safety hooks are adapted directly from [BluePilot bp-7.0 e1d051d7ba270261b4455068bd68f1a58db15a4a](https://github.com/BluePilotDev/bluepilot/commit/e1d051d7ba270261b4455068bd68f1a58db15a4a), principally by Alan Polk, with John Christman, Jacob Neulight, Nathan Ingraham, Praeuner and other BluePilot, sunnypilot and comma contributors. Exact source paths and intentionally stricter safety checks are recorded in that folder. Both upstream `LICENSE` and `LICENSE.md` notices are retained. Its layout follows Alan Polk's September 2026 library proposal; this is a StarPilot adapter, not a published BluePilot library release.
## RAV4 Continental ARS510 receiver
The optional ACC-target and fused-track receivers and DBC preserve firestar5683's Dom integration from `b220a37f4bd119e402225a7aa16a7cd84878d0af` (Sea Turtles, 2026-10-13) and its ARS510 source attribution to ds-sebastian/ars510-radar `fb5b29c2b5999f34afeb6f7ac494a73949d66c67`, copyright 2026 ARS510 radar contributors. The complete MIT permission and warranty notice is retained in THIRD_PARTY_NOTICES.md. Startup/settings ownership is adapted to the current StarPilot architecture; no original firmware binaries or safety permissions are copied.
+1 -11
View File
@@ -8,17 +8,7 @@ See the [project overview](README.md) for the current development entry points a
The entries below are retained upstream release notes. They describe openpilot releases, not completed StarPilot 7.0 milestones.
Version 0.11.3 (2026-XX-XX)
========================
* Autospeed
* On supported cars, openpilot will intelligently decide what speed to drive based on the scene, including:
* stopping at red lights, and going at green lights
* slowing down for curves
* adjusting speed based on surrounding traffic
* This behavior was previously available with openpilot's "Experimental mode", but is now out of beta and always enabled
* With openpilot deciding which speed to drive, the car's cruise speed is treated as a maximum speed openpilot will not exceed
Version 0.11.2 (2026-10-14)
Version 0.11.2 (2026-08-12)
=======================
* New driving model
* Big model with 880M parameters
+5 -9
View File
@@ -287,8 +287,6 @@ def prune_cache_dir(target=None, source=None, env=None):
# ********** start building stuff **********
SConscript(['opendbc_repo/opendbc/can/SConscript'])
# Build common module
SConscript(['openpilot/common/SConscript'])
Import('_common')
@@ -296,7 +294,10 @@ common = [_common, 'json11', 'zmq']
Export('common')
# Build messaging (cereal + msgq + socketmaster + their dependencies)
SConscript(['msgq_repo/SConscript'])
# Enable swaglog include in dependency sources
env_swaglog = env.Clone()
env_swaglog['CXXFLAGS'].append('-DSWAGLOG="\\"common/swaglog.h\\""')
SConscript(['msgq_repo/SConscript'], exports={'env': env_swaglog})
SConscript(['openpilot/cereal/SConscript'])
@@ -327,17 +328,12 @@ SConscript([
'openpilot/selfdrive/locationd/SConscript',
'openpilot/selfdrive/modeld/SConscript',
'openpilot/selfdrive/ui/SConscript',
'openpilot/starpilot/audio/native/SConscript',
'openpilot/starpilot/models/runtime_bridge/SConscript',
])
# Replay is also the isolated on-device viewer engine.
if GetOption('extras') or arch == "comma_arm64":
SConscript(['openpilot/tools/replay/SConscript'])
# Build desktop-only tools
if GetOption('extras') and arch != "comma_arm64":
SConscript([
'openpilot/tools/replay/SConscript',
'openpilot/tools/cabana/SConscript',
'openpilot/tools/jotpluggler/SConscript',
])
-28
View File
@@ -84,31 +84,3 @@ BluePilot bp-dev e22afa6be9b881fa784c92ebb316db47728a3d81. Its exact source and
contributors are recorded in the library CREDITS.md; the same root/opendbc MIT
and Custom MIT notices coexist unchanged at that revision. This adds attribution,
not a license-scope decision or permission grant.
## Continental ARS510 radar
The optional RAV4 radar decoders and `toyota_tss2_ars510.dbc` are adapted from
<https://github.com/ds-sebastian/ars510-radar> at
`fb5b29c2b5999f34afeb6f7ac494a73949d66c67`.
MIT License
Copyright (c) 2026 ARS510 radar contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+2 -3
View File
@@ -8,9 +8,8 @@
Run openpilot's large driving models on a computer connected to your comma.
</p>
**Jetlink is experimental.** It requires a comma running an openpilot fork
with Jetlink support: zoompilot's
[`main` release](https://github.com/zoompilot/zoompilot/tree/main), or StarPilot.
**Jetlink is experimental.** It requires zoompilot's
[`develop` branch](https://github.com/zoompilot/zoompilot/tree/develop).
If the link drops or lags while engaged, the comma says **TAKE CONTROL** and
stays engaged on the small model. Be ready to take over.
+1 -1
View File
@@ -10,4 +10,4 @@ jetlink: run openpilot's large driving models on an attached Jetson.
# so does the release check (macos/scripts/check-version.sh). The comma runs
# jetlink from a checkout, not an install, so this is also all it has. The Swift
# reads it from Pinned.swift: run JetlinkKit/Scripts/make_pins.py after a bump.
__version__ = '0.8.7'
__version__ = '0.8.3'
+23 -61
View File
@@ -102,16 +102,12 @@ class JetlinkClient:
# the newest model output read off the link, whichever call read it: what
# a frame whose own reply is late publishes again (JetlinkModelState.end)
self.last_output: np.ndarray | None = None
# frames sent and not yet answered, oldest first. More than one after a
# reply a frame gave up on (infer_end with hold)
# frames sent and not yet answered, oldest first. More than one while
# frames go out without being waited for (drain), or after a reply a
# frame gave up on (infer_end with hold)
self._in_flight: deque[Sent] = deque()
# seqs sent with no reply wanted (leave): an ERROR to one is not this link's
self._no_reply_wanted: set[int] = set()
# frames the server never answered this session, for the log: a lost
# datagram is one, and to the caller a held frame like any late reply
self.frames_lost = 0
# take a server's offer of frame datagrams (_take_datagrams); False keeps frames on the stream, for A/B runs
self.allow_datagrams = True
# -- construction ---------------------------------------------------------
@@ -132,8 +128,8 @@ class JetlinkClient:
"""This end is the USB gadget, over a gadget another process owns.
The owner holds ep0 and the UDC bind for as long as the link is enabled,
so a drive starting or ending is not an unplug the host has to recover
from. See FfsTransport.borrowed.
so a drive starting or ending is no longer an unplug the host has to
recover from. See FfsTransport.borrowed.
"""
from jetlink.transport.ffs import FfsTransport
return cls(FfsTransport.borrowed(mount, udc, bounce=bounce, owner_gadget=owner_gadget), **kw)
@@ -214,8 +210,8 @@ class JetlinkClient:
self._dispatch(msg)
continue
if msg.msg_type == P.Msg.INFER_RESP and any(f.seq == msg.seq for f in self._in_flight):
# the answer to a frame that gave up waiting for it (a held frame):
# read like any other, then on to what is wanted
# the answer to a frame nobody waited for, or one a frame gave up
# waiting for: read like any other, then on to what is wanted
self._take_reply(msg)
continue
# A reply a caller gave up on. Drop it, or every frame reads one behind.
@@ -242,20 +238,8 @@ class JetlinkClient:
# its last output is nothing the new model may publish
self._in_flight.clear()
self.last_output = None
self.frames_lost = 0
self.t.send_json(P.Msg.HELLO_REQ, seq, {'client': client})
resp = json.loads(bytes(self._expect(P.Msg.HELLO_RESP, seq, timeout).payload))
self._take_datagrams(resp)
return resp
def _take_datagrams(self, hello: dict) -> None:
"""Frames as datagrams when the server offers them (only a phone's server,
over the cable). A session that offers none puts frames back on the link."""
port, token = hello.get('frame_port'), hello.get('frame_token')
if port and token is not None and self.allow_datagrams and self.t.use_datagrams(int(port), int(token)):
log.info("frames go to the server as datagrams (port %d)", int(port))
else:
self.t.stop_datagrams()
return json.loads(bytes(self._expect(P.Msg.HELLO_RESP, seq, timeout).payload))
def state(self, timeout: float = 2.0) -> dict:
seq = self._next_seq()
@@ -395,51 +379,36 @@ class JetlinkClient:
def infer_begin(self, warped: np.ndarray, packed: np.ndarray, frame_id: int = 0,
reset: bool = False, want_state: bool = False, deadline: float | None = None,
skip_if_busy: bool = False) -> int | None:
silence: float | None = None) -> int:
"""Send a frame and return immediately with its sequence number.
Split from infer_end so the caller can work while the Jetson is busy;
openpilot publishes chestnutState in that window. A frame whose reply
infer_end stopped waiting for is read by the next drain or infer_end. A
host that has answered none of them for the client's deadline is gone,
not slow, and the link is done here, before another frame goes out to
it. `deadline` bounds this frame's send.
`skip_if_busy` says the caller can do without this frame's reply: it
holds the last output, as JetlinkModelState.end does. A frame the link
cannot take without waiting for the host to drain earlier ones is then
not sent, and None comes back: the link is as good as it was
(Transport.try_send). Over a phone's cable it goes as datagrams instead,
where a lost one is a frame the server never answers, and the next reply
takes it out of flight (_take_reply).
openpilot publishes chestnutState in that window. A frame nobody calls
infer_end for is read by the next drain or infer_end. A host that has
answered none of them for `silence` (the client's deadline by default)
is gone, not slow, and the link is done here, before another frame goes
out to it. `deadline` bounds this frame's send.
"""
if self.spec is None:
raise LinkError("ensure_engine() first")
if self.dead:
raise LinkError("link previously failed")
if self.waiting_for() > self.deadline:
silence = self.deadline if silence is None else silence
if self.waiting_for() > silence:
self.dead = True
raise LinkError(f"no answer to {len(self._in_flight)} frames in {self.deadline:.1f}s; link abandoned")
raise LinkError(f"no answer to {len(self._in_flight)} frames in {silence:.1f}s; link abandoned")
warped = _as_bytes(warped, self.spec.warped_nbytes, 'warped')
packed = _as_bytes(packed, self.spec.packed_nbytes, 'packed')
seq = self._next_seq()
flags = ((P.Flag.RESET_QUEUES if reset else 0) | (P.Flag.WANT_STATE if want_state else 0)
| (P.Flag.WANT_HIDDEN if self.want_hidden else 0))
parts = (P.pack_infer_req(frame_id, flags), warped, packed)
sent = Sent(seq, frame_id, flags, time.monotonic())
timeout = self.deadline if deadline is None else deadline
try:
if skip_if_busy and self.t.datagrams:
self.t.send_datagrams(P.Msg.INFER_REQ, seq, parts)
elif skip_if_busy:
if not self.t.try_send(P.Msg.INFER_REQ, seq, parts, timeout=timeout):
return None # the seq is skipped; replies are matched by seq, not counted
else:
self.t.send(P.Msg.INFER_REQ, seq, parts, timeout=timeout)
self._in_flight.append(Sent(seq, frame_id, flags, time.monotonic()))
self.t.send(P.Msg.INFER_REQ, seq, (P.pack_infer_req(frame_id, flags), warped, packed),
timeout=self.deadline if deadline is None else deadline)
except LinkError:
self.dead = True
raise
self._in_flight.append(sent)
return seq
def infer_end(self, seq: int, deadline: float | None = None, hold: float | None = None) -> np.ndarray | None:
@@ -507,15 +476,12 @@ class JetlinkClient:
def _take_reply(self, msg: Message) -> np.ndarray:
"""One INFER_RESP off the stream: checked against the frame it answers,
its timings and telemetry kept, the frame and any older one taken out of
flight. Raises NotFinite on Status.NOT_FINITE, the link kept, and with
the link dead on anything else but a good answer."""
flight. Raises, with the link dead, on anything but a good answer."""
while self._in_flight and self._in_flight[0].seq != msg.seq:
# an older frame the server never answered: it answers in order, so a
# frame that fell out of its session (a hello between), or whose
# datagrams did not all arrive, is not coming
# an older frame the server never answered: it answers in order, and a
# frame that fell out of its session (a hello between) is not coming
skipped = self._in_flight.popleft()
log.warning('frame %d (seq %d) was never answered', skipped.frame_id, skipped.seq)
self.frames_lost += 1
if not self._in_flight:
self.dead = True
raise LinkError(f'inference response seq {msg.seq} answers no frame sent')
@@ -525,10 +491,6 @@ class JetlinkClient:
raise LinkError('inference response is missing its header')
fid, status, gpu_us, queue_us, total_us = P.unpack_infer_resp(msg.payload)
self.last_timings = (gpu_us, queue_us, total_us)
if status == P.Status.NOT_FINITE and fid == frame_id:
# the server's session goes on, keeping nothing of the frame
# (Session.infer), so the link stays and last_output is the one before
raise P.NotFinite(f"inference output is not finite: NOT_FINITE (frame {fid})")
if status != P.Status.OK:
self.dead = True
raise LinkError(f"inference failed: {_name(P.Status, status)} (frame {fid})")
+39 -54
View File
@@ -12,8 +12,8 @@ Holding ep0 needs sysfs and a unix socket; `openpilot.common.swaglog` costs
28 MB because it drags numpy, capnp and zmq in to publish a log line, and
`openpilot.common.params` imports swaglog, so a module that touches either
prices the owner out of being minimal. Measured on the comma: python plus this
plus the FunctionFS transport is 10.4 MB against 47.5 MB with openpilot's
modules imported.
plus the FunctionFS transport is 10.4 MB against 47.5 MB for the daemon that
imported the world.
Nothing in jetlink.comma may import openpilot or knows a param: the settings
come from jetlink.openpilot.settings, over the directory and the keys the
@@ -146,7 +146,7 @@ UDC_PATH = Path(UDC_SYSFS)
# the network function jetlink-root.sh gadget --ios adds after ffs.jetlink. Its
# netdev is not usb0, which the modem holds, but whatever the kernel names it
NET_FUNCTION = 'ncm.usb0'
# written by jetlink/comma/jetlink-root.sh gadget, which the owner runs when the
# written by scripts/comma/jetlink-root.sh gadget, which the owner runs when the
# link is on and there is no gadget: "ok", or "error: <reason>"
GADGET_STATUS = Path("/dev/shm/jetlink-gadget")
# the owner's own: "error: the lender could not listen: <reason>" while nothing
@@ -158,17 +158,15 @@ CC_ORIENTATION = Path('/sys/class/power_supply/usb/typec_cc_orientation')
# sleep. Presence comes from this, not the UDC; a marker whose writer is dead is
# a leftover from a kill
DORMANT = Path("/dev/shm/jetlink-dormant")
# hardwared's request to power the Jetson off; see request_shutdown
# hardwared's request to power the Jetson off; see backend.shutdown
SHUTDOWN_REQUEST = Path("/dev/shm/jetlink-shutdown")
# the owner's status record: the owner's state in one file, rewritten whole
# every step, which makes it the owner's heartbeat too. A clean stop removes it
# the owner's status record: everything the readers used to take from the
# files above one by one, rewritten whole every step, which makes it the
# owner's heartbeat too. A clean stop removes it
STATUS = Path("/dev/shm/jetlink/status.json")
# how old that record may be before a reader takes its owner for gone: six of
# its 0.5 s steps. A wait inside a step renews it (Owner.beat)
HEARTBEAT_TIMEOUT = 3.0
# how the owner's crash-loop backoff starts its record's error, which
# status.Status.reason_code reads it by
BACKING_OFF = "the service keeps stopping"
# the owner's start times, for its crash-loop backoff (see the owner)
STARTS = Path("/dev/shm/jetlink/starts.json")
# held with flock by the owner for its whole life, so there is only ever one
@@ -176,15 +174,11 @@ OWNER_LOCK = Path("/dev/shm/jetlink/owner.lock")
# what the server's last hello said (lending.SERVER_FIELDS), kept apart from
# the status record so a clean stop leaves it for the next owner
SERVER = Path("/dev/shm/jetlink/server.json")
# the joining model's runtime record, rewritten by a thread in modeld once a
# second and on every handover: whether the big model drives, for a reader
# with no modelV2 (status.runtime). modeld's close() leaves it stopping
MODELD = Path("/dev/shm/jetlink/modeld.json")
# how often modeld rewrites that record without a handover
RECORD_PERIOD = 1.0
# how long a host that stopped reading configured still counts as there: long
# enough to bridge a USB3 link recovery passing through "addressed", and a
# bounce made on purpose when a host will not enumerate (wait_for_host)
# how long a host that stopped reading configured still counts as there. The
# owner holds the gadget for as long as the link is enabled, so presence no
# longer blinks at every handover; what is left to bridge is a USB3 link
# recovery passing through "addressed", and a bounce made on purpose when a
# host will not enumerate (wait_for_host)
PRESENCE_HOLD = 5.0
@@ -205,31 +199,22 @@ def write_record(path: Path, record) -> None:
def clear_leftovers(directory: Path) -> None:
"""The temporaries a writer killed between its write and its rename left
in `directory`, for the owner's start. Only a dead writer's (write_record
names them by pid): modeld writes its runtime record there too, and a
live one's is mid-write."""
in `directory`. For the owner's start, when nothing else writes there."""
try:
for leftover in directory.glob('.*.json.*'):
pid = leftover.name.rsplit('.', 1)[-1]
if not (pid.isdigit() and pid_alive(int(pid))):
leftover.unlink(missing_ok=True)
leftover.unlink(missing_ok=True)
except OSError:
pass
def read_record(path: Path) -> dict | None:
"""A JSON record write_record wrote, or None: missing, or not a record."""
try:
record = json.loads(read(path))
except ValueError:
return None
return record if isinstance(record, dict) else None
def owner_status() -> dict | None:
"""The owner's status record, or None without one: no owner has run since
boot, the last one stopped cleanly, or it is one too old to write it."""
return read_record(STATUS)
try:
record = json.loads(read(STATUS))
except ValueError:
return None
return record if isinstance(record, dict) else None
def owner_alive(record: dict) -> bool:
@@ -331,8 +316,8 @@ def wait_for_host(timeout: float, bounce=None, should_stop=None, report=None, mo
"""Wait for the Jetson to enumerate us, bouncing a bus that stalled.
The gadget stays bound throughout. An unbind is an unplug as the far end sees
it, and while one boots it takes ~50 s a cycle, so an unbind on a timer can
land on a box seconds from enumerating.
it, and while one boots it takes ~50 s a cycle: doing that on a timer landed
an unplug on a box that was seconds from enumerating.
The one case that needs an edge is a host that took the bind as a wake, reset
the bus and stopped. The UDC then sits in default or addressed with the CC
@@ -435,15 +420,9 @@ def dormant() -> bool:
pid = int(read(DORMANT))
except ValueError:
return False
return pid_alive(pid)
def pid_alive(pid) -> bool:
if not isinstance(pid, int) or isinstance(pid, bool) or pid <= 0:
return False # 0 and below name process groups, never one process
try:
os.kill(pid, 0)
except (ProcessLookupError, OverflowError, ValueError):
except ProcessLookupError:
return False
except PermissionError:
pass # alive, just not ours to signal
@@ -451,15 +430,7 @@ def pid_alive(pid) -> bool:
def request_shutdown(reason: str) -> bool:
"""Hand the owner a power-off request, whole or not at all: a partial one (a
full /dev/shm) is one the owner cannot read, so it never takes it. False,
and logged, when it cannot be written."""
try:
write_record(SHUTDOWN_REQUEST, {'reason': reason})
return True
except OSError:
log.exception("jetlink: could not write the shutdown request")
return False
return _write(SHUTDOWN_REQUEST, json.dumps({'reason': reason}), "write the shutdown request")
def pending_shutdown() -> str | None:
@@ -470,8 +441,22 @@ def pending_shutdown() -> str | None:
"""
if not SHUTDOWN_REQUEST.exists():
return None
record = read_record(SHUTDOWN_REQUEST)
return None if record is None else str(record.get('reason', ''))
try:
return str(json.loads(read(SHUTDOWN_REQUEST)).get('reason', ''))
except ValueError:
return None
def await_shutdown(timeout: float, poll: float = 0.25) -> bool:
"""Wait for the owner's run to take a shutdown request. False if nobody did
within `timeout`, and then the request is withdrawn."""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if not SHUTDOWN_REQUEST.exists():
return True
time.sleep(poll)
finish_shutdown()
return False
def finish_shutdown() -> None:
+6 -6
View File
@@ -7,8 +7,8 @@ See the LICENSE file in the root directory for more details.
Who may do endpoint IO on the gadget, while one process owns it throughout.
The owner (owner.py) holds ep0 for as long as the link is on, and nothing else
ever does: to the Jetson, a change of who holds ep0 is an unplug, a fresh
libusb open and a fresh server session.
ever does: when two processes took turns at it, every change of owner was an
unplug, a fresh libusb open and a fresh server session as the Jetson saw it.
What still has to change hands is the right to read the endpoint files.
FunctionFS keeps a queued read queued until something completes it, so a second
@@ -119,10 +119,10 @@ class Loan:
def bounce(self) -> bool:
"""Ask the owner to take the gadget down and put it back up.
The only thing that completes a FunctionFS write nobody is reading is the
unbind, and the unbind belongs to whoever holds ep0. Called on writes the
host has stopped taking (FfsTransport._abort_write), so the re-enumeration
it costs is not the expensive part.
The only thing that dequeues a FunctionFS write nobody is reading is the
unbind, and the unbind belongs to whoever holds ep0. Called from the write
watchdog on a link that is already 15 s stuck, so the re-enumeration it
costs is not the expensive part.
"""
with self._lock:
if self._closed:
+35 -128
View File
@@ -71,8 +71,6 @@ GADGET_SETUP_BACKOFF = 60.0
GADGET_HELD_RETRY = 2.0
# between attempts to bring the gadget's network interface up after a bind
NET_BACKOFF = 5.0
# jetlink-root.sh rps attempts a bind before the owner stops asking (ensure_steered)
STEER_TRIES = 6
# between attempts to listen for borrowers after one failed
LENDER_BACKOFF = 30.0
# between runs of the worker that found nothing to do. It costs a couple of
@@ -102,8 +100,8 @@ LOG_BYTES = 1 << 20
def logger(path: Path) -> logging.Logger:
"""The owner's log, to stderr and `path`: swaglog costs 28 MB, so the owner
keeps its own. The worker's lines go to the drive's log; these stay for
reading afterwards."""
keeps its own. The worker's lines go to the drive; these are for a bench
session, which wants them after the drive."""
log = logging.getLogger('jetlink.owner')
log.setLevel(logging.INFO)
handlers: list[logging.Handler] = [logging.StreamHandler()]
@@ -176,17 +174,11 @@ class Owner:
self.dormant = False
# the link mode the root tuning was applied for, None while stock
self.tuned: str | None = None
# the tuning put back since the link was last on: None not yet asked,
# False a restore failed (release_tuning)
self.released: bool | None = None
# the draw from the USB-C port: cut by this owner, and whether it was last
# settled for the link on (steer_draw)
self.cut = False
self.draw_for: bool | None = None
# when there was last something to do. The hold runs from here, not from
# process start: this daemon lives across ignitions, so a hold measured
# from its birth would expire once and then release the gadget on every
# wake with the jetson still coming up
# process start: this daemon is not restarted at ignition any more, so a
# hold measured from its birth expires once and never applies again, and
# every wake released the gadget a second later with the jetson still
# coming up
self.idle_since = time.monotonic()
self.next_attempt = 0.0
self.next_gadget_attempt = 0.0
@@ -202,9 +194,6 @@ class Owner:
self._peer: str | None = None # the phone published as dialed in
self.net_ready = False # usb0 configured for this bind
self.next_net_attempt = 0.0
self.steered = False # usb0's receive work read back on core 6, since the drive started
self.next_steer = 0.0
self.steer_tries = 0 # this bind's, up to STEER_TRIES
self.lender_failed = False # said once, until it listens again
self.next_lender = 0.0
self.worker: subprocess.Popen | None = None
@@ -366,26 +355,6 @@ class Owner:
return
self.next_net_attempt = now + NET_BACKOFF
self.net_ready = gadget.net_up()
self.steered, self.next_steer, self.steer_tries = False, 0.0, 0
def ensure_steered(self, offroad: bool) -> None:
"""The cable's receive work onto core 6 (jetlink-root.sh rps), once a
drive has started: net's own steer, at the bind, reads back 00 while
hardwared has the big cores parked, and the kernel never adds them later.
Retried on NET_BACKOFF, STEER_TRIES times a bind: a core 6 that stays
offline is no reason to run sudo every few seconds for a whole drive."""
if offroad or self.steered or not self.net_ready or self.steer_tries >= STEER_TRIES:
return
now = time.monotonic()
if now < self.next_steer:
return
self.next_steer = now + NET_BACKOFF
self.steer_tries += 1
self.steered = root.run('rps')
if self.steered:
gadget.log.warning("jetlink: the cable's receive work is on core 6")
elif self.steer_tries == STEER_TRIES:
gadget.log.warning("jetlink: the cable's receive work stays where the kernel puts it until the next bind")
def settle(self) -> None:
"""Put the gadget back to bound with nothing open on it.
@@ -475,8 +444,8 @@ class Owner:
# the far end may still be waking; give it the hold before letting go
self.idle_since = time.monotonic()
# after the run, not before it: a run writes JetlinkSpec itself, so a mark
# taken at spawn would always differ by the time it exits and every
# successful provision would start a second one
# taken at spawn always differs by the time it exits and every successful
# provision started a second one
self.note_marks()
return False
@@ -633,88 +602,35 @@ class Owner:
self.close_link()
self.stop_worker()
self.wake()
self.release_tuning()
if self.tuned:
root.run('vm', 'restore')
root.run('udc', 'restore')
if self.tuned == 'ios':
root.run('draw', 'on', timeout=root.PORT_TIMEOUT)
self.tuned = None
self.port.off()
return
# before the gadget: a host is never drawn from while the link comes up
self.steer_draw(True)
self.link_step(mode == 'ios')
self.released = None
if self.tuned != mode:
# jetlink-root.sh vm: the recording VM tuning the gadget's reads need;
# udc: the USB device side kept on for a host that powers the port
# (port.py). While the link is on. After the step, so the first gadget
# does not wait on them. Put back only when the link is not wanted
# (release_tuning)
# (port.py); draw, iOS only: no current drawn from the port, which an
# iPhone cannot supply. While the link is on. After the step, so the
# first gadget does not wait on them. Put back only when the link is
# turned off or leaves iOS, never on exit: manager stops this at
# ignition, just as the contention starts, and a phone plugged in before
# it is back should still come up. A start assumes the draw stock, so a
# cut a dead iOS owner left lasts until the reboot that clears it
if self.tuned is None:
root.run('vm', 'apply')
root.run('udc', 'apply')
if mode == 'ios':
if root.run('draw', 'off', timeout=root.PORT_TIMEOUT):
gadget.log.info("jetlink: drawing no current from the USB-C port, for an iPhone")
elif self.tuned == 'ios':
root.run('draw', 'on', timeout=root.PORT_TIMEOUT)
self.tuned = mode
def link_wanted(self) -> bool:
"""Is the link still wanted at a stop: set on, and no chestnut on the
port? Settings.mode() reads an unreadable param as 'off', as manager
does, so that is not wanted. The except only guards a bug: anything
raised in the stop's finally would skip the teardown after it. manager's
own chestnut test is chestnut_present(), cached 2 s, and this walks the
USB ids itself, so a chestnut mid-re-enumeration can read as absent here
and keep the tuning for that drive; accepted."""
try:
return self.settings.mode() != 'off' and not port.chestnut_attached(self.port.chestnut_ids)
except Exception:
gadget.log.exception("jetlink: could not tell whether the link is still wanted; keeping its tuning")
return True
def release_tuning(self, final: bool = False) -> bool:
"""The link is not wanted, turned off or a chestnut on the port: the
port, its draw and the root tuning go back as AGNOS boots them, at a step
and at the stop manager makes of it, which often comes before a step has
seen the setting. Any other stop keeps them for the owner manager starts
next, which applies them again; a reboot clears them.
The draw first, while a hold still keeps the far end on the port: letting
go of one can drop it for a DRP toggle, and an empty port read in that
moment would give the draw back with a host still in. It is given back
only on an empty port, so it stays cut with a chestnut on, which the
comma powers and where the cut does nothing. The tuning is restored
whatever `tuned` says, since an owner before this one may have applied
it: once when the link goes, and again at the stop (final) if that
failed. On the port's short timeout, since at a stop it comes before the
FunctionFS close, inside manager's 5 s. True when the draw is still cut."""
self.steer_draw(False)
self.port.off()
if self.released is None or (final and not self.released):
vm = root.run('vm', 'restore', timeout=root.PORT_TIMEOUT)
self.released = root.run('udc', 'restore', timeout=root.PORT_TIMEOUT) and vm
if self.released:
self.tuned = None
return self.cut
def steer_draw(self, want: bool) -> None:
"""jetlink-root.sh draw: with the link on, no current drawn from the USB-C
port, whatever is on it. The comma runs off its own power port, and drawing
from the host's only costs it: a 3X browns out the instant it draws from a
laptop (PMIC power-off reason UVLO), and an iPhone sags under the draw and
never sees a device. The launch script cuts it at boot (jetlink-root.sh
boot); this keeps it cut. A forced voter, so it holds for every plug after.
Given back only on an empty port, so the link turned off with a host still
in never lets the charger draw from it. Kept on a stop with the link
wanted, like the tuning (release_tuning). A start assumes the draw stock,
so the link off gives it back on an empty port even uncut here: an owner
before this one may have cut it, and letting go is safe to repeat."""
if want == self.draw_for:
return
if want:
self.cut = root.run('draw', 'off', timeout=root.PORT_TIMEOUT)
if self.cut:
gadget.log.info("jetlink: drawing no current from the USB-C port")
else:
if port.power_role() != 'none':
return # a host is still in; looked at again next step
root.run('draw', 'on', timeout=root.PORT_TIMEOUT)
self.cut = False
# settled, or failed, which is not retried every step
self.draw_for = want
def ensure_lender(self) -> None:
"""Listen for borrowers, or say why not and try again later.
@@ -754,7 +670,6 @@ class Owner:
return
self.attached = gadget.host_attached()
self.watch_the_port()
self.ensure_steered(offroad)
# before the worker gate: hardwared waits 25 s for this and a build in
# flight takes minutes, so a shutdown request cannot queue behind one
@@ -762,9 +677,9 @@ class Owner:
if reason is not None and not self.lender.lent:
if self.worker_running():
if self.shutting_down:
# the run asking the jetson. Stopping it here would restart it every
# half-second step, less than it takes to start, so nothing would
# ever ask and hardwared would give up after its 25 s
# the run asking the jetson. Stopping it here restarted it every
# step, half a second, less than it takes to start: on the bench
# nothing ever asked, and hardwared gave up after its 25 s
return
self.stop_worker()
if time.monotonic() < self.next_shutdown_run:
@@ -946,7 +861,7 @@ class Owner:
A power-off request ends it: hardwared waits 25 s for the owner to take
one, and the Jetson on its own supply stays on otherwise."""
self.backing_off = (f"{gadget.BACKING_OFF} ({died} times in {CRASH_WINDOW / 60:.0f} min), "
self.backing_off = (f"the service keeps stopping ({died} times in {CRASH_WINDOW / 60:.0f} min), "
f"waiting {until - time.monotonic():.0f} s before starting it again")
gadget.log.warning("jetlink: %s", self.backing_off)
self.publish_status()
@@ -966,7 +881,7 @@ class Owner:
still holds the gadget) would clear that owner's records, overwrite its
status record and release its port hold. The kernel lets an flock go with
the process however it dies, and the fd is not inherited by a run.
Without a /dev/shm to lock in, the owner runs unlocked."""
Without a /dev/shm to lock in, the owner runs as it did before."""
try:
gadget.OWNER_LOCK.parent.mkdir(parents=True, exist_ok=True)
fd = os.open(gadget.OWNER_LOCK, os.O_RDWR | os.O_CREAT, 0o644)
@@ -1022,17 +937,9 @@ class Owner:
time.sleep(max(0.0, POLL - (time.monotonic() - started)))
finally:
# first, inside manager's 5 s: it stops this when a chestnut turns up,
# and the comma has to host that. The tuning and the draw go with the
# port when the link is not wanted (release_tuning)
if self.link_wanted():
self.port.off()
elif self.release_tuning(final=True):
if port.power_role() == 'source':
gadget.log.info("jetlink: the comma powers what is on the USB-C port; its draw stays cut, "
"which does nothing there, until a reboot")
else:
gadget.log.warning("jetlink: a host is still on the USB-C port; the comma draws nothing from it "
"until a reboot")
# and the comma has to host that. The sysctls stay: a stop here is where
# a drive begins
self.port.off()
self.lender.stop()
# about this process's lender; with no owner there is nothing to borrow
# anyway, and a chestnut may be why it stopped
+2 -2
View File
@@ -11,8 +11,8 @@ host. An A-to-C cable settles which by construction: the A end only pulls CC
up, so the comma can only be the sink and the device. A C-to-C cable does not.
Both ends are dual role, and the comma can come out the source and the host:
facing a Mac, nothing enumerates; facing an iPhone or a Jetson's own USB-C
port, the far end enumerates as a device (a Jetson on C-to-C reads
"Powered cable w/ sink" and enumerates as 0955:7020).
port, the far end enumerates as a device. Carrot's Jetson on C-to-C read
"Powered cable w/ sink" and enumerated as 0955:7020.
While the link is on, a chestnut is the only thing the comma should host on
this port. So once the comma has been the source for a few seconds with no
+4 -4
View File
@@ -4,7 +4,7 @@ Copyright (c) 2026-, Zeph Leggett.
This file is part of jetlink and is licensed under the MIT License.
See the LICENSE file in the root directory for more details.
Root on the comma: jetlink/comma/jetlink-root.sh under sudo -n, and nothing
Root on the comma: scripts/comma/jetlink-root.sh under sudo -n, and nothing
else. The script leaves its own record of the gadget and its network in
/dev/shm (jetlink-gadget, jetlink-net), so a False here only says the call
failed; the reason is in the log line and, for the gadget, in those files.
@@ -27,9 +27,9 @@ log = logging.getLogger('jetlink.comma')
AGNOS = os.path.isfile('/AGNOS')
# Next to this file, so the package alone is all the comma runs. A path
# through the <openpilot>/jetlink symlink is fine: bash only opens it.
SCRIPT = Path(__file__).with_name('jetlink-root.sh')
# Resolved, so that <openpilot>/jetlink, a symlink into jetlink_repo, still
# leads to the checkout's scripts.
SCRIPT = Path(__file__).resolve().parents[2] / 'scripts' / 'comma' / 'jetlink-root.sh'
# sudo, configfs, and for iOS nmcli and dnsmasq
TIMEOUT = 30.0
+41 -100
View File
@@ -11,42 +11,35 @@ The fork implements Openpilot (interface.py) once, in an adapter module that
holds every openpilot import, binds it here (bind) and calls the methods of
the Jetlink that comes back. That is the whole contract between the two repos:
the names in __all__, the Jetlink methods and Status, which
tests/openpilot/test_api.py and test_interface.py pin, and the facade the
fork's hook lines call them through (hooks.py) and the kit its CI runs
(testing.py). API names its version: an adapter declares the one it was
written to, hooks.Hooks refuses one it does not accept, and the fork treats
that as jetlink being absent, an offroad alert and no link rather than a
crash. Everything else in this package is internal to jetlink and may change
at any time.
tests/openpilot/test_api.py and test_interface.py pin. API names its version:
the adapter checks it exactly and treats any other value as jetlink being
absent, an offroad alert and no link rather than a crash. Everything else in
this package is jetlink's own and changes without notice.
API changes only for a breaking change to the contract: removing or renaming
anything in it, changing what one of its members means, or making an optional
member required. A new Status field, a new Jetlink method, or a new interface
member that jetlink reads with getattr and a fallback, is additive and keeps
it. A bump keeps the previous API working for one minor release, so a fork can
move its pin and then its adapter. Each release that changes the contract
says how in its CHANGELOG section (Forks, docs/publishing.md).
API changes only for a breaking change to the contract. A new Status field, a
new Jetlink method, or a new interface member that jetlink reads with getattr
and a fallback, is additive and keeps it.
Imported by the resident gadget owner, so nothing heavy at module level: the
parts a heavy process uses are imported when it first uses them.
"""
from __future__ import annotations
import threading
import time
from jetlink.comma import gadget
from jetlink.openpilot.events import Flags, SwapEvents
from jetlink.openpilot.interface import MODES, STATES, Keys, ModelFace, Openpilot, OwnerConfig, conformance
from jetlink.openpilot.parts import Parts, for_this_process
from jetlink.openpilot.status import Status
API = 3
API = 1
__all__ = ['API', 'MODES', 'STATES', 'Jetlink', 'Keys', 'ModelFace', 'Openpilot', 'OwnerConfig', 'Status', 'bind',
'conformance', 'SHUTDOWN_TIMEOUT', 'Flags', 'SwapEvents']
'conformance']
# how long a power-off request waits for the owner's run to shut the Jetson
# down before it is withdrawn. Wake from suspend is ~8 s to a server
# how long a caller waits for the owner's run to shut the Jetson down. Wake
# from suspend is ~8 s to a server
SHUTDOWN_TIMEOUT = 25.0
@@ -69,11 +62,6 @@ class Jetlink:
self._failures: dict[str, str] = {}
# when this process asked for the Jetson to power off, until it was taken
self._shutdown_asked: float | None = None
# shutdown_step: has it run in this process, and has it said yes
self._stepped = False
self._may_power_off = False
# the joining model's handovers as handed_over() last saw them
self._handovers = 0
def _read(self, what: str, read, fallback):
"""read(), or fallback(error) when it raises. The readers never raise:
@@ -109,21 +97,6 @@ class Jetlink:
mode = self._mode()
return self._read('the status', lambda: status.read(self._parts, mode), lambda error: status.failed(error, mode))
def runtime(self):
"""What modeld runs (status.Runtime), the part of status() that changes
while driving, for a loop: the setting, the chestnut test and modeld's
record, nothing else. The chestnut test may walk the USB devices in sysfs,
once per parts.CHESTNUT_TTL (2 s), as status() does. None where
status().running is None for want of the link: off, or beside a chestnut."""
from jetlink.openpilot import status
return status.runtime() if self.enabled() else None
def model_state(self, ref: str) -> str | None:
"""One catalog model for the picker's list: 'ready' (built on the Jetson),
'downloaded' (on the comma) or None. Asked when a list opens."""
from jetlink.openpilot import status
return self._read('a model\'s state', lambda: status.model_state(self._parts, ref), lambda error: None)
def reason(self) -> str | None:
"""Why the link the user asked for cannot run: hardwared's offroad alert.
None with the link off. The files the gadget and the build leave, nothing
@@ -164,10 +137,6 @@ class Jetlink:
def attach(self, small, cam_w: int, cam_h: int):
"""Join the link to modeld, once the camera is up and `small` is built:
the joining model, which drives as `small` until the Jetson is there.
Before every run() it asks the adapter's in_control() (is openpilot or
MADS in control? it swaps only while not), and modeld writes onto it
`frame_drop_ratio` (its share of dropped camera frames, which hands a
lagging large model back).
None unless prepare() said yes in this process: without it the GPU's
thread would start on modeld's realtime core. If the joining model cannot
@@ -182,23 +151,6 @@ class Jetlink:
self._log.exception("jetlink load failed")
return small
def handed_over(self, model) -> bool:
"""modeld, after every run(): did it change which model drives? The
joining model counts its swaps and hand-backs (JoinedModel.handovers),
and the stall of one is not lag, nor are the drops of its frame. Any
other model has no count and never hands over. Since the last call in
this process, which runs one modeld loop."""
handovers = getattr(model, 'handovers', 0)
changed, self._handovers = handovers != self._handovers, handovers
return changed
def state(self, model) -> str:
"""What modeld publishes as the big model's state: the model's
big_model_state as it is, one of STATES on a joining model, and 'none' for
a model without one. A fork's capnp enum holds exactly STATES, so a value
outside it would raise on the frame thread."""
return getattr(model, 'big_model_state', 'none')
def request_shutdown(self, reason: str = '') -> bool:
"""The device is powering off for good: ask for the Jetson to go down
with it, and return at once. True when the request now waits for the
@@ -226,47 +178,9 @@ class Jetlink:
the Jetson powered off later, when nobody wants it off, and the log says
the request went unanswered."""
try:
return self._pending()
pending = gadget.SHUTDOWN_REQUEST.exists()
except Exception:
return False
def shutdown_step(self, should_shutdown: bool, reason: str = '') -> bool:
"""hardwared, every loop: may the device power off now? True once it may
(put DoShutdown), and from then on. False while should_shutdown is False
and nothing has been asked. The first True asks once (request_shutdown):
True at once when there was nothing to ask, otherwise once the owner has
taken the request, or SHUTDOWN_TIMEOUT after asking, when it is withdrawn.
Once asked, should_shutdown is not read again, and the caller keeps no
timer. A stat a call. Never raises: a failure once a power-off is wanted
is a yes, since a jetlink bug must not keep a comma on to drain the car's
battery, and one before then is a no."""
if self._may_power_off:
return True
try:
first, self._stepped = not self._stepped, True
# a request this process did not write: a partial one, or one from a
# hardwared that has since restarted. Left, the owner would power the
# Jetson off whenever it next saw one, maybe a drive later
if first and self._shutdown_asked is None:
gadget.finish_shutdown()
if self._shutdown_asked is None and not should_shutdown:
return False
asked = self._shutdown_asked is not None or self.request_shutdown(reason)
self._may_power_off = not asked or not self._pending()
except Exception:
self._log.exception("jetlink: the shutdown step failed")
if self._shutdown_asked is not None:
# nobody follows the request any more, so it must not stay for the owner
try:
gadget.finish_shutdown()
except Exception:
pass
self._may_power_off = should_shutdown or self._shutdown_asked is not None
return self._may_power_off
def _pending(self) -> bool:
"""shutdown_pending(), raising."""
pending = gadget.SHUTDOWN_REQUEST.exists()
asked, now = self._shutdown_asked, time.monotonic()
if asked is None:
return pending
@@ -281,12 +195,39 @@ class Jetlink:
self._shutdown_asked = None
return pending
def shutdown(self, reason: str = '', timeout: float = SHUTDOWN_TIMEOUT) -> None:
"""request_shutdown(), then wait up to `timeout` for the owner to take it,
on a thread abandoned at the deadline: for a caller with nothing to do
meanwhile. hardwared keeps publishing, and uses the two above."""
if not self.enabled():
return
def request():
try:
self._request_shutdown(reason, timeout)
except Exception:
self._log.exception("jetlink: shutdown request failed")
t = threading.Thread(target=request, name='accelerator-shutdown', daemon=True)
t.start()
t.join(timeout)
if t.is_alive():
self._log.warning("jetlink: shutdown request still pending after %.0f s, going on without it", timeout)
def _request_shutdown(self, reason: str, timeout: float) -> None:
if not self._ask_for_power_off(reason):
return
if gadget.await_shutdown(timeout):
self._log.warning("jetlink: shutdown request handed to the jetson")
else:
self._log.warning("jetlink: nobody took the shutdown request within %.0f s", timeout)
def _ask_for_power_off(self, reason: str) -> bool:
"""Hand the request to the owner: nothing else can touch the link, and
the request is a file (gadget.SHUTDOWN_REQUEST) the owner's step looks
for before anything else. Skipped when no Jetson is known to be there
(dormant counts as there). A run busy in a long provision is stopped for
it; shutdown_pending()'s deadline covers a Jetson that never answers."""
it; the caller's timeout covers a Jetson that never answers."""
from jetlink.openpilot.status import Presence
if not self.enabled():
return False
-115
View File
@@ -1,115 +0,0 @@
"""
Copyright (c) 2026-, Zeph Leggett.
This file is part of jetlink and is licensed under the MIT License.
See the LICENSE file in the root directory for more details.
The onroad events of a big model that joins mid-drive, as flags. A native big
model loads before the first modelV2; this one joins onto a modelV2 the small
model already publishes, and can leave and come back. Each fork's selfdrived
calls SwapEvents.update once a tick and maps the flags onto its own events.
It swaps in only while nothing is in control, so the driver is told once when
it is ready (offer) and re-engages to use it. For a second after a swap nothing
engages (no_entry) while the big model builds its history and proves it keeps
up (the joining model hands back on a held frame in that second), and the chime
at the end of it says the driver can. When it leaves, the small model drives on
and the driver is told (link_lost); nothing disengages.
The fork's own chime on a rising modelV2.big, a native big model's, must not
fire for a joined one, or it says engage now at the start of the no-entry
second: gate it on the state being 'none', or at least drop it on no_entry
(zoompilot drops it on no_entry or chime, then adds it on chime).
Either way the switch costs modeld a frame or two, and a frame the camera
dropped is an invalid pose from the model and a localizer soft disable on the
next tick (TAKE CONTROL IMMEDIATELY). So a switch is `settling` for a second, which selfdrived must
treat as it does a native big model's load: hold the comm and localization
errors that frame raises.
"""
from __future__ import annotations
from typing import NamedTuple
# how long each lasts, in seconds; SwapEvents counts them in the caller's ticks.
# The link-lost warning covers about the time the small model takes to refill
# its history, the no-entry about 20 frames of the big model's. An enable inside
# that second is refused, and pressed again after the chime
OFFER_S = 3.
HANDBACK_S = 5.
SWITCHING_S = 1.
class Flags(NamedTuple):
offer: bool # ready while something is in control: re-engage to switch
link_lost: bool # it fell while in control; the small model drives on
no_entry: bool # the second after a swap: refuse an engagement
chime: bool # the end of that second: the big model drives, engage now
settling: bool # the second after a switch either way: forgive its dropped frame
def _tick(left: int) -> tuple[bool, int]:
return left > 0, max(left - 1, 0)
class SwapEvents:
def __init__(self, tick_s: float):
# checked once here: update() runs in selfdrived's loop and never raises
if not tick_s > 0 or round(SWITCHING_S / tick_s) < 1:
raise ValueError(f"SwapEvents needs the loop period in seconds, short enough to count a {SWITCHING_S:g} s switch: {tick_s}")
self.offer_ticks = round(OFFER_S / tick_s)
self.handback_ticks = round(HANDBACK_S / tick_s)
self.switching_ticks = round(SWITCHING_S / tick_s)
self.offered = False
self.running = False
# the state as last seen fresh: a gap in the fork's status or message holds
# it, never reads as the link going
self._state = 'none'
# ticks left of each event, and of the settling after a switch either way
self._offer = self._handback = self._switching = self._settle = 0
@property
def settling(self) -> bool:
return self._settle > 0
def update(self, fresh: bool, big: bool, in_control: bool, state: str) -> Flags:
"""One tick of the caller's loop, in hooks.SwapTracker.update's order.
`fresh`: modelV2 and the fork's state message are seen, alive and valid
this tick. `big`: modelV2.big, from a modelV2 alive and valid this tick.
`in_control`: the fork's meaning of something steering or able to
(zoompilot: openpilot or MADS, MADS even paused), the same rule the swap
waits on (the adapter's engaged()). `state`: the joining model's
big_model_state, its STATES name (str() of a capnp enum, never its .raw
number, which matches no state)."""
# a stale state neither offers the switch nor rearms the offer
if fresh:
self._state = state
if self._state != 'ready' or big:
# ended by the swap as well as by the link going: an offer still on
# screen after the switch reads as if it had not happened
self.offered = False
self._offer = 0
elif in_control and not self.offered and self._handback == 0:
# with nothing in control it swaps at once and the chime says so. Once
# per readiness, not at every stop, and after the link-lost warning,
# which would hide it
self.offered = True
self._offer = self.offer_ticks
# a chestnut dropping modelV2.big is its own fork's failure event, not this
running = bool(big) and self._state != 'none'
if running != self.running:
self._settle = self.switching_ticks + 1 # counted down below, this tick included
self._switching = self.switching_ticks + 1 if running else 0 # the no-entry, then the chime
if not running and in_control:
self._handback = self.handback_ticks
self.running = running
self._settle = max(self._settle - 1, 0)
if not in_control:
self._handback = 0
offer, self._offer = _tick(self._offer)
link_lost, self._handback = _tick(self._handback)
switching, self._switching = _tick(self._switching)
return Flags(offer, link_lost, switching and self._switching > 0, switching and self._switching == 0, self.settling)
-292
View File
@@ -1,292 +0,0 @@
"""
Copyright (c) 2026-, Zeph Leggett.
This file is part of jetlink and is licensed under the MIT License.
See the LICENSE file in the root directory for more details.
What a fork's hook lines call: Jetlink's methods, behind a net that never
raises.
manager, hardwared, the UI and the model manager call these on every device,
link on or off, and modeld on every drive: whatever jetlink does wrong turns
the link off and is logged, and never takes one of them down. Each hook has
the answer it gets with the link off, and gives it on any failure. They are
not all False: attach() keeps modeld's model, extend_catalog() the catalog,
and shutdown_step() says yes to a power-off that is wanted, since a broken
jetlink must never keep a comma on to drain the car's battery. selfdrived's
swap events (swap_events()) answer no event at all.
The fork's guard, which builds Hooks or else Off inside a function, and its
own fallback for no jetlink at all: docs/porting.md, "The adapter module".
Off is for a jetlink that imports but cannot be used (another API); link_on()
is the fork's stdlib read of the setting, so the reason a link cannot run is
told only to someone who turned it on.
Imported by every process that imports the fork's hook module, the resident
gadget owner among them: the standard library and jetlink's light half only.
Construction checks the API and nothing more; the adapter is built and bound
on the first hook that needs it, once per process. handed_over() and state()
read the model alone and never bind. With link_on() saying the link is off,
the only hooks that bind are BINDS_WITH_LINK_OFF's; every other one answers
as the link off, modeld's prepare() among them.
"""
from __future__ import annotations
import threading
from collections.abc import Callable
from jetlink.openpilot import API, Flags, Jetlink, SwapEvents, bind
# the APIs a fork's adapter may be written to: today's, and after a bump the
# previous one too, for one minor release (jetlink/openpilot/__init__.py)
COMPATIBLE = frozenset({API})
# the hooks that ask jetlink with link_on() saying off. status(): the UI shows
# the big model's name from the model library. shutdown_step(): hardwared
# clears, from boot, a power-off request it did not write; and the request
# and its wait, for a link turned off while one is pending. The model
# manager's: the catalog is hardware and the library, not the setting
BINDS_WITH_LINK_OFF = frozenset({'status', 'shutdown_step', 'request_shutdown', 'shutdown_pending', 'model_state',
'should_extend_catalog', 'extend_catalog'})
# what selfdrived is told when the swap events cannot run: none at all
NO_EVENTS = Flags(False, False, False, False, False)
# how often the swap events read modeld's runtime record without a state of the fork's own
RUNTIME_POLL_S = 0.2
def _text(error: BaseException) -> str:
"""An error as a log line says it, from an error whose str() may raise too."""
try:
return f"{type(error).__name__}: {error}"
except Exception:
return type(error).__name__
class _Hooks:
"""The hooks and the answer each gets with the link off. _call(name,
default, *args) is Jetlink.<name>(*args), or default with the link off;
_run() the same without asking link_on()."""
def __init__(self, reason: str = '', link_on: Callable[[], bool] | None = None):
self._why = reason
self._link_on = link_on
# the joining model's handovers as handed_over() last saw them, in this process
self._handovers = 0
self._prepared = False
def _link(self) -> bool | None:
"""link_on()'s answer, or None without one or for one that raises, which
counts as on: the hook asks jetlink, as without link_on()."""
try:
return None if self._link_on is None else bool(self._link_on())
except Exception:
return None
def _call(self, name: str, default, *args):
if name not in BINDS_WITH_LINK_OFF and self._link() is False:
return default
return self._run(name, default, *args)
def _run(self, name: str, default, *args):
return default
def should_run(self, started, params, CP) -> bool:
"""manager's should_run for the gadget owner."""
return self._call('enabled', False)
def status(self):
return self._call('status', None)
def reason(self) -> str | None:
"""Why jetlink cannot run here, told to someone who turned the link on,
else Jetlink.reason's. The call first: a bind that fails sets the why.
Without link_on() the why is told; with one that raises, nobody is."""
on = True if self._link_on is None else self._link()
if on is False:
return None
answer = self._run('reason', None)
return self._why if self._why and on else answer
def prepare(self, chestnut: bool) -> bool:
"""modeld, before config_realtime_process. A chestnut runs the large
model itself and never asks jetlink."""
self._prepared = bool(not chestnut and self._call('prepare', False))
return self._prepared
def attach(self, model, small, cam_w: int, cam_h: int):
"""modeld, once `small` is built: the model to run, `model` unless the link
joined. Binds nothing unless prepare() said yes in this process."""
if not self._prepared:
return model
# past prepare()'s yes: the setting has had its say for this modeld
joined = self._run('attach', None, small, cam_w, cam_h)
return model if joined is None else joined
# Jetlink's own two, which read the model and self._handovers alone: modeld
# asks every frame, so they never bind
def handed_over(self, model) -> bool:
try:
return bool(Jetlink.handed_over(self, model))
except Exception:
return False
def state(self, model) -> str:
try:
return Jetlink.state(self, model)
except Exception:
return 'none'
def runtime(self):
"""What modeld runs, from its runtime record (Jetlink.runtime): a
status.Runtime, or None with the link off or unusable, or a chestnut
fitted. Cheap enough for selfdrived's loop: two small files, and the
chestnut test's USB walk once per parts.CHESTNUT_TTL, as status()."""
return self._call('runtime', None)
def shutdown_step(self, should_shutdown: bool, reason: str = '') -> bool:
return self._call('shutdown_step', should_shutdown, should_shutdown, reason)
def request_shutdown(self, reason: str = '') -> bool:
return self._call('request_shutdown', False, reason)
def shutdown_pending(self) -> bool:
return self._call('shutdown_pending', False)
def model_state(self, ref: str) -> str | None:
return self._call('model_state', None, ref)
def should_extend_catalog(self) -> bool:
return self._call('should_extend_catalog', False)
def extend_catalog(self, catalog: dict) -> dict:
return self._call('extend_catalog', catalog, catalog)
def swap_events(self, tick_s: float) -> SwapTracker:
"""selfdrived's swap-event tracker, updated every `tick_s` (events.SwapEvents)."""
return SwapTracker(None, None, tick_s)
class Off(_Hooks):
"""The link off on every hook; reason() says why, when the link is on."""
class Hooks(_Hooks):
"""Jetlink over the fork's adapter, for an adapter written to `api`.
adapter_factory() makes it, on first use. Raises at once for an API this
jetlink does not accept. `log` is a logger, or a function that returns one,
for what fails before the adapter's log is there."""
def __init__(self, api: int, adapter_factory: Callable[[], object], *, link_on: Callable[[], bool] | None = None,
log=None):
if api not in COMPATIBLE:
raise ValueError(f"jetlink package API {API}, this build expects {api}")
super().__init__('', link_on)
self._factory = adapter_factory
# the adapter's once bound; until then the fork's, or the standard library's
self._log = log
self._bound = None
self._binding = threading.Lock()
# hook -> the failure last logged for it, cleared by a call that works
self._failed: dict[str, str] = {}
def _jetlink(self):
"""Bound once, the failure included: one Jetlink per process, so that
prepare() and attach() reach the same one."""
if self._bound is None:
with self._binding:
if self._bound is None:
self._bound = self._bind()
return self._bound or None
def _bind(self):
try:
op = self._factory()
self._log = getattr(op, 'log', None) or self._log
return bind(op)
except Exception as e:
self._why = f"jetlink failed to start: {_text(e)}"
self._report("jetlink: %s", self._why)
return False
def _report(self, msg: str, *args) -> None:
"""A failure to the log, in the except block that caught it. A log that
fails is let go: nothing a hook does may raise."""
try:
if not hasattr(self._log, 'exception'):
try:
log = self._log() # a factory; None or a logger raises here too
except Exception:
log = None
if not hasattr(log, 'exception'):
import logging
log = logging.getLogger('jetlink')
self._log = log
self._log.exception(msg, *args)
except Exception:
pass
def _net(self, name: str, default, fn, *args):
"""fn(*args), or default when it raises. Logged once per distinct error
(the UI would log a failing status five times a second), again after a
call that works."""
try:
result = fn(*args)
except Exception as e:
text = _text(e)
if self._failed.get(name) != text:
self._failed[name] = text
self._report("jetlink: %s() failed: %s", name, text)
return default
self._failed.pop(name, None)
return result
def _run(self, name: str, default, *args):
def call():
jetlink = self._jetlink()
return default if jetlink is None else getattr(jetlink, name)(*args)
return self._net(name, default, call)
def swap_events(self, tick_s: float) -> SwapTracker:
"""Bookkeeping alone, so it binds nothing: selfdrived builds it once,
link on or off."""
return SwapTracker(self._net('swap_events', None, SwapEvents, tick_s), self, tick_s)
class SwapTracker:
"""events.SwapEvents behind the net: update() is no event at all when it
fails, and settling False; the failure is logged as the hooks' are. Inert
without a tracker."""
def __init__(self, events: SwapEvents | None, hooks: Hooks | None, tick_s: float):
self._events = events
self._hooks = hooks
self._ok = events is not None
# runtime() a few times a second: modeld rewrites its record once a second
self._every = max(round(RUNTIME_POLL_S / tick_s), 1) if events is not None else 1
self._ticks = 0
self._runtime = None
@property
def settling(self) -> bool:
try:
return self._ok and bool(self._events.settling)
except Exception:
return False
def update(self, fresh: bool, big: bool, in_control: bool, state: str | None = None) -> Flags:
"""SwapEvents.update. `state` is the fork's own message's, where it has
one (zoompilot: modelDataV2SP.acceleratorState); without it the tracker
reads modeld's runtime record (hooks.runtime()), and a tick without a
live one is not fresh."""
if self._events is None:
return NO_EVENTS
if state is None:
if self._ticks == 0:
self._runtime = self._hooks.runtime()
self._ticks = (self._ticks + 1) % self._every
state = getattr(self._runtime, 'state', None)
fresh = fresh and state is not None
flags = self._hooks._net('swap_events.update', None, self._events.update, fresh, big, in_control, state or 'none')
self._ok = flags is not None
return NO_EVENTS if flags is None else flags
+20 -83
View File
@@ -17,7 +17,7 @@ The interface is split by process. The resident gadget owner gets data only
(OwnerConfig), no callbacks, so it stays at the standard library and about
10 MB. The heavy processes get the adapter object and each uses its side:
StatusSide in manager, the UI, hardwared and the model manager, WorkerSide in
the provisioning run, ModelSide in modeld.
the provisioning run, ModelSide in modeld, BuildSide in the warp build.
The standard library only: the owner imports this module.
"""
@@ -82,8 +82,7 @@ class ModelFace:
# The sides are typing Protocols so a type checker can hold the adapter to
# them; conformance() is the check both repos run. A parameter jetlink passes
# by keyword is keyword-only here, and only those names must match. The
# members in OPTIONAL are ones jetlink reads with getattr and a fallback.
# by keyword is keyword-only here, and only those names must match.
class StatusSide(Protocol):
"""Every reader: manager, the UI, hardwared, the model manager."""
@@ -104,11 +103,10 @@ class StatusSide(Protocol):
"""Is comma's chestnut fitted? A USB walk; jetlink caches the answer."""
def camera(self) -> tuple[int, int, int, int]:
"""This device's (cam_w, cam_h, model_w, model_h): the warp modeld loads."""
"""This device's (cam_w, cam_h, model_w, model_h): the warp modeld asks for."""
def warp_path(self, cam_w: int, cam_h: int, model_w: int, model_h: int) -> Path:
"""Where modeld's warp for this geometry is: openpilot's driving warp
pickle, which warp.Warps.load reads."""
"""Where the build puts the warp for this geometry."""
class WorkerSide(StatusSide, Protocol):
@@ -129,98 +127,39 @@ class WorkerSide(StatusSide, Protocol):
class ModelSide(WorkerSide, Protocol):
"""modeld: comma's model face, whether anything is in control, and the
structured log line the link's telemetry goes to. What modeld writes onto
the model every frame is Jetlink.attach's."""
"""modeld: comma's model face, the engagement a swap waits out, and the
structured log line the link's telemetry goes to."""
def model_face(self) -> ModelFace:
"""What a ModelState for comma's large model has to carry."""
def in_control(self) -> bool:
"""Is openpilot or MADS in control? The joining model asks on modeld's
frame thread before every frame and swaps the large model in only while
not. True when it cannot tell; never raises."""
def engagement(self) -> Callable[[int], bool]:
"""A poller, made on the thread that calls it: (timeout_ms) -> engaged. Waits
up to timeout_ms for news and answers True when controls are engaged, or when
that is not known."""
def event(self, name: str, **fields: Any) -> None:
"""A structured log line (cloudlog.event)."""
def small_reset(self, small: Any) -> Callable[[], None] | None:
"""Optional (OPTIONAL): the reset of the fork's small model for a hand-back, or None
for jetlink's own (warp.prepare_reset, which knows stock modeld's,
modeld_v2's and a ModelState with GPU queues and numpy inputs). Asked once
on modeld's main thread before the frame loop. The reset runs on the
frame thread at a hand-back, so it zeroes the history in place and
allocates and compiles nothing; warp.reset_queues makes one."""
class BuildSide(Protocol):
"""scons: comma's warp graph."""
def make_warp(self, cam_w: int, cam_h: int, model_w: int, model_h: int) -> tuple[Callable[..., Any], int]:
"""The warp graph for this geometry, and the size of the NV12 frame it
reads. Imports comma's graph module before anything of tinygrad's."""
class Openpilot(ModelSide, Protocol):
class Openpilot(ModelSide, BuildSide, Protocol):
"""The whole adapter: one object implements every side."""
# What modeld hands Jetlink.attach and gets back: not the adapter's, pinned so
# a fork knows what its ModelState needs and what it may read off the result.
class SmallModel(Protocol):
"""The fork's own ModelState, whatever its shape. jetlink calls run()
positionally and writes attributes onto it (lat_delay, PLANPLUS_CONTROL);
the rest is optional, read with getattr: the frame's desire key and numpy
inputs (modeld_v2), the lateral delay a joining large model starts on, and
sunnypilot's planplus gain."""
desire_key: str
numpy_inputs: Any
lat_delay: float
PLANPLUS_CONTROL: float
def run(self, bufs: Any, transforms: Any, inputs: Any, after_enqueue: Callable[[], None] | None = None) -> Any:
"""One frame's outputs, as modeld's ModelState.run."""
class JoinedModel(Protocol):
"""What attach returns and modeld reads off and writes onto every frame.
frame_drop_ratio is written, lat_delay read and written. lat_delay and the
face (get_action_from_model through LONG_SMOOTH_SECONDS, modeld_v2's reads)
follow the model that drives, like any name it does not define, so over a
stock modeld's small model they are there only while the large one drives."""
handovers: int # moves on every swap and hand-back; modeld forgives that frame's drops
big_model_state: str # one of STATES
chestnut: bool # modelV2.big
get_action_from_model: Callable[..., Any]
constants: Any
LAT_SMOOTH_SECONDS: float
LONG_SMOOTH_SECONDS: float
frame_drop_ratio: float # modeld's share of dropped camera frames, before every run()
lat_delay: float
def run(self, bufs: Any, transforms: Any, inputs: Any, after_enqueue: Callable[[], None] | None = None) -> Any:
"""One frame's outputs, from whichever model drives."""
def close(self) -> None:
"""Let go of the link at modeld's exit."""
# The members of each protocol an implementation may lack: jetlink reads them
# with getattr and a fallback. conformance() passes an object without one and
# holds one with it to its signature, and adding one keeps API. Kept out of
# the protocols, where a name would be a member itself; a type checker still
# asks for these, as typing has no optional member
OPTIONAL: dict[type, tuple[str, ...]] = {
ModelSide: ('small_reset',),
SmallModel: ('desire_key', 'numpy_inputs', 'lat_delay', 'PLANPLUS_CONTROL'),
JoinedModel: ('get_action_from_model', 'constants', 'LAT_SMOOTH_SECONDS', 'LONG_SMOOTH_SECONDS', 'lat_delay'),
}
def optional(protocol: type) -> frozenset[str]:
"""The members of `protocol` an implementation may lack, inherited ones included."""
return frozenset(name for cls in protocol.__mro__ for name in OPTIONAL.get(cls, ()))
def load_adapter(module: str) -> Openpilot:
"""The adapter an adapter module makes. An adapter module is the fork's
module that holds every openpilot import jetlink needs: its top level
imports only the standard library, and it has adapter() -> Openpilot and
owner_config() -> OwnerConfig. The entry points that run as their own
process (the provisioning run) are told its name."""
process (the provisioning run, the warp build) are told its name."""
return importlib.import_module(module).adapter()
@@ -262,11 +201,9 @@ def conformance(obj: object, protocol: type) -> list[str]:
jetlink pins the interface with it, and the fork's tests run it against the
real adapter."""
problems = []
may_lack = optional(protocol)
for name, expected in members(protocol).items():
if not hasattr(obj, name):
if name not in may_lack:
problems.append(f"{name}: missing")
problems.append(f"{name}: missing")
continue
if expected is None:
continue
+282 -301
View File
@@ -15,41 +15,36 @@ So modeld is handed the small model it already loaded, and the Jetson is
swapped in underneath once the link, the engine and the warp are all there.
modeld re-reads `model` every frame, and `modelV2.big` keeps its meaning.
Until the swap the comma runs its own model and nothing else; the link idles
on a keepalive ping. The large model is built on the swap frame and starts
from a reset history, as at every modeld start. The second after the swap is
its proof: the fork holds engagement off for it, and a large model that holds
a frame in it (model_state.PROVING_FRAMES) hands back before anyone can
engage. While it drives, a reply that is not back in time is not waited for:
the previous frame's output is published again (a held frame,
From the join on, every frame goes to the Jetson whether or not it drives:
the small model drives on, and the large model's history and hidden state are
current and warm when the swap comes, so its first driving frame costs what
every frame costs. Those shadow frames are also the proof: the large model
is offered only once the host has answered the last PROVING_FRAMES of them
in time (model_state.keeping_up), so a host cold from a reconnect, or one
that is simply too slow, proves itself with the small model driving and not
at the wheel. The swap itself changes which model's output is published and
nothing else. While it drives, a reply that is not back in time is not
waited for: the previous frame's output is published again (a held frame,
model_state.HOLD_FRAME) and the camera frame is not dropped. Too many holds
hand the drive back to the small model, as a lost link does.
No frames go to the host while the small model drives. Sending them costs
the comma ~9 ms of every frame for as long as the window stays shut (a whole
drive with MADS on), which starves the driver monitoring model under
selfdrived's frequency floor (commIssue).
Two rules the swap keeps:
- tinygrad work happens on modeld's thread. The joining thread does link IO
only; building the JetlinkModelState unpickles a TinyJit, and doing that
next to the small model running frames on the same device is not safe.
- never swap while the plan is steering. The two models disagree by ~195 m of
planned path, and the swap costs a frame or two. The adapter says before
every frame whether anything is in control (in_control), from openpilot's
own messages; standstill alone is not enough, longitudinal control may still
hold the brake.
planned path, and the swap costs a frame or two. Require fresh, fully
disengaged controls; standstill alone is not enough, longitudinal control
may still hold the brake.
"""
from __future__ import annotations
import os
import threading
import time
from jetlink import protocol as P
from jetlink.comma import gadget
from jetlink.openpilot.model_state import JetlinkModelState
from jetlink.transport.priority import background_thread
# after a join fails or the large model dies mid-drive. Long enough not to
@@ -61,25 +56,39 @@ REJOIN_DELAY = 5.0
REJOIN_DELAY_MAX = 60.0
STABLE_SECONDS = 60.0
# after a join that held: a link that ran for minutes and then went is a USB
# drop, and the host re-enumerates a rebound gadget in under a second
# drop, and the host re-enumerates a rebound gadget in under a second. The
# rest of the old delay was the driver's time, not the Jetson's
REJOIN_DELAY_QUICK = 1.0
# failures in a streak retried that quickly before the doubling starts. A
# demote is mostly one late frame, a tail of the link or the host and not a
# dead one, and a full backoff costs 5 to 60 s on the small model. A link
# that fails on its heels this many times is backed off as usual
# dead one: on a phone's cable each one cost 5 to 60 s on the small model,
# five times in nine minutes on an iPad (2026-10-03). A link that fails on
# its heels this many times is backed off as before
QUICK_RETRIES = 3
# how often a link that is ready but has nowhere to land gets checked, and
# how long the check may take; both off the frame loop
KEEPALIVE_PERIOD = 10.0
PING_TIMEOUT = 2.0
# after a link lost while it only shadowed: no swap, demote or alert to pay
# for, so no doubling. A phone that dropped its first frames again and again
# pulsed the icon for 1 to 60 s between tries (2026-10-04)
SHADOW_REJOIN_DELAY = 2.0
# how often a backoff looks at the gadget. A host that configures it again
# after it went away is a replug, which the backoff is not for: a replugged
# Jetson is back in under half a second
# after it went away is a replug, which the backoff is not for: one waited 16 s
# for a Jetson that was back in 0.4 (2026-09-29)
REPLUG_POLL = 0.25
# link drops in one drive before the status names the cable. One is chance;
# repeated drops are almost always the USB-C port losing its host, and nothing
# else the driver can see says so
# link drops in one drive before the status names the cable. One is weather;
# the 2026-09-07 evening drive had six in twelve minutes, every one the
# USB-C port losing its host, and nothing the driver could see said so
DROPS_TO_BLAME_CABLE = 2
ENGAGEMENT_POLL_MS = 100
ENGAGEMENT_MAX_AGE = 0.25
# a large-model frame is ~30 ms and the worst seen on the current stack ~55.
# One that took LATE_FRAME, or a second past SLOW_FRAME within LAG_WINDOW of
# the last, is a fault and is handled as a loss: modeld would otherwise count
# the dropped camera frames into modeldLagging with the slow model still
# steering. With a late reply held rather than waited for (model_state), a
# frame this long is the comma's own stall, a warp or a send, or a frame with
# nothing to hold yet. A frame past link.INFERENCE_TIMEOUT never returns; it fails
LATE_FRAME = 0.1
SLOW_FRAME = 0.075
LAG_WINDOW = 10.0
# A host slower than the camera but never that slow (an iPhone that has warmed
# up: 55 ms a frame drops one camera frame in ten) still has modeld skipping
# frames, and selfdrived soft-disables on modeldLagging once modeld's filtered
@@ -89,58 +98,54 @@ DROPS_TO_BLAME_CABLE = 2
# a second within about 6.5 s is not. modeld forgives the frame of a handover
# too, so the drops that decided it never reach selfdrived
DROP_LIMIT = 0.0075
# the first frames after every swap are never counted as slow: the first after
# a join carries the history reset, and a Mac's is ~100 ms of CoreML warm-up
SETTLING_FRAMES = 3
# the small model drives the first frames of every modeld start, even with the
# large model ready: its first run in a process costs ~1.3 s, which the first
# fallback frame would pay (25 dropped frames, commIssue) if the large model
# drove from frame one. Nothing is in control at a modeld start
# fallback frame paid (25 dropped frames, commIssue) when the large model had
# driven from frame one. Nothing is in control at a modeld start
SMALL_WARMUP_FRAMES = 3
# the panel's line while a ready link waits for the window
READY = 're-engage to switch'
# shadowing this long without keeping up, the panel says the link is too slow
# to drive rather than 'checking' for the drive. The proof goes on: a host
# that warms up late still gets offered
SLOW_LINK_AFTER = 10.0
# the panel's line while the large model shadows: once it is built, once it
# keeps up, and once it has not for SLOW_LINK_AFTER
CHECKING = 'checking the link keeps up'
PROVEN = 're-engage to switch'
SLOW = 'too slow to drive'
# why a demote happened, as the log says it
LOST = 'lost jetlink'
BEHIND = 'jetlink fell behind'
# a NaN or an inf in an output: a host that answers, wrongly. A lag and not a
# drop, the link is kept, and the next swap starts the host from a reset
BAD_OUTPUT = 'jetlink output not finite'
# what modeld_v2 builds the frame's inputs from: the small bundle's desire
# name and optional slots, read off the small model whichever drives
SMALL_READS = ('desire_key', 'numpy_inputs')
# why a built model was let go before it drove
NOT_UP = 'could not bring up the large model'
# and as the leave says it to the server (protocol.Msg.LEAVE)
LEAVING = {LOST: P.LEAVE_LOST, BEHIND: P.LEAVE_BEHIND}
# the demotes that count as lag, which says nothing about the cable
LAGS = (BEHIND, BAD_OUTPUT)
# and as the panel's one line says it while the join tries again, with the
# `why` of its progress code (status.PROGRESS_CODES 'reconnecting')
RECONNECTING = {LOST: ('reconnecting', 'lost'), BEHIND: ('fell behind, reconnecting', 'behind'),
NOT_UP: ('load failed, reconnecting', 'load_failed'),
BAD_OUTPUT: ('bad output, reconnecting', 'bad_output')}
# how long a model name the catalog had not resolved stays the sha's
NAME_RETRY = 30.0
# and as the panel's one line says it while the join tries again
RECONNECTING = {LOST: 'reconnecting', BEHIND: 'fell behind, reconnecting', NOT_UP: 'load failed, reconnecting'}
class JoiningModelState:
"""Duck-types openpilot's modeld ModelState, with a second one inside.
`progress` is where the join says what it is waiting on (status.Progress),
`log` is cloudlog on a comma, and `in_control` the adapter's, asked before
every frame; without one, the in_control last written stands. `name` is
the catalog's name for a model's sha256, for the runtime record.
`engagement` makes the poller the swap window reads (the adapter's), and
`log` is cloudlog on a comma.
"""
def __init__(self, small, connect, build, prepare=None, reset_small=None, *, progress, log, in_control=None,
name=None):
def __init__(self, small, connect, build, prepare=None, reset_small=None, *, progress, engagement, log):
self._small = small
self._active = small
self._connect = connect
self._build = build
self._reset_small = reset_small
self._progress = progress
self._engagement = engagement
self._log = log
# whatever the swap would otherwise do on the frame loop, done now on
# modeld's main thread before the frame loop exists; at the swap it costs
# modeld's main thread before the frame loop exists; at the swap it cost
# a 1.5 s frame
if prepare is not None:
try:
@@ -151,17 +156,14 @@ class JoiningModelState:
log.exception("jetlink: could not prepare the large model ahead of the swap")
raise
# handed over by the joining thread, consumed by the swap frame, which
# handed over by the joining thread, consumed by the next frame, which
# builds the large model state from it. Only ever assigned under the lock
self._joined: tuple[object, object] | None = None
# a link is ready and waiting for the window. Kept true while a keepalive
# ping has _joined out for a moment
self._available = False
# the large model state, from the swap until it hands back: always _active.
# Only the frame thread assigns it
# the large model state once built: fed every frame, driving when it is
# _active. Only the frame thread assigns it
self._big = None
# what the frame thread let go of and why (LOST, BEHIND, BAD_OUTPUT, or a
# build that failed), for the join thread to say goodbye to and close or keep.
# what the frame thread let go of and why (LOST, BEHIND, or a build that
# failed), for the join thread to say goodbye to and close or keep.
# Assigned under the lock
self._retired: tuple[object, str] | None = None
self._lock = threading.Lock()
@@ -180,24 +182,29 @@ class JoiningModelState:
# changes of the model that drives, and decisions to change it: modeld
# compares it across run() (handovers)
self._handovers = 0
# set on a frame the large model fell behind on; the next frame demotes
# set on a frame the large model fell behind on; the next frame demotes.
# _slow_at is when the last slow frame was, for the second strike
self._lagging = False
self._slow_at: float | None = None
# modeld's share of dropped camera frames, for the frame about to run
self._frame_drop_ratio = 0.0
# set by the large model's first frame: the join thread clears the
# progress, which is params IO and not the frame thread's
self._announce = False
# frames the small model has run since start
# frames each model has run: the small one's since start, the large one's
# since it swapped in
self._small_frames = 0
self._big_frames = 0
# the panel's last line about the large model shadowing (CHECKING, PROVEN
# or SLOW). A report is a param write on the frame thread, so only a
# change is reported
self._proving = CHECKING
# whether the host had let go of the gadget when the last link was lost,
# and whether this failure streak has already skipped a backoff for a replug
self._host_left = False
self._replugged = False
# in control until the adapter says otherwise, so a swap can never happen
# on no information
self._in_control = True
self._ask_in_control = in_control
# assume engaged and moving until a message says otherwise, so a swap can
# never happen on no information
self._engaged = True
self._engagement_updated = 0.0
self._stop = threading.Event()
# whether the large model has produced a frame; a first inference that
@@ -205,20 +212,10 @@ class JoiningModelState:
# modelDataV2SP, not a param: a chestnut's load is over once, this never is
self._loading = True
# the runtime record's: when the large model's last answer (not a held
# frame) came, and a wake for its writer on a hand-back and the first
# answer. The frame thread assigns them and nothing else; the writer
# thread reads the rest itself. The path is taken now, so a test's stays
# its own
self._output_at = 0.0
self._record = threading.Event()
self._record_path = gadget.MODELD
self._name = name
self._thread = threading.Thread(target=self._join_loop, daemon=True)
self._thread.start()
self._writer = threading.Thread(target=self._write_loop, daemon=True)
self._writer.start()
self._threads = [threading.Thread(target=self._join_loop, daemon=True),
threading.Thread(target=self._watch_engagement, daemon=True)]
for t in self._threads:
t.start()
# -- what modeld reads ------------------------------------------------------
# A read not defined here follows the model that is driving (__getattr__):
@@ -226,16 +223,21 @@ class JoiningModelState:
# function, which sunnypilot keeps on the ModelState so a custom small bundle
# can carry its own. Swaps and demotes happen inside run(), so what is read
# after it belongs to the model whose output it is. What the loop writes
# lands on both, but for what only this reads. The frame's inputs are built
# for the small bundle (SMALL_READS, off the small model whichever drives);
# the large model reads the desire under any name and the loop always
# supplies what it needs, so a swap can land on any frame
# lands on both. The frame's inputs are built for the small bundle (its
# desire name, its optional slots); the large model reads the desire under
# any name and the loop always supplies what it needs, so a swap can land on
# any frame
@property
def big_model_available(self) -> bool:
"""Connected and waiting to switch: what big_model_state calls ready, and
what the driver is offered."""
return not self._stop.is_set() and self._available and self._active is self._small
"""Built, keeping up with the frames it shadows, and waiting to switch:
what big_model_state calls ready, and what the driver is offered."""
return not self._stop.is_set() and self._shadowing and self._big.keeping_up
@property
def _shadowing(self) -> bool:
# a large model built and fed every frame, with the small model driving
return self._big is not None and self._active is self._small
@property
def chestnut(self) -> bool:
@@ -269,6 +271,14 @@ class JoiningModelState:
# read by the status publisher on every send, so the telemetry follows the link
return getattr(self._active, 'client', None)
@property
def desire_key(self) -> str:
return self._small.desire_key
@property
def numpy_inputs(self):
return self._small.numpy_inputs
@property
def lat_delay(self):
return self._active.lat_delay
@@ -298,36 +308,25 @@ class JoiningModelState:
@frame_drop_ratio.setter
def frame_drop_ratio(self, value):
# modeld's frameDropPerc / 100, written before every run(). Read here, by
# the lag rule; no small model reads it
# the lag rule; it lands on the small model too, as every write does
self._frame_drop_ratio = value
@property
def in_control(self) -> bool:
return self._in_control
@in_control.setter
def in_control(self, value):
# openpilot or MADS in control, by openpilot's own messages, asked before
# every run(): the swap waits for False. Not a thread polling selfdrived:
# that wakes ~300 times a second and takes 6 to 11 % of a core inside modeld
self._in_control = value
self._small.frame_drop_ratio = value
def __getattr__(self, name):
# Only for names this class does not define, so a read that a comma or
# sunnypilot sync adds is not an AttributeError on the frame thread (modeld
# re-raises it and is dead for the drive). Reads only; a write lands on
# this object alone, so each one modeld makes has its own setter above
# Only for names this class does not define. Without it, a comma or
# sunnypilot sync that adds one read was an AttributeError on the frame
# thread, which modeld re-raises: modeld dead for the drive, on jetlink
# devices only. Reads only; a new write lands here and nowhere else, so
# each one modeld makes has its own setter above
if name.startswith('_'):
# also what keeps __init__ from recursing before _active exists
raise AttributeError(name)
# a small model without one of SMALL_READS has none, whichever drives
return getattr(self._small if name in SMALL_READS else self._active, name)
return getattr(self._active, name)
# -- the frame path ---------------------------------------------------------
def run(self, bufs, transforms, inputs, after_enqueue=None):
if self._ask_in_control is not None:
self.in_control = self._ask_in_control()
self._adopt()
if self._lagging:
# the last frame was the large model's last, published as it came
self._lagging = False
@@ -337,71 +336,132 @@ class JoiningModelState:
self._log.warning("jetlink: modeld dropped %.2f %% of camera frames behind the large model, "
"the small model drives from this one", self._frame_drop_ratio * 100)
self._demote(BEHIND)
self._note_proving()
self._maybe_swap()
if self._active is self._small:
self._small_frames += 1
return self._small.run(bufs, transforms, inputs, after_enqueue)
return self._run_small(bufs, transforms, inputs, after_enqueue)
return self._run_big(bufs, transforms, inputs, after_enqueue)
def _run_small(self, bufs, transforms, inputs, after_enqueue):
"""The small model's frame, and the large model's too when there is one:
warped now, sent once the small model's own work is on the GPU (its
after_enqueue), so the send overlaps it. A link that fails here is lost
before it drove: let go and reopened, with the small model's frame
unharmed."""
big = self._big
if big is None:
return self._small.run(bufs, transforms, inputs, after_enqueue)
try:
big.prepare(bufs, transforms, inputs)
except Exception:
self._log.exception("jetlink: the large model's link failed with the small model driving, reopening")
self._retire(LOST)
return self._small.run(bufs, transforms, inputs, after_enqueue)
failed: list[BaseException] = []
def enqueued():
try:
big.send()
except Exception as e: # never through the small model's run: modeld would make it fatal
failed.append(e)
if after_enqueue is not None:
after_enqueue()
result = self._small.run(bufs, transforms, inputs, enqueued)
if not failed and not big.sent:
# a small model that did not call back; send now
try:
big.send()
except Exception as e:
failed.append(e)
if not failed:
# the proof: would this frame have been held? (model_state.check_shadow)
try:
big.check_shadow()
except Exception as e:
failed.append(e)
if failed:
self._log.error("jetlink: the large model's link failed with the small model driving, reopening",
exc_info=failed[0])
self._retire(LOST)
return result
def _run_big(self, bufs, transforms, inputs, after_enqueue):
big = self._big
started = time.monotonic()
try:
result = big.run(bufs, transforms, inputs, after_enqueue)
except Exception as e:
except Exception:
failed = time.monotonic()
self._log.exception("jetlink: large model failed mid-drive, back to the small model")
self._demote(BAD_OUTPUT if isinstance(e, P.NotFinite) else LOST)
self._demote(LOST)
demoted = time.monotonic()
# re-run the frame rather than propagate: modeld's fallback is permanent,
# this one is retryable. after_enqueue is dropped, the large model may
# already have called it
result = self._small.run(bufs, transforms, inputs, None)
# all of it is one modeld frame, 42 to 107 ms measured with the small
# model warm. The dropped camera frames are forgiven
# all of it is one modeld frame, 42 to 107 ms on the 2026-09-29 drives
# with the small model warm. The dropped camera frames are forgiven
# (modeld sees the model change), but the log should say which part it was
done = time.monotonic()
self._log.warning("jetlink: fallback frame %.0f ms: link %.0f, demote %.0f, small model %.0f",
(done - started) * 1e3, (failed - started) * 1e3,
(demoted - failed) * 1e3, (done - demoted) * 1e3)
return result
if not getattr(big, 'held', False):
self._output_at = time.monotonic()
took = time.monotonic() - started
self._big_frames += 1
if self._loading:
# a connected engine can still fail its first inference; only announce
# readiness after a frame the caller can publish
self._loading = False
self._announce = True
self._rejoin.set()
self._record.set()
self._progress.clear()
self._log.warning("jetlink: large model joined mid-drive, modelV2.big is now true")
if big.behind:
# this frame's output is published as it came; its stall is forgiven now
self._lagging = True
self._handed_over()
self._log.warning("jetlink: large model %s, the small model drives from the next", big.behind)
if self._big_frames > SETTLING_FRAMES:
why = getattr(big, 'behind', None) or (f'took {took * 1e3:.0f} ms' if self._fell_behind(took) else None)
if why:
# this frame's output is published as it came; its stall is forgiven now
self._lagging = True
self._handovers += 1
self._log.warning("jetlink: large model %s, the small model drives from the next", why)
return result
def _maybe_swap(self) -> None:
"""Build the large model state from a join that has landed and swap it in,
on this thread, which is where everything tinygrad touches must happen.
Its first frame resets the host's history. A build that fails is backed
off like a demote, or one that fails the same way every time is a connect
and a build per second for the drive."""
if self._joined is None or self._in_control or self._small_frames < SMALL_WARMUP_FRAMES:
def _fell_behind(self, took: float) -> bool:
if took > LATE_FRAME:
return True
if took <= SLOW_FRAME:
return False
now = time.monotonic()
second = self._slow_at is not None and now - self._slow_at <= LAG_WINDOW
self._slow_at = now
return second
@property
def _window_open(self) -> bool:
# standstill does not make an active longitudinal controller safe to swap
fresh = 0 <= time.monotonic() - self._engagement_updated < ENGAGEMENT_MAX_AGE
return fresh and not self._engaged
def _adopt(self) -> None:
"""Build the large model state from a join that has landed, on this
thread, which is where everything tinygrad touches must happen. From the
next frame on it is fed every frame (_run_small) and swapped in when the
window opens. A build that fails is backed off like a demote, or one that
fails the same way every time is a connect and a build per second for
the drive."""
if self._joined is None:
return
with self._lock:
joined, self._joined = self._joined, None
if joined is not None:
self._available = False
if joined is None:
return
client, spec = joined
try:
t0 = time.monotonic()
big = self._build(client, spec)
big.lat_delay = getattr(self._small, 'lat_delay', big.lat_delay)
self._log.warning("jetlink: built the large model state in %.0f ms", (time.monotonic() - t0) * 1000)
big.lat_delay = self._small.lat_delay
self._log.warning("jetlink: built the large model state in %.0f ms; it runs every frame from here",
(time.monotonic() - t0) * 1000)
except Exception:
self._log.exception("jetlink: could not bring up the large model, staying small")
with self._lock:
@@ -409,33 +469,63 @@ class JoiningModelState:
self._back_off()
return
self._big = big
self._proving = CHECKING # the join thread said so at link ready
# the link is held from here, shadowing or driving: a shadow that ran for
# minutes and then went starts a new streak, as a drive that did
self._joined_at = time.monotonic()
# no wake: the record says the swap with the first answer, which wakes it
def _note_proving(self) -> None:
"""The panel's line while the large model shadows, on a change only:
PROVEN once it keeps up, SLOW once it has not for SLOW_LINK_AFTER, and
CHECKING again after a hand-back. The first driven frame clears it."""
if not self._shadowing:
return
if self._big.keeping_up:
line = PROVEN
elif time.monotonic() - self._joined_at > SLOW_LINK_AFTER:
line = SLOW
else:
line = CHECKING
if line != self._proving:
self._proving = line
self._log.warning("jetlink: large model %s", line)
self._report('connect', line)
def _maybe_swap(self) -> None:
if not self._shadowing:
return
if not self._big.keeping_up or not self._window_open or self._small_frames < SMALL_WARMUP_FRAMES:
return
# nothing to build and nothing to reset (the module docstring says why)
self._big_frames = 0
self._slow_at = None
self._handovers += 1
self._active = big
self._active = self._big
def _demote(self, why: str) -> None:
"""Back to the small model, from a reset history. On the frame thread, so
nothing here waits: the join thread reads the port, reports and closes
the link, moments later. Counted as a lost link or a hand-back for lag,
and backed off."""
the link, moments later."""
self._retire(why) # while _shadowing still tells a drive from a shadow
self._active = self._small
self._handovers += 1
self._loading = True
if self._reset_small is not None:
self._reset_small()
def _retire(self, why: str) -> None:
"""Let the large model go, driving or not, for the join thread to say
goodbye to and close or keep (_close_retired). Counted as a lost link or
a hand-back for lag, and backed off."""
shadow = self._shadowing
big, self._big = self._big, None
if why in LAGS:
if why == BEHIND:
self._lags += 1
else:
self._drops += 1
with self._lock:
self._retired = (big, why)
self._back_off()
self._active = self._small
self._handed_over()
self._loading = True
if self._reset_small is not None:
self._reset_small()
def _handed_over(self) -> None:
self._handovers += 1
self._record.set()
self._back_off(shadow)
def _close_retired(self, retired, why: str) -> None:
"""Say goodbye over the retired large model's link, then close it if it
@@ -467,13 +557,14 @@ class JoiningModelState:
except Exception:
self._log.exception("jetlink: could not say why the link is left")
def _back_off(self) -> None:
def _back_off(self, shadow: bool = False) -> None:
"""Push the next attempt out, further each time one fails on its heels.
Each failed cycle is a swap frame, a demote frame and the alerts that go
with them. The first QUICK_RETRIES failures of a streak are retried in
REJOIN_DELAY_QUICK; the ones after double from REJOIN_DELAY. A join that
held for STABLE_SECONDS starts a new streak.
held for STABLE_SECONDS starts a new streak. A link lost while it only
shadowed waits SHADOW_REJOIN_DELAY.
"""
held = time.monotonic() - self._joined_at if self._joined_at else 0.0
stable = bool(self._joined_at) and held > STABLE_SECONDS
@@ -481,7 +572,9 @@ class JoiningModelState:
if stable:
self._replugged = False # a new streak may skip a backoff for a replug again
self._joined_at = 0.0
if self._failures <= QUICK_RETRIES:
if shadow:
delay = SHADOW_REJOIN_DELAY
elif self._failures <= QUICK_RETRIES:
delay = REJOIN_DELAY_QUICK
else:
delay = min(REJOIN_DELAY * 2 ** (self._failures - QUICK_RETRIES - 1), REJOIN_DELAY_MAX)
@@ -492,18 +585,17 @@ class JoiningModelState:
# -- background -------------------------------------------------------------
def _report(self, stage: str, msg: str, code: str | None, args: dict | None = None) -> None:
def _report(self, stage: str, msg: str) -> None:
"""Tell the UI what the join is waiting on.
Offroad the record carries a provisioning run's progress; without this, a
Jetson that is not plugged in looks like one about to load. No fraction to
give, and the panel does not invent one.
Jetson that is not plugged in looked like one six seconds from loading. No
fraction to give, and the panel does not invent one.
"""
# the count is the diagnosis. A link that runs for minutes and then goes,
# again and again, is the cable, and the cable is the one thing the
# driver can do something about
self._progress.report(stage, 0.0, msg, drops=self._drops if self._drops >= DROPS_TO_BLAME_CABLE else 0,
code=code, args=args)
self._progress.report(stage, 0.0, msg, drops=self._drops if self._drops >= DROPS_TO_BLAME_CABLE else 0)
def _note_link_loss(self, why: str) -> None:
"""Off the frame thread: what the comma's USB-C port sees now, next to the
@@ -518,8 +610,7 @@ class JoiningModelState:
# has still been replugged
self._host_left = not gadget.host_attached()
self._log.warning("jetlink: %s, %s; drop %d, lag %d this drive", why, port, self._drops, self._lags)
msg, code = RECONNECTING[why]
self._report('connect', msg, 'reconnecting', {'why': code})
self._report('connect', RECONNECTING[why])
def _join_loop(self) -> None:
"""Open the link and get the engine ready. No tinygrad in here."""
@@ -534,16 +625,6 @@ class JoiningModelState:
# a dead link, and close() sets this. An idle wake per second is not
# free on modeld's core
self._rejoin.wait()
if self._announce:
# the large model's first frame is out. Before the retired one below,
# whose reconnecting line is newer; woken for this alone, wait again.
# A demote or close sets its own flag before the event, so clearing
# the event and then looking misses neither
self._announce = False
self._progress.clear()
self._rejoin.clear()
if self._retired is None and not self._stop.is_set():
continue
with self._lock:
retired, self._retired = self._retired, None
if retired is not None:
@@ -560,20 +641,16 @@ class JoiningModelState:
self._rejoin.clear()
if self._wait_out_back_off():
return
self._report('connect', 'waiting for jetlink', 'waiting_for_host')
self._report('connect', 'waiting for jetlink')
try:
# the connect can take minutes when the picked model still has to be
# built, so it is handed the flag close() sets rather than polled
client, spec = self._connect(self._stop.is_set)
except Exception as e:
# expected while the Jetson boots, so no stack trace. A failure that
# knows when it is worth asking again says so (link.ModelMissing), and
# what the panel should show meanwhile
delay = getattr(e, 'retry_after', REJOIN_DELAY)
if waiting := getattr(e, 'waiting', None):
self._report('connect', waiting, getattr(e, 'code', None))
self._log.warning("jetlink: not joined yet (%s), retrying in %.0fs", e, delay)
if self._stop.wait(delay):
# expected while the Jetson boots. Not exception(): a stack trace every
# 5 s for the first minute of every drive is noise
self._log.warning("jetlink: not joined yet (%s), retrying in %.0fs", e, REJOIN_DELAY)
if self._stop.wait(REJOIN_DELAY):
return
self._rejoin.set()
continue
@@ -581,11 +658,9 @@ class JoiningModelState:
if self._stop.is_set():
client.close()
return
self._available = True
self._joined = (client, spec)
self._log.warning("jetlink: link ready, waiting for a window to swap")
self._report('connect', READY, 'ready_waiting_window')
self._keep_alive()
self._log.warning("jetlink: link ready, the large model shadows from the next frame")
self._report('connect', CHECKING)
def _wait_out_back_off(self) -> bool:
"""Until the next attempt is due, or a host configures the gadget again
@@ -604,117 +679,21 @@ class JoiningModelState:
host_left = host_left or not attached
return self._stop.is_set()
def _keep_alive(self) -> None:
"""Ping a link that is waiting for a swap window.
On a drive with no stop and no disengage that is the whole drive, and a
host that reboots in there would otherwise be found at the swap: a build
on a dead link, a demote and the backoff, all on modeld's thread.
The client is taken out of _joined for the ping and put back after, so the
frame loop sees a whole one or none, and never waits on the lock.
"""
def _watch_engagement(self) -> None:
background_thread() # off modeld's realtime core; see _join_loop
# made on this thread: the poller's sockets (a SubMaster) belong to the
# thread that made them
engaged = self._engagement()
while not self._stop.is_set():
if self._rejoin.wait(KEEPALIVE_PERIOD):
return
with self._lock:
joined, self._joined = self._joined, None
if joined is None:
return # swapped in on a frame, or closed under us
try:
joined[0].ping(timeout=PING_TIMEOUT)
except Exception as e:
self._available = False
self._log.warning("jetlink: the link died before it could be used (%s), reopening", e)
self._note_link_loss(LOST)
try:
joined[0].close()
except Exception:
self._log.exception('jetlink: closing the dead link')
self._back_off()
return
with self._lock:
stopped = self._stop.is_set()
if not stopped:
self._joined = joined
if stopped:
# close() ran during the ping and found nothing to close
self._say_leaving(joined[0], P.LEAVE_STOPPED)
joined[0].close()
return
def _write_loop(self) -> None:
"""The runtime record (gadget.MODELD), once a second, on a hand-back and on
the first answer after a swap: tmpfs IO, the link record and a catalog
read for the name, so never on the frame thread, and off modeld's realtime
core first (_join_loop). The last one, after close(), is stopping.
status.runtime reads pid, at, stopping, state, active and last_output_at;
handovers, held, sha256, name, checkpoint, transport, drops and lags have
no reader in jetlink yet, they are for forks (StarPilot's sha and
checkpoint, a drive's counts)."""
background_thread()
pid, names, kinds = os.getpid(), {}, {}
failing = False
while True:
# before the record is built: a wake from here on is for the next one
self._record.clear()
stopping = self._stop.is_set()
try:
gadget.write_record(self._record_path, self._runtime(pid, names, kinds, stopping))
failing = False
except Exception:
if not failing: # once, not every second
self._log.exception("jetlink: could not write the runtime record")
failing = True
if stopping:
return
self._record.wait(gadget.RECORD_PERIOD)
def _runtime(self, pid: int, names: dict, kinds: dict, stopping: bool) -> dict:
big = self._big
state = self.big_model_state
swapped_at = self._joined_at # after the state: a swap sets it before _active
if state == 'running' and self._output_at < swapped_at:
# swapped in and not answered yet: the swap is still landing, not a
# model that stopped answering
state = 'joining'
spec = getattr(big, 'spec', None)
sha = getattr(spec, 'sha256', None)
if big is not None and swapped_at not in kinds:
# once per swap: the gadget the link runs on does not change under it
kinds.clear()
kinds[swapped_at] = gadget.link_kind()
return {'pid': pid, 'at': time.monotonic(), 'stopping': stopping, 'state': state,
'handovers': self._handovers, 'active': big is not None and self._active is big,
'last_output_at': self._output_at, 'held': bool(getattr(big, 'held', False)),
'sha256': sha, 'name': self._name_of(sha, names) if sha else None,
'checkpoint': getattr(spec, 'checkpoint', None),
'transport': kinds.get(swapped_at) if big is not None else None, 'drops': self._drops,
'lags': self._lags}
def _name_of(self, sha: str, names: dict) -> str:
"""The catalog's name, kept once it is one. The sha's stand-in (a row not
resolved yet, or a lookup that failed) is asked again after NAME_RETRY."""
name, retry_at = names.get(sha, (None, 0.0))
if time.monotonic() >= retry_at:
try:
name = self._name(sha) if self._name is not None else None
except Exception:
name = None
real = bool(name) and name != sha[:16]
name = name if real else sha[:16]
names[sha] = (name, float('inf') if real else time.monotonic() + NAME_RETRY)
return name
# rechecked on every poll, including ones with no news: that is when
# alive turns false. The frame thread expires the answer too
self._engaged = engaged(ENGAGEMENT_POLL_MS)
self._engagement_updated = time.monotonic()
def close(self) -> None:
first = not self._stop.is_set() # the leave is said once; cleanups close twice
self._stop.set()
self._available = False
self._rejoin.set()
# after the stop: the writer's next record is its last, and says stopping
self._record.set()
self._writer.join(1.0)
self._progress.clear()
with self._lock:
joined, self._joined = self._joined, None
@@ -739,16 +718,19 @@ def join(parts, cam_w: int, cam_h: int, small) -> JoiningModelState:
what the link will hand back; another geometry is rejected.
"""
from jetlink.openpilot import link as links
# here rather than in build(): the import then costs modeld's main thread
# before the frame loop, not the frame the swap lands on
from jetlink.openpilot.model_state import JetlinkModelState
op = parts.op
face = op.model_face()
ready: dict = {}
link = links.Link(parts.log)
def prepare():
from jetlink.openpilot.warp import Warp, prepare_reset
from jetlink.openpilot.warp import prepare_reset, warm
# the gadget first, so the Jetson enumerates while the warp loads. Left to
# the join thread the bind lands ~3 s later, behind the small model's
# first frame, and can stall a frame by hundreds of ms
# the join thread the bind landed ~3 s later, behind the small model's
# first frame, and one ignition had a 655 ms frame during the bind
links.present_early(link, background_thread)
cached = parts.spec.load()
if cached is not None:
@@ -757,9 +739,9 @@ def join(parts, cam_w: int, cam_h: int, small) -> JoiningModelState:
else:
geometry = parts.warps.geometry()[2:]
try:
# the fork's own reset of its small model, else jetlink's from its shape
ready['reset_small'] = getattr(op, 'small_reset', lambda small: None)(small) or prepare_reset(small)
warp = Warp(parts.warps.load(cam_w, cam_h, *geometry), face.frame_size(cam_w, cam_h))
ready['reset_small'] = prepare_reset(small)
warp = parts.warps.load(cam_w, cam_h, *geometry)
warm(warp, face.frame_size(cam_w, cam_h))
except Exception:
link.close()
raise
@@ -770,11 +752,10 @@ def join(parts, cam_w: int, cam_h: int, small) -> JoiningModelState:
warp = ready.get('warp') if ready.get('geometry') == (img_w * 2, img_h * 2) else None
if warp is None:
raise RuntimeError('no prepared warp for the server model geometry')
return JetlinkModelState(client, spec, warp, face=face, log=parts.log, event=op.event)
return JetlinkModelState(cam_w, cam_h, client, spec, warp, face=face, log=parts.log, event=op.event)
def connect(should_stop=None):
return links.open_link(parts, link, should_stop)
return JoiningModelState(small, connect, build, prepare, reset_small=lambda: ready['reset_small'](),
progress=parts.progress, log=parts.log, in_control=op.in_control,
name=parts.models.name_for)
progress=parts.progress, engagement=op.engagement, log=parts.log)
+20 -81
View File
@@ -31,11 +31,9 @@ CONNECT_TIMEOUT = 45.0
CONNECT_DELAY = 0.5
# four frame periods, and four times the slowest frame seen. A host that goes
# silent with no USB edge (a hung server, a phone's cable) costs one frame this
# long plus the small model's, inside modelV2's 0.5 s alive limit; at 0.5 s
# here commIssue flashes on top of the fallback
# long plus the small model's, inside modelV2's 0.5 s alive limit; 0.5 s here
# flashed commIssue on top of the fallback
INFERENCE_TIMEOUT = 0.2
# between joins while neither end has the picked model and nothing else is built
MODEL_WAIT = 60.0
# how long the load may wait for the early gadget bind; a provisioning run may
# still be letting go of the endpoints
PRESENT_TIMEOUT = 5.0
@@ -45,16 +43,6 @@ PRESENT_TIMEOUT = 5.0
BUILD_TIMEOUT = 1800.0
class ModelMissing(RuntimeError):
"""Neither end has the picked model and nothing else is built: only a
provisioning run's download changes that, so the join asks again after
`retry_after` rather than every few seconds, and the panel says `waiting`
(joining._join_loop)."""
retry_after = MODEL_WAIT
waiting = 'big model not downloaded yet'
code = 'model_missing' # waiting's progress code (status.PROGRESS_CODES)
def _left(deadline: float | None, default: float) -> float:
"""What is left of an absolute `deadline`, or `default` without one."""
return default if deadline is None else max(0.0, deadline - time.monotonic())
@@ -105,7 +93,7 @@ class Link:
def open(self, deadline: float | None = None):
"""The client, opening one if we have not got one yet, or if the one held
is dead: a big model retired after a link loss closes its client, and
reusing it fails the next attempt with EBADF."""
reusing it failed the next attempt with EBADF, 5 s after every loss."""
if self.client is not None and self.client.dead:
self.close()
if self.client is None:
@@ -230,7 +218,7 @@ def identity(parts, entry: dict) -> tuple[str, int]:
sha256, nbytes = entry.get('oid'), entry.get('size')
if sha256 and nbytes:
return sha256, int(nbytes)
parts.progress.report('connect', 0.0, 'finding model', code='finding_model')
parts.progress.report('connect', 0.0, 'finding model')
return parts.models.resolve_pointer(entry['ref'])
@@ -241,16 +229,6 @@ def identity(parts, entry: dict) -> tuple[str, int]:
_hashed: dict[tuple[str, int, int], str] = {}
def _hash_key(path: Path, st) -> tuple[str, int, int]:
return str(path), st.st_size, st.st_mtime_ns
def remember_hash(path: Path, sha256: str) -> None:
"""A file whose hash is already proven, as a download's is on the way in:
its upload need not read the gigabyte again (verified_upload)."""
_hashed[_hash_key(path, path.stat())] = sha256
def verified_upload(log, model_path: Path | None, sha256: str, nbytes: int) -> Path | None:
"""The file to upload, once its hash is proven to match the registry: under
a sha the bytes do not have, the Jetson's plan would lie about its contents."""
@@ -262,7 +240,7 @@ def verified_upload(log, model_path: Path | None, sha256: str, nbytes: int) -> P
log.error("jetlink: %s is %d bytes, the registry says %d; not uploading it",
model_path.name, st.st_size, nbytes)
return None
key = _hash_key(model_path, st)
key = (str(model_path), st.st_size, st.st_mtime_ns)
have = _hashed.get(key)
if have is None:
from jetlink.spec import sha256_file
@@ -297,41 +275,13 @@ def ensure(parts, client, sha256: str, nbytes: int, model_path: Path | None, *,
return spec
def stand_in(parts, hello: dict, pick: str, built) -> tuple[str, int] | None:
"""The model to ask for in place of the pick, as (sha256, nbytes), or None
for the pick itself. `built` is the spec record's ready spec, or None.
The server's hello says what it has: the model it has loaded and the ones it
has built. The pick wins whenever the server has it. Otherwise the model it
has loaded drives, so a phone set to its own model serves that rather than
whatever the comma last built on some other server; then the comma's own
record of the last model built, if this server has it too. A server that lists nothing has only the record
to go on."""
cached = hello.get('cached_models')
have = None
if isinstance(cached, list):
loaded = hello.get('loaded')
have = set(cached) | ({loaded} if loaded else set())
if pick in have:
return None
if loaded:
# its size names it to the server, and only a catalog model resolved
# here, or the record, has one
nbytes = built.nbytes if built is not None and built.sha256 == loaded else parts.models.size_for(loaded)
if nbytes:
return loaded, nbytes
if built is not None and built.sha256 != pick and (have is None or built.sha256 in have):
return built.sha256, built.nbytes
return None
def open_link(parts, link: Link, should_stop=None):
"""Get a client and a spec. Link IO only, so it is safe off modeld's thread;
everything that touches tinygrad stays in the joining state's build.
The picked model is built here if the Jetson has not got it, minutes with
the small model driving: a provisioning run works offroad only, so a model
picked in the driveway would otherwise cost the whole drive. Only reached with
picked in the driveway cost the whole drive otherwise. Only reached with
the small model driving (the join loop stops asking once joined), so a build
here never unloads an engine that is steering. Whatever this attempt cannot
use stays on `link`, still open, for the next one.
@@ -352,34 +302,23 @@ def open_link(parts, link: Link, should_stop=None):
hello.get('engine_state'), str(hello.get('loaded'))[:16])
link.note_server(hello)
sha256, nbytes = identity(parts, selected)
path = parts.models.shipped_model_path()
built = parts.spec.ready_spec()
standin = stand_in(parts, hello, sha256, built)
if standin is not None:
# by design, a model already built drives until the
# pick is fetched and built, which a provisioning run does parked, rather
# than the small model. Building here would unload it, and uploading
# would share the link with its frames
parts.log.warning("jetlink: %s is not ready yet, %s drives until it is", selected.get('name', sha256[:16]),
parts.models.name_for(standin[0]))
(sha256, nbytes), path = standin, None
elif built is None or built.sha256 != sha256:
if not parts.spec.engine_ready_for(sha256):
parts.log.warning("jetlink: %s is not built yet, building it with the small model driving",
selected.get('name', sha256[:16]))
# normally one round trip, since the provisioning run left the engine loaded. A
# server that restarted reloads from the plan cache, 13 to 25 s; one
# that has never seen this model builds it, 102 to 294 s
spec = ensure(parts, client, sha256, nbytes, path, progress=parts.progress.report_with_eta,
should_stop=should_stop)
selected.get('name', sha256[:16]))
try:
# normally one round trip, since the provisioning run left the engine loaded. A
# server that restarted reloads from the plan cache, 13 to 25 s; one
# that has never seen this model builds it, 102 to 294 s
spec = ensure(parts, client, sha256, nbytes, parts.models.shipped_model_path(),
progress=parts.progress.report_with_eta, should_stop=should_stop)
except EngineMissing:
# neither end has the bytes. Fetching them needs the internet and a
# gigabyte of it, which is a parked job; clear the record so the next
# parked period provisions again
parts.spec.clear_ready()
raise
client.deadline = INFERENCE_TIMEOUT
return client, spec
except EngineMissing as e:
# neither end has the bytes. Fetching them is a provisioning run's job;
# clear the record if it named this model, so the next one does. The link
# is fine and stays on `link`: closing it leaves a read queued that the
# next attempt's open has to bounce the gadget for
parts.spec.clear_ready(sha256)
raise ModelMissing(str(e)) from e
except BaseException:
link.close()
raise
+147 -113
View File
@@ -13,8 +13,8 @@ frames and context in, 18452 floats out, and holds no control state.
The split is at `run_policy`. The warp stays on the comma's GPU: its input is a
2 MB camera buffer already there and its output the 393 KB the link carries
anyway. The history queues live on the Jetson, shipping them would cost ~10 MB
a frame instead of ~0.5 MB. modeld runs its warp as a JIT of its own, and
the link runs the same one; see warp.
a frame instead of ~0.5 MB. Upstream fused warp and policy into one JIT, so
the fork's build compiles a standalone warp; see warp.
A model that keeps its own history (openpilot #38916, Cinque Terre V3 on)
takes the same warped frame and the same scalars; its hidden state never
@@ -22,7 +22,8 @@ leaves the Jetson, so there is no prev_feat to send back. The spec says which.
What openpilot's modeld reads off a ModelState comes from the fork's adapter
(ModelFace): comma's constants, parser, smoothing and action function.
tinygrad is warp.Warp's to import: only modeld has it.
tinygrad is imported when a model is built, never at module level: only
modeld and the build have it.
"""
from __future__ import annotations
@@ -34,7 +35,7 @@ from dataclasses import dataclass
import numpy as np
from jetlink.protocol import NotFinite
from jetlink.openpilot.warp import call_warp
SEND_RAW_PRED = os.getenv('SEND_RAW_PRED')
SLOW_FRAME = 0.05 # the full 20 Hz budget, not just the largest outliers
@@ -43,7 +44,7 @@ SLOW_FRAME = 0.05 # the full 20 Hz budget, not just the largest outliers
# would be dropped: modeld's own work around run() takes the rest of the 50 ms.
# A held frame is a plan one frame old, which the 10 s plan and the actuator
# delay both dwarf, where a dropped frame is the same stale plan plus a count
# toward selfdrived's modeldLagging. None waits the client's full deadline
# toward selfdrived's modeldLagging. None waits the client's deadline, as before
HOLD_FRAME = 0.046
# Held frames the large model may make before it is behind rather than
# tailing: this many in a row, or more than HOLDS_ALLOWED within HOLD_WINDOW.
@@ -56,16 +57,12 @@ HOLD_FRAME = 0.046
HOLDS_IN_A_ROW = 5
HOLDS_ALLOWED = 20
HOLD_WINDOW = 10.0
# The first frames after a swap, which the fork holds engagement off for (a
# second at 20 Hz): a single held frame among them is behind. A host cold from
# a reconnect proves itself here, before anyone can engage, rather than at the
# wheel, where a slow host's return is a soft disable within a second
PROVING_FRAMES = 20
# the first of them never hand back, held or not: the first after a join
# carries the history reset, and a Mac's is ~100 ms of CoreML warm-up. A
# single slow frame later is the comma's own stall, a warp or a send, which
# handing back would not fix; only held frames say the host is behind
SETTLING_FRAMES = 3
# How long a host may answer none of its shadow frames before the link is
# abandoned, where a driving one gets the client's deadline
# (link.INFERENCE_TIMEOUT, 0.2 s). The small model drives meanwhile, and a
# phone's first frames after a join are Neural Engine warm-up: at 0.2 s an
# iPhone 18 Pro lost 11 of 18 joins inside 0.33 s (2026-10-04)
SHADOW_SILENCE = 1.0
# frames between asks for the server's telemetry, which rides on the response:
# every second one, as modeld sent a chestnut's state (20 Hz over 10 Hz)
TELEMETRY_EVERY = 2
@@ -73,6 +70,12 @@ TELEMETRY_EVERY = 2
TELEMETRY_PERIOD = 1.0
# frames Trips keeps the timings of: a minute at 20 Hz
TRIPS_KEPT = 1200
# shadow frames in a row that would not have been held (check_shadow) before
# the large model may drive (joining reads `keeping_up`): a second at 20 Hz.
# A host cold from a reconnect proves itself with the small model driving,
# not at the wheel, where an iPhone's every return was a soft disable
# within a second (2026-10-04)
PROVING_FRAMES = 20
class Trips:
@@ -83,6 +86,7 @@ class Trips:
budget it cannot measure. Bounded to the last TRIPS_KEPT frames."""
def __init__(self):
self.sent = 0 # frames sent; the ones never ended ran with the small model driving
self.frames = 0 # frames waited for and published
self.over = 0 # frames past SLOW_FRAME, the whole 20 Hz budget
self.held = 0 # frames whose reply was late and published the previous one
@@ -98,9 +102,9 @@ class Trips:
self._server_ms.append(server_us / 1e3)
def summary(self) -> dict:
"""frames, over, held, span_s, and over the frames kept: p50_ms, p99_ms,
max_ms of the whole frame and server_ms, the server's mean total."""
out = {'frames': self.frames, 'over': self.over, 'held': self.held,
"""frames, over, held, shadowed, span_s, and over the frames kept: p50_ms,
p99_ms, max_ms of the whole frame and server_ms, the server's mean total."""
out = {'frames': self.frames, 'over': self.over, 'held': self.held, 'shadowed': self.sent - self.frames,
'span_s': round(time.monotonic() - self.started, 1)}
if self._whole_ms:
whole = sorted(self._whole_ms)
@@ -111,12 +115,12 @@ class Trips:
@dataclass
class Frame:
"""One frame on its way to the host: when its warp started, was launched
and finished, then its seq and send time once sent. The bytes are the
warp's output (Warp.output)."""
"""One frame on its way to the host: when the warp and the readback
finished, the bytes the wire takes, then its seq and send time once sent."""
t0: float
t1: float
t2: float
data: object
seq: int | None = None
sent: float = 0.0
telemetry: bool = False
@@ -130,14 +134,18 @@ class JetlinkModelState:
action function off the ModelState; stock modeld has them as module
constants. This is comma's large model, so they are comma's.
A frame is prepare(), send() and end(); run() is all three, as modeld calls
it. The joining model (joining.JoiningModelState) reads `behind` after each
frame.
A frame is prepare(), send() and end(). run() is all three, as modeld calls
it. The joining model (joining.JoiningModelState) prepares and sends every
frame the small model drives and never ends them, so the host's model is
warm and current at the swap; it reads `behind` after each driven frame.
"""
prev_desire: np.ndarray # for tracking the rising edge of the pulse
def __init__(self, client, spec, warp, *, face, log, event):
def __init__(self, cam_w: int, cam_h: int, client, spec, warp, *, face, log, event):
from tinygrad.device import Device
from tinygrad.tensor import Tensor
self._tensor = Tensor
self._log = log
# (name, **fields): a structured log line, cloudlog.event on a comma
self._event = event
@@ -153,13 +161,19 @@ class JetlinkModelState:
self.client = client
self.spec = spec
# Over a phone's cable, hand the socket the warp's GPU mapping itself: the
# kernel copies it while the first segments are already on the wire, where
# copying it here first held the send back 2.5 ms. On the Mac stand-in the
# comma's side of a frame was 0.6 ms faster at p50 and 2 ms at p99. USB
# keeps the host copy it was measured with. The client's transport says
# which, as the owner lent it
self.send_from_gpu = client.t.link_info().get('kind') == 'cable'
# not chestnut hardware, but the same role: modelV2.big, the UI and the
# model manager key off this flag
self.chestnut = True
# a warp.Warp, loaded and warmed ahead: its first call links the JIT, and
# this runs on modeld's frame thread. Frames go out of its output as they
# are, over either transport
# a warm warp, loaded ahead: the first call costs ~2 s and this can run on
# modeld's frame thread (warp.warm)
self.warp = warp
self.input_shapes = spec.input_shapes
@@ -168,16 +182,26 @@ class JetlinkModelState:
# from the same field
self.frame_skip = spec.frame_skip
# compile_modeld.make_input_queues' packed_npy_inputs, minus the warp's
# inputs and the GPU queues the server owns; for a stateful graph, minus
# prev_feat too
self.packed = np.zeros(spec.packed_nelem, dtype=np.float32)
self.npy = {k: self.packed[at].reshape(shape) for k, (at, shape) in spec.packed_layout.items()}
# the warp's own two inputs stay separate NPY tensors; everything past the
# warp is the server's. Writing the numpy array is what feeds the JIT
self.npy = {'tfm': np.zeros((3, 3), dtype=np.float32),
'big_tfm': np.zeros((3, 3), dtype=np.float32)}
self.warp_inputs = {k: Tensor(v, device='NPY').realize() for k, v in self.npy.items()}
# compile_modeld.make_input_queues' packed_npy_inputs, minus the GPU queues
# the server owns; for a stateful graph, minus prev_feat too
self.packed = np.zeros(spec.packed_nelem, dtype=np.float32)
self.npy.update({k: self.packed[at].reshape(shape) for k, (at, shape) in spec.packed_layout.items()})
# read once: it must be the device the cached JIT was compiled against
self.warp_dev = Device.DEFAULT
self.prev_desire = np.zeros(face.desire_len, dtype=np.float32)
self.parser = face.parser()
self.frame_size = face.frame_size(cam_w, cam_h)
# the camera buffers modeld hands over, whatever the graph calls its inputs
self.vision_input_names = ['img', 'big_img']
self.full_frames: dict = {}
self._blob_cache: dict = {}
self._need_reset = True
self._frame_id = 0
self._last_logged = 0.0
@@ -194,8 +218,8 @@ class JetlinkModelState:
self._holds: deque[float] = deque()
self._holds_in_a_row = 0
self.behind: str | None = None
# whether the last frame published the one before it: the runtime record's
self.held = False
# shadow frames in a row that would not have been held
self._kept_up = 0
def slice_outputs(self, model_outputs: np.ndarray, output_slices: dict[str, slice]) -> dict[str, np.ndarray]:
return {k: model_outputs[np.newaxis, v] for k, v in output_slices.items()}
@@ -212,22 +236,28 @@ class JetlinkModelState:
return
self._last_logged = now
self._event("jetlinkTelemetry", dead=bool(self.client.dead), **telemetry)
send = getattr(self.client.t, 'last_send', None)
if send:
self._event("jetlinkSend", nonce=self.client.nonce, frame=self._frame_id, totals=self.client.t.send_totals.copy(), **send)
def prepare(self, bufs: dict, transforms: dict[str, np.ndarray], inputs: dict[str, np.ndarray]) -> None:
"""Take this frame up: start the warp, and while the GPU runs it, read
what the host has answered so far (a held frame's late reply) and pack
the rest of the inputs. send() puts it on the link and end() waits for
its answer."""
t0 = time.perf_counter()
self.warp.start(_address(bufs['img']), _address(bufs['big_img']), transforms['img'], transforms['big_img'])
t1 = time.perf_counter()
# the GPU is warping; nothing below is an input to it
"""Take this frame up: read what the host has answered so far, then warp
the frame for it. send() puts it on the link and end() waits for its
answer. A frame sent and never ended ran on the host with the small model
driving (a shadow frame); its timings reach the log here, at the next."""
frame = self._frame
if frame is not None and frame.seq is not None and (self._frame_id <= 3 or frame.sent - frame.t0 > SLOW_FRAME / 2):
# the frame before was sent and never ended: a shadow frame
self._log.warning("jetlink: shadow frame %d warp %.1f data %.1f send %.1f ms; %d unanswered", self._frame_id,
(frame.t1 - frame.t0) * 1e3, (frame.t2 - frame.t1) * 1e3, (frame.sent - frame.t2) * 1e3,
self.client.unanswered)
self.client.drain()
self.log_telemetry()
for key in bufs.keys():
ptr = np.frombuffer(bufs[key].data, dtype=np.uint8).ctypes.data
cache_key = (key, ptr)
if cache_key not in self._blob_cache:
self._blob_cache[cache_key] = self._tensor.from_blob(ptr, (self.frame_size,), dtype='uint8', device=self.warp_dev)
self.full_frames[key] = self._blob_cache[cache_key]
# Model decides when action is completed, so desire input is just a pulse triggered on rising edge.
# Under whichever name the loop keyed it: stock modeld's desire_pulse, or a modeld_v2 bundle's own
desire = inputs[next(k for k in inputs if k.startswith('desire'))]
@@ -236,105 +266,124 @@ class JetlinkModelState:
self.prev_desire[:] = desire
self.npy['traffic_convention'][:] = inputs['traffic_convention']
self.npy['action_t'][:] = inputs['action_t']
self.npy['tfm'][:, :] = transforms['img'][:, :]
self.npy['big_tfm'][:, :] = transforms['big_img'][:, :]
# free again once infer_begin returns: the gadget's io_submit and the
# cable's sendmsg have copied it by then, before the next warp writes it
self.warp.wait()
self._frame = Frame(t0, t1, time.perf_counter())
t0 = time.perf_counter()
warped = call_warp(self.warp, **self.warp_inputs, frame=self.full_frames['img'], big_frame=self.full_frames['big_img'])
t1 = time.perf_counter()
# .data() rather than .numpy(): same ~2.5 ms mean (a write-combined GPU
# mapping read), but no per-frame allocation and no 52 ms outlier. The
# mapping is safe to send: infer_begin returns only once the socket has
# copied all of it, before the next warp can write it
if self.send_from_gpu:
data = warped._buffer().as_memoryview(allow_zero_copy=True)
else:
data = warped.data()
self._frame = Frame(t0, t1, time.perf_counter(), data)
def send(self, want_telemetry: bool = False) -> None:
def send(self, want_telemetry: bool = False, silence: float | None = SHADOW_SILENCE) -> None:
"""The prepared frame to the host. It asks for the server's telemetry
when the log is due for it, or when told to: modeld asks on the frames it
would have sent a chestnut's state on.
With an output to hold, a frame the link cannot take without waiting for
the host to drain earlier ones is not sent, and end() holds it: the same
plan one frame old a late reply publishes, where waiting stalls the frame
loop on a host that is behind."""
would have sent a chestnut's state on. Sent from outside, it is a shadow
frame and a quiet host has `silence`; run() sends with the client's
deadline."""
frame = self._frame
self._frame_id += 1
frame.telemetry = want_telemetry or time.monotonic() - self._last_logged >= TELEMETRY_PERIOD
try:
frame.seq = self.client.infer_begin(self.warp.output, self.packed, self._frame_id, reset=self._need_reset,
want_state=frame.telemetry, skip_if_busy=self._can_hold)
except Exception:
self._log.warning("jetlink: frame %d send failed: %s", self._frame_id, getattr(self.client.t, 'last_send', {}))
raise
frame.seq = self.client.infer_begin(frame.data, self.packed, self._frame_id, reset=self._need_reset,
want_state=frame.telemetry, silence=silence)
frame.sent = time.perf_counter()
if frame.seq is not None:
self._need_reset = False
frame.data = None
self._need_reset = False
self.trips.sent += 1
@property
def sent(self) -> bool:
"""Has the prepared frame gone out?"""
return self._frame is not None and self._frame.seq is not None
def end(self, after_enqueue: Callable[[], None] | None = None) -> dict[str, np.ndarray]:
"""The sent frame's output, as modeld blocks on a chestnut's, but not past
HOLD_FRAME once there is an output to publish again: a frame that long
is a dropped camera frame. Only a stall past the client's deadline, or an
output that is not finite, raises, into the joining state's demotion to
the small model."""
is a dropped camera frame. Only a stall past the client's deadline raises,
into the joining state's demotion to the small model."""
frame, self._frame = self._frame, None
# publish health while the Jetson works
if after_enqueue is not None:
after_enqueue()
waiting_from = time.perf_counter()
if frame.seq is None:
model_output = None # not sent (see send): held
else:
try:
model_output = self.client.infer_end(frame.seq, hold=self._hold_left(frame) if self._can_hold else None)
except NotFinite:
# answered, so the leave's summary counts it
self.trips.record(time.perf_counter() - frame.t0, self.client.last_timings[2])
raise
hold = None
if HOLD_FRAME and self.client.last_output is not None:
hold = self._hold_left(frame)
model_output = self.client.infer_end(frame.seq, hold=hold)
t4 = time.perf_counter()
held = self.held = model_output is None
held = model_output is None
self.behind = self._note_hold(held)
self.trips.record(t4 - frame.t0, self.client.last_timings[2])
outputs = self._outputs(self.client.last_output if held else model_output)
self.trips.record(t4 - frame.t0, self.client.last_timings[2])
# a frame past the budget is a dropped camera frame; send against reply
# says which end it was. Held frames are a steady few on a marginal host,
# so after the first they go to the log at TELEMETRY_PERIOD
log_hold = held and (self.trips.held <= 3 or t4 - self._last_hold_logged >= TELEMETRY_PERIOD)
if self._frame_id <= 3 or t4 - frame.t0 > SLOW_FRAME or log_hold:
note = ''
if held:
self._last_hold_logged = t4
note = f', held ({self.trips.held} so far{", not sent" if frame.seq is None else ""})'
# persisted on the comma so a drive can separate server execution from
# receive stalls once the Jetson is offline; server total excludes USB
gpu_us, queue_us, total_us = self.client.last_timings
receive = getattr(self.client.t, 'last_receive', {})
# over the cable: how frames go, how many never came back, and what the
# kernel dropped on the way in (softnet), since the link came up
drops = getattr(self.client.t, 'net_drops', lambda: None)()
cable = (f"; frames {'as datagrams' if self.client.t.datagrams else 'on the stream'}, "
f"{self.client.frames_lost} lost, net drops {drops}" if drops is not None else '')
self._log.warning("jetlink: frame %d warp %.1f data %.1f send %.1f wait %.1f ms%s; "
"server gpu %.1f queue %.1f total %.1f ms; ffs maxima prepare %.1f read_wait %.1f handoff %.1f ms%s",
"server gpu %.1f queue %.1f total %.1f ms; ffs maxima prepare %.1f read_wait %.1f handoff %.1f ms",
self._frame_id, (frame.t1 - frame.t0) * 1e3, (frame.t2 - frame.t1) * 1e3,
(frame.sent - frame.t2) * 1e3, (t4 - waiting_from) * 1e3,
note,
f', held ({self.trips.held} so far)' if held else '',
gpu_us / 1e3, queue_us / 1e3, total_us / 1e3, receive.get('prepare', 0.0) * 1e3,
receive.get('read_wait', 0.0) * 1e3, receive.get('handoff', 0.0) * 1e3, cable)
receive.get('read_wait', 0.0) * 1e3, receive.get('handoff', 0.0) * 1e3)
return outputs
def run(self, bufs: dict, transforms: dict[str, np.ndarray],
inputs: dict[str, np.ndarray], after_enqueue: Callable[[], None] | None = None) -> dict[str, np.ndarray]:
self.prepare(bufs, transforms, inputs)
self.send(after_enqueue is not None)
self.send(after_enqueue is not None, silence=None)
return self.end(after_enqueue)
@property
def _can_hold(self) -> bool:
"""Is there an output to publish again instead of waiting (HOLD_FRAME)?"""
return bool(HOLD_FRAME) and self.client.last_output is not None
def keeping_up(self) -> bool:
"""Would the host have driven the last PROVING_FRAMES shadow frames
without a hold? What the joining model swaps on; nothing while it drives."""
return self._kept_up >= PROVING_FRAMES
def check_shadow(self) -> None:
"""Score the shadow frame just sent as end() would have, now that the
small model's own frame is done: kept up when its reply is back HOLD_FRAME
or less after its warp began, waited for until then as end() waits. The
joining model calls this once per shadow frame; a reply not back by then
is read by the next prepare(), and starts the proof over. The shadow
frame went out after the small model's work was queued, later than a
driven one, so a host that passes here does not hold at the wheel."""
frame = self._frame
if frame is None or frame.seq is None:
return
self.client.drain()
left = self._hold_left(frame)
if not self.client.unanswered:
kept = left >= 0
elif self.client.unanswered == 1 and left > 0:
kept = self.client.infer_end(frame.seq, hold=left) is not None
else:
# past the hold, or behind older frames: replies come in order, so
# this one cannot make it, and waiting would only hold the small model
kept = False
self._kept_up = self._kept_up + 1 if kept else 0
@staticmethod
def _hold_left(frame) -> float:
"""What is left of the hold for `frame`, from its warp's start."""
return HOLD_FRAME - (time.perf_counter() - frame.t0)
return (HOLD_FRAME or SLOW_FRAME) - (time.perf_counter() - frame.t0)
def _note_hold(self, held: bool) -> str | None:
"""Why the large model should hand back after this frame, if it should:
any held frame among its first PROVING_FRAMES past SETTLING_FRAMES,
HOLDS_IN_A_ROW held frames running, or more than HOLDS_ALLOWED in
HOLD_WINDOW. None while it is keeping up."""
if not held:
@@ -346,10 +395,6 @@ class JetlinkModelState:
while now - self._holds[0] > HOLD_WINDOW:
self._holds.popleft()
self._holds_in_a_row += 1
if self._frame_id <= SETTLING_FRAMES:
return None
if self._frame_id <= PROVING_FRAMES:
return f'held frame {self._frame_id} of the first {PROVING_FRAMES}'
if self._holds_in_a_row >= HOLDS_IN_A_ROW:
return f'held {self._holds_in_a_row} frames in a row'
if len(self._holds) > HOLDS_ALLOWED:
@@ -361,17 +406,10 @@ class JetlinkModelState:
gives the same dict again: the parser's softmax works in place on it."""
if self._parsed is not None and self._parsed[0] is model_output:
return self._parsed[1]
# the server checks first (Status.NOT_FINITE, which the client raises as
# this). This is for one that did not, or bytes the link changed: a raw
# inf is a corrupt output even where the parser would clip it finite. A
# sum allocates nothing, and any NaN or inf makes it one; these outputs
# are far from overflow
if not np.isfinite(model_output.sum()):
raise NotFinite(f"frame {self._frame_id}: the output is not finite")
# the non-finite check runs on the server (Status.NOT_FINITE -> LinkError),
# so modeld's big->small failover fires as it does for a chestnut
outputs_dict = self.parser.parse_outputs(self.slice_outputs(model_output, self.output_slices))
# a finite output can still parse to inf, by a parser that does not clip
if not np.isfinite(sum(float(value.sum()) for value in outputs_dict.values())):
raise NotFinite(f"frame {self._frame_id}: the parsed output is not finite")
self.spec.feed_back(self.packed, model_output)
if SEND_RAW_PRED:
outputs_dict['raw_pred'] = model_output.copy()
self._parsed = (model_output, outputs_dict)
@@ -381,7 +419,3 @@ class JetlinkModelState:
"""Let go of the link; the joining state calls this on a model it retires."""
self.client.close()
def _address(buf) -> int:
"""Where a camera buffer modeld hands over (a VisionBuf) keeps its bytes."""
return np.frombuffer(buf.data, dtype=np.uint8).ctypes.data
+14 -85
View File
@@ -34,11 +34,6 @@ _TRAILING_DATE = re.compile(r' \([A-Za-z]+ \d{1,2}, \d{4}\)$')
# the index and the slot are JSON params, and the UI names the active model
# every frame; the status line can lag a new pick by this long
INDEX_TTL = 2.0
# A big model's download over a car's connection: the waits between the
# attempts of one provisioning run, one more attempt than waits. Each carries
# on from the .part the last left, so a stall costs the wait, not the bytes
# (lfs.lfs_download)
DOWNLOAD_RETRY_DELAYS = (5.0, 15.0, 30.0, 60.0, 120.0)
class Models:
@@ -82,14 +77,11 @@ class Models:
return known[ref]['oid'], int(known[ref]['size'])
oid, size = self.fetch_pointer(ref)
known[ref] = {'oid': oid, 'size': size}
self._keep_pointers(known)
self.op.log.warning("jetlink: %s is %s, %d MB", ref[:10], oid[:16], size >> 20)
return oid, size
def _keep_pointers(self, known: dict[str, dict]) -> None:
# blocking: the next lookup reads this back, and a put still in flight would be lost under it
self.op.put(self.op.keys.pointers, known, block=True)
self._index_cache = None
self.op.log.warning("jetlink: %s is %s, %d MB", ref[:10], oid[:16], size >> 20)
return oid, size
@staticmethod
def _row(name: str, ref: str, known: dict[str, dict]) -> dict:
@@ -128,7 +120,7 @@ class Models:
"""What runs with no pick: jetlink's default big model, else the newest the
catalog lists. Not the chestnut's default, which is the model in the tree.
A fork with no catalog at all (Keys.catalog None) gets jetlink's default by
its ref; one whose catalog has not been fetched yet waits for it."""
its ref; one whose catalog has not been fetched yet waits for it, as ever."""
if self.op.keys.catalog is None:
return self._row(DEFAULT_BIG_MODEL_NAME, DEFAULT_BIG_MODEL_REF, self._index()[1])
models = self.model_index()
@@ -162,18 +154,17 @@ class Models:
"""One file per model, so switching back does not re-download."""
return f"{model['oid'][:16]}.onnx"
def model_path(self, model: dict) -> Path:
"""Where a model's ONNX lives, whether or not it is there yet."""
return self.model_dir() / self.model_file_name(model)
def shipped_model_path(self) -> Path | None:
"""The chosen large model, if it has been fetched. Keyed on the oid, not
the in-tree pointer, which moves with upstream syncs; the size is the cheap
check that the file is the one we mean."""
model = self.selected_model()
if model is None or not self.has_file(model):
if model is None or not model['oid']:
return None
return self.model_path(model)
path = self.model_dir() / self.model_file_name(model)
if path.is_file() and path.stat().st_size == model['size']:
return path
return None
def lfs_endpoints(self) -> list[str]:
"""Where a model's bytes are asked for, nearest first: the LFS server this
@@ -191,43 +182,19 @@ class Models:
pass
return out + [e for e in LFS_ENDPOINTS if e not in out]
def fetch_shipped_model(self, progress=None, should_stop=None, retrying=None) -> Path | None:
def fetch_shipped_model(self, progress=None, should_stop=None) -> Path | None:
"""Download the chosen large model if it is not here yet; None when nothing
is chosen. The registry streams it to a .part file and hashes it on the way,
so only the whole model ever takes the name.
A transfer that fails is tried again after each of DOWNLOAD_RETRY_DELAYS,
carrying on from the .part it left; once they are spent the .part outlives
the run for the next one. Each attempt asks the LFS server again:
the address it hands out expires. `retrying(error, delay)` hears of each
wait; a run that is stopped stops waiting."""
so only the whole model ever takes the name."""
from jetlink.registry.catalog import NetworkError
from jetlink.registry.lfs import Pointer
from jetlink.registry.lfs import Pointer, lfs_download, lfs_resolve
model = self.selected_model()
if model is None or not model['oid']:
return None
dest = self.model_path(model)
if self.has_file(model):
dest = self.model_dir() / self.model_file_name(model)
if dest.is_file() and dest.stat().st_size == model['size']:
return dest
pointer = Pointer(model['oid'], int(model['size']))
for delay in (*DOWNLOAD_RETRY_DELAYS, None):
try:
return self._fetch_once(pointer, dest, progress, should_stop)
except NetworkError as e:
if delay is None:
raise
self.op.log.warning("jetlink: download of %s failed (%s), trying again in %.0f s", pointer.oid[:16], e, delay)
if retrying is not None:
retrying(e, delay)
end = time.monotonic() + delay
while time.monotonic() < end:
if should_stop is not None and should_stop():
raise
time.sleep(min(1.0, end - time.monotonic()))
def _fetch_once(self, pointer, dest: Path, progress, should_stop) -> Path:
from jetlink.registry.catalog import NetworkError
from jetlink.registry.lfs import lfs_download, lfs_resolve
for endpoint in self.lfs_endpoints():
href = lfs_resolve(endpoint, pointer)
if href is None:
@@ -236,30 +203,6 @@ class Models:
return lfs_download(href, pointer, dest, progress=progress, should_stop=should_stop)
raise NetworkError(f"no LFS server has {pointer.oid[:16]}")
def _row_for(self, oid: str) -> dict | None:
"""The catalog model with this oid, among those we have resolved."""
return next((m for m in self.model_index() if m['oid'] == oid), None)
def name_for(self, oid: str) -> str:
"""The catalog's name for the model with this oid, or the oid's first 16
characters when it lists none we have resolved."""
row = self._row_for(oid)
return row['name'] if row is not None else oid[:16]
def size_for(self, oid: str) -> int | None:
"""The byte count of the catalog model with this oid, once resolved here."""
row = self._row_for(oid)
return row['size'] if row is not None else None
def has_file(self, model: dict) -> bool:
"""Is this model's ONNX on the comma, whole? Its size is the cheap check."""
if not model.get('oid') or not model.get('size'):
return False
try:
return self.model_path(model).stat().st_size == model['size']
except OSError:
return False
# -- the model manager's catalog --------------------------------------------
def big_catalog(self, catalog: dict) -> dict:
@@ -269,10 +212,7 @@ class Models:
Never raises: a probe that fails keeps what the last one found."""
try:
from jetlink.registry.catalog import fetch_catalogs, merge_catalogs
fetched = fetch_catalogs()
self.keep_pins(fetched.get('pointers', {}))
merged = merge_catalogs([catalog, fetched], selector=self.op.catalog_selector)
merged.pop('pointers', None) # the pins are kept above, never in the model manager's catalog
merged = merge_catalogs([catalog, fetch_catalogs()], selector=self.op.catalog_selector)
added = len(merged.get('bundles', [])) - len(catalog.get('bundles', []))
if added <= 0:
return catalog
@@ -282,17 +222,6 @@ class Models:
self.op.log.exception("jetlink: could not check for newer catalogs")
return self._found_before(catalog)
def keep_pins(self, pins: dict[str, dict]) -> None:
"""Writes the list's pins over the kept pointers (see catalog.pinned_pointers)."""
known = self.pointers()
changed = {ref: pin for ref, pin in pins.items() if known.get(ref) != pin}
if not changed:
return
known.update(changed)
self._keep_pointers(known)
for ref, pin in changed.items():
self.op.log.warning("jetlink: %s is pinned to %s", ref[:10], pin['oid'][:16])
def _found_before(self, catalog: dict) -> dict:
"""The catalog with the models the last probe found folded in again, from the
model manager's cached copy: every entry there the fetched catalog does not
+17 -62
View File
@@ -84,66 +84,30 @@ class ProvisioningRun:
# -- provisioning ---------------------------------------------------------
def fetch_model(self):
"""Download the picked large model, if the comma has not got it whole.
"""Download the pinned large model, once.
Minutes on a slow link, so it reports progress and stops when the owner
stops the run; it is the one call in a run that blocks for long. A stall is
retried within the run and resumed from where it stopped, by this run or
the next (Models.fetch_shipped_model); a run gives up only once those
retries are spent, and the .part waits for the next one.
stops the run; it is the one call in a run that blocks for long.
"""
if self.fetch_failed:
return None
parts = self.parts
model = parts.models.selected_model() or {}
size = model.get('size') or 0
got = 0.0
def progress(frac: float) -> None:
nonlocal got
got = frac
got_mb, total_mb = int(frac * size) >> 20, size >> 20
parts.progress.report('download', frac, f"downloading {got_mb} of {total_mb} MB" if size else 'downloading',
code='downloading', args={'got_mb': got_mb, 'total_mb': total_mb} if size else None)
def retrying(error, delay: float) -> None:
parts.progress.report('download', got, f"download stalled, retrying in {delay:.0f} s",
code='download_stalled', args={'retry_s': round(delay)})
try:
path = parts.models.fetch_shipped_model(progress=progress, should_stop=lambda: self.stop, retrying=retrying)
path = self.parts.models.fetch_shipped_model(
progress=lambda frac: self.parts.progress.report('download', frac, 'downloading'),
should_stop=lambda: self.stop,
)
except Exception:
if self.stop:
self.log.warning("jetlink: download stopped, the next run carries on from where it got to")
else:
# the owner starts another run after its backoff, which resumes the .part
self.log.exception("jetlink: could not fetch the large model")
parts.progress.report('failed', 1.0, 'download failed, retrying later', code='download_failed')
self.log.exception("jetlink: could not fetch the large model")
self.parts.progress.report('failed', 1.0, 'could not download the large model')
# one attempt per run; retrying a gigabyte on a loop is worse than staying small
self.fetch_failed = True
return None
if path is not None and model.get('oid'):
link.remember_hash(path, model['oid']) # the download proved it; the upload need not read it again
return path
def needs_download(self) -> bool:
"""Does the comma have to fetch the picked model before a Jetson can build
it? Only while its engine is not known to be built and the file is not here.
Asked without the Jetson: a download needs only the internet, and a car
whose Jetson goes off with the ignition has no Jetson while parked."""
parts = self.parts
entry = parts.models.selected_model()
if entry is None:
return False
sha256, _ = link.identity(parts, entry)
if parts.spec.engine_ready_for(sha256):
return False
return parts.models.shipped_model_path() is None
def provision(self) -> bool:
"""Make the Jetson ready for the selected model; a host is attached. run()
has fetched the file already unless the spec record said the model was
built; a server that lost it since asks for the bytes, and they are
fetched then."""
"""Make the Jetson ready for the selected model; a host is attached. The
file is fetched only when the server asks for the bytes: the Jetson keeps
its own copy of every ONNX and never prunes it."""
from jetlink.client import EngineMissing
parts = self.parts
entry = parts.models.selected_model()
@@ -160,7 +124,7 @@ class ProvisioningRun:
self.log.warning("jetlink: provisioning %s (%d MB, sha %s)",
entry.get('name', sha256[:16]), nbytes >> 20, sha256[:16])
parts.progress.report('connect', 0.0, 'connecting', code='connecting')
parts.progress.report('connect', 0.0, 'connecting')
def ensure(path):
return link.ensure(parts, self.client, sha256, nbytes, path, progress=parts.progress.report_with_eta,
@@ -173,15 +137,15 @@ class ProvisioningRun:
except EngineMissing:
# the server has nothing to build from, and neither have we: fetch the
# model and hand it over in this run. The owner lends the link for the
# whole run; leaving after the download would put the build a
# WORKER_BACKOFF later
# whole run, so leaving after the download only put the build a
# WORKER_BACKOFF later, five minutes of a finished download doing nothing
if model_path is not None or (model_path := self.fetch_model()) is None:
raise
# a download of minutes can outlast the session; a hello starts a new one
self.hello()
spec = ensure(model_path)
parts.progress.report('ready', 1.0, 'engine ready', code='engine_ready')
parts.progress.report('ready', 1.0, 'engine ready')
self.log.warning("jetlink: engine ready for %s", spec.sha256[:16])
return True
@@ -265,24 +229,15 @@ class ProvisioningRun:
finished = False
try:
# the bytes first while no host is on the cable: a parked car whose
# Jetson is switched with the ignition never has one to wait for. A host
# that is there is asked first, since one with the model built needs
# nothing from the comma; provision() fetches it if the host has not
if self.needs_download() and not gadget.host_attached() and self.fetch_model() is None:
return False
if not self.open_link():
return False
if not self.wait_for_jetson():
self.log.warning("jetlink: no jetson within %.0f s, leaving it for the next run", WAKE_TIMEOUT)
if self.parts.models.shipped_model_path() is not None:
self.parts.progress.report('waiting', 0.0, 'downloaded, builds when the jetson is on',
code='downloaded_waiting_host')
return False
finished = self.provision()
except Exception:
self.log.exception("jetlink: provisioning failed")
self.parts.progress.report('failed', 1.0, 'see the log', code='failed')
self.parts.progress.report('failed', 1.0, 'see the log')
finally:
self.close_link()
return finished
+9 -23
View File
@@ -4,7 +4,7 @@ Copyright (c) 2026-, Zeph Leggett.
This file is part of jetlink and is licensed under the MIT License.
See the LICENSE file in the root directory for more details.
The spec of the last model a server built (the pick, or a stand-in for it), and whether its engine is built, in one record.
The picked model's spec, and whether its engine is built, in one record.
Reading the shapes and output slices means parsing a 766 MB ONNX. The server
does that when it builds the engine and answers with the spec; provisioning
@@ -13,7 +13,7 @@ record also says whether the server has built the engine for the sha it names,
so the spec and the readiness can never name different models. It must
survive a reboot, or every ignition rebuilds a 160 s engine.
Stored as a param (Keys.spec), through the adapter's get and put.
A param today (Keys.spec), through the adapter's get and put.
"""
from __future__ import annotations
@@ -27,12 +27,11 @@ class SpecRecord:
value = self.op.get(self.op.keys.spec)
return value if isinstance(value, dict) else None
def load(self, d: dict | None = None):
"""The recorded ModelSpec, or None if there is not a usable one. `d` is
the record, when the caller has read it already."""
def load(self):
"""The recorded ModelSpec, or None if there is not a usable one."""
from jetlink.spec import ModelSpec
try:
d = self._raw() if d is None else d
d = self._raw()
return ModelSpec.from_dict(d) if d else None
except Exception:
self.op.log.exception("jetlink: cached spec is unreadable")
@@ -44,25 +43,12 @@ class SpecRecord:
def engine_ready_for(self, sha256: str | None) -> bool:
"""Has the server built the engine for this model? Params only."""
return bool(sha256) and self.built_sha() == sha256
def built_sha(self) -> str | None:
"""ready_spec()'s sha256, without making the spec of it."""
d = self._raw()
return d.get('sha256') if d is not None and d.get('ready') is True else None
return bool(sha256) and d is not None and d.get('sha256') == sha256 and d.get('ready') is True
def ready_spec(self):
"""The last model whose engine the server built, whichever model is picked
now: what drives while a new pick is still being fetched or built. None
without one."""
d = self._raw()
return self.load(d) if d is not None and d.get('ready') is True else None
def clear_ready(self, sha256: str | None = None) -> None:
def clear_ready(self) -> None:
"""The engine is no longer known to be built. The spec stays: it still
sizes the warp, and the next provisioning run asks again. With `sha256`,
only if the record is that model's: a new pick the server has nothing of
says nothing about the last model it built, which still drives."""
sizes the warp, and the next provisioning run asks again."""
d = self._raw()
if d is not None and d.get('ready') and (sha256 is None or d.get('sha256') == sha256):
if d is not None and d.get('ready'):
self.op.put(self.op.keys.spec, {**d, 'ready': False})
+24 -189
View File
@@ -22,24 +22,13 @@ from jetlink.comma import gadget
# -- progress -----------------------------------------------------------------
# Written by the provisioning run and the joining state, read by the UI: a
# param (Keys.progress), because the writer is another process.
# param today (Keys.progress), because the writer is another process.
# Each report is a file write, and the UI reads it at 5 Hz. An upload reports
# once per 4 MB chunk, 440 of them for a 1.7 GB model, and onroad that is IO a
# recording would have to share the disk with.
PROGRESS_MIN_INTERVAL = 0.25
# what a progress record's `code` may be, for a fork that words the line
# itself; `msg` stays the English one. `args`, where a code has them:
# downloading and upload {got_mb, total_mb}, download_stalled {retry_s},
# reconnecting {why: lost, behind, load_failed or bad_output}, build {eta_s}.
# From upload on they are stage names, and the server's stages are its own:
# a server can add one, so a fork falls back to `msg` for a code it does not
# know. A server message with no stage has no code
PROGRESS_CODES = ('waiting_for_host', 'finding_model', 'model_missing', 'ready_waiting_window', 'reconnecting',
'connecting', 'downloading', 'download_stalled', 'download_failed', 'downloaded_waiting_host',
'engine_ready', 'failed', 'upload', 'patch', 'parse', 'convert', 'compile', 'build', 'load', 'warm')
def estimated_build_seconds(size: int) -> int:
"""Orin Nano Super, TensorRT 10.3: the 766 MB models built in 102 to 166 s,
@@ -68,28 +57,21 @@ class Progress:
return None
return value if isinstance(value, dict) else None
def report(self, stage: str, frac: float, msg: str = '', drops: int = 0, code: str | None = None,
args: dict | None = None) -> None:
def report(self, stage: str, frac: float, msg: str = '', drops: int = 0) -> None:
"""Never raises: called from except handlers. A fraction's ticks are held to
4 Hz within a stage; a report with no fraction (a join's state) and the end
of a stage always go through, so the panel's last word is never dropped. A
join says 'waiting for jetlink' and 're-engage to switch' moments apart;
join says 'waiting for jetlink' and 'checking the link keeps up' 50 ms apart;
holding the second left the first on the panel for the rest of the drive.
`msg` is the panel's one short line; `drops`, once the link has dropped often
enough to blame the cable, is the panel's to word; `code` and `args` say
`msg` for a fork with its own words (PROGRESS_CODES)."""
enough to blame the cable, is the panel's to word."""
last_stage, last_at = self._last
now = time.monotonic()
if 0.0 < frac < 1.0 and stage == last_stage and now - last_at < PROGRESS_MIN_INTERVAL:
return
self._last = (stage, now)
record = {'stage': stage, 'frac': round(frac, 4), 'msg': msg, 'drops': drops}
if code is not None:
record['code'] = code
if args:
record['args'] = args
try:
self.op.put(self.op.keys.progress, record)
self.op.put(self.op.keys.progress, {'stage': stage, 'frac': round(frac, 4), 'msg': msg, 'drops': drops})
except Exception:
self.op.log.exception("jetlink: could not report progress")
@@ -103,14 +85,11 @@ class Progress:
"""Progress, with how long the build still has to run, from the model's
size. Only the build: the upload reports MB of MB and a connect has nothing
to predict."""
args = None
size = self._size() if self._size is not None and stage in ('build', 'upload') else None
if size and stage == 'build':
left = estimated_build_seconds(size) * max(0.0, 1.0 - frac)
msg, args = _eta(left), {'eta_s': round(left)}
elif size:
args = {'got_mb': int(frac * size) >> 20, 'total_mb': size >> 20}
self.report(stage, frac, msg, code=stage or None, args=args)
if stage == 'build':
size = self._size() if self._size is not None else None
if size:
msg = _eta(estimated_build_seconds(size) * max(0.0, 1.0 - frac))
self.report(stage, frac, msg)
# -- what is on the other end -------------------------------------------------
@@ -189,13 +168,8 @@ def usb_port() -> str | None:
# -- whether the link can run --------------------------------------------------
# the offroad alert's text for a device whose checkout has no warp for its camera
# the offroad alert's text for a device whose build made no warp for its camera
NO_WARP = "no warp built for this camera"
# how a failed snapshot starts its reason (the owner's backoff: gadget.BACKING_OFF)
STATUS_FAILED = "jetlink status failed"
# Status.reason_code: the reason's kind, for a fork with its own words. Any
# reason that is none of the others is the gadget's, its text the detail
REASON_CODES = ('no_warp', 'service_stopped', 'backing_off', 'status_failed', 'gadget')
def unavailable(parts, record: dict | None) -> str | None:
@@ -213,61 +187,21 @@ def unavailable(parts, record: dict | None) -> str | None:
return None if parts.warps.built() else NO_WARP
def _built_for_the_pick(parts) -> bool:
spec = parts.spec.load()
selected = parts.models.selected_model()
return (spec is not None and selected is not None and spec.sha256 == selected['oid']
and parts.spec.engine_ready_for(spec.sha256))
def reason(parts, mode: str) -> str | None:
"""Why the link cannot run, for someone who asked for it only: with it off,
a device that cannot present the gadget simply does not offer the feature."""
return unavailable(parts, owner_record()[0]) if parts.enabled(mode) else None
# -- what modeld runs ------------------------------------------------------------
# a big model drives while its last answer was this recent when the record
# was written: five frames at 20 Hz, so a held frame or two still counts
OUTPUT_FRESH = 0.25
# and while the record itself is this recent: a modeld that froze stops
# reading as driving within it. Two writer periods, since the writer runs at
# background priority and a late write must not read as a hand-back; below the
# owner's HEARTBEAT_TIMEOUT, which `running` keeps
ACTIVE_FRESH = 2 * gadget.RECORD_PERIOD
class Runtime(NamedTuple):
state: str | None = None # the joining model's state (STATES)
active: bool = False # the big model drives modeld's frames now
sha256: str | None = None # and which model it is, while it does
checkpoint: str | None = None
def runtime() -> Runtime:
"""The joining model's state, whether the big model drives now and which
model it is, from modeld's runtime record (gadget.MODELD); Runtime() without
a live one: no modeld with the link since boot, or one that stopped, died
or stopped writing (gadget.HEARTBEAT_TIMEOUT, as the owner's). Never raises."""
record = gadget.read_record(gadget.MODELD)
try:
if record is None or record.get('stopping') or not gadget.owner_alive(record) \
or not gadget.pid_alive(record.get('pid')):
return Runtime()
# the answer's age when written: the writer wakes once a second, the frames 20 times
answered = record['at'] - record.get('last_output_at', 0.0) < OUTPUT_FRESH
active = bool(record.get('active')) and answered and time.monotonic() - record['at'] < ACTIVE_FRESH
def text(key: str, shown: bool = True) -> str | None:
value = record.get(key)
return value if shown and isinstance(value, str) else None
return Runtime(text('state'), active, text('sha256', active), text('checkpoint', active))
except (TypeError, KeyError):
return Runtime()
# -- the snapshot ----------------------------------------------------------------
# Status.phase, in the order it is decided (see there). Plain strings, and
# like the codes only ever added to: a fork maps one it does not know to its
# own default
PHASES = ('unavailable', 'off', 'active', 'waiting', 'reconnecting', 'failed', 'preparing', 'connecting', 'prepared')
class Status(NamedTuple):
"""Everything a reader shows, taken at once. Fields are only ever added."""
enabled: bool # the setting is on and no chestnut is fitted
@@ -280,87 +214,11 @@ class Status(NamedTuple):
progress: dict | None # {stage, frac, msg} while provisioning or joining
model: str | None # the big model it will run: the pick, else jetlink's default
default_model: str | None # jetlink's default, named like the chestnut's (no build date)
standin: str | None = None # the last model built, which drives while the pick is not ready
running: str | None = None # the joining model's state (STATES) in a live modeld, by its record (runtime)
active: bool = False # the big model drives modeld's frames now, by the same record
running_sha256: str | None = None # the model it drives with, while active
running_checkpoint: str | None = None
@property
def runnable(self) -> bool:
"""Is a big model built to run: the pick, or the stand-in for it?"""
return self.ready or self.standin is not None
@property
def active_model(self) -> str | None:
"""The big model that runs: the pick once it is built, else the stand-in."""
return self.model if self.ready else self.standin
@property
def shown(self) -> bool:
"""Anything of the link's to show: it is on, a host is there, or progress."""
return self.enabled or self.present or self.progress is not None
@property
def holds_port(self) -> bool:
"""The link holds the comma's USB port, so ADB cannot have it."""
return self.enabled
@property
def port_state(self) -> str | None:
"""The port line's: 'connected' with a host on the link, else the CC
pin's 'empty' or 'device' (a cable with something live behind it, it
cannot say what); None where the kernel does not say."""
if self.present:
return 'connected'
if self.port is None:
return None
return 'empty' if self.port == 'empty' else 'device'
@property
def reason_code(self) -> str | None:
"""`reason` as one of REASON_CODES, None without one."""
reason = self.reason
if reason is None:
return None
if reason == NO_WARP:
return 'no_warp'
if reason == STOPPED:
return 'service_stopped'
if reason.startswith(gadget.BACKING_OFF):
return 'backing_off'
return 'status_failed' if reason.startswith(STATUS_FAILED) else 'gadget'
@property
def phase(self) -> str:
"""One of PHASES, what a panel's state chip says, decided in that order.
Why the link cannot run comes first (a failed snapshot has the reason
with enabled False), then the setting, then what a live modeld says it
runs, then the progress stage, then the records. Without a live record
the progress line decides, and it can be a drive old: a modeld that is
never closed leaves its join's last 'connect' line, which reads
connecting offroad, as the icon reads loading. A big model that stopped
answering while it drives is reconnecting: it is about to hand back. A
build inside a join is preparing; a connect is connecting, a join's or a
provisioning run's."""
if self.reason is not None:
return 'unavailable'
if not self.enabled:
return 'off'
if self.active:
return 'active'
if self.running == 'ready':
return 'waiting'
if self.running in ('retrying', 'running'):
return 'reconnecting'
stage = str((self.progress or {}).get('stage') or '')
if stage == 'failed':
return 'failed'
if stage not in ('', 'ready', 'connect'):
return 'preparing'
if self.running == 'joining' or stage == 'connect':
return 'connecting'
return 'prepared' if self.runnable else 'preparing'
"""model, once it can run."""
return self.model if self.ready else None
def icon(self, started: bool, model_seen: bool, running_big: bool, state: str) -> str:
"""The chestnut icon's state for the link: a ChestnutState value.
@@ -376,7 +234,7 @@ class Status(NamedTuple):
return 'disconnected'
if stage and stage != 'ready':
return 'failed' if stage == 'failed' else 'loading'
return 'ready' if self.runnable else 'uncompiled'
return 'ready' if self.ready else 'uncompiled'
if model_seen and running_big:
return 'active'
@@ -389,7 +247,7 @@ class Status(NamedTuple):
# nothing is in control, so a driver who stays engaged keeps the small model
if state == 'ready':
return 'waiting'
if not self.runnable:
if not self.ready:
return 'uncompiled'
if state == 'running':
return 'active'
@@ -400,44 +258,21 @@ def read(parts, mode: str) -> Status:
"""Everything at once, over the link setting the caller read: each file once."""
record, live = owner_record()
enabled = parts.enabled(mode)
live_model = runtime() if enabled else Runtime()
reason = unavailable(parts, record) if enabled else None
# the last model the server built; the pick's, or the stand-in that drives
# until the pick is built (link.open_link)
built = parts.spec.built_sha() if enabled and reason is None else None
selected = parts.models.selected_model() if built is not None else None
ready = built is not None and selected is not None and built == selected['oid']
return Status(
enabled=enabled,
mode=mode,
transport=link_transport(mode, live),
present=bool(live.get('present')) if live is not None else parts.presence.present(),
port=usb_port(),
ready=ready,
ready=enabled and reason is None and _built_for_the_pick(parts),
reason=reason,
progress=parts.progress.read(),
model=parts.models.selected_model_name(),
default_model=parts.models.default_model_name(),
standin=parts.models.name_for(built) if built is not None and not ready else None,
running=live_model.state,
active=live_model.active,
running_sha256=live_model.sha256,
running_checkpoint=live_model.checkpoint,
)
def model_state(parts, ref: str) -> str | None:
"""One catalog model, for the picker's list: 'ready' when the Jetson has
built it (the spec record), 'downloaded' when the comma has its file, else
None. Records and one stat; a model never resolved has neither."""
model = next((m for m in parts.models.model_index() if m['ref'] == ref), None)
if model is None or not model.get('oid'):
return None
if parts.spec.engine_ready_for(model['oid']):
return 'ready'
return 'downloaded' if parts.models.has_file(model) else None
def failed(error: str, mode: str) -> Status:
"""What a reader gets when the snapshot itself failed: nothing to show, the
setting as it is, and, for someone who asked for the link, the failure where
@@ -447,4 +282,4 @@ def failed(error: str, mode: str) -> Status:
def failure(error: str, mode: str) -> str | None:
return f"{STATUS_FAILED}: {error}" if mode != 'off' else None
return f"jetlink status failed: {error}" if mode != 'off' else None
-266
View File
@@ -1,266 +0,0 @@
"""
Copyright (c) 2026-, Zeph Leggett.
This file is part of jetlink and is licensed under the MIT License.
See the LICENSE file in the root directory for more details.
The Openpilot interface over stock openpilot's paths: the base a fork's
adapter subclasses, overriding only what its fork does differently.
The one jetlink module that imports openpilot, and only inside a method, where
it is used: the top level is the standard library and interface.py, because
the resident owner imports this module (through the fork's owner_config()).
Which method runs where matters. owner_config() imports only
openpilot.common.basedir (os alone); anything under common.hardware loads
capnp, swaglog loads zmq, and model_face() imports modeld itself, so it is for
modeld alone, where that module is already loaded.
Data are class attributes, behaviour small methods, each a fork may override;
one override point per value: a fork sets KEYS and CATALOG_SELECTOR, and the
interface's keys and catalog_selector read them. Both are jetlink's contract:
renaming or removing one, or changing what one means, is a breaking change
and bumps API. scripts/upstream (the upstream-paths CI job) checks the paths
and literals here against commaai/openpilot master weekly.
A fork's adapter module defines its subclass lazily, as its top level must
import without a jetlink checkout: docs/porting.md, "The adapter module".
Defined there, the class's module names the adapter module the provisioning
run is started with, so it needs no ADAPTER_MODULE.
"""
from __future__ import annotations
import os
import time
from functools import cached_property
from pathlib import Path
# root.AGNOS is openpilot's own test for a comma (common.hardware.AGNOS), which
# that package answers only after loading capnp; root is on the owner's path
from jetlink.comma import root
from jetlink.openpilot.interface import Keys, ModelFace, OwnerConfig
# how long a Params or SubMaster that failed to load is left before the next
# try: in_control() asks on every realtime frame
RETRY_S = 5.0
# what the base engaged() reads, by service: only those in IN_CONTROL_SERVICES
_ENGAGED = {'carControl': ('enabled', 'latActive', 'longActive'), 'selfdriveState': ('enabled', 'active')}
# what marks stock modeld's ModelState, which small_reset() zeroes (and its prev_desire)
_MODEL_STATE = ('input_queues', 'state_pairs', 'packed_input')
class Stock:
"""jetlink.openpilot.interface.Openpilot over stock openpilot (the 0.11 layout)."""
# the fork's adapter module, which the provisioning run imports; None for
# the module the subclass is defined in
ADAPTER_MODULE: str | None = None
# the canonical names, which a fork declares in params_keys.h. No model
# manager upstream: no big-model slot or catalog, so jetlink runs its default
KEYS = Keys(link='JetlinkLink', offroad='IsOffroad', progress='JetlinkProgress', spec='JetlinkSpec',
pointers='JetlinkModelPointers', charge_phone='JetlinkChargePhone')
# the model manager's catalog version; none upstream
CATALOG_SELECTOR = 0
# comma's chestnut, running and in its ROM (common.hardware.usb's
# CHESTNUT_USB_IDS + CHESTNUT_ROM_USB_IDS, which CI holds this to): written
# out, as the owner cannot import that package
CHESTNUT_IDS = frozenset({(0xADD1, 0x0001), (0x3801, 0x0001), (0x174C, 0x2464), (0x174C, 0x2463)})
# what in_control() reads; carState too, so a dead card counts as in control.
# It goes with engaged(): the base reads only the services listed here, so a
# fork that adds one (zoompilot's carControlSP) overrides engaged() to read it
IN_CONTROL_SERVICES = ('carState', 'carControl', 'selfdriveState')
# modeld's driving warp, which its SConscript compiles per camera
WARP_DIR = 'openpilot/selfdrive/modeld/models'
WARP_NAME = 'driving_warp_{cam_w}x{cam_h}_tinygrad.pkl'
OWNER_LOG = Path('/data/log/jetlink-owner.log')
def __init__(self):
# nothing imported here: owner_config() builds one in the owner
self._stores: dict[Path, object] = {}
self._params_after = 0.0
self._sm = None
self._sm_after = 0.0
self._in_control_error: str | None = None
# -- the readers --------------------------------------------------------------
# the interface's members, over the class attributes a fork sets
@property
def keys(self) -> Keys:
return self.KEYS
@property
def catalog_selector(self) -> int:
return self.CATALOG_SELECTOR
@cached_property
def log(self):
from openpilot.common.swaglog import cloudlog
return cloudlog
@cached_property
def basedir(self) -> Path:
# the checkout's root, which holds .lfsconfig and the openpilot package
from openpilot.common.basedir import BASEDIR
return Path(BASEDIR)
def params_dir(self) -> Path:
# params.cc and hw.h's rule: PARAMS_ROOT, else /data/params on a comma and
# $HOME/.comma<OPENPILOT_PREFIX>/params elsewhere, then /<OPENPILOT_PREFIX,
# or d>. HOME unset gives /.comma, as the C++ concatenation does. The
# environment only, per call, so it follows a prefix as Params does
prefix = os.environ.get('OPENPILOT_PREFIX', '')
store = os.environ.get('PARAMS_ROOT')
if store is None:
store = '/data/params' if root.AGNOS else os.environ.get('HOME', '') + '/.comma' + prefix + '/params'
return Path(store) / os.environ.get('OPENPILOT_PREFIX', 'd')
def _params(self):
# one Params per store: making one costs more than the read on a comma, and
# the UI reads several five times a second. By store, for a test's prefix
where = self.params_dir()
store = self._stores.get(where)
if store is None:
# one that failed to load is not tried again for RETRY_S: get() is on the UI's frame
if time.monotonic() < self._params_after:
raise RuntimeError("openpilot's Params failed to load, retrying shortly")
try:
from openpilot.common.params import Params
store = self._stores[where] = Params()
except Exception:
self._params_after = time.monotonic() + RETRY_S
raise
return store
def get(self, key: str):
# hardwared's and the UI's threads read here, which UnknownKeyName (a
# params library older than the key) must not take down
try:
return self._params().get(key)
except Exception:
return None
def chestnut_present(self) -> bool:
# any chestnut id, its ROM's included, the owner's walk: never the port
# beside one, even one modeld ignores until its firmware matches
from jetlink.comma.port import chestnut_attached
return chestnut_attached(self.CHESTNUT_IDS)
def camera(self) -> tuple[int, int, int, int]:
# modeld takes the size from camerad's buffer; a comma four's os04c10 is
# the 1344x760 one, every other device's the 1928x1208
from openpilot.common.hardware import HARDWARE
from openpilot.common.transformations.camera import _ar_ox_fisheye, _os_fisheye
from openpilot.common.transformations.model import MEDMODEL_INPUT_SIZE
camera = _os_fisheye if HARDWARE.get_device_type() == 'mici' else _ar_ox_fisheye
return camera.width, camera.height, *MEDMODEL_INPUT_SIZE
def warp_path(self, cam_w: int, cam_h: int, model_w: int, model_h: int) -> Path:
# scons builds it for the one model input size, which jetlink checks the warp makes
return self.basedir / self.WARP_DIR / self.WARP_NAME.format(cam_w=cam_w, cam_h=cam_h, model_w=model_w, model_h=model_h)
# -- the provisioning run --------------------------------------------------------
def put(self, key: str, value, *, block: bool = False) -> None:
self._params().put(key, value, block=block)
def remove(self, key: str) -> None:
self._params().remove(key)
def model_root(self) -> Path:
# sunnypilot's Paths.model_root(), where there is one; upstream has none,
# and the same place it would be
from openpilot.common.hardware.hw import Paths
if hasattr(Paths, 'model_root'):
return Path(Paths.model_root())
return Path('/data/media/0/models') if root.AGNOS else Path(Paths.comma_home()) / 'media' / '0' / 'models'
# -- modeld ---------------------------------------------------------------------
def parser(self):
"""A new Parser for comma's large model's outputs."""
from openpilot.selfdrive.modeld.parse_model_outputs import Parser
return Parser()
def model_constants(self):
"""The ModelConstants modeld reads off the model."""
from openpilot.selfdrive.modeld.constants import ModelConstants
return ModelConstants
def action_function(self):
"""modeld's get_action_from_model."""
from openpilot.selfdrive.modeld.modeld import get_action_from_model
return get_action_from_model
def model_face(self) -> ModelFace:
from openpilot.selfdrive.modeld.constants import ModelConstants
from openpilot.selfdrive.modeld.modeld import LAT_SMOOTH_SECONDS, LONG_SMOOTH_SECONDS
from openpilot.system.camerad.cameras.nv12_info import get_nv12_info
# desire_len is comma's large model's, which is stock modeld's
return ModelFace(parser=self.parser, frame_size=lambda w, h: get_nv12_info(w, h)[3],
desire_len=ModelConstants.DESIRE_LEN, constants=self.model_constants(),
lat_smooth_seconds=LAT_SMOOTH_SECONDS, long_smooth_seconds=LONG_SMOOTH_SECONDS,
get_action_from_model=self.action_function())
def engaged(self, sm) -> bool:
"""Is openpilot in control, off fresh and valid IN_CONTROL_SERVICES?
carControl's enabled or either axis, or selfdriveState's enabled or active,
of those services the fork lists."""
return any(getattr(sm[service], field) for service, fields in _ENGAGED.items()
if service in self.IN_CONTROL_SERVICES for field in fields)
def in_control(self) -> bool:
# on modeld's frame thread before every frame: a poll that never waits, and
# a service late or invalid, or any failure, holds the swap off, never modeld
try:
if self._sm is None:
# one try per RETRY_S, not one per frame
if time.monotonic() < self._sm_after:
return True
self._sm_after = time.monotonic() + RETRY_S
from openpilot.cereal import messaging
self._sm = messaging.SubMaster(list(self.IN_CONTROL_SERVICES))
services = self.IN_CONTROL_SERVICES
self._sm.update(0)
result = not (self._sm.all_alive(services) and self._sm.all_valid(services)) or self.engaged(self._sm)
except Exception as e:
# once per distinct error: this runs 20 times a second
if self._in_control_error != (text := f"{type(e).__name__}: {e}"):
self._in_control_error = text
try:
self.log.error("jetlink: in_control() failed: %s", text)
except Exception:
pass
return True
self._in_control_error = None
return bool(result)
def event(self, name: str, **fields) -> None:
self.log.event(name, **fields)
def small_reset(self, small):
"""Stock modeld's ModelState, as its warmup leaves it: the recurrent state
on the GPU, the packed host inputs and the desire edge, zeroed in place.
None for another shape, so jetlink's own reset runs."""
if not all(hasattr(small, name) for name in _MODEL_STATE):
return None
from jetlink.openpilot.warp import reset_for_stock
return reset_for_stock(small, getattr(small, 'prev_desire', None))
# -- the adapter module's, for jetlinkd --------------------------------------------
@classmethod
def owner_config(cls) -> OwnerConfig:
"""What jetlinkd needs, data only: openpilot.common.basedir is the one import."""
from openpilot.common.basedir import BASEDIR
op = cls()
# the provisioning run starts in the checkout with it on the path, where
# the launch script links jetlink in
return OwnerConfig(params_dir=op.params_dir(), keys=op.keys, chestnut_ids=cls.CHESTNUT_IDS,
adapter=cls.ADAPTER_MODULE or cls.__module__, cwd=Path(BASEDIR), env={'PYTHONPATH': BASEDIR},
log_file=cls.OWNER_LOG)
@classmethod
def main(cls) -> None:
"""jetlinkd: hold the USB gadget until manager stops this process."""
from jetlink.openpilot.owner import main
main(cls.owner_config())
-452
View File
@@ -1,452 +0,0 @@
"""
Copyright (c) 2026-, Zeph Leggett.
This file is part of jetlink and is licensed under the MIT License.
See the LICENSE file in the root directory for more details.
A kit a fork runs in its own CI, against its real adapter and modelds, and
the fakes jetlink's own tests share.
check_adapter(module) the adapter module: every side of Openpilot, a top
level of the standard library alone, an owner_config()
that imports nothing heavy, an in_control() that
answers a bool at once
check_seam(path) a modeld's main(): prepare() before it goes realtime,
attach() once the small model is built and its
result run, handed_over() read
check_fallback(obj) the fork's own hooks for no usable jetlink: each
answers as Off does, over HOOK_CALLS
install_guard() an audit hook a conftest starts and stops around each
test, which records a reach for the machine's own
params, /dev/shm records, sysfs or sudo
FakeJoinedModel, FakeSubMaster, RecordingLog, and tinygrad over numpy
(fake_tinygrad)
The checks return their problems, [] when there are none, so one assertion
shows them all: assertEqual(check_adapter(adapter_module), []).
Not on the owner's path, and no test framework or numpy at module level: a
fork imports this from its tests only.
"""
from __future__ import annotations
import ast
import json
import os
import subprocess
import sys
import time
from collections.abc import Iterable
from pathlib import Path
from types import ModuleType
from jetlink.openpilot.interface import Openpilot, conformance
# jetlink's own heavy half, which the owner must not reach: numpy, and urllib
# with ssl and email behind it
HEAVY_JETLINK = ('jetlink.client', 'jetlink.spec', 'jetlink.queues', 'jetlink.registry')
# -- the adapter module ----------------------------------------------------------
_PROBE = '''
import importlib, json, sys
before = set(sys.modules)
module = importlib.import_module(sys.argv[1])
found = {'import': sorted(set(sys.modules) - before)}
before = set(sys.modules)
try:
config = module.owner_config()
found['owner_config'] = sorted(set(sys.modules) - before)
from jetlink.openpilot.interface import OwnerConfig
found['config'] = isinstance(config, OwnerConfig)
except Exception as e:
found['error'] = f'{type(e).__name__}: {e}'
print(json.dumps(found))
'''
def _foreign(names, own: str, heavy, allowed=()) -> list[str]:
"""What is neither the standard library nor jetlink's light half, nor of
the fork's package `own`, nor `allowed`: by the outermost module of each."""
found = {n for n in names if n not in allowed and (n.startswith((*heavy, *HEAVY_JETLINK)) or
n.split('.')[0] not in (*sys.stdlib_module_names, 'jetlink', own))}
return sorted(n for n in found if not any(n.startswith(f + '.') for f in found))
def check_adapter(module: ModuleType, *, heavy: tuple[str, ...] = ()) -> list[str]:
"""The adapter module's problems. Its top level imports the standard
library alone, as manager and the resident owner import it. owner_config()
may import more: anything of the fork's own package that pulls in no third
party, so 'openpilot.common.params' passes unless it is named in `heavy`. A
fork should name its heavy modules there (params, swaglog, cereal). Both
in a fresh interpreter on this one's path, so a module this process has
loaded already counts."""
name = module.__name__
problems = [f"{name}.{member}(): missing" for member in ('adapter', 'owner_config')
if not callable(getattr(module, member, None))]
if problems:
return problems
# on this interpreter's path, as a fork's CI has set it up
try:
run = subprocess.run([sys.executable, '-c', _PROBE, name], capture_output=True, text=True, timeout=120,
env={**os.environ, 'PYTHONPATH': os.pathsep.join(p for p in sys.path if p)})
# the probe's line is the last: the module may print as it imports
found = json.loads(run.stdout.strip().splitlines()[-1])
except subprocess.TimeoutExpired:
return [f"{name} did not import and build its owner_config() in a fresh interpreter within 120 s"]
except (IndexError, ValueError):
return [f"{name} does not import in a fresh interpreter: {run.stderr.strip()[-500:]}"]
# its own packages' __init__s, which run first
if top := _foreign(found['import'], '', heavy, {name.rsplit('.', i)[0] for i in range(name.count('.') + 1)}):
problems.append(f"{name}'s top level imports {top}: manager and the resident owner import it")
if 'error' in found:
problems.append(f"owner_config() raised {found['error']}")
elif owner := _foreign(found['owner_config'], name.split('.')[0], heavy):
problems.append(f"owner_config() imports {owner}: the resident owner calls it")
if found.get('config') is False:
problems.append("owner_config(): not an OwnerConfig")
try:
op = module.adapter()
except Exception as e:
return [*problems, f"adapter() raised {type(e).__name__}: {e}"]
problems += [f"adapter: {p}" for p in conformance(op, Openpilot)]
if not callable(getattr(op, 'in_control', None)):
return problems
try:
op.in_control() # the first may set up what it reads
start = time.monotonic()
answer, took = op.in_control(), time.monotonic() - start
except Exception as e:
return [*problems, f"in_control() raised {type(e).__name__}: {e}"]
if not isinstance(answer, bool):
problems.append(f"in_control() answered {answer!r}, not a bool")
if took > 0.05:
problems.append(f"in_control() took {took * 1e3:.0f} ms: modeld asks before every frame")
return problems
# -- the modeld seam ---------------------------------------------------------------
def check_seam(modeld_path: str | Path) -> list[str]:
"""The problems of a modeld's main() with the hooks, read off its source
(importing a modeld costs tinygrad and a vision stream). The hooks are
found as the receiver of attach(model, small, cam_w, cam_h): the fork's
module (jetlink_adapter.attach) or its Hooks (HOOKS.attach) alike. Calls
are ordered as the source reads, so one in a branch that never runs (an
`if False:`) counts as made. state() is the fork's to publish where it has
a message for it, so it is not asked for."""
path = Path(modeld_path)
main = next((n for n in ast.parse(path.read_text(), filename=str(path)).body
if isinstance(n, ast.FunctionDef) and n.name == 'main'), None)
if main is None:
return [f"{path.name}: no main()"]
def at(n: ast.AST) -> tuple[int, int]:
return n.lineno, n.col_offset
nodes = list(ast.walk(main))
calls = sorted((n for n in nodes if isinstance(n, ast.Call)), key=at)
def called(name: str, facade: str | None = None) -> list[tuple[int, int]]:
return [at(c) for c in calls if (c.func.attr if isinstance(c.func, ast.Attribute) else getattr(c.func, 'id', None)) == name
and (facade is None or isinstance(c.func, ast.Attribute) and ast.unparse(c.func.value) == facade)]
def stored(target: str) -> list[tuple[int, int]]:
return [at(n) for n in nodes if isinstance(getattr(n, 'ctx', None), ast.Store) and ast.unparse(n) == target]
attach = next((c for c in calls if isinstance(c.func, ast.Attribute) and c.func.attr == 'attach'
and len(c.args) + len(c.keywords) == 4), None)
if attach is None:
return [f"{path.name}: main() never calls attach(model, small, cam_w, cam_h)"]
facade = ast.unparse(attach.func.value)
problems, prepare, realtime = [], called('prepare', facade), called('config_realtime_process')
if not prepare:
problems.append(f"main() never calls {facade}.prepare()")
elif realtime and prepare[0] >= realtime[0]:
problems.append("prepare() comes after config_realtime_process(): the GPU's thread would inherit the realtime core")
elif prepare[0] >= at(attach):
problems.append("attach() comes before prepare(), so the link never joins")
if not realtime:
problems.append("main() never calls config_realtime_process()")
small = attach.args[1] if len(attach.args) > 1 else next((k.value for k in attach.keywords if k.arg == 'small'), None)
if isinstance(small, ast.Name) and not any(p < at(attach) for p in stored(small.id)):
problems.append(f"attach() comes before {small.id} is built")
# the name its result is bound to, which the frame loop must run
bound = next((t.id for n in nodes if isinstance(n, (ast.Assign, ast.AnnAssign)) and n.value is attach
for t in getattr(n, 'targets', [getattr(n, 'target', None)]) if isinstance(t, ast.Name)), None)
if bound is None:
problems.append("attach()'s result is dropped: modeld would never run the joined model")
elif not any(p > at(attach) for p in called('run', bound)):
problems.append(f"attach()'s result, {bound}, is never run: modeld would keep driving the model it had")
elif not any(p > at(attach) for p in stored(f'{bound}.frame_drop_ratio')):
problems.append(f"main() never writes {bound}.frame_drop_ratio: a host that drops camera frames would never hand back")
if not called('handed_over', facade):
problems.append(f"main() never calls {facade}.handed_over(model)")
return [f"{path.name}: {p}" for p in problems]
# -- the fork's fallback ------------------------------------------------------------------
_MODEL = object()
# every hook a fork's lines may call, and the arguments each is checked with:
# the one list of them (jetlink's tests hold Hooks to it)
HOOK_CALLS = {'should_run': [(False, None, None)], 'status': [()], 'reason': [()], 'prepare': [(False,), (True,)],
'attach': [(_MODEL, object(), 1928, 1208)], 'handed_over': [(_MODEL,)], 'state': [(_MODEL,)],
'shutdown_step': [(False,), (True, 'test')], 'request_shutdown': [('test',)], 'shutdown_pending': [()],
'model_state': [('f' * 40,)], 'should_extend_catalog': [()], 'extend_catalog': [({'bundles': []},)],
'runtime': [()], 'swap_events': [(0.01,)]}
def check_fallback(fallback, hooks: Iterable[str] | None = None, *, link_on=None) -> list[str]:
"""The problems of the fork's own hooks for a jetlink that is missing or
too old to have Off: each of `hooks`, the ones the fork's lines call (by
default every hook it has, so pass them to have a missing one named),
there and answering as Off does, the link off. reason() is the fork's to
word, a str or None, and None while `link_on()`, the fork's read of the
setting, says off: nobody who left the link off is nagged. swap_events()
may be None where the fork's lines check for it. link_on() reads the
params store, so the test sets PARAMS_ROOT or OPENPILOT_PREFIX first, or
it reads the machine's own."""
from jetlink.openpilot.events import Flags
from jetlink.openpilot.hooks import Off
def flags(tracker):
found = tracker.update(True, False, True, state='ready')
return tuple(bool(getattr(found, f)) for f in Flags._fields), bool(tracker.settling)
problems = []
for name in hooks if hooks is not None else [n for n in HOOK_CALLS if hasattr(fallback, n)]:
if name not in HOOK_CALLS or not callable(getattr(fallback, name, None)):
problems.append(f"{name}(): missing" if name in HOOK_CALLS else f"{name}(): not a hook")
continue
for args in HOOK_CALLS[name]:
try:
got, want = getattr(fallback, name)(*args), getattr(Off(), name)(*args)
if name == 'swap_events':
got, want = (want, want) if got is None else (flags(got), flags(want))
except Exception as e:
problems.append(f"{name}() raised {type(e).__name__}: {e}")
continue
if name == 'reason':
try:
off = link_on is not None and not link_on()
except Exception as e:
problems.append(f"link_on() raised {type(e).__name__}: {e}")
continue
same = got is None or (isinstance(got, str) and not off)
else:
same = got is want if any(want is a for a in args) else (type(got), got) == (type(want), want)
if not same:
problems.append(f"{name}() answered {got!r}, where the link off is {want!r}")
return problems
# -- the guard ------------------------------------------------------------------------
# the machine's own params store, jetlink's records, sysfs, FunctionFS and a
# PC's openpilot params
FORBIDDEN = ('/data/', '/dev/shm/', '/dev/ffs', '/sys/', str(Path.home() / '.comma') + os.sep)
# read by a library on import, never jetlink's own reach: tinygrad's CPU quota
ALLOWED = ('/sys/fs/cgroup/cpu.max',)
# the audit events Guard reads a path from, and the argument it is
_PATH_ARG = {'open': 0, 'os.listdir': 0, 'os.scandir': 0, 'socket.connect': 1}
class Guard:
"""Records, between start() and stop(), every open, listing, socket connect
or root-script run that reaches a forbidden path. Recorded, not refused:
many of the paths are read inside `except OSError`, so a refusal would only
hide the reach. A test that reaches the live params store can clear a
real setting such as JetlinkSpec."""
def __init__(self, forbidden: tuple[str, ...] = FORBIDDEN):
self.forbidden = tuple(forbidden)
self.touched: list[str] | None = None
def start(self) -> None:
self.touched = []
def stop(self) -> list[str]:
"""What was reached since start(), once each."""
touched, self.touched = self.touched or [], None
return sorted(set(touched))
def __call__(self, event: str, args) -> None:
touched = self.touched
if touched is None:
return
if event == 'subprocess.Popen':
# the root script is sudo on a comma
argv = args[1] if len(args) > 1 and isinstance(args[1], (list, tuple)) else []
if (hit := next((str(a) for a in argv if 'jetlink-root.sh' in str(a) or str(a) == 'sudo'), None)) is not None:
touched.append(f"{event} {hit}")
return
index = _PATH_ARG.get(event)
value = args[index] if index is not None and len(args) > index else None
if isinstance(value, (str, bytes, os.PathLike)):
path = os.path.abspath(os.fsdecode(value))
if (path + os.sep).startswith(self.forbidden) and path not in ALLOWED:
touched.append(f"{event} {path}")
def install_guard(forbidden: tuple[str, ...] = FORBIDDEN) -> Guard:
"""A Guard, installed as this process's audit hook (for good: Python has no
removing one). Install once, in a conftest, and start and stop it around
each test."""
guard = Guard(forbidden)
sys.addaudithook(guard)
return guard
# -- fakes ---------------------------------------------------------------------------
class FakeJoinedModel:
"""What attach() returns, for a fork's modeld tests: drives as `small`;
hand_over() is a swap or a hand-back."""
def __init__(self, small=None, *, state: str = 'joining', chestnut: bool = False, constants=None):
self.small = small
self.handovers = 0
self.big_model_state = state
self.chestnut = chestnut
self.constants = constants
self.LAT_SMOOTH_SECONDS = self.LONG_SMOOTH_SECONDS = self.frame_drop_ratio = self.lat_delay = 0.0
self.runs, self.closed = 0, False
def hand_over(self, state: str = 'running') -> None:
self.handovers += 1
self.big_model_state = state
def run(self, bufs, transforms, inputs, after_enqueue=None):
self.runs += 1
return self.small.run(bufs, transforms, inputs, after_enqueue) if self.small is not None else {}
def get_action_from_model(self, *args):
return ('action', args)
def close(self) -> None:
self.closed = True
class _Blank:
"""A message never received, as cereal's SubMaster holds one: every field
reads falsy, a nested one too."""
def __getattr__(self, name: str):
if name.startswith('__'):
raise AttributeError(name)
return self
def __bool__(self) -> bool:
return False
class FakeSubMaster:
"""cereal's SubMaster over fixed messages, for an adapter's engaged() and
in_control() tests: what Stock and StarPilot's adapter read of it. A
service given a message is seen, alive and valid; any other is none of
them until update() receives it, and reads as a blank message whose every
field is falsy. A test changes `data`, `alive` and `valid` as it goes.
update() keeps its timeouts: in_control() is on modeld's frame thread and
must never wait (all 0). In for the real one: mock.patch.object(messaging,
'SubMaster', lambda services: FakeSubMaster(services, **messages)). Unlike
cereal's: a service is seen from construction, not from its first receive;
a blank field is falsy but equals nothing (a capnp default equals 0 or
False); and there is no updated, frame, freq_ok or logMonoTime."""
def __init__(self, services: Iterable[str] = (), **messages):
self.services = list(dict.fromkeys([*services, *messages]))
self.data = dict(messages)
self.seen = {s: s in messages for s in self.services}
self.alive = dict(self.seen)
self.valid = dict(self.seen)
self.timeouts: list[float] = []
def update(self, timeout: float = 100) -> None:
self.timeouts.append(timeout)
for s in self.services:
self.seen[s] = self.seen[s] or (s in self.data and self.alive[s])
def all_alive(self, service_list=None) -> bool:
return all(self.alive[s] for s in service_list or self.services)
def all_valid(self, service_list=None) -> bool:
return all(self.valid[s] for s in service_list or self.services)
def __getitem__(self, service: str):
if service not in self.seen:
raise KeyError(service) # not subscribed, as cereal's
return self.data.get(service, _BLANK)
_BLANK = _Blank()
def _level(name: str):
def log(self, msg, *args, **kwargs):
try:
text = str(msg) % args if args else str(msg)
except (TypeError, ValueError):
text = f"{msg} {args}"
self.records.append((name, text))
return log
class RecordingLog:
"""cloudlog's face, keeping every line: (level, message with its arguments)."""
debug, info, warning, error, exception, critical = map(_level, ('debug', 'info', 'warning', 'error', 'exception', 'critical'))
def __init__(self):
self.records: list[tuple[str, str]] = []
def lines(self, level: str | None = None) -> list[str]:
return [text for lvl, text in self.records if level is None or lvl == level]
def has(self, fragment: str, level: str | None = None) -> bool:
return any(fragment in text for text in self.lines(level))
# The fork's tinygrad is not jetlink's to install. What jetlink calls of it is
# small, and these stand-ins do it with numpy; the fork's tests run the real one.
class FakeTensor:
def __init__(self, data=None, device=None, dtype=None):
import numpy as np
self.array = np.asarray(data) if data is not None else np.zeros(0)
self.device = device
def realize(self, *others):
return self
def assign(self, value):
self.array[...] = value
return self
def numpy(self):
return self.array
class FakeDevice:
DEFAULT = 'CPU'
def fake_jit(fn=None, prune=False):
"""TinyJit as a decorator or a wrapper: the function as it is."""
return (lambda f: f) if fn is None else fn
def fake_tinygrad() -> dict[str, ModuleType]:
"""sys.modules entries for what jetlink imports of tinygrad; patch them in
with mock.patch.dict(sys.modules, fake_tinygrad())."""
from types import SimpleNamespace
def module(name, **attrs):
m = ModuleType(name)
m.__dict__.update(attrs)
return m
return {
'tinygrad': module('tinygrad', Tensor=FakeTensor, TinyJit=fake_jit, Device=FakeDevice),
'tinygrad.tensor': module('tinygrad.tensor', Tensor=FakeTensor),
'tinygrad.device': module('tinygrad.device', Device=FakeDevice),
'tinygrad.helpers': module('tinygrad.helpers', PARALLEL=SimpleNamespace(value=8)),
'tinygrad.engine': module('tinygrad.engine'),
'tinygrad.engine.jit': module('tinygrad.engine.jit', TinyJit=fake_jit),
}
+177 -197
View File
@@ -4,82 +4,92 @@ Copyright (c) 2026-, Zeph Leggett.
This file is part of jetlink and is licensed under the MIT License.
See the LICENSE file in the root directory for more details.
The comma-side warp: openpilot's own, where it lives, what loads it, how the
frame loop runs it (Warp), and the small model's reset for a fallback.
The comma-side warp JIT: how it is built, where it lives, what loads it, and
the small model's reset for a fallback.
The warp stays on the comma (see model_state). openpilot's modeld runs it as
a JIT of its own, driving_warp_{w}x{h}_tinygrad.pkl, one per camera, and the
link runs the same pickle: both camera frames in as one (2, frame size) uint8
input_frame, their transforms as one (2, 3, 3) M_inv, and out the
(2, 6, 128, 256) uint8 frame the link sends. The fork's adapter says where it
is. Nothing compiles one at runtime: a device without one runs the small
model.
The warp stays on the comma (see model_state), and upstream's fused run_model
JIT (openpilot #38684) has no warp to borrow, so comma's make_warp graph is
JIT-compiled as a build target: the fork's build runs
`python -m jetlink.openpilot.warp` once per camera it builds for. A source
build makes the one for its own camera, a prebuilt release one for every
camera it installs on. Nothing compiles one at runtime: in modeld the ~9 s
compile would hold back the first frame on every ignition, and in a
provisioning run, which only runs offroad, it was lost to ignition. A device
without one runs the small model.
load() is what stands between a bad pickle and the car.
tinygrad is the fork's, and only modeld has it: every import of it here is
inside the function that needs it.
tinygrad is the fork's, and only modeld and the build have it: every import of
it here is inside the function that needs it.
"""
from __future__ import annotations
import ctypes
import argparse
import pickle
import sys
from pathlib import Path
# what the warp's TinyJit was captured with: sorted(kwargs) of modeld's call
WARP_INPUT_NAMES = ['M_inv', 'input_frame']
# KGSL allocation flags (msm_kgsl.h) for memory the GPU's accesses snoop the
# CPU's caches on, mapped write-back: KGSL_MEMFLAGS_IOCOHERENT, which
# tinygrad's kgsl bindings lack, and KGSL_CACHEMODE_WRITEBACK (3 << 26). See
# Warp
COHERENT_WRITEBACK = (1 << 31) | (3 << 26)
# what TinyJit records for the keyword call in call_warp: sorted(kwargs)
WARP_INPUT_NAMES = ['big_frame', 'big_tfm', 'frame', 'tfm']
def call_warp(warp, tfm, big_tfm, frame, big_frame):
"""Call a warp JIT. Every caller goes through here, capture included.
TinyJit names inputs from enumerate(args) plus sorted(kwargs) and refuses a
call whose names differ from the capture. A positional compile and a keyword
call raised JitError on the first frame of a drive.
"""
return warp(tfm=tfm, big_tfm=big_tfm, frame=frame, big_frame=big_frame)
def init_device(log) -> None:
"""Bring the GPU up now, on the caller's thread, and turn tinygrad's
compile pool off for the process.
"""Bring the GPU up now, on the caller's thread.
tinygrad initialises the device on its first kernel run and spawns a libusb
event thread doing it. A thread created after config_realtime_process(7, 54)
inherits SCHED_FIFO 54 and the core-7 pin and preempts the frame loop,
enough to drop ~5% of frames. Failure is not a reason to refuse the accelerator.
inherits SCHED_FIFO 54 and the core-7 pin and preempts the frame loop; that
cost 5% of frames once. Failure is not a reason to refuse the accelerator.
"""
try:
# modeld never needs the pool (engine/worker.py), and its PARALLEL idle
# interpreters' handler threads would inherit FIFO 54 on core 7
from tinygrad.helpers import PARALLEL
PARALLEL.value = 0
except Exception:
log.exception("jetlink: could not turn tinygrad's compile pool off")
try:
from tinygrad.tensor import Tensor
Tensor([0.0]).realize()
except Exception:
log.exception("jetlink: could not bring the gpu up before modeld goes realtime")
# the same trap for tinygrad's compile pool (engine/worker.py): created on
# the first compile, after modeld goes realtime, its handler threads sat at
# SCHED_FIFO 54 on core 7. An older tinygrad or PARALLEL=0 is not a failure
try:
from tinygrad.engine.worker import get_worker_pool
get_worker_pool()
except Exception:
log.exception("jetlink: could not start tinygrad's compile pool before modeld goes realtime")
class Warps:
"""The warps this device's checkout has, where the fork's adapter says."""
"""The warps the build made for this device, as the fork's adapter says where."""
def __init__(self, op):
self.op = op
# nothing makes a warp at runtime and the checkout's are there before
# manager, so the answer holds for the life of the process; the UI asks
# at 5 Hz
# nothing compiles a warp at runtime and the build runs before manager, so
# the answer holds for the life of the process; the UI asks at 5 Hz
self._built: bool | None = None
def geometry(self) -> tuple[int, int, int, int]:
"""(cam_w, cam_h, model_w, model_h) for this device: the warp modeld
loads. If the link's model disagrees, load() raises and the drive is
small-model."""
"""(cam_w, cam_h, model_w, model_h) for this device: the choice modeld's
build makes, so the warp built is the one modeld asks for. If they
disagree, load() raises and the drive is small-model."""
return tuple(self.op.camera())
def path(self, cam_w: int, cam_h: int, model_w: int, model_h: int) -> Path:
return Path(self.op.warp_path(cam_w, cam_h, model_w, model_h))
def is_cached(self, cam_w: int, cam_h: int, model_w: int, model_h: int) -> bool:
"""Is there a warp for this geometry? Presence only: a pickle from an
incompatible tinygrad raises in load()."""
"""Is there a warp for this geometry?
Presence only. Staleness is the build's job: the target depends on tinygrad
and the capture sources. A pickle from an incompatible tinygrad raises in load().
"""
return self.path(cam_w, cam_h, model_w, model_h).is_file()
def built(self) -> bool:
@@ -89,185 +99,155 @@ class Warps:
self._built = self.is_cached(*self.geometry())
return self._built
def load(self, cam_w: int, cam_h: int, model_w: int, model_h: int) -> dict:
"""The warp as openpilot pickles it: {'run': its TinyJit, 'input_specs':
{name: (shape, dtype, device)}}. Raises if it is not there or is not one
the frame loop can run.
def load(self, cam_w: int, cam_h: int, model_w: int, model_h: int):
"""The built warp JIT. Raises if it is not there or is stale.
modeld's big-model load is wrapped in the fallback to the small model, and a
warp that cannot be trusted must not reach the car.
"""
path = self.path(cam_w, cam_h, model_w, model_h)
if not path.is_file():
raise RuntimeError(f"no warp for {cam_w}x{cam_h} -> {model_w}x{model_h} at {path}")
with open(path, 'rb') as f:
if not self.is_cached(cam_w, cam_h, model_w, model_h):
raise RuntimeError(f"no warp built for {cam_w}x{cam_h} -> {model_w}x{model_h}; "
"the fork's build makes it (python -m jetlink.openpilot.warp)")
with open(self.path(cam_w, cam_h, model_w, model_h), 'rb') as f:
warp = pickle.load(f)
if not isinstance(warp, dict) or not {'run', 'input_specs'} <= warp.keys():
raise RuntimeError(f"{path.name} is not a driving warp: {type(warp).__name__}")
# a JIT pickled before TinyJit captured loads fine and computes nothing; one
# captured under other names raises JitError on the first frame of a drive
captured = getattr(warp['run'], 'captured', None)
# captured with a different call convention raises JitError on the first
# frame of a drive. Both have happened
captured = getattr(warp, 'captured', None)
if captured is None:
raise RuntimeError("the warp was pickled before it captured; it computes nothing")
raise RuntimeError("cached warp was pickled before it captured; it computes nothing")
names = list(getattr(captured, 'expected_names', []))
if names != WARP_INPUT_NAMES:
raise RuntimeError(f"the warp expects {names}, the frame loop passes {WARP_INPUT_NAMES}")
specs = warp['input_specs']
frames, tfm = tuple(specs['input_frame'][0]), tuple(specs['M_inv'][0])
if len(frames) != 2 or frames[0] != 2 or tfm != (2, 3, 3):
raise RuntimeError(f"the warp takes frames {frames} and transforms {tfm}, not two of each")
made = tuple(getattr(captured.ret, 'shape', ()))
if made != (2, 6, model_h // 2, model_w // 2):
raise RuntimeError(f"the warp makes {made}, not a {model_w}x{model_h} model's input")
raise RuntimeError(f"cached warp expects {names}, call_warp passes {WARP_INPUT_NAMES}")
return warp
class Warp:
"""The warp as modeld's frame loop runs it: start() with two camera buffers
and their transforms, wait(), and `output` holds the warped frame, where the
link sends it from.
def warm(warp, frame_size: int) -> None:
"""Run a loaded warp JIT until it is cheap to call. `frame_size` is the
camera's NV12 buffer size (ModelFace.frame_size).
openpilot's warp takes both frames as one buffer, so start() copies the
camera buffers in, as modeld does, and the transforms beside them. Two
things the JIT alone does not do, measured on the comma:
- Its output lives in GPU memory the CPU reads through its cache. tinygrad
maps a QCOM buffer write-combined, which the CPU reads uncached: copying
the 393 KB out takes 2.9 ms, the kernel's copy straight from that mapping
7.3 ms. The Adreno 630 is IO-coherent, so KGSL memory flagged IOCOHERENT
and write-back is right to read once the GPU is done: 0.22 ms for the
kernel's copy, the GPU's own time unchanged. It moves before the first
call, which links the JIT to its buffers' addresses.
- TinyJit prepares and checks every call's inputs, a graph rewrite per
input: 0.97 ms of a 1.93 ms call. Every frame's inputs are the same two
buffers, which start() writes into, so they are checked once, by the
warm-up's calls through TinyJit, and a frame replays the capture.
The warm-up, the link of the JIT and its first runs, is paid here rather
than on modeld's frame loop.
Measured: loading 0.3 s, the first call 1.9 s, the second 5 ms. Paid on
modeld's frame loop that was ~26 dropped frames and 16 s of modeldLagging
after every join.
"""
def __init__(self, warp: dict, frame_size: int):
import numpy as np
from tinygrad.device import Device
from tinygrad.tensor import Tensor
jit, specs = warp['run'], warp['input_specs']
(frames_shape, _, _), (tfm_shape, _, _) = specs['input_frame'], specs['M_inv']
# what it reads of each camera buffer: modeld's frame_copy_size, short of
# the buffer's end
self._frame_size = frames_shape[1]
if self._frame_size > frame_size:
raise RuntimeError(f"the warp reads {self._frame_size} bytes a frame, past a {frame_size} byte camera buffer")
out = jit.captured.ret.uop.base.buffer
self._device = out.device
if self._device.startswith('QCOM'):
_make_coherent(out)
# the two buffers every frame passes, written in place by start()
self._tensors = {name: Tensor(np.zeros(shape, dtype=dtype), device=device).realize()
for name, (shape, dtype, device) in specs.items()}
for _ in range(2):
jit(**self._tensors)
self.wait = Device[self._device].synchronize
self.wait()
self.output = out.as_memoryview(allow_zero_copy=True)
self._frames = np.frombuffer(self._view('input_frame'), dtype=np.uint8).reshape(frames_shape)
self._at = [self._frames[i].ctypes.data for i in range(2)]
self._tfm = np.frombuffer(self._view('M_inv'), dtype=np.float32).reshape(tfm_shape)
self._replay = jit.captured
self._inputs = [self._tensors[name].uop.base for name in WARP_INPUT_NAMES]
def _view(self, name: str) -> memoryview:
"""The CPU's view of an input buffer, not a copy of it."""
return self._tensors[name].uop.base.buffer.as_memoryview(allow_zero_copy=True)
def start(self, frame: int, big_frame: int, tfm, big_tfm) -> None:
"""Warp the camera buffers at these addresses under their transforms.
`output` is the frame once wait() returns, until the next start(). The
GPU is idle here: every start() is waited for before the next."""
ctypes.memmove(self._at[0], frame, self._frame_size)
ctypes.memmove(self._at[1], big_frame, self._frame_size)
self._tfm[0] = tfm
self._tfm[1] = big_tfm
self._replay(self._inputs, {})
def _make_coherent(out) -> None:
"""Move a QCOM buffer to KGSL memory flagged IOCOHERENT and write-back.
Raises if the kernel keeps either flag back (the allocation's record holds
its answer): write-back memory that is not coherent reads stale frames."""
import numpy as np
from tinygrad.device import Device
dev = Device[out.device]
mem = dev._gpu_alloc(out.nbytes, flags=COHERENT_WRITEBACK)
if mem.meta[0].flags & COHERENT_WRITEBACK != COHERENT_WRITEBACK:
dev._gpu_free(mem)
raise RuntimeError(f"the GPU driver kept back IO-coherent memory (flags {mem.meta[0].flags:#x})")
out.deallocate()
out.allocate(opaque=mem)
from tinygrad.tensor import Tensor
def reset_queues(queues, arrays, prev_desire=None):
"""A small model's reset, captured now: `queues`, its tensors on the GPU,
zeroed by one TinyJit, then `prev_desire` and each numpy array in `arrays`
in place. Every buffer keeps its identity, which the model's own JIT is
linked to, and the reset allocates and compiles nothing, so it may run on
the frame thread. What an adapter's small_reset returns."""
from tinygrad import Tensor, TinyJit
queues, arrays = tuple(queues), tuple(arrays)
@TinyJit
def clear():
Tensor.realize(*(q.assign(0) for q in queues))
# TinyJit captures on its second call, and one with nothing to run raises
# there. Its kernels compile here, after init_device() turned the compile
# pool off
if queues:
for _ in range(3):
clear()
def reset():
if queues:
clear()
if prev_desire is not None:
prev_desire.fill(0)
for array in arrays:
array.fill(0)
return reset
def reset_for_stock(state, prev_desire=None):
"""The reset of stock modeld's ModelState layout, captured now, as its warmup
zeroes it: the recurrent state on the GPU (input_queues named by
state_pairs) and the one packed host buffer (packed_input), and the desire
edge. `state` is the ModelState, or a modeld_v2 native bundle's adapter,
which keeps the same layout."""
return reset_queues([state.input_queues[name] for name in state.state_pairs], (state.packed_input,), prev_desire)
frames = [np.zeros(frame_size, dtype=np.uint8) for _ in range(2)]
blobs = [Tensor.from_blob(f.ctypes.data, (frame_size,), dtype='uint8', device=Device.DEFAULT) for f in frames]
eye = [np.eye(3, dtype=np.float32) for _ in range(2)]
tfm, big_tfm = (Tensor(e, device='NPY').realize() for e in eye)
for _ in range(2):
call_warp(warp, tfm, big_tfm, blobs[0], blobs[1]).realize()
Device.default.synchronize()
def prepare_reset(model):
"""Capture the small model's reset before driving, keeping its JIT's buffer identities.
Its history is stale after the Jetson ran, so a fallback starts from the
zero history modeld's warmup leaves it at. Nothing is allocated or compiled
on the failure frame, so never through modeld_v2's reset_warmup_buffers(),
which compiles on it. jetlink's own, for an adapter without small_reset
(interface.ModelSide).
Its history is stale after the Jetson ran, so a fallback starts from the same
zero history as modeld startup. Nothing is allocated or compiled on the
failure frame.
The small model is whatever the fork's modeld runs, and each keeps its
history its own way. Stock modeld's ModelState, and a modeld_v2 native
bundle's adapter: the recurrent state on the GPU (state_pairs) and one
packed host buffer (packed_input), as their warmup zeroes them. A modeld_v2
legacy bundle's adapter: history queues on the GPU, and numpy inputs. Any
other ModelState with input_queues (StarPilot's): every queue on the GPU,
and its numpy inputs (numpy_inputs, or npy); its own _reset_state() makes
new tensors, which its JIT is not linked to. The fork's tests pin those names.
The small model is whatever bundle the user picked, stock modeld's or a
modeld_v2 one, so the history is whatever GPU queues it has; the NPY tensors
are zeroed through their numpy views. Duck-typed on openpilot's ModelState:
input_queues, numpy_inputs (modeld_v2) or npy (modeld), prev_desire. The
fork's tests pin those names.
"""
adapter = getattr(model, 'adapter', None)
keeper = model if adapter is None else adapter
prev_desire = getattr(model, 'prev_desire', None)
if adapter.is_native if adapter is not None else hasattr(model, 'state_pairs'):
return reset_for_stock(keeper, prev_desire)
queues = [q for q in keeper.input_queues.values() if q.device != 'NPY']
arrays = (keeper.numpy_inputs if hasattr(keeper, 'numpy_inputs') else keeper.npy).values()
return reset_queues(queues, arrays, prev_desire)
from tinygrad import Tensor, TinyJit
queues = tuple(q for q in model.input_queues.values() if q.device != 'NPY')
npy = model.numpy_inputs if hasattr(model, 'numpy_inputs') else model.npy
@TinyJit
def clear():
Tensor.realize(*(q.assign(0) for q in queues))
for _ in range(3):
clear()
def reset():
clear()
model.prev_desire.fill(0)
for array in npy.values():
array.fill(0)
return reset
# -- the build ------------------------------------------------------------------
def compile_warp(graph, frame_size: int, out: Path) -> Path:
"""JIT the warp graph and pickle it to `out`. Holds the GPU while it runs.
Three runs before pickling: TinyJit captures on the second call, and a
pickle taken earlier is an empty jit that silently does nothing.
"""
import numpy as np
from tinygrad.device import Device
from tinygrad.engine.jit import TinyJit
from tinygrad.tensor import Tensor
warp_jit = TinyJit(graph, prune=True)
# one set of input tensors: TinyJit captures against the buffers it is
# first handed. Random so nothing constant-folds
rng = np.random.default_rng(42)
tfm_npy, big_tfm_npy = np.eye(3, dtype=np.float32), np.eye(3, dtype=np.float32)
tfm = Tensor(tfm_npy, device='NPY')
big_tfm = Tensor(big_tfm_npy, device='NPY')
frame = Tensor.randint(frame_size, low=0, high=256, dtype='uint8', device=Device.DEFAULT).realize()
big_frame = Tensor.randint(frame_size, low=0, high=256, dtype='uint8', device=Device.DEFAULT).realize()
for _ in range(3):
tfm_npy[:] = rng.standard_normal((3, 3)).astype(np.float32)
big_tfm_npy[:] = rng.standard_normal((3, 3)).astype(np.float32)
call_warp(warp_jit, tfm, big_tfm, frame, big_frame).realize()
Device.default.synchronize()
out = Path(out)
out.parent.mkdir(parents=True, exist_ok=True)
# through a temporary: a compile killed mid-write leaves nothing under the
# name load() opens
tmp = out.with_suffix('.pkl.tmp')
with open(tmp, 'wb') as f:
pickle.dump(warp_jit, f)
tmp.replace(out)
return out
def size(text: str) -> tuple[int, int]:
"""'WxH' as (w, h)."""
w, sep, h = text.lower().partition('x')
if not sep:
raise argparse.ArgumentTypeError(f"expected WxH, not {text!r}")
try:
return int(w), int(h)
except ValueError:
raise argparse.ArgumentTypeError(f"expected WxH, not {text!r}") from None
def main(argv: list[str] | None = None) -> None:
"""Build the warp for one camera, as the fork's build runs it."""
from jetlink.openpilot.interface import load_adapter
p = argparse.ArgumentParser(prog='python -m jetlink.openpilot.warp', description=main.__doc__)
p.add_argument('--adapter', required=True, help="the fork's adapter module")
p.add_argument('--camera', type=size, required=True, help='camera resolution, WxH')
p.add_argument('--model', type=size, required=True, help='model input, WxH')
p.add_argument('--output', type=Path, required=True)
args = p.parse_args(argv)
(cam_w, cam_h), (model_w, model_h) = args.camera, args.model
op = load_adapter(args.adapter)
print(f"Compiling jetlink warp for {cam_w}x{cam_h} -> {model_w}x{model_h}...")
# before anything of tinygrad's is imported here: comma's graph module
# patches tinygrad's firmware fetch as it loads
graph, frame_size = op.make_warp(cam_w, cam_h, model_w, model_h)
out = compile_warp(graph, frame_size, args.output)
print(f" Saved to {out}")
if __name__ == "__main__":
sys.exit(main())
+1 -56
View File
@@ -15,7 +15,6 @@ vectored write out, one read into a preallocated buffer back.
"""
from __future__ import annotations
import functools
import struct
from enum import IntEnum
@@ -43,7 +42,7 @@ GADGET_TX_ALIGN = 16 * 1024
# The comma's gadget as a USB host finds it: these IDs (pid.codes' test
# allocation; get a real PID before distributing this), which
# jetlink/comma/jetlink-root.sh presents, then the link's interface by its class
# scripts/comma/jetlink-root.sh presents, then the link's interface by its class
# triple (transport/ffs.py), because the comma set to iOS presents a composite
# gadget and a reordered config could move the number. Endpoint addresses are
# read from the descriptors, never assumed: FunctionFS renumbers them at bind.
@@ -115,54 +114,6 @@ INFER_RESP_SIZE = struct.calcsize(INFER_RESP_FMT)
_infer_resp = struct.Struct(INFER_RESP_FMT)
# A frame over the phone's cable as UDP datagrams (TcpTransport.use_datagrams):
# the INFER_REQ's stream bytes, header and pad included, cut into equal pieces
# of at most DATAGRAM_PAYLOAD, each behind a DATAGRAM_HEADER. The comma's kernel
# cuts a datagram into IP fragments after netfilter and routing, so the stack's
# per-packet work runs once a datagram, about 7 times a frame, where TCP runs it
# for each of ~272 segments: measured 7.35 vs 4.3 ms a frame.
# The server offers it in HELLO_RESP (frame_port, frame_token) on the cable only.
# A datagram with another token is from an earlier session and is dropped.
DATAGRAM_MAGIC = 0x4D52464A # b'JFRM'
# magic, token, seq, offset, total: where this piece goes in a message of `total` bytes
DATAGRAM_HEADER_FMT = '<IIIII'
DATAGRAM_HEADER_SIZE = struct.calcsize(DATAGRAM_HEADER_FMT)
# UDP carries 65507 bytes at most; equal pieces of up to this, 7 for a big model's frame
DATAGRAM_PAYLOAD = 65000
_datagram_header = struct.Struct(DATAGRAM_HEADER_FMT)
def pack_datagram_header(token: int, seq: int, offset: int, total: int) -> bytes:
return _datagram_header.pack(DATAGRAM_MAGIC, token, seq, offset, total)
def pack_datagram_header_into(buf, token: int, seq: int, offset: int, total: int) -> None:
_datagram_header.pack_into(buf, 0, DATAGRAM_MAGIC, token, seq, offset, total)
def unpack_datagram_header(buf) -> tuple[int, int, int, int]:
"""(token, seq, offset, total); ProtocolError for anything not ours."""
magic, token, seq, offset, total = _datagram_header.unpack_from(buf)
if magic != DATAGRAM_MAGIC:
raise ProtocolError(f"bad datagram magic 0x{magic:08x}")
return token, seq, offset, total
def datagram_pieces(total: int) -> tuple[tuple[int, int], ...]:
"""(offset, size) of each datagram for a message of `total` bytes: as few as
DATAGRAM_PAYLOAD allows, equal to within a byte. The same every frame of a
model, so kept."""
return _pieces(total, DATAGRAM_PAYLOAD)
@functools.lru_cache(maxsize=8)
def _pieces(total: int, payload: int) -> tuple[tuple[int, int], ...]:
n = max(1, -(-total // payload))
base, extra = divmod(total, n)
sizes = [base + (1 if i < extra else 0) for i in range(n)]
return tuple((sum(sizes[:i]), size) for i, size in enumerate(sizes))
class ProtocolError(RuntimeError):
pass
@@ -202,9 +153,3 @@ class Status(IntEnum):
BAD_SHAPE = 2
INFER_FAILED = 3
NOT_FINITE = 4 # model produced NaN/Inf; caller must fall back
class NotFinite(ValueError):
"""A frame's output with a NaN or an inf in it, by the server's word
(Status.NOT_FINITE) or the comma's own check: the host answered, with
nothing to drive on, and the link is as good as it was. Not a LinkError."""
+4 -3
View File
@@ -11,8 +11,8 @@ openpilot folds these into the tinygrad JIT on the GPU. Shipping the history
across the link would cost ~10 MB a frame, so the queues live on the server and
the comma sends only the newest warped frame and the packed scalars. The hidden
state the graph returns stays there too: the next frame pushes it into the
features queue, where modeld's prev_feat went. tests/test_queues.py holds this
to openpilot's tinygrad original, and the staging conformance fixture
features queue, where modeld's prev_feat went. The fork's tests/test_queues.py
checks this against the tinygrad original, and the staging conformance fixture
(JetlinkKit/Scripts/make_conformance_fixtures.py) holds the Swift server's
queues to this, bit for bit.
@@ -59,11 +59,12 @@ class PolicyQueues:
did. The queues stay until the client resets them."""
self.prev_feat[...] = 0
def after_run(self, outputs: dict[str, np.ndarray]) -> None:
def after_run(self, outputs: dict[str, np.ndarray], dest=None) -> None:
"""Keep this frame's hidden state for the next one.
Only after a frame whose outputs are all finite: modeld fed back only
what infer_end returned, and it raises on NOT_FINITE and on a failed run.
`dest` is unused; the fork's test_queues still passes it.
"""
self.prev_feat.reshape(-1)[...] = np.asarray(outputs[DRIVING_OUTPUT], np.float32).reshape(-1)[self._hidden]
+3 -25
View File
@@ -33,11 +33,6 @@ CATALOG_URL = CATALOG_URL_TEMPLATE.format(version=CATALOG_VERSION)
# the first that is not there, so a model published later is listed without a
# release of this package. See merge_catalogs.
PROBE_LIMIT = 10
# zoompilot's own big models: ones sunnypilot does not list (yet), such as
# previews from comma's open pull requests, in the catalog's shape. Merged
# after sunnypilot's catalogs, so their entry for the same commit wins. An
# edit on main reaches every comma within the hour, with no release.
EXTRA_CATALOG_URL = 'https://raw.githubusercontent.com/zoompilot/jetlink/refs/heads/main/catalog/extra_big_models.json'
# The selector version the fork requires (REQUIRED_JSON_VERSION on the comma).
# It is a string in the JSON; bundles at any other version describe fields we
# would misread.
@@ -142,9 +137,8 @@ def fetch_catalog(url: str = CATALOG_URL, timeout: float = CATALOG_TIMEOUT, open
def fetch_catalogs(opener=None) -> dict:
"""The pinned catalog merged with every one sunnypilot has published since,
whose commits the server runs too, and then zoompilot's extra models. The
probe stops at the first version that is not there, and a missing extra list
adds nothing. Any other failure raises: a short list would drop a model
whose commits the server runs too. The probe stops at the first version that
is not there. Any other failure raises: a short list would drop a model
found only past it from whatever the caller caches, so the caller keeps its
last complete one instead."""
found = [fetch_catalog(CATALOG_URL, opener=opener)]
@@ -153,23 +147,7 @@ def fetch_catalogs(opener=None) -> dict:
found.append(fetch_catalog(CATALOG_URL_TEMPLATE.format(version=v), opener=opener))
except NotFound:
break
try:
extra = fetch_catalog(EXTRA_CATALOG_URL, opener=opener)
except NotFound:
extra = {}
merged, pins = merge_catalogs([*found, extra]), pinned_pointers(extra)
return {**merged, 'pointers': pins} if pins else merged
def pinned_pointers(data: dict) -> dict[str, dict]:
"""The extra list's well-formed pins, {ref: {'oid', 'size'}}: the ONNX jetlink
runs for that commit whatever a lookup said. See catalog/README.md."""
pins = data.get('pointers')
if not isinstance(pins, dict):
return {}
return {ref: {'oid': p['oid'], 'size': p['size']} for ref, p in pins.items()
if is_ref(ref) and isinstance(p, dict) and is_sha256(p.get('oid'))
and type(p.get('size')) is int and p['size'] > 0}
return merge_catalogs(found)
def merge_catalogs(catalogs: list[dict], selector: int = REQUIRED_SELECTOR_VERSION) -> dict:
+42 -172
View File
@@ -13,19 +13,13 @@ ask a jetlink server for. The bytes themselves are on comma's LFS servers,
which is GitLab and not GitHub: each is asked in turn because which one has an
object varies with the model's age.
Since openpilot #38930 a commit can ship a precompiled tinygrad
Since openpilot #38930 (2026-09-16) a commit can ship a precompiled tinygrad
pkl instead, with no ONNX in the tree at all; Cinque Terre V3 is one. Its
subject names the export ("Use f78ed37d for the precompiled eGPU driving
model") and the ONNX is in comma's HuggingFace model repo, in the folder that
id starts. sunnypilot's model builds find it the same way. That repo speaks
the LFS batch protocol too, so it is one more endpoint to ask.
Not every such commit names its export. ResAction's pull request (#39037)
adds the ONNX in one commit and replaces it with the pkl in the
next, subject "compiled", and a squash merge's subject only names the pull
request. The pointer is still in a diff: the one the commit's own patch
deletes, or the last one its pull request's patch carries.
The zoompilot fork downloads through this module too, asking its own
.lfsconfig endpoint first. No openpilot imports.
"""
@@ -50,14 +44,6 @@ BIG_ONNX = 'big_driving_supercombo.onnx'
POINTER_URL = 'https://raw.githubusercontent.com/commaai/openpilot/{ref}/openpilot/selfdrive/modeld/models/' + BIG_ONNX
# a commit's subject without the API, whose anonymous limit a car behind CGNAT shares
COMMIT_PATCH_URL = 'https://github.com/commaai/openpilot/commit/{ref}.patch'
# every commit of a pull request, one patch after another, for a squash merge's subject
PULL_PATCH_URL = 'https://github.com/commaai/openpilot/pull/{number}.patch'
# how much of a patch is read for the pointers in its diffs; a model commit's is 3 KB
PATCH_MAX = 256 << 10
# the diff header of the big ONNX, wherever the tree keeps selfdrive
_ONNX_DIFF = re.compile(r'^diff --git a/\S*selfdrive/modeld/models/' + re.escape(BIG_ONNX) + r' ')
# a squash merge's subject ends with its pull request
_PULL = re.compile(r'\(#([0-9]+)\)$')
DRIVING_MODELS_REPO = 'commaai/openpilot_driving_models'
DRIVING_MODELS_TREE_URL = f'https://huggingface.co/api/models/{DRIVING_MODELS_REPO}/tree/main'
LFS_ENDPOINTS = (
@@ -67,9 +53,6 @@ LFS_ENDPOINTS = (
)
# an export folder is a uuid; the subject names its first eight
_EXPORT_ID = re.compile(r'\b[0-9a-f]{8}\b')
# how comma's model commits name an export: the whole uuid, or its first eight
# and the checkpoint ('8d736683/12864'). A bare eight could be any hex word
_NAMED_EXPORT = r'\b{}[-/]'
LFS_MEDIA_TYPE = 'application/vnd.git-lfs+json'
POINTER_TIMEOUT = 10.0
CONNECT_TIMEOUT = 30.0
@@ -112,7 +95,7 @@ def parse_pointer_text(text: str) -> Pointer | None:
def fetch_pointer(ref: str, timeout: float = POINTER_TIMEOUT, opener=None) -> Pointer:
"""The oid and size of the ONNX at a comma commit: in its tree, or for a
commit that ships a precompiled pkl instead, the ONNX it was built from."""
commit that ships a precompiled pkl instead, the export its subject names."""
try:
text = http_get(POINTER_URL.format(ref=ref), timeout, opener, POINTER_MAX).decode('utf-8', 'replace')
except NotFound:
@@ -123,11 +106,10 @@ def fetch_pointer(ref: str, timeout: float = POINTER_TIMEOUT, opener=None) -> Po
return pointer
def patch_subjects(text: str) -> list[str]:
"""Every commit's subject in a patch, in order: one for a commit's, one per
commit for a pull request's."""
lines = text.splitlines()
subjects = []
def commit_subject(ref: str, timeout: float = POINTER_TIMEOUT, opener=None) -> str:
"""A comma commit's subject line, from the head of its patch."""
url = COMMIT_PATCH_URL.format(ref=ref)
lines = http_get(url, timeout, opener, POINTER_MAX).decode('utf-8', 'replace').splitlines()
for i, line in enumerate(lines):
if line.startswith('Subject:'):
subject = [line.removeprefix('Subject:').strip()]
@@ -136,42 +118,8 @@ def patch_subjects(text: str) -> list[str]:
if not cont[:1].isspace() or not cont.strip():
break
subject.append(cont.strip())
subjects.append(re.sub(r'^\[PATCH[^\]]*\]\s*', '', ' '.join(subject)))
return subjects
def diff_pointer(text: str) -> Pointer | None:
"""The big ONNX's pointer in a patch's diffs, the last one wins: the pointer a
diff gives the file, or for a diff that deletes it, the pointer it had. A
commit that swaps the ONNX for a pkl deletes the ONNX the pkl was built from."""
found = None
section: list[str] | None = None
def close():
nonlocal found
if section is not None:
added = parse_pointer_text('\n'.join(x[1:] for x in section if x.startswith('+') and not x.startswith('+++')))
removed = parse_pointer_text('\n'.join(x[1:] for x in section if x.startswith('-') and not x.startswith('---')))
found = added or removed or found
for line in text.splitlines():
# each diff, and each commit of a pull request's patch, ends the one before
if line.startswith(('diff --git ', 'From ')):
close()
section = [] if _ONNX_DIFF.match(line) else None
elif section is not None:
section.append(line)
close()
return found
def commit_subject(ref: str, timeout: float = POINTER_TIMEOUT, opener=None) -> str:
"""A comma commit's subject line, from the head of its patch."""
url = COMMIT_PATCH_URL.format(ref=ref)
subjects = patch_subjects(http_get(url, timeout, opener, POINTER_MAX).decode('utf-8', 'replace'))
if not subjects:
raise RegistryError(f"{url} has no subject line")
return subjects[0]
return re.sub(r'^\[PATCH[^\]]*\]\s*', '', ' '.join(subject))
raise RegistryError(f"{url} has no subject line")
def _tree(path: str, timeout: float, opener) -> list[dict]:
@@ -180,68 +128,15 @@ def _tree(path: str, timeout: float, opener) -> list[dict]:
def fetch_export_pointer(ref: str, timeout: float = POINTER_TIMEOUT, opener=None) -> Pointer:
"""The big ONNX a precompiled-pkl commit was built from. In turn: the export
its subject names, in comma's model repo; the pointer its own diff deletes;
and for a squash merge, the last pointer its pull request carried, or the
export one of that pull request's commits names.
The newest export named wins: a pull request's commits are in order, and
Mountain Dew's (#39047) named three before the one it shipped. One that is
not in the model repo yet stops the search rather than falling back to an
older one, a different model."""
url = COMMIT_PATCH_URL.format(ref=ref)
patch = http_get(url, timeout, opener, PATCH_MAX).decode('utf-8', 'replace')
subjects = patch_subjects(patch)
if not subjects:
raise RegistryError(f"{url} has no subject line")
subject = subjects[0]
ids = _newest_first(subject)
pointer, missing = _export_pointer(ref, ids, subject, timeout, opener)
if pointer is not None:
return pointer
tried = [f"its subject {_not_uploaded(missing)}" if missing else f"no folder in {DRIVING_MODELS_REPO} for {', '.join(ids)}" if ids
else 'its subject names no export']
if (pointer := diff_pointer(patch)) is not None:
log.info("%s deletes %s %s", ref[:10], BIG_ONNX, pointer.oid[:16])
return pointer
tried.append(f"its diff has no {BIG_ONNX}")
if (pull := _PULL.search(subject)) is not None:
number = pull.group(1)
pull_patch = http_get(PULL_PATCH_URL.format(number=number), timeout, opener, PATCH_MAX).decode('utf-8', 'replace')
if (pointer := diff_pointer(pull_patch)) is not None:
log.info("%s merged #%s, whose last %s is %s", ref[:10], number, BIG_ONNX, pointer.oid[:16])
return pointer
if missing is None: # an export the subject names and the repo lacks stops the search here too
pull_subjects = ' '.join(patch_subjects(pull_patch))
pull_ids = [i for i in _newest_first(pull_subjects) if i not in ids]
pointer, pull_missing = _export_pointer(ref, pull_ids, pull_subjects, timeout, opener)
if pointer is not None:
return pointer
tried.append(f"pull request #{number} {_not_uploaded(pull_missing)}" if pull_missing else f"nor does pull request #{number}")
raise RegistryError(f"{ref[:10]} has no {BIG_ONNX}: {'; '.join(tried)} ({subject!r})")
def _newest_first(text: str) -> list[str]:
"""The eight-hex ids `text` names, the last mention first."""
return list(dict.fromkeys(reversed(_EXPORT_ID.findall(text))))
def _not_uploaded(export: str) -> str:
return f"names export {export}, which is not in {DRIVING_MODELS_REPO} yet"
def _export_pointer(ref: str, ids: list[str], subject: str, timeout: float, opener) -> tuple[Pointer | None, str | None]:
"""The ONNX in the first export folder one of `ids` starts, and None; or
None and the first id `subject` names as an export that has no folder, where
the search stops; or None and None when no id has a folder."""
"""The big ONNX a precompiled-pkl commit was built from, in comma's model repo."""
subject = commit_subject(ref, timeout=timeout, opener=opener)
ids = list(dict.fromkeys(_EXPORT_ID.findall(subject)))
if not ids:
return None, None
raise RegistryError(f"{ref[:10]} has no {BIG_ONNX} and its subject names no export: {subject!r}")
folders = [e['path'] for e in _tree('', timeout, opener) if e.get('type') == 'directory']
for export in ids:
matches = [f for f in folders if f.startswith(export)]
if not matches:
if re.search(_NAMED_EXPORT.format(export), subject):
return None, export
continue
if len(matches) > 1:
raise RegistryError(f"{ref[:10]}: {export} starts {len(matches)} folders in {DRIVING_MODELS_REPO}")
@@ -257,8 +152,8 @@ def _export_pointer(ref: str, ids: list[str], subject: str, timeout: float, open
if not is_sha256(oid) or not isinstance(size, int) or size <= 0:
raise RegistryError(f"{files[0]['path']} is not an lfs object")
log.info("%s names export %s: %s", ref[:10], export, files[0]['path'])
return Pointer(oid, size), None
return None, None
return Pointer(oid, size)
raise RegistryError(f"{ref[:10]}: no folder in {DRIVING_MODELS_REPO} for {', '.join(ids)}")
def lfs_resolve(endpoint: str, pointer: Pointer, timeout: float = CONNECT_TIMEOUT, opener=None) -> str | None:
@@ -299,50 +194,43 @@ def lfs_download(href: str, pointer: Pointer, dest: Path, progress: ProgressFn |
"""Stream to a .part file, hashing as we go, and only then take the name.
A half-written model must never sit where the next start would hand it to a
backend to build from. A transfer that stalls or is stopped keeps its .part,
and the next call carries on from it with a Range request: 755 MB over a
car's connection need not arrive in one go. Only bytes that prove wrong are dropped.
backend to build from.
"""
opener = opener or urllib.request.urlopen
dest = Path(dest)
dest.parent.mkdir(parents=True, exist_ok=True)
free = shutil.disk_usage(dest.parent).free
if free < pointer.size + FREE_SLACK:
raise RegistryError(f"need {pointer.size >> 20} MB for the model, {free >> 20} MB free")
part = dest.with_name(dest.name + '.part')
digest = hashlib.sha256()
written = _resume_from(part, pointer.size, digest)
free = shutil.disk_usage(dest.parent).free
if free < pointer.size - written + FREE_SLACK:
raise RegistryError(f"need {(pointer.size - written) >> 20} MB more for the model, {free >> 20} MB free")
written = 0
# Whole percent only: a gigabyte at 4 MB a chunk would call this a few
# hundred times and the callback may write a param or a socket line.
reported = -1
if written < pointer.size:
request = urllib.request.Request(href, headers={'Range': f'bytes={written}-'}) if written else href
try:
with opener(request, timeout=CONNECT_TIMEOUT) as response:
if written and response.status != 206:
# the server ignored the Range and sends the whole object
log.warning("%s: no partial content, downloading from the start", pointer.oid[:16])
digest, written = hashlib.sha256(), 0
with open(part, 'ab' if written else 'wb') as out:
while True:
if should_stop is not None and should_stop():
raise RegistryError('download stopped')
chunk = response.read(CHUNK)
if not chunk:
break
out.write(chunk)
digest.update(chunk)
written += len(chunk)
if progress is not None and pointer.size:
percent = int(100 * written / pointer.size)
if percent != reported:
reported = percent
progress(min(1.0, written / pointer.size))
except RegistryError:
raise
except Exception as e:
raise NetworkError(f"could not download {pointer.oid[:16]}: {e}") from e
try:
with opener(href, timeout=CONNECT_TIMEOUT) as response, open(part, 'wb') as out:
while True:
if should_stop is not None and should_stop():
raise RegistryError('download cancelled')
chunk = response.read(CHUNK)
if not chunk:
break
out.write(chunk)
digest.update(chunk)
written += len(chunk)
if progress is not None and pointer.size:
percent = int(100 * written / pointer.size)
if percent != reported:
reported = percent
progress(min(1.0, written / pointer.size))
except RegistryError:
part.unlink(missing_ok=True)
raise
except Exception as e:
part.unlink(missing_ok=True)
raise NetworkError(f"could not download {pointer.oid[:16]}: {e}") from e
if written != pointer.size:
part.unlink(missing_ok=True)
@@ -355,21 +243,3 @@ def lfs_download(href: str, pointer: Pointer, dest: Path, progress: ProgressFn |
if progress is not None:
progress(1.0)
return dest
def _resume_from(part: Path, size: int, digest) -> int:
"""How much of the model an earlier attempt left in `part`, hashed into
`digest`; 0, with the file gone, for anything that cannot be a prefix."""
try:
have = part.stat().st_size
except OSError:
return 0
if have > size:
part.unlink(missing_ok=True)
return 0
buf = bytearray(CHUNK)
view = memoryview(buf)
with open(part, 'rb') as f:
while n := f.readinto(buf):
digest.update(view[:n])
return have
+7 -3
View File
@@ -14,14 +14,14 @@ openpilot #38916. feat_dim is prod(fb[2:]); older forks use fb[2], which gives
32 rather than 16384 for the big model's (1,32,32,512) features_buffer.
tests/test_queues.py catches the drift.
The stateful one (openpilot #38916; Cinque Terre V3 onwards)
The stateful one (openpilot #38916, 2026-09-15; Cinque Terre V3 onwards)
carries its history in the graph: the newest warped frame goes in as new_img,
each queue goes in as state_<q> and comes back advanced as next_state_<q>, and
the hidden state never leaves the graph.
Either way the wire carries the newest frame and three scalars, and the reply
the outputs less hidden_state (protocol 3): the queued graph's server feeds its
own hidden state back, where an older modeld sends prev_feat.
own hidden state back, which modeld did through prev_feat until then.
"""
from __future__ import annotations
@@ -32,6 +32,7 @@ from dataclasses import dataclass
from functools import cached_property
from pathlib import Path
from jetlink.onnx_meta import parse_file
from jetlink.protocol import INFER_REQ_SIZE, INFER_RESP_SIZE
# openpilot ModelConstants; duplicated so the server needs no openpilot import
@@ -140,6 +141,10 @@ class ModelSpec:
def packed_nelem(self) -> int:
return sum(math.prod(s) for s in self.packed_shapes.values())
def feed_back(self, packed, output) -> None:
"""Nothing: the server feeds the hidden state back itself (protocol 3).
Kept for the modeld glue that still calls it after every frame."""
@property
def packed_nbytes(self) -> int:
return self.packed_nelem * 4 # float32
@@ -238,7 +243,6 @@ def sha256_file(path: str, bufsize: int = 1 << 20) -> tuple[str, int]:
def spec_from_onnx(path: str, frame_skip: int = DEFAULT_FRAME_SKIP,
sha256: str | None = None, nbytes: int | None = None) -> ModelSpec:
from jetlink.onnx_meta import parse_file # the scripts' and fixtures': modeld and the owner never parse a graph
meta = parse_file(path)
if sha256 is None or nbytes is None:
sha256, nbytes = sha256_file(path)
-222
View File
@@ -1,222 +0,0 @@
"""
Copyright (c) 2026-, Zeph Leggett.
This file is part of jetlink and is licensed under the MIT License.
See the LICENSE file in the root directory for more details.
Linux native AIO (io_submit), for the gadget's endpoint writes.
A synchronous FunctionFS write holds the endpoint's one usb_request, and
epfile->mutex with it, until the host has taken every byte, so a frame sent as
32 KB writes leaves the bus idle between them: 1.4 ms a frame at 13 writes,
2.8 ms at 25, against one 512 KB write. The AIO path in
ffs_epfile_io gives each iocb its own request, copies the iocb's bytes into it
and queues it before io_submit moves on, then lets go of the mutex. dwc3 puts
every queued request on its TRB ring, so the bus streams them back to back,
and the caller's buffers are free again the moment io_submit returns.
Only what FfsTransport needs: one context, PWRITEV iocbs, and reaping. Linux
only: the comma is aarch64, and CI's x86_64 runs the real thing too.
"""
from __future__ import annotations
import ctypes
import functools
import os
import platform
import time
import numpy as np
# io_setup, io_destroy, io_submit, io_getevents
_SYSCALLS = {'aarch64': (0, 1, 2, 4), 'x86_64': (206, 207, 209, 208)}
IOCB_CMD_PWRITEV = 8
# iovecs one iocb may gather. A request is cut from the message's parts and
# padding; the most it spans is the header, the inference header, the frame,
# the packed inputs and the padding.
IOVECS = 8
class _Iocb(ctypes.Structure):
# struct iocb, little-endian (aio_key before aio_rw_flags)
_fields_ = [('data', ctypes.c_uint64), ('key', ctypes.c_uint32), ('rw_flags', ctypes.c_uint32),
('opcode', ctypes.c_uint16), ('reqprio', ctypes.c_int16), ('fildes', ctypes.c_uint32),
('buf', ctypes.c_uint64), ('nbytes', ctypes.c_uint64), ('offset', ctypes.c_int64),
('reserved2', ctypes.c_uint64), ('flags', ctypes.c_uint32), ('resfd', ctypes.c_uint32)]
class _Event(ctypes.Structure):
_fields_ = [('data', ctypes.c_uint64), ('obj', ctypes.c_uint64), ('res', ctypes.c_int64), ('res2', ctypes.c_int64)]
class _Iovec(ctypes.Structure):
_fields_ = [('base', ctypes.c_void_p), ('len', ctypes.c_size_t)]
class _Timespec(ctypes.Structure):
_fields_ = [('sec', ctypes.c_long), ('nsec', ctypes.c_long)]
class _PyBuffer(ctypes.Structure):
# CPython's Py_buffer, stable since 3.3
_fields_ = [('buf', ctypes.c_void_p), ('obj', ctypes.c_void_p), ('len', ctypes.c_ssize_t),
('itemsize', ctypes.c_ssize_t), ('readonly', ctypes.c_int), ('ndim', ctypes.c_int),
('format', ctypes.c_char_p), ('shape', ctypes.c_void_p), ('strides', ctypes.c_void_p),
('suboffsets', ctypes.c_void_p), ('internal', ctypes.c_void_p)]
_get_buffer = ctypes.pythonapi.PyObject_GetBuffer
_get_buffer.argtypes = [ctypes.py_object, ctypes.POINTER(_PyBuffer), ctypes.c_int]
_get_buffer.restype = ctypes.c_int
_release_buffer = ctypes.pythonapi.PyBuffer_Release
_release_buffer.argtypes = [ctypes.POINTER(_PyBuffer)]
_release_buffer.restype = None
def address(buf) -> tuple[int, int]:
"""Where a contiguous buffer's bytes start, and how many there are;
read-only ones (bytes) included, which ctypes.c_char.from_buffer refuses.
Valid while the caller holds `buf` and nothing resizes it."""
view = _PyBuffer()
_get_buffer(buf, ctypes.byref(view), 0) # PyBUF_SIMPLE: contiguous bytes; raises otherwise
try:
return view.buf or 0, view.len
finally:
_release_buffer(ctypes.byref(view))
class Layout:
"""A message whose spans are `lengths` bytes long, cut into requests of
`size` bytes: for each piece, the request it is in, its iovec within that
request's IOVECS, the span it gathers from and where in it, and its length.
Python loops over the pieces only here; a message of the same shape (every
frame of a link) reuses it through layout()."""
def __init__(self, lengths: tuple[int, ...], size: int):
request, slot, span, offset, length, pieces = [], [], [], [], [], []
r = filled = 0
for j, n in enumerate(lengths):
at = 0
while n:
take = min(n, size - filled)
if len(pieces) == r:
pieces.append(0)
if pieces[r] == IOVECS:
raise ValueError(f'a request gathers more than {IOVECS} buffers')
request.append(r)
slot.append(pieces[r])
span.append(j)
offset.append(at)
length.append(take)
pieces[r] += 1
at, n, filled = at + take, n - take, filled + take
if filled == size:
r, filled = r + 1, 0
self.count = len(pieces)
# where each request's pieces start, and the iovec words they fill
self.first = [0]
for c in pieces:
self.first.append(self.first[-1] + c)
self.words = 2 * (IOVECS * np.array(request, dtype=np.int64) + np.array(slot, dtype=np.int64))
self.span = np.array(span, dtype=np.int64)
self.offset = np.array(offset, dtype=np.uint64)
self.length = np.array(length, dtype=np.uint64)
self.pieces = np.array(pieces, dtype=np.uint64)
@functools.lru_cache(maxsize=8)
def layout(lengths: tuple[int, ...], size: int) -> Layout:
return Layout(lengths, size)
class Aio:
"""A kernel AIO context of `depth` iocbs writing to `fd`, used from one
thread at a time.
submit() queues requests of a Layout, gathered from the spans' addresses
(`bases`), and returns how many the kernel queued; the kernel has copied
those bytes, so the caller may reuse its buffers at once. reap() returns
each finished request's result, bytes written or a negative errno, in no
promised order. The iocb and iovec arrays are reused across submits: the
kernel copies both before io_submit returns.
"""
def __init__(self, fd: int, depth: int):
self._setup, self._destroy, self._submit, self._getevents = _SYSCALLS[platform.machine()]
self._libc = ctypes.CDLL(None, use_errno=True)
self._syscall = self._libc.syscall
self._syscall.restype = ctypes.c_long
self.depth = depth
self._ctx = ctypes.c_ulong(0)
self._call(self._setup, ctypes.c_long(depth), ctypes.byref(self._ctx))
self._iocbs = (_Iocb * depth)()
self._ptrs = (ctypes.POINTER(_Iocb) * depth)(*(ctypes.pointer(c) for c in self._iocbs))
self._iovecs = (_Iovec * (depth * IOVECS))()
iovecs, size = ctypes.addressof(self._iovecs), ctypes.sizeof(_Iovec)
for i, cb in enumerate(self._iocbs):
# everything but the token, the iovec count and the iovecs is fixed
cb.opcode, cb.fildes, cb.buf = IOCB_CMD_PWRITEV, fd, iovecs + i * IOVECS * size
# The per-frame fields as 64-bit words, written with numpy: iocb word 4 is
# the iovec count, an iovec is (address, length), and an event is (data,
# iocb, result, result2).
self._iocb_words = np.frombuffer(self._iocbs, dtype=np.uint64)
self._counts = np.arange(depth) * 8 + 4
self._iovec_words = np.frombuffer(self._iovecs, dtype=np.uint64)
self._events = (_Event * depth)()
self._event_words = memoryview(self._events).cast('B').cast('q')
self._timeout = _Timespec()
def _call(self, nr: int, *args) -> int:
ret = self._syscall(ctypes.c_long(nr), *args)
if ret < 0:
err = ctypes.get_errno()
raise OSError(err, os.strerror(err))
return ret
address = staticmethod(address)
layout = staticmethod(layout)
def submit(self, plan: Layout, bases: list[int], start: int, count: int) -> int:
"""Queue `count` of the plan's requests from `start`, in order, gathered
from the spans at `bases`. Raises OSError only when the kernel took none
of them; a short count means the next one failed, and nothing of it, or of
any after it, was queued."""
if count > self.depth:
raise ValueError(f'{count} requests for a context of {self.depth}')
bases = np.array(bases, dtype=np.uint64)
lo, hi = plan.first[start], plan.first[start + count]
words = plan.words[lo:hi] - 2 * IOVECS * start
self._iovec_words[words] = bases[plan.span[lo:hi]] + plan.offset[lo:hi]
self._iovec_words[words + 1] = plan.length[lo:hi]
self._iocb_words[self._counts[:count]] = plan.pieces[start:start + count] # nbytes counts iovecs for PWRITEV
return self._call(self._submit, self._ctx, ctypes.c_long(count), self._ptrs)
def reap(self, min_nr: int, timeout: float) -> list[int]:
"""Finished requests' results, waiting up to `timeout` s (0: not at all)
for at least `min_nr`; `min_nr` must not exceed what is queued, or this
waits out the whole timeout. The GIL is released while it waits (ctypes).
A signal ends io_getevents with EINTR, and nothing retries a ctypes call
(PEP 475 is os.* only), so this does: modeld's frame thread takes a
SIGUSR2 from msgq ~160 times a second."""
end = time.monotonic() + timeout
while True:
left = max(0.0, end - time.monotonic())
self._timeout.sec = int(left)
self._timeout.nsec = int((left - int(left)) * 1e9)
try:
n = self._call(self._getevents, self._ctx, ctypes.c_long(min_nr), ctypes.c_long(self.depth),
self._events, ctypes.byref(self._timeout))
except InterruptedError:
if time.monotonic() >= end:
return []
continue
return self._event_words[2:4 * n:4].tolist()
def close(self) -> None:
"""Destroy the context. The kernel cancels what is still queued and waits,
uninterruptibly, for it to finish: only call this with nothing in flight,
or after the endpoint has been disabled, which completes everything."""
if self._ctx.value:
ctx, self._ctx = self._ctx, ctypes.c_ulong(0)
self._call(self._destroy, ctx)
+59 -57
View File
@@ -79,33 +79,6 @@ class Transport(ABC):
def send(self, msg_type: int, seq: int, parts=(), flags: int = 0, timeout: float | None = None) -> None:
"""Send one message. `parts` is an iterable of buffers, sent as one message."""
def try_send(self, msg_type: int, seq: int, parts=(), timeout: float | None = None) -> bool:
"""send(), unless the link would first make the caller wait for the peer
to take earlier messages: then False, and nothing was sent. Only a
transport that queues its writes can tell (FfsTransport); the rest send,
within `timeout` as send() does, and say True."""
self.send(msg_type, seq, parts, timeout=timeout)
return True
@property
def datagrams(self) -> bool:
"""Do frames the caller can do without go as datagrams? Only a phone's
cable, once its server offered them (TcpTransport.use_datagrams)."""
return False
def use_datagrams(self, port: int, token: int) -> bool:
"""Send those frames to the server's UDP `port` behind its `token`, where
the link can (TcpTransport); False and nothing changes elsewhere."""
return False
def stop_datagrams(self) -> None:
"""Frames on the link itself again."""
def net_drops(self) -> int | None:
"""Packets the kernel dropped on their way in since the link came up, on a
link that goes through the network stack (TcpTransport); None elsewhere."""
return None
@abstractmethod
def recv(self, timeout: float | None = None) -> Message:
"""Block for one message. Raises LinkTimeout if `timeout` elapses."""
@@ -188,23 +161,34 @@ class RxBuffer:
class StreamTransport(Transport):
"""Framing over any ordered byte stream.
TCP and FunctionFS are both streams, so they both need exactly this.
TCP, USB bulk and FunctionFS are all streams, so they all need exactly this.
Subclasses supply only the two primitives that differ.
"""
# Pad sent messages to a multiple of this; 0 uses the one-byte PADDED rule.
# The peer's messages always use the PADDED rule: only the gadget pads to a
# burst, and only a USB host (the Swift server) reads what it sends. See
# protocol.GADGET_TX_ALIGN.
# Extra capacity past the current message, for transports whose reads need a
# minimum buffer size (a bulk OUT endpoint wants a whole packet).
read_slack = 0
# Bulk endpoints reject a read whose buffer is not a whole number of packets.
# 0 means "no constraint" (TCP).
packet_size = 0
read_chunk = 1 << 20
# Pad sent messages to a multiple of this and expect the peer's padded
# likewise; 0 uses the one-byte PADDED rule. See protocol.GADGET_TX_ALIGN.
tx_align = 0
rx_align = 0
# Largest single write. FunctionFS allocates a contiguous buffer per writev,
# so a 4 MB upload chunk hits ENOMEM on a fragmented device where a few
# hundred KB of inference does not. 0 means no cap (TCP).
write_chunk = 0
def __init__(self, rx_size: int = 1 << 20):
self.rx = RxBuffer(rx_size)
self._desynced = False
self._send_deadline: float | None = None
def _write_timeout(self) -> float | None:
def _write_timeout(self, default: float | None = None) -> float | None:
if self._send_deadline is None:
return None
return default
remaining = self._send_deadline - time.monotonic()
if remaining <= 0:
raise LinkError('send timed out; link abandoned')
@@ -212,10 +196,9 @@ class StreamTransport(Transport):
# -- primitives a subclass must provide ----------------------------------
@abstractmethod
def _write(self, bufs: list[memoryview]) -> int:
"""Write from one or more buffers. Returns bytes written (may be partial).
A transport that overrides send (FfsTransport) need not provide it."""
raise NotImplementedError
"""Write from one or more buffers. Returns bytes written (may be partial)."""
@abstractmethod
def _read_into(self, dest: memoryview, timeout: float | None) -> int:
@@ -228,21 +211,6 @@ class StreamTransport(Transport):
# -- framing -------------------------------------------------------------
def send(self, msg_type: int, seq: int, parts=(), flags: int = 0, timeout: float | None = None) -> None:
bufs = self._frame(msg_type, seq, parts, flags)
self._send_deadline = None if timeout is None else time.monotonic() + timeout
try:
while bufs:
self._write_timeout()
n = self._write(bufs)
if n <= 0:
raise LinkError("peer went away during send")
bufs = advance(bufs, n)
finally:
self._send_deadline = None
def _frame(self, msg_type: int, seq: int, parts, flags: int) -> list[memoryview]:
"""One message as the buffers that go on the wire, in order: the header,
`parts`, then the padding. Nothing is copied."""
# cast('B'): slicing a float32 view in advance() would step by elements
bufs = [memoryview(p).cast('B') for p in parts]
length = sum(b.nbytes for b in bufs)
@@ -255,8 +223,23 @@ class StreamTransport(Transport):
# A bulk transfer only ends on a short packet; see protocol.PACKET_MULTIPLE.
flags |= P.Flag.PADDED
bufs.append(memoryview(_PAD)[:1])
bufs.insert(0, memoryview(P.pack_header(msg_type, seq, length, flags)))
return bufs
header = P.pack_header(msg_type, seq, length, flags)
bufs.insert(0, memoryview(header))
self._send_deadline = None if timeout is None else time.monotonic() + timeout
try:
while bufs:
self._write_timeout()
n = self._write(take(bufs, self.write_chunk) if self.write_chunk else bufs)
if n <= 0:
raise LinkError("peer went away during send")
bufs = advance(bufs, n)
finally:
self._send_deadline = None
def _clamp_read(self, dest: memoryview) -> int:
"""How many bytes this transport may ask for in one read."""
n = min(dest.nbytes, self.read_chunk)
return (n // self.packet_size) * self.packet_size if self.packet_size else n
def _fill(self, need: int, timeout: float | None) -> None:
"""Read until `need` bytes are buffered, or the deadline passes.
@@ -268,10 +251,15 @@ class StreamTransport(Transport):
whatever has already arrived and nothing else: that is how a caller
polls for a reply without blocking (JetlinkClient.drain).
"""
self.rx.reserve(need)
self.rx.reserve(need + self.read_slack)
end = None if timeout is None else time.monotonic() + timeout
while self.rx.available < need:
dest = self.rx.writable()[:need - self.rx.available]
dest = self.rx.writable()[:self._read_limit(need - self.rx.available)]
if self._clamp_read(dest) == 0:
# No room for a whole packet: every read returns 0 and this loop spins
# while the peer blocks. read_slack is too small; say so, do not hang.
raise LinkError(f"no room to read the rest of a {need} byte message "
f"({self.rx.available} in hand); read_slack too small")
remaining = None if end is None else max(0.0, end - time.monotonic())
n = self._read_into(dest, remaining)
if n:
@@ -279,6 +267,17 @@ class StreamTransport(Transport):
elif remaining == 0.0:
raise LinkTimeout(f"only {self.rx.available} of {need} bytes arrived in time")
def _read_limit(self, missing: int) -> int:
"""`missing` bytes of the current message, rounded up to a whole packet.
Keeps the USB host in sync: the gadget's messages are burst-aligned, so a
read for exactly what is left never stays outstanding past the end of a
message. Reading further desynced about once in 400 frames.
"""
if self.packet_size:
return -(-missing // self.packet_size) * self.packet_size
return missing
def recv(self, timeout: float | None = None) -> Message:
if self._desynced:
raise LinkError("stream desynced; the link must be reopened")
@@ -297,7 +296,10 @@ class StreamTransport(Transport):
raise LinkError(f"protocol error, link unusable: {e}") from e
# the remainder of the budget, so one recv cannot block for twice what it
# was given
pad = 1 if flags & P.Flag.PADDED else 0
if self.rx_align:
pad = -(P.HEADER_SIZE + length) % self.rx_align
else:
pad = 1 if flags & P.Flag.PADDED else 0
self._fill(P.HEADER_SIZE + length + pad,
None if end is None else max(0.0, end - time.monotonic()))
self.rx.take(P.HEADER_SIZE)
+134 -263
View File
@@ -11,7 +11,7 @@ has CONFIG_USB_F_FS built in, while a host needs no kernel driver at all, which
an L4T rootfs stripped of the gadget modules does not have. See
docs/transport.md.
Bring-up (jetlink/comma/jetlink-root.sh gadget does all of this):
Bring-up (scripts/comma/jetlink-root.sh gadget does all of this):
configfs gadget -> functions/ffs.jetlink -> mount -t functionfs
write descriptors + strings to ep0 -> ep1 (OUT) and ep2 (IN) appear
echo <udc> > UDC
@@ -28,8 +28,9 @@ import time
from collections import deque
from jetlink import protocol as P
from jetlink.transport.base import UDC_SYSFS, LinkError, LinkTimeout, StreamTransport, udc_speed, usb_link_info
from jetlink.transport.base import UDC_SYSFS, LinkError, LinkTimeout, StreamTransport, take, udc_speed, usb_link_info
from jetlink.transport.priority import background_thread, widen_affinity
from jetlink.transport.watchdog import WriteWatchdog
# --- FunctionFS ABI -------------------------------------------------------
@@ -48,19 +49,17 @@ USB_ENDPOINT_XFER_BULK = 0x02
EP_OUT = 0x01 # host -> device
EP_IN = 0x82 # device -> host
# FunctionFS kmallocs a contiguous buffer per read and frees it on return. 16 KB
# is an order-2 page allocation, which this kernel retries without bound under
# memory pressure; vm.extra_free_kbytes (jetlink-root.sh) keeps that from
# stalling: no reclaim in 3,396 reads while recording. 8 KB would come from
# the slab like the writes, but a reply is 8,356 bytes, so it would take a
# second read: +0.3 ms a frame at p50.
# FunctionFS kmallocs a contiguous buffer per read: order-5 and order-6 failures
# in ffs_epfile_read_iter stalled replies 100-300 ms, and order-2 does not fail.
# Do not go smaller to dodge the rare slow success: one page per read is ~19
# syscalls for a 74 KB reply and cost ~6 ms a frame.
READ_CHUNK = 16 * P.USB_MAX_PACKET
# How much the reader may queue before it stops. Only bounds memory if the
# consumer stalls: inference never needs more than one response.
MAX_QUEUED = 512 << 10
MAX_QUEUED = 8 << 20
# Read buffers kept for reuse. Steady state has one or two in flight, so this is
# only a ceiling for a transient backlog. 8 * 16 KB = 128 KB.
FREE_BUFS = 8
# only a ceiling for a transient backlog. 16 * 16 KB = 256 KB.
FREE_BUFS = 16
# Just below modeld's frame loop (54), so the loop wins a contended core and the
# reader still preempts everything else. See _raise_reader_priority.
READER_RT_PRIORITY = 51
@@ -75,23 +74,11 @@ EP_READY_TIMEOUT = 10.0
# EP_READY_TIMEOUT covers the claim that follows.
EP_OPEN_TIMEOUT = 10.0
# FunctionFS writes cannot time out: a request sits queued until the host
# FunctionFS writes cannot time out: the request sits queued until the host
# drains it, measured at 90 s while a Jetson booted and three minutes on a
# drive. Only unbinding the UDC completes it (ESHUTDOWN). A send waiting for
# the host to take earlier writes waits this long by default.
# drive. Unbinding the UDC dequeues it, the writev returns ESHUTDOWN, and the
# caller retries. A real write is ~3.6 ms.
WRITE_TIMEOUT = 15.0
# Bytes the IN endpoint may hold that the host has not taken. A whole frame is
# 459 KB, so a frame may go out behind one the host has not finished, and a
# third may not (try_send refuses it). The one behind covers a completion the
# kernel has not posted yet as well: FunctionFS posts each from a SCHED_OTHER
# kworker. Uploads stream through it in windows.
QUEUED_LIMIT = 1 << 20
# How long closing waits for queued writes to reach the host (a LEAVE) before
# it drops the gadget to complete them, and how long it then waits for that.
# An endpoint file whose request is still queued stays open in the kernel, and
# the next open of it answers EBUSY.
CLOSE_FLUSH = 0.5
ABORT_DRAIN = 2.0
# where the comma's gadget lives: jetlink-root.sh gadget builds it there and
# mounts its FunctionFS instance here
GADGET = '/sys/kernel/config/usb_gadget/jetlink'
@@ -104,13 +91,10 @@ READER_JOIN_TIMEOUT = 1.0
REBIND_SETTLE = 0.5
_NOT_READY = (errno.EIO, errno.ESHUTDOWN, errno.ENODEV)
# A signal inside a synchronous FunctionFS transfer is not a retry:
# ffs_epfile_io dequeues the request and returns EINTR whatever part of it the
# host already took, os.readv/os.writev re-issue the whole call, and the stream
# has lost or repeated bytes. modeld's msgq raises SIGUSR2 ~160 times a second,
# enough to fail a synchronous-write link about once per 30 s of driving.
# The reader masks every signal for its life. Writes are AIO, which never
# waits inside a transfer, so the frame thread needs no mask.
# A signal inside a FunctionFS transfer is not a retry: ffs_epfile_io returns
# EINTR mid-transfer, os.writev re-issues the whole call, and the host sees the
# start of the message twice. modeld's msgq raises SIGUSR2 ~160 times a second,
# which cost one link failure per 30 s of driving. Nothing here waits on one.
_IO_SIGNALS = signal.valid_signals()
@@ -154,30 +138,20 @@ def build_strings(name: str = 'jetlink') -> bytes:
class FfsTransport(StreamTransport):
"""The gadget transport. Reads run on their own thread; writes are AIO.
"""The gadget transport. Reads run on their own thread.
FunctionFS ignores O_NONBLOCK once the host has enabled the endpoint, so a
read on the caller's thread cannot honour a deadline: a 0.5 s timeout was
measured returning after 14 s. The reader hands whole chunks over a condition
variable the caller can wait on.
A message goes out as AIO requests of write_chunk bytes, all submitted in
one io_submit on the caller's thread (see aio.py): the kernel has copied the
bytes when it returns, the bus streams the requests back to back, and the
caller never waits inside a transfer. What it can wait on is room
(QUEUED_LIMIT): a host that is not taking data fills it, which a frame
answers by holding (try_send) and anything else by waiting, then dropping
the gadget, under its deadline.
variable the caller can wait on. Writes stay on the caller's thread; a host
that is not reading is a dead link either way.
"""
# FunctionFS kmallocs one contiguous buffer per request, and AIO holds a
# frame's worth at once. A whole frame in one needs order-7 pages, which
# recording rollover leaves none of; 57 x 32 KB needs 15 order-3 blocks at
# once, and an aged comma can have as few as 3 free: io_submit then stalls
# ~94 ms in compaction. 8 KB is the largest size
# SLUB serves from a slab cache (kmalloc-8192), which never compacts: it
# falls back to order-1 slabs. A whole number of 1 KB packets, so no request
# ends on a short packet.
write_chunk = P.GADGET_TX_ALIGN // 2
read_chunk = READ_CHUNK
# dwc3 resends a TRB about once in 400 frames. A message that fits one writev
# is replayed whole and the receiver drops it by seq; split across two writes
# the replay lands mid-stream and the link is lost. The largest inference
# request is 400 KB padded, and only uploads, which sha256 covers, are bigger.
write_chunk = 512 * P.USB_MAX_PACKET
tx_align = P.GADGET_TX_ALIGN
def __init__(self, mount: str = MOUNT, gadget: str | None = None,
@@ -211,7 +185,7 @@ class FfsTransport(StreamTransport):
read a dead link left queued keeps them busy: see _open_after_bounce); ep0
never does, which is why the owner keeps it.
The ep0 rule still holds and `udc` is what keeps it: the endpoints are
The ep0 rule is unchanged and `udc` is what keeps it: the endpoints are
still opened only once that controller reads configured. `bounce` is how a
write with no reader is freed, since the unbind that frees it belongs to
the owner. `owner_gadget` is the last resort for when the owner does not
@@ -232,9 +206,9 @@ class FfsTransport(StreamTransport):
def _prepare(self, mount: str, gadget: str | None) -> None:
# This end only receives replies: an INFER_RESP is 8 KB, or 74 KB when
# WANT_HIDDEN asks for the whole output, and the upload goes the other way.
# 128 KB leaves margin the memory-tight comma can spare, and RxBuffer grows
# 256 KB leaves margin the memory-tight comma can spare, and RxBuffer grows
# past it on demand.
super().__init__(rx_size=128 << 10)
super().__init__(rx_size=256 << 10)
self.mount = mount
self.gadget = gadget
self.bound_udc: str | None = None
@@ -243,6 +217,7 @@ class FfsTransport(StreamTransport):
self.ep0 = self.ep_out = self.ep_in = -1
self._state_fd = -1 # held-open UDC 'state' fd; see _udc_state
self._ready_deadline: float | None = None
self._read_size = READ_CHUNK
self._cv = threading.Condition()
self._chunks: deque[tuple[memoryview, float, float, float]] = deque()
self._free: deque[bytearray] = deque() # read buffers the consumer handed back; see _read_loop
@@ -252,18 +227,9 @@ class FfsTransport(StreamTransport):
self._closing = False
self._had_host = False
self._open_lock = threading.Lock()
self.last_send: dict = {}
self.send_totals: dict = {}
self._write_aborted = False
self._reader: threading.Thread | None = None
# the AIO writes: the context once ep2 is open, holding QUEUED_LIMIT in
# requests; requests submitted and reaped over its life, each write_chunk
# bytes (_requests), and when each submit's last one went in; and the
# first failure, which ends the link (see _queue)
self._tx_lock = threading.Lock()
self._aio = None
self._submitted = self._reaped = 0
self._batches: deque[tuple[int, float]] = deque()
self._tx_error: str | None = None
self._write_guard = WriteWatchdog(self._abort_write)
def _udc_state(self) -> str | None:
"""The controller's gadget state, off a held-open fd.
@@ -333,11 +299,6 @@ class FfsTransport(StreamTransport):
raise
self.ep_out = self._open_after_bounce(deadline)
self.ep_in = os.open(os.path.join(self.mount, 'ep2'), os.O_RDWR)
try:
self._aio = _open_aio(self.ep_in, QUEUED_LIMIT // self.write_chunk)
except OSError as e:
self._close_fds(('ep_in', 'ep_out'), None)
raise LinkError(f"no AIO context for the gadget's writes: {e}") from e
self._had_host = True
self._reader = threading.Thread(target=self._read_loop, name='jetlink-ffs-read', daemon=True)
self._reader.start()
@@ -353,8 +314,9 @@ class FfsTransport(StreamTransport):
def _open_after_bounce(self, deadline: float) -> int:
"""ep1 answered EBUSY: a transport this process closed after its link died
left a read queued on it, and FunctionFS keeps that read, and the endpoint
with it, until an unbind completes it, as when a Jetson's server
restarts mid-drive. The unbind is the owner's, so a borrower asks for one, then opens once the
with it, until an unbind completes it. Seen when a Jetson's server
restarted mid-drive: some rejoins met EBUSY until the drive ended. The
unbind is the owner's, so a borrower asks for one, then opens once the
host has the gadget back and the stale read has let go."""
log.warning("jetlink: ep1 is held by a read the last link left queued, asking the owner to bounce the gadget")
if not self._bounce():
@@ -374,8 +336,8 @@ class FfsTransport(StreamTransport):
An exchange leaves a read queued on the endpoint, and FunctionFS keeps it
queued until something completes it: nobody else may read that endpoint
until it does, and the only thing that dequeues it is the unbind. So the
owner gives the endpoints up like this and binds straight back (one
re-enumeration) without ever letting go of ep0, which is the whole point,
owner gives the endpoints up like this and binds straight back - one
re-enumeration - without ever letting go of ep0, which is the whole point,
since the gadget exists only while somebody holds it.
False when there is nothing to give up. Afterwards this transport is bare
@@ -384,7 +346,6 @@ class FfsTransport(StreamTransport):
if self.gadget is None or self._closing or self.ep_out < 0:
return False
udc, self._had_host = self.bound_udc, False
self._settle_writes()
self.unbind() # completes the queued read with ESHUTDOWN
reader, self._reader = self._reader, None
if reader is not None and reader is not threading.current_thread():
@@ -400,7 +361,7 @@ class FfsTransport(StreamTransport):
self._cv.notify_all()
self.rx.start = self.rx.end = 0
self._ready_deadline = None
self._tx_error = None
self._write_aborted = False
time.sleep(REBIND_SETTLE)
self.bind(udc)
return True
@@ -468,186 +429,88 @@ class FfsTransport(StreamTransport):
_close_quietly(self._state_fd)
self._state_fd = -1
def _abort_write(self, why: str = 'USB send exceeded its deadline') -> None:
"""Drop the gadget so the writes the host is not taking complete, with
ESHUTDOWN, and the link with them. Nothing else completes a queued
FunctionFS request (see WRITE_TIMEOUT), and one left queued holds its
endpoint file open in the kernel under the next user of the gadget.
def _shrink(self, attr: str) -> bool:
"""Halve a transfer size after the kernel refused to allocate for one.
ENOMEM here is contiguous memory, so it tracks fragmentation: a size that
worked at boot can fail an hour in, which is what a 1.7 GB model does to a
comma. Slow beats broken, and both directions need it.
"""
self._set_error(f'{why}; link abandoned')
self.last_send['aborted'] = True
self.send_totals['aborts'] = self.send_totals.get('aborts', 0) + 1
log.warning("jetlink: %s, dropping the gadget to free the queued writes", why)
floor = (self.packet_size or P.USB_MAX_PACKET) * 16
current = getattr(self, attr)
if current <= floor:
return False
setattr(self, attr, max(floor, current // 2))
if attr == '_read_size':
self._free.clear() # pooled buffers are the old, larger size now
log.warning("jetlink: gadget could not allocate for %s, dropping to %d KB",
attr, getattr(self, attr) >> 10)
return True
def _shrink_write(self) -> bool:
return self._shrink('write_chunk')
def _abort_write(self) -> None:
"""Take the link down so a write nobody is reading returns. See WRITE_TIMEOUT.
On the watchdog thread: the one stuck in writev cannot act.
"""
self._write_aborted = True
log.warning("jetlink: no reader for %.3f s, dropping the gadget to free the write",
getattr(self, '_write_budget', WRITE_TIMEOUT))
if self._bounce is None:
return self.unbind()
# A borrowed gadget: the unbind belongs to whoever owns ep0, so ask. If
# nobody answers (the owner died, or is wedged itself), take the gadget
# down from here anyway: a re-enumeration costs one rejoin, where requests
# nothing completes cost the endpoint until the comma reboots.
# A borrowed gadget: the unbind that dequeues this belongs to whoever owns
# ep0, so ask. If nobody answers - the owner died, or is wedged itself -
# take the gadget down from here anyway. This runs while a frame thread is
# inside a writev the kernel will never return from on its own, and a
# re-enumeration costs one rejoin where a wedged modeld costs the whole
# drive, the small model included.
if self._bounce():
return
log.error("jetlink: the gadget's owner did not answer; dropping the link from here")
self.unbind(self._owner_gadget)
# -- writes: AIO ---------------------------------------------------------
def send(self, *args, **kwargs) -> None:
# Reset the quantum at each message: a latched shrink would split every
# later request across two writes and re-arm the dwc3 replay (see
# write_chunk). The pressure that forced it is usually gone by the next
# frame.
self.write_chunk = type(self).write_chunk
super().send(*args, **kwargs)
def try_send(self, msg_type: int, seq: int, parts=(), timeout: float | None = None) -> bool:
"""Queue one message unless that means waiting for the host: False, and
nothing sent, while writes it has not taken leave no room for this one
(QUEUED_LIMIT). A frame waits for nothing here, `timeout` included; the
caller holds instead (JetlinkModelState)."""
bufs = self._frame(msg_type, seq, parts, 0)
def _write(self, bufs: list[memoryview]) -> int:
self._ensure_epfiles()
return self._queue(bufs, None, wait=False)
def send(self, msg_type: int, seq: int, parts=(), flags: int = 0, timeout: float | None = None) -> None:
bufs = self._frame(msg_type, seq, parts, flags)
self._ensure_epfiles()
self._queue(bufs, None if timeout is None else time.monotonic() + timeout, wait=True)
def _queue(self, bufs: list[memoryview], deadline: float | None, wait: bool) -> bool:
"""Put one framed message on the IN endpoint as requests of write_chunk
bytes. Returns once the kernel holds every byte, not once the host has
taken them; a failure among earlier requests is this send's LinkError.
`wait` waits for room until `deadline` (WRITE_TIMEOUT from now by
default), then drops the gadget; without it there is no waiting, and
False says the message was not sent."""
started = time.monotonic()
deadline = started + WRITE_TIMEOUT if deadline is None else deadline
total = sum(b.nbytes for b in bufs)
stats = self.last_send = {'bytes': total, 'requests': 0, 'submit_ms': 0.0, 'wait_ms': 0.0,
'backlog_kb': 0, 'backlog_ms': 0.0, 'chunk': self.write_chunk,
'refused': False, 'aborted': False, 'errno': None}
self._write_budget = self._write_timeout(WRITE_TIMEOUT)
# See _IO_SIGNALS: a signal here duplicates data on the wire.
was = signal.pthread_sigmask(signal.SIG_BLOCK, _IO_SIGNALS)
try:
with self._tx_lock:
if self._closing or self._aio is None:
raise LinkError('gadget closing')
self._collect()
if self._tx_error is not None:
raise LinkError(self._tx_error)
if queued := self._submitted - self._reaped:
# what the host had not taken when this message came: the measure
# of a host falling behind, which nothing else on this end can see
stats['backlog_kb'] = queued * self.write_chunk >> 10
stats['backlog_ms'] = (started - self._batches[0][1]) * 1e3
if not wait and (queued + total // self.write_chunk) > self._aio.depth:
stats['refused'] = True
return False
# the views in `bufs` keep these addresses valid until io_submit returns
spans = [self._aio.address(b) for b in bufs if b.nbytes]
plan = self._aio.layout(tuple(n for _, n in spans), self.write_chunk)
self._submit_message(plan, [a for a, _ in spans], deadline, stats)
return True
if not self._write_guard.arm(self._write_budget):
raise LinkError('gadget write watchdog already expired or closed')
while True:
try:
n = os.writev(self.ep_in, bufs)
self._had_host = True
return n
except OSError as e:
if self._write_aborted:
# Say whose doing it was: ENODEV alone reads like a cable falling
# out of a socket.
raise LinkError(f"gadget write had no reader for {self._write_budget:.3f}s") from e
# FunctionFS submits a write as one request, so a failed writev put
# nothing on the wire and is safe to retry.
if e.errno in _NOT_READY and self._wait_for_host_ready():
continue
if e.errno == errno.ENOMEM and self._shrink_write():
# A short write is fine: send() loops until the message is out.
bufs = take(bufs, self.write_chunk)
continue
raise LinkError(f"gadget write failed: {e}{self._udc_note()}") from e
finally:
self._count(stats)
def _submit_message(self, plan, bases: list[int], deadline: float, stats: dict) -> None:
"""io_submit the message's requests (an aio.Layout over the spans at
`bases`) as room allows: all at once for a frame, in windows for an upload
chunk larger than the context."""
done = 0
while done < plan.count:
room = self._aio.depth - (self._submitted - self._reaped)
if room <= 0:
self._wait_for_room(deadline, stats)
continue
done += self._submit(plan, bases, done, min(room, plan.count - done), stats)
def _submit(self, plan, bases: list[int], start: int, count: int, stats: dict) -> int:
"""io_submit `count` of the message's requests from `start`; how many the
kernel queued. FunctionFS reports a failed write as its completion
(_collect): io_submit itself fails only on a bad context or iocb."""
t0 = time.monotonic()
try:
n = self._aio.submit(plan, bases, start, count)
except OSError as e:
stats['errno'] = e.errno
why = f"gadget write failed: {e}{self._udc_note()}"
self._set_error(why)
raise LinkError(why) from e
now = time.monotonic()
stats['submit_ms'] += (now - t0) * 1e3
stats['requests'] += n
self._submitted += n
self._batches.append((self._submitted, now))
return n
def _wait_for_room(self, deadline: float, stats: dict) -> None:
"""Wait for the host to take queued writes, until `deadline`."""
now = time.monotonic()
if now >= deadline:
self._abort_write(f'the host took no USB data for {now - self._batches[0][1]:.2f} s')
raise LinkError(self._tx_error)
self._collect(1, deadline - now)
stats['wait_ms'] += (time.monotonic() - now) * 1e3
if self._tx_error is not None:
raise LinkError(self._tx_error)
def _set_error(self, why: str) -> None:
"""The link's first write failure, which every send after it raises."""
if self._tx_error is None:
self._tx_error = why
def _collect(self, min_nr: int = 0, timeout: float = 0.0) -> None:
"""Reap finished writes, waiting up to `timeout` for `min_nr` of them. The
first that did not write all of its bytes is the link's failure."""
queued = self._submitted - self._reaped
if not queued:
return
results = self._aio.reap(min(min_nr, queued), timeout)
self._reaped += len(results)
while self._batches and self._batches[0][0] <= self._reaped:
self._batches.popleft() # what is left starts with the oldest still queued
chunk = self.write_chunk
if results.count(chunk) != len(results):
bad = next(r for r in results if r != chunk)
self._set_error(f"gadget write failed: {os.strerror(-bad) if bad < 0 else f'{bad} of {chunk} bytes'}"
f"{self._udc_note()}")
def _count(self, stats: dict) -> None:
"""Session totals, which keep a slow send between the 1 Hz log samples."""
totals = self.send_totals
totals['messages'] = totals.get('messages', 0) + 1
for key in ('bytes', 'requests', 'refused'):
totals[key] = totals.get(key, 0) + stats[key]
for key, top in (('submit_ms', 'max_submit_ms'), ('wait_ms', 'max_wait_ms'), ('backlog_ms', 'max_backlog_ms')):
totals[top] = max(totals.get(top, 0.0), stats[key])
def _settle_writes(self) -> None:
"""Before the endpoint files go: give queued writes CLOSE_FLUSH to reach
the host (a LEAVE), drop the gadget to complete the rest, and destroy the
AIO context once nothing is left in it. io_destroy waits, uninterruptibly,
for what it cancels, and FunctionFS cannot cancel a request while the bus
is suspended (dwc3 refuses in LPM), so it is only called on an empty
context; otherwise the context is left to the process."""
if self._aio is None:
return
if not self._tx_lock.acquire(timeout=CLOSE_FLUSH):
# a send is waiting for room the host is not making; this frees it
self._abort_write('closing a link whose host stopped taking USB data')
if not self._tx_lock.acquire(timeout=ABORT_DRAIN):
log.error("jetlink: a send still holds the gadget's writes; leaving their AIO context to the process")
return
try:
self._drain(CLOSE_FLUSH)
if self._submitted > self._reaped:
self._abort_write(f'{self._submitted - self._reaped} USB writes still queued at close')
self._drain(ABORT_DRAIN)
if self._submitted > self._reaped:
log.error("jetlink: %d USB writes never completed; leaving their AIO context to the process",
self._submitted - self._reaped)
else:
self._aio.close()
self._aio = None
self._submitted = self._reaped = 0
self._batches.clear()
finally:
self._tx_lock.release()
def _drain(self, seconds: float) -> None:
end = time.monotonic() + seconds
while self._submitted > self._reaped and (left := end - time.monotonic()) > 0:
self._collect(1, left)
completed = self._write_guard.disarm()
signal.pthread_sigmask(signal.SIG_SETMASK, was)
if not completed:
raise LinkError('gadget write had no reader before deadline; link abandoned')
# -- the reader thread ---------------------------------------------------
@@ -665,8 +528,8 @@ class FfsTransport(StreamTransport):
"""Realtime priority, so a completed read leaves the endpoint at once.
The creator has dropped to SCHED_OTHER, and under recording plus onroad load
the wait to be scheduled after a transfer measures 17 ms mean, 23 ms max,
over budget on read_wait alone. The reader copies a chunk and blocks again, so it
the wait to be scheduled after a transfer was 17 ms mean, 23 ms max - over
budget on read_wait alone. The reader copies a chunk and blocks again, so it
never holds a core. Best effort, and open-coded because jetlink must not
import openpilot.
"""
@@ -676,14 +539,15 @@ class FfsTransport(StreamTransport):
pass
def _read_loop(self) -> None:
# See _IO_SIGNALS.
# As in _write: an interrupted read drops the packets it already took. This
# thread handles no signals, so mask them for its whole life.
signal.pthread_sigmask(signal.SIG_BLOCK, _IO_SIGNALS)
self._widen_affinity()
self._raise_reader_priority()
# Fill the pool up front: recycling alone leaves a gap while a reply's
# chunks arrive and the consumer is a scheduling beat behind, and an
# allocation there reclaims under memory pressure (24 ms, over budget).
self._free.extend(bytearray(READ_CHUNK) for _ in range(FREE_BUFS))
self._free.extend(bytearray(self._read_size) for _ in range(FREE_BUFS))
while not self._closing:
with self._cv:
while self._queued >= MAX_QUEUED and not self._closing:
@@ -702,16 +566,19 @@ class FfsTransport(StreamTransport):
# Reuse a buffer the consumer handed back: a fresh bytearray here was
# measured stalling 20+ ms mid-frame under memory pressure, and the read
# cannot start until it returns. Only this thread pops the free list.
buf = self._free.popleft() if self._free else bytearray(READ_CHUNK)
buf = self._free.popleft() if self._free else bytearray(self._read_size)
read_started = time.monotonic()
try:
# A multiple of the packet size: the OUT endpoint rejects anything else.
# Multiples of the packet size only: the OUT endpoint rejects anything
# else, and _read_size is only ever halved from one.
got = os.readv(self.ep_out, [buf])
except OSError as e:
if self._closing:
return
if e.errno in _NOT_READY and self._wait_for_host_ready():
continue
if e.errno == errno.ENOMEM and self._shrink('_read_size'):
continue
self._fail(f"gadget read failed: {e}{self._udc_note()}")
return
if got == 0:
@@ -754,9 +621,10 @@ class FfsTransport(StreamTransport):
else:
self._chunks.popleft()
# Copied out, so return the buffer to the pool. chunk.obj survives a
# reslice; skip anything not from this pool, such as a test's bytes.
# reslice; skip anything not from this pool at the current size, such
# as a shrunk buffer or a test's bytes.
buf = chunk.obj
if type(buf) is bytearray and len(buf) == READ_CHUNK and len(self._free) < FREE_BUFS:
if type(buf) is bytearray and len(buf) == self._read_size and len(self._free) < FREE_BUFS:
self._free.append(buf)
self._queued -= n
self._cv.notify_all()
@@ -786,7 +654,7 @@ class FfsTransport(StreamTransport):
The endpoints answer ENODEV both before a host configures the gadget and
after it disconnects, and only the first deserves the grace period; giving
it to the second costs a mid-drive disconnect 10 s, about 200 frames. The UDC
it to the second cost 10 s and 201 frames on a mid-drive disconnect. The UDC
state separates them, and it is set in the interrupt that disabled the
endpoint, so it is current when the failed call returns.
"""
@@ -797,7 +665,18 @@ class FfsTransport(StreamTransport):
def close(self) -> None:
self._closing = True
self._settle_writes()
self._write_guard.close()
self._write_guard.thread.join(READER_JOIN_TIMEOUT)
if self._write_guard.thread.is_alive():
# It is stuck inside its own abort, which is already unbinding the
# gadget. This used to raise, to keep a late abort from unbinding a
# gadget the next owner had bound - but the raise came before a single
# fd was closed, so ep0 stayed open for the life of the process and
# every descriptor write after it answered ESRCH. unbind() is
# idempotent, so a late abort finds nothing left to let go of; the leak
# was the worse half by far.
log.warning("jetlink: gadget watchdog teardown did not finish in %.0f s, "
"closing the endpoints anyway", READER_JOIN_TIMEOUT)
# Unbinding disables the endpoints, which completes the reader's pending
# request with ESHUTDOWN and lets the thread exit before its fd goes away.
self.unbind()
@@ -828,14 +707,6 @@ class FfsTransport(StreamTransport):
_close_quietly(fd)
def _open_aio(fd: int, depth: int):
"""The AIO context for ep2's writes (aio.Aio). Imported on first use: the
resident gadget owner imports this module and never writes, and ctypes is
not free."""
from jetlink.transport.aio import Aio
return Aio(fd, depth)
def _close_quietly(fd: int) -> None:
# Runs on its own thread when a read never came back; that thread inherits
# the creator's scheduling, which in modeld is the frame loop's.
+8 -81
View File
@@ -12,19 +12,13 @@ the vendor interface instead. See docs/transport.md.
One sendmsg per message (header and body in one segment train, NODELAY set)
and reads straight into the reusable receive buffer, so the steady state does
not allocate on the wire. Over the cable, frames can go as UDP datagrams
instead (use_datagrams, protocol.DATAGRAM_MAGIC); everything else stays here.
not allocate on the wire.
"""
from __future__ import annotations
import logging
import socket
from collections.abc import Iterator
from jetlink import protocol as P
from jetlink.transport.base import LinkError, StreamTransport, advance, take, udc_speed, usb_link_info
log = logging.getLogger('jetlink.tcp')
from jetlink.transport.base import LinkError, StreamTransport, udc_speed, usb_link_info
DEFAULT_PORT = 5599
# The comma's end of the USB network link a phone dials (jetlink-root.sh gadget --ios).
@@ -38,9 +32,6 @@ class TcpTransport(StreamTransport):
self.sock = sock
self._timeout: float | None = -1.0 # force the first settimeout
_tune(sock)
# (socket connected to the server's frame port, its token); see use_datagrams
self._udp: tuple[socket.socket, int] | None = None
self._drops_at = _softnet_dropped()
def on_the_cable(self) -> bool:
"""Is either end the comma's cable address? Then this is a phone's USB
@@ -54,54 +45,6 @@ class TcpTransport(StreamTransport):
def link_info(self) -> dict:
return usb_link_info('cable', udc_speed()) if self.on_the_cable() else {'kind': 'tcp'}
def net_drops(self) -> int:
"""Every CPU's receive backlog drops since this link came up
(/proc/net/softnet_stat): a receive core starved of time shows here
before anywhere else. -1 where it cannot be read."""
now = _softnet_dropped()
return -1 if now < 0 or self._drops_at < 0 else now - self._drops_at
# -- frames as datagrams ---------------------------------------------------
def use_datagrams(self, port: int, token: int) -> bool:
"""Send frames the caller can do without (send_datagrams) to the peer's
UDP `port` from now on, behind the server's `token`: what its HELLO_RESP
offers over the cable. False, and nothing changes, anywhere else: off the
cable a lost datagram has more than a USB link between it and the phone.
Blocking: the 4 MB send buffer takes a frame whole."""
self.stop_datagrams()
if not self.on_the_cable():
return False
try:
udp = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 4 << 20)
udp.connect((self.sock.getpeername()[0], port))
except OSError as e:
log.warning("jetlink: no frame datagrams to port %d (%s); frames stay on TCP", port, e)
return False
self._udp = (udp, token)
return True
def stop_datagrams(self) -> None:
udp, self._udp = self._udp, None
if udp is not None:
udp[0].close()
@property
def datagrams(self) -> bool:
return self._udp is not None
def send_datagrams(self, msg_type: int, seq: int, parts=(), flags: int = 0) -> None:
"""The message as `datagrams`, one sendmsg each, once use_datagrams took
an offer. A send the kernel refuses (the phone's server gone) fails the
link, as a refused send on the stream does."""
udp, token = self._udp
try:
for piece in datagrams(self._frame(msg_type, seq, parts, flags), token, seq):
udp.sendmsg(piece)
except OSError as e:
raise LinkError(f"frame datagrams failed: {e}") from e
@classmethod
def connect(cls, host: str, port: int = DEFAULT_PORT, timeout: float = 5.0) -> TcpTransport:
return cls(socket.create_connection((host, port), timeout=timeout))
@@ -114,11 +57,16 @@ class TcpTransport(StreamTransport):
srv.listen(backlog)
return srv
@classmethod
def accept(cls, srv: socket.socket) -> tuple[TcpTransport, tuple]:
conn, addr = srv.accept()
return cls(conn), addr
@classmethod
def listen_once(cls, host: str = '0.0.0.0', port: int = DEFAULT_PORT,
timeout: float | None = None) -> tuple[TcpTransport, tuple]:
"""Take exactly one incoming connection and stop listening: a phone that
dials us, for the benchmark and parity scripts run from a Mac."""
dials us, for the bench and parity scripts run from a Mac."""
srv = cls.listen(host, port)
try:
srv.settimeout(timeout)
@@ -161,7 +109,6 @@ class TcpTransport(StreamTransport):
return n
def close(self) -> None:
self.stop_datagrams()
# shut down first: a close alone keeps the connection up while another
# process holds a copy of the socket (the comma's gadget owner holds the
# phone's dial), and the peer hears nothing until that copy goes too
@@ -172,26 +119,6 @@ class TcpTransport(StreamTransport):
pass
def _softnet_dropped(path: str = '/proc/net/softnet_stat') -> int:
"""The kernel's receive backlog drops, all CPUs: the second column, in hex."""
try:
with open(path) as f:
return sum(int(line.split()[1], 16) for line in f if line.strip())
except (OSError, ValueError, IndexError):
return -1
def datagrams(bufs: list[memoryview], token: int, seq: int) -> Iterator[list]:
"""A message's stream bytes `bufs` as protocol.datagram_pieces, each its
header and views into `bufs`. The header's buffer is reused: send or copy
each datagram before taking the next."""
total = sum(b.nbytes for b in bufs)
header = bytearray(P.DATAGRAM_HEADER_SIZE)
for offset, size in P.datagram_pieces(total):
P.pack_datagram_header_into(header, token, seq, offset, total)
yield [header, *take(advance(bufs, offset), size)]
def _tune(sock: socket.socket) -> None:
# NODELAY is the one that matters: without it the header and the body can be
# split across an RTT. Guarded anyway; a transport must not die in setsockopt.
@@ -0,0 +1,71 @@
"""
Copyright (c) 2026-, Zeph Leggett.
This file is part of jetlink and is licensed under the MIT License.
See the LICENSE file in the root directory for more details.
"""
from __future__ import annotations
import threading
import time
from jetlink.transport.priority import background_thread
class WriteWatchdog:
"""One worker per link, with cancellation serialized against expiry.
Once expiry wins, the link cannot be armed again. The callback runs outside
the lock: unbinding a broken USB controller must not block disarm().
"""
def __init__(self, abort):
self._abort = abort
self._cv = threading.Condition()
self._deadline = None
self._expired = False
self._closed = False
self.thread = threading.Thread(target=self._run, name='jetlink-write-guard', daemon=True)
self.thread.start()
def arm(self, budget: float) -> bool:
with self._cv:
if self._expired or self._closed:
return False
if self._deadline is not None:
raise RuntimeError('concurrent writes on one link')
self._deadline = time.monotonic() + budget
self._cv.notify()
return True
def disarm(self) -> bool:
with self._cv:
self._deadline = None
self._cv.notify()
return not self._expired and not self._closed
def close(self) -> None:
with self._cv:
self._closed = True
self._deadline = None
self._cv.notify()
def _run(self) -> None:
# This inherits the creator's policy and pin, which in modeld is the frame
# loop's SCHED_FIFO 54 on one core.
background_thread()
with self._cv:
while not self._closed:
if self._deadline is None:
self._cv.wait()
continue
remaining = self._deadline - time.monotonic()
if remaining > 0:
self._cv.wait(remaining)
continue
self._expired = True
self._deadline = None
break
else:
return
self._abort()
-5
View File
@@ -24,16 +24,11 @@ dev = ["pytest>=7", "ruff", "onnx>=1.16"]
[tool.setuptools.packages.find]
include = ["jetlink*"]
# The root script the comma runs (jetlink/comma/root.py), so a wheel has it too
[tool.setuptools.package-data]
"jetlink.comma" = ["jetlink-root.sh"]
[tool.setuptools.dynamic]
version = { attr = "jetlink.__version__" }
[tool.pytest.ini_options]
testpaths = ["tests"]
markers = ["real_aio: uses Linux AIO itself, not tests/aio_fakes.FakeAio"]
# The fork's rule set (sunnypilot's pyproject.toml), so code moves between the
# two without reformatting, with two differences the fork does not allow:
@@ -6,25 +6,23 @@
# Everything jetlink does as root on the comma, in one place. The owner runs it
# through jetlink/comma/root.py, which is sudo -n and a timeout:
#
# sudo jetlink/comma/jetlink-root.sh gadget # the gadget for a Jetson or a Mac
# sudo jetlink/comma/jetlink-root.sh gadget --ios # the gadget for an iPhone
# sudo jetlink/comma/jetlink-root.sh net # after a bind, iOS only: address, DHCP
# sudo jetlink/comma/jetlink-root.sh rps # the cable's receive work onto core 6, read back
# sudo jetlink/comma/jetlink-root.sh check # what is there now
# sudo jetlink/comma/jetlink-root.sh teardown
# sudo jetlink/comma/jetlink-root.sh port hold|off # the USB-C port held as the device, or let go
# sudo jetlink/comma/jetlink-root.sh port device|reset # ask the far end to host over USB PD, or reset PD
# sudo jetlink/comma/jetlink-root.sh port source # ask the far end for the source role: the comma charges it
# sudo jetlink/comma/jetlink-root.sh udc apply|restore # the USB device controller kept a device and its debug log off, or stock
# sudo jetlink/comma/jetlink-root.sh udc start|stop # its device side turned on, or off
# sudo jetlink/comma/jetlink-root.sh vm apply|restore # the link's VM tuning, or the stock values
# sudo jetlink/comma/jetlink-root.sh draw off|on # no current drawn from the port, or the charger's own
# sudo jetlink/comma/jetlink-root.sh boot SETTING # at boot: draw off while the Jetlink setting's file is not 0
# sudo scripts/comma/jetlink-root.sh gadget # the gadget for a Jetson or a Mac
# sudo scripts/comma/jetlink-root.sh gadget --ios # the gadget for an iPhone
# sudo scripts/comma/jetlink-root.sh net # after a bind, iOS only: address, DHCP
# sudo scripts/comma/jetlink-root.sh check # what is there now
# sudo scripts/comma/jetlink-root.sh teardown
# sudo scripts/comma/jetlink-root.sh port hold|off # the USB-C port held as the device, or let go
# sudo scripts/comma/jetlink-root.sh port device|reset # ask the far end to host over USB PD, or reset PD
# sudo scripts/comma/jetlink-root.sh port source # ask the far end for the source role: the comma charges it
# sudo scripts/comma/jetlink-root.sh udc apply|restore # the USB device controller kept a device, or stock
# sudo scripts/comma/jetlink-root.sh udc start|stop # its device side turned on, or off
# sudo scripts/comma/jetlink-root.sh vm apply|restore # the link's VM tuning, or the stock values
# sudo scripts/comma/jetlink-root.sh draw off|on # no current drawn from the port (iOS), or the charger's own
#
# For the comma four and the comma 3X only. Both are SDM845 on the same AGNOS
# kernel (4.9, dwc3 at a600000.dwc3), with configfs, FunctionFS and NCM built
# in and no modules to load.
# The 3X is assumed from the kernel it shares with the four; only the four is tested.
# The 3X is assumed from the kernel it shares with the four; only the four has been on the bench.
#
# gadget: the USB gadget, chosen by the comma's Jetlink setting. The
# comma is the USB device and the host runs the model; docs/transport.md says
@@ -46,7 +44,7 @@
# $NET_STATUS_FILE; it never fails the gadget.
set -euo pipefail
GADGET=${JETLINK_GADGET:-/sys/kernel/config/usb_gadget/jetlink}
GADGET=/sys/kernel/config/usb_gadget/jetlink
FFS_MOUNT=/dev/ffs-jetlink
FFS_NAME=jetlink
CONFIGFS=/sys/kernel/config
@@ -70,25 +68,6 @@ DHCP_RANGE=192.168.60.2,192.168.60.254,10m
DNSMASQ_PID=/dev/shm/jetlink-dnsmasq.pid
DNSMASQ_IF=/dev/shm/jetlink-dnsmasq.if
DNSMASQ_LEASES=/dev/shm/jetlink-usb0.leases
SYS_NET=${JETLINK_SYS_NET:-/sys/class/net}
# The cable's receive work (RPS): core 6, the one big core openpilot runs no
# real-time thread on. It clocks out the rest of each frame and takes the
# phone's reply, and under load 4.9 hands it to ksoftirqd, which waits behind
# any SCHED_FIFO thread on its core. With f0 (cores 4-7) each connection hashed
# to one of them: 4 is card/controlsd/selfdrived, 5 the ui, 87 and 77 % busy on
# a drive. A ui-like FIFO 53 load on the receive core took the link from 7.6 to
# 12.4 ms (p90 36 to 47 ms round trip); on core 6, 7.7.
# The kernel keeps only the cores online at the write, so with the big cores
# parked it reads 00: the owner steers again once a drive starts.
RPS_CPUS=40
# No connection tracking on the cable. AGNOS builds conntrack and the legacy
# tables in, so every packet to and from the phone would be tracked and walk four
# empty tables: NOTRACK in raw for the cable's netdev takes 0.37 ms off a TCP
# frame's send, and covers the TCP that frame
# datagrams fall back to. The rules carry a comment and are replaced as a set:
# the netdev's name moves between binds, and usb0 is the modem's when it has one.
IPTABLES=${JETLINK_IPTABLES:-iptables-legacy}
TRACK_TAG=jetlink
# port: the charger's DISABLE_POWER_ROLE_SWITCH voter on the PMI8998, the one
# role lever that holds across plugs, and the policy engine's own USB PD
@@ -106,18 +85,11 @@ DUAL_ROLE=${JETLINK_DUAL_ROLE:-/sys/class/dual_role_usb/otg_default}
# its data lines only after USB PD, and at boot no gadget is bound for 10 s.
UDC_GLUE=${JETLINK_UDC_GLUE:-/sys/devices/platform/soc/a600000.ssusb}
PE_PARAMS=${JETLINK_PE_PARAMS:-/sys/module/policy_engine/parameters}
# And Qualcomm's FunctionFS logs four lines to an IPC debug ring for every USB
# request, inside io_submit on modeld's frame thread: about 200 a frame, into
# a buffer only debugfs reads. Turned off while recording, the frame's
# io_submit goes from 1.20 to 0.88 ms and frames from 21.14 to 20.86 ms at
# p50. 1 turns it off; stock is 0.
FFS_LOG_OFF=${JETLINK_FFS_LOG_OFF:-/sys/kernel/debug/ipc_logging/f_fs/log_disable}
USB_PSY=${JETLINK_USB_PSY:-/sys/class/power_supply/usb}
# draw: the charger's USB_ICL voter, the input current limit on the port. As
# the sink the charger draws up to 900 mA from whatever powers the port, though
# the comma runs on its own supply. A 3X browns out the moment it draws from a
# laptop (PMIC power-off reason UVLO). An iPhone powers what it plugs into with
# the comma runs on its own supply. An iPhone powers what it plugs into with
# little current; under that draw its supply sags, the charger's weak-charger
# and reverse-boost handlers cut and restore the input in a loop, and the phone
# never sees a device. Forced to 0 the input is suspended: the port still reads
@@ -127,24 +99,21 @@ USB_ICL_VOTER=${JETLINK_USB_ICL_VOTER:-/sys/kernel/debug/pmic-votable/USB_ICL}
# vm: loggerd's dirty pages pile up until the kernel reclaims them
# synchronously, right while a FunctionFS transfer allocates its buffer: gadget
# reads stall 200-350 ms and the big model falls back. Capping dirty memory
# reads stalled 200-350 ms and the big model fell back. Capping dirty memory
# keeps that reclaim cheap. System-wide, since the gadget read shares the
# kernel with every writer.
# Not vm.min_free_kbytes: a 128 MB floor takes about three times that out of
# MemAvailable, 360 MB on a 3.6 GB comma, and openpilot's LOW MEMORY alert
# reads MemTotal-MemAvailable against 90 %, so a drive at 80 % would alert. vm.extra_free_kbytes (Android's) instead has kswapd keep that much
# more free ahead of the allocators, leaving the direct-reclaim floor where it
# is: 32 MB took 3 minutes of recording from 49 direct-reclaim stalls (the
# longest 148 ms) to none, for 2.7 points on that gauge, as the kernel takes
# it out of MemAvailable about 2.5 times; 64 MB adds nothing but costs 4.5.
# A drive sits near 71 %.
# reads MemTotal-MemAvailable against 90 %. Drives at 80 % showed 90 and
# alerted. The caps alone are not yet re-measured against the stall.
# And the socket buffer caps, for the iPhone's TCP link: transport/tcp.py asks
# for 4 MB, which turns autotuning off, and the kernel clamps it to these, 224
# KB on AGNOS and so 448 KB of buffer. A 393 KB frame in 1448-byte segments does
# not fit, so the send waits on the phone's ACKs: 6 to 50 ms, following its
# round trip. With 4 MB the frame goes in one write and the send takes ~4 ms. Only a socket that asks gets more; one set up
# not fit, so the send waited on the phone's ACKs: 6 to 50 ms, following its
# round trip. With 4 MB the frame goes in one write and the send took ~4 ms
# (a tester's comma, 2026-10-04). Only a socket that asks gets more; one set up
# before the apply keeps the old cap until the phone reconnects.
VM_SYSCTLS=(vm.dirty_bytes=16777216 vm.dirty_background_bytes=8388608 vm.extra_free_kbytes=32768
VM_SYSCTLS=(vm.dirty_bytes=16777216 vm.dirty_background_bytes=8388608
net.core.wmem_max=4194304 net.core.rmem_max=4194304)
PROC_SYS=${JETLINK_PROC_SYS:-/proc/sys}
# the stock values to write back, one key=value a line, recorded by the first
@@ -152,7 +121,7 @@ PROC_SYS=${JETLINK_PROC_SYS:-/proc/sys}
SYSCTL_PREV=${JETLINK_SYSCTL_PREV:-/dev/shm/jetlink-sysctl-prev}
usage() {
echo "usage: $0 gadget [--ios] | net | rps | check | teardown | port hold|off|device|reset|source|sink | udc apply|restore|start|stop | vm apply|restore | draw off|on | boot SETTING" >&2
echo "usage: $0 gadget [--ios] | net | check | teardown | port hold|off|device|reset|source|sink | udc apply|restore|start|stop | vm apply|restore | draw off|on" >&2
exit 2
}
@@ -191,45 +160,10 @@ dnsmasq_alive() {
net_ifname() {
local name
name=$(cat "$GADGET/functions/$NET_FN/ifname" 2>/dev/null || true)
[[ -n "$name" && -d "$SYS_NET/$name" ]] || return 1
[[ -n "$name" && -d "/sys/class/net/$name" ]] || return 1
echo "$name"
}
# The cable's receive work onto RPS_CPUS; 0 only when the mask reads it back.
steer() {
local dev=$1 mask got
mask="$SYS_NET/$dev/queues/rx-0/rps_cpus"
[[ -w "$mask" ]] || { echo "jetlink: $dev has no receive steering to set" >&2; return 1; }
put "$RPS_CPUS" "$mask" "could not steer $dev's receive work" || return 1
got=$(cat "$mask" 2>/dev/null || true)
if [[ -z "$got" ]] || (( 16#$got != 16#$RPS_CPUS )); then
echo "jetlink: $dev's receive steering reads ${got:-nothing}, not $RPS_CPUS (core 6 offline?)" >&2
return 1
fi
}
# Every raw-table rule of ours gone, each found by its comment. Never fatal;
# fails only without the tool, which untrack reads as nothing to do.
retrack() {
local rule
command -v "$IPTABLES" >/dev/null 2>&1 || return 1
while read -r rule; do
[[ -n "$rule" ]] || continue
# word splitting is the point: one rule's arguments
# shellcheck disable=SC2086
"$IPTABLES" -w -t raw $rule 2>/dev/null || true
done < <("$IPTABLES" -w -t raw -S 2>/dev/null | sed -n "/--comment $TRACK_TAG /s/^-A /-D /p")
return 0
}
# The cable's traffic, both ways, out of conntrack: ours replaced by these two.
untrack() {
local dev=$1
retrack || return 0
"$IPTABLES" -w -t raw -I PREROUTING -i "$dev" -m comment --comment "$TRACK_TAG" -j CT --notrack 2>/dev/null || true
"$IPTABLES" -w -t raw -I OUTPUT -o "$dev" -m comment --comment "$TRACK_TAG" -j CT --notrack 2>/dev/null || true
}
# The comma's end of the cable network. Idempotent, and never fatal. The owner
# runs it through net once it has bound the UDC and the interface has appeared.
net_up() {
@@ -251,9 +185,11 @@ net_up() {
net_status "error: $why"; echo "jetlink: $why" >&2
return 0
fi
# parked big cores read back 00 here; the owner steers again onroad
steer "$dev" 2>/dev/null || true
untrack "$dev"
# Steer receive processing onto the big cores: the comma's little cores add
# milliseconds to a 460 KB frame.
if [[ -w "/sys/class/net/$dev/queues/rx-0/rps_cpus" ]]; then
echo f0 > "/sys/class/net/$dev/queues/rx-0/rps_cpus" 2>/dev/null || true
fi
# DHCP for the phone, and only that: no router (option 3) and no DNS (option 6),
# so the phone keeps its default route over Wi-Fi. No DNS service (--port=0).
# dnsmasq binds the interface by name, so a netdev that came back under a new
@@ -286,7 +222,6 @@ net_down() {
kill "$pid" 2>/dev/null || true
fi
rm -f "$DNSMASQ_PID" 2>/dev/null || true
retrack || true
}
# The network function and its DHCP server, gone: for teardown, and for a USB
@@ -362,7 +297,7 @@ cmd_gadget() {
echo 0x02 > bDeviceSubClass
echo 0x01 > bDeviceProtocol
else
# the plain gadget: class per interface
# the plain gadget, as it has always been: class per interface
echo 0x0100 > bcdDevice
echo 0x00 > bDeviceClass
echo 0x00 > bDeviceSubClass
@@ -446,13 +381,6 @@ cmd_net() {
net_up
}
# Not fail(): that is the gadget's record.
cmd_rps() {
local dev
dev=$(net_ifname) || { echo "jetlink: no cable network interface to steer" >&2; exit 1; }
steer "$dev" || exit 1
}
cmd_teardown() {
local f linked=0
net_remove
@@ -571,22 +499,20 @@ cmd_port() {
# it off again once that host is gone.
cmd_udc() {
case "${1:-}" in
apply) udc_set Y 1 || exit 1 ;;
restore) udc_set N 0 || exit 1 ;;
apply) udc_compliance Y || exit 1 ;;
restore) udc_compliance N || exit 1 ;;
start) put peripheral "$UDC_GLUE/mode" "could not start the USB device controller ($UDC_GLUE/mode)" || exit 1 ;;
stop) put none "$UDC_GLUE/mode" "could not stop the USB device controller ($UDC_GLUE/mode)" || exit 1 ;;
*) usage ;;
esac
}
# The compliance modes to $1 and FunctionFS's debug log switch to $2, each on
# its own, so one the kernel lacks does not keep the others.
udc_set() {
# Both knobs, each on its own, so one the kernel lacks does not keep the other.
udc_compliance() {
local failed=0 knob
for knob in "$PE_PARAMS/usb_compliance_mode" "$UDC_GLUE/usb_compliance_mode"; do
put "$1" "$knob" "could not set $knob" || failed=1
done
put "$2" "$FFS_LOG_OFF" "could not set $FFS_LOG_OFF" || failed=1
return $failed
}
@@ -612,50 +538,41 @@ release_voter() {
{ echo "jetlink: could not release $1" >&2; return 1; }
}
# A key's value, or nothing when it cannot be read.
sysctl_read() {
local value=""
{ read -r value < "$PROC_SYS/${1//.//}"; } 2>/dev/null || true
echo "$value"
}
# One key per write, so a value the kernel rejects does not take the rest with it.
sysctl_write() {
put "$2" "$PROC_SYS/${1//.//}" "could not set $1=$2"
}
# apply records the stock values, once a boot, and writes the link's; restore
# writes the record back and drops it, and without one does nothing. Either
# is safe to repeat. A reboot resets them.
# Applied while the link is on, so a device with the link off runs stock
# values. Only restore undoes them, never an exit: a process that restored on
# exit and stopped at ignition would hand every drive the stock values. A
# reboot resets them.
vm_apply() {
local pair key value had="" rec="" failed=0
# A key already in the record keeps its first value: a second apply must
# not record our own. One missing from it (added by a newer jetlink
# since this boot's first apply) is recorded now, while it is stock.
[[ -e "$SYSCTL_PREV" ]] && had=$(cat "$SYSCTL_PREV" 2>/dev/null)
for pair in "${VM_SYSCTLS[@]}"; do
key=${pair%%=*}
if [[ $'\n'"$had" == *$'\n'"$key="* || $'\n'"$had" == *$'\n'"${key%_bytes}_ratio="* ]]; then
continue
local pair key value rec="" failed=0
if [[ ! -e "$SYSCTL_PREV" ]]; then
for pair in "${VM_SYSCTLS[@]}"; do
key=${pair%%=*}
value=""
{ read -r value < "$PROC_SYS/${key//.//}"; } 2>/dev/null || true
if [[ "$value" == 0 && "$key" == *_bytes ]]; then
# Stock AGNOS runs the dirty limits in ratio mode, so the *_bytes keys
# read 0, and the kernel silently drops a 0 written back to one.
# Writing the ratio key is what zeroes the bytes key, so that is the
# one to put back.
key=${key%_bytes}_ratio
value=""
{ read -r value < "$PROC_SYS/${key//.//}"; } 2>/dev/null || true
fi
if [[ -n "$value" ]]; then
rec+="$key=$value"$'\n'
else
echo "jetlink: could not read $key" >&2
fi
done
if [[ -n "$rec" ]]; then
{ printf '%s' "$rec" > "$SYSCTL_PREV" && chmod 0644 "$SYSCTL_PREV"; } 2>/dev/null ||
echo "jetlink: could not record the previous sysctls in $SYSCTL_PREV" >&2
fi
value=$(sysctl_read "$key")
if [[ "$value" == 0 && "$key" == *_bytes ]]; then
# Stock AGNOS runs the dirty limits in ratio mode, so the *_bytes keys
# read 0, and the kernel silently drops a 0 written back to one.
# Writing the ratio key is what zeroes the bytes key, so that is the
# one to put back.
key=${key%_bytes}_ratio
value=$(sysctl_read "$key")
fi
if [[ -n "$value" ]]; then
rec+="$key=$value"$'\n'
else
echo "jetlink: could not read $key" >&2
fi
done
if [[ -n "$rec" ]]; then
{ printf '%s' "$rec" >> "$SYSCTL_PREV" && chmod 0644 "$SYSCTL_PREV"; } 2>/dev/null ||
echo "jetlink: could not record the previous sysctls in $SYSCTL_PREV" >&2
fi
for pair in "${VM_SYSCTLS[@]}"; do
sysctl_write "${pair%%=*}" "${pair#*=}" || failed=1
@@ -688,9 +605,9 @@ cmd_vm() {
esac
}
# The port's input current while the link is on; see USB_ICL_VOTER. off forces
# the limit to 0 and on gives the charger its own limit back; either is safe
# to repeat. A reboot clears it.
# The port's input current while the link is iOS; see USB_ICL_VOTER. off forces
# the limit to 0 and on gives the charger its own limit back. Like udc and vm,
# undone only by on, never at an exit; a reboot clears it.
cmd_draw() {
case "${1:-}" in
off) force_voter "$USB_ICL_VOTER" 0 "to stop drawing from the port" || exit 1 ;;
@@ -699,30 +616,16 @@ cmd_draw() {
esac
}
# The fork's launch script, before manager: with the link on, draw nothing from
# the port from boot rather than from the owner's first step, ~20 s later. The
# comma runs off its own power port. SETTING is the Jetlink setting's params
# file; anything but 0 is on, a missing file is off.
cmd_boot() {
[[ $# -eq 1 ]] || usage
local mode
mode=$(cat "$1" 2>/dev/null) || return 0
[[ -n "$mode" && "$mode" != 0 ]] || return 0
cmd_draw off
}
cmd=${1:-}
if [[ $# -gt 0 ]]; then shift; fi
case "$cmd" in
gadget) cmd_gadget "$@" ;;
net) cmd_net ;;
rps) cmd_rps ;;
check) cmd_check ;;
teardown) cmd_teardown ;;
port) cmd_port "$@" ;;
udc) cmd_udc "$@" ;;
vm) cmd_vm "$@" ;;
draw) cmd_draw "$@" ;;
boot) cmd_boot "$@" ;;
*) usage ;;
esac
-2
View File
@@ -83,8 +83,6 @@ function launch {
mv "$DIR" /data/safe_staging/old_openpilot
mv "${STAGING_ROOT}/finalized" "$DIR"
cd "$DIR"
# releases before the openpilot/ move can't swap AGNOS on newer AGNOS, use ours
[ ! -L system/hardware/tici/agnos.json ] && /data/safe_staging/old_openpilot/openpilot/common/hardware/comma/agnos.py --verify system/hardware/tici/agnos.json && sudo reboot
echo "Restarting launch script ${LAUNCHER_LOCATION}"
unset AGNOS_VERSION
-2
View File
@@ -178,8 +178,6 @@ type Offline struct {
overlap u.Curry[float64]
grid *wayGrid // padded way boxes by cell, see way_index.go
endpoints map[m.Position][]int32 // way indices by first/last node
wayBoxes []m.Box // per-way bounding box, by way index
wayLanes []uint8 // per-way lane tag, by way index
}
func (o *Offline) _box() m.Box {
+2 -34
View File
@@ -348,19 +348,14 @@ func (w *Way) OnWay(location log.GpsLocationData, distanceMultiplier float32) (O
res.OnWay = false
return res, nil
}
max_dist := MatchDistance(location, w.Width(), distanceMultiplier)
if !MayBeWithin(box, pos, max_dist) {
// No point of the way can be close enough; skip the per-segment
// distance. The result is the same OnWay=false the scan would give.
res.OnWay = false
return res, nil
}
d, err := w.DistanceFrom(pos)
res.Distance = d
if err != nil {
res.OnWay = false
return res, errors.Wrap(err, "could not get distance to way")
}
max_dist := max(location.HorizontalAccuracy(), 5) + w.Width()
max_dist *= distanceMultiplier
if d.Distance < max_dist {
res.OnWay = true
@@ -374,33 +369,6 @@ func (w *Way) OnWay(location log.GpsLocationData, distanceMultiplier float32) (O
return res, nil
}
// MatchDistance is the distance from a way, in metres, within which OnWay
// accepts a fix: the fix's accuracy (at least 5 m) plus the road width,
// scaled by the caller's multiplier.
func MatchDistance(location log.GpsLocationData, width float32, distanceMultiplier float32) float32 {
return (max(location.HorizontalAccuracy(), 5) + width) * distanceMultiplier
}
// MayBeWithin is false only when every point of box is at least maxDist
// metres from pos by DistanceTo's haversine. Validated way nodes, and so
// every segment point DistanceFrom measures to, lie inside the way's box.
// The pads are twice the bound: the haversine distance is at least R*dLat
// and at least (2/pi)*R*cos(maxLat)*dLon.
func MayBeWithin(box m.Box, pos m.Position, maxDist float32) bool {
dist := float64(maxDist)
latPad := 2 * dist / ms.R * ms.TO_DEGREES
maxLat := max(math.Abs(box.MinPos.Lat()), math.Abs(box.MaxPos.Lat()), math.Abs(pos.Lat()))
if !(maxLat < 89) {
return true // too close to a pole for the longitude bound; measure instead
}
lonPad := 2 * dist / (ms.R * math.Cos(maxLat*ms.TO_RADIANS)) * ms.TO_DEGREES
padded := m.Box{
MinPos: m.NewPosition(box.MinPos.Lat()-latPad, box.MinPos.Lon()-lonPad),
MaxPos: m.NewPosition(box.MaxPos.Lat()+latPad, box.MaxPos.Lon()+lonPad),
}
return padded.PosInside(pos)
}
func (w *Way) BearingAlignment(location log.GpsLocationData) (float32, error) {
pos := m.NewPosition(location.Latitude(), location.Longitude())
d, err := w.DistanceFrom(pos)
-22
View File
@@ -3,9 +3,7 @@ package maps
import (
"math"
"pfeifer.dev/mapd/cereal/log"
m "pfeifer.dev/mapd/math"
ms "pfeifer.dev/mapd/settings"
)
// Dense tiles hold hundreds of thousands of ways. The road matcher used to
@@ -75,12 +73,10 @@ func buildWayGrid(tile m.Box, ways []m.Box) *wayGrid {
func (o *Offline) buildWayIndexes() {
count := o.Ways.Len()
boxes := make([]m.Box, count)
lanes := make([]uint8, count)
o.endpoints = make(map[m.Position][]int32)
for i := range count {
raw := o.waysRaw.At(i)
boxes[i] = m.Box{MinPos: m.NewPosition(raw.MinLat(), raw.MinLon()), MaxPos: m.NewPosition(raw.MaxLat(), raw.MaxLon())}
lanes[i] = raw.Lanes()
nodes, err := raw.Nodes()
if err != nil || nodes.Len() < 2 {
continue
@@ -94,24 +90,6 @@ func (o *Offline) buildWayIndexes() {
}
}
o.grid = buildWayGrid(o.Box(), boxes)
o.wayBoxes, o.wayLanes = boxes, lanes
}
// WayMayMatch is false only when way i cannot pass OnWay(location,
// distanceMultiplier): the same box and match distance OnWay uses, read from
// the index so a candidate that cannot match is never decoded. true when the
// tile has no index.
func (o *Offline) WayMayMatch(i int, location log.GpsLocationData, distanceMultiplier float32) bool {
if i < 0 || i >= len(o.wayBoxes) {
return true
}
lanes := o.wayLanes[i]
if lanes == 0 {
lanes = 2
}
width := float32(lanes) * ms.Settings.DefaultLaneWidth // as Way.Width
return MayBeWithin(o.wayBoxes[i], m.NewPosition(location.Latitude(), location.Longitude()),
MatchDistance(location, width, distanceMultiplier))
}
// WaysNear returns the indices, in tile order, of every way whose padded box
+1 -13
View File
@@ -104,18 +104,6 @@ func (s *State) ResetRoadEvidence() {
s.lastMapLoadAttempt = time.Time{}
}
// expireRoadEvidence drops everything derived from an expired fix but keeps
// the verified tile. A dense tile can take longer to load than the fix TTL;
// discarding it here made every following fix reload it, so the loop never
// published a match (~600 ms per load, ~60 % CPU, in Las Vegas 2026-10-15).
// Tile currency is still checked every loop by invalidateChangedTile.
func (s *State) expireRoadEvidence() {
s.clearRoadDerived()
s.roadFixMonoTime = 0
s.Position = m.Position{}
s.DistanceSinceLastPosition = 0
}
func (s *State) invalidateChangedTile() bool {
if s.Data.Loaded && !s.Data.SourceCurrentFor(s.SnapshotID) {
s.ResetRoadEvidence()
@@ -141,7 +129,7 @@ func (s *State) ExpireGpsEvidence() {
func (s *State) expireGpsEvidenceAt(now uint64) {
if !cereal.GpsFixFreshAt(cereal.GpsSource(s.GpsSource), s.GpsMonoTime, now) && s.RoadStatus != custom.MapdOut_SampleStatus_noGps {
s.expireRoadEvidence()
s.ResetRoadEvidence()
s.RoadStatus = custom.MapdOut_SampleStatus_noGps
}
}
+2 -14
View File
@@ -298,8 +298,8 @@ func TestSlowTileLoadExpiresGpsBeforePublication(t *testing.T) {
return tile, nil
}, time.Unix(0, 0))
event, out := decodedOutput(t, &s)
if event.Valid() || event.LogMonoTime() != boot || out.RoadStatus() != custom.MapdOut_SampleStatus_noGps || out.GpsMonoTime() != fix || out.GpsSource() != custom.MapdOut_GpsSource_external || out.SourceGeneration() != 7 || out.ProducerSession() == 0 || out.SpeedLimit() != 0 || out.WayId() != 0 || s.DistanceSinceLastPosition != 0 || !s.Data.Loaded {
t.Fatalf("slow loader published stale road or dropped the verified tile: valid=%v status=%v source=%v gps=%d generation=%d limit=%f", event.Valid(), out.RoadStatus(), out.GpsSource(), out.GpsMonoTime(), out.SourceGeneration(), out.SpeedLimit())
if event.Valid() || event.LogMonoTime() != boot || out.RoadStatus() != custom.MapdOut_SampleStatus_noGps || out.GpsMonoTime() != fix || out.GpsSource() != custom.MapdOut_GpsSource_external || out.SourceGeneration() != 7 || out.ProducerSession() == 0 || out.SpeedLimit() != 0 || out.WayId() != 0 || s.DistanceSinceLastPosition != 0 || s.Data.Loaded {
t.Fatalf("slow loader published stale road: valid=%v status=%v source=%v gps=%d generation=%d limit=%f", event.Valid(), out.RoadStatus(), out.GpsSource(), out.GpsMonoTime(), out.SourceGeneration(), out.SpeedLimit())
}
_, seg, err := capnp.NewMessage(capnp.SingleSegment(nil))
if err != nil {
@@ -320,18 +320,6 @@ func TestSlowTileLoadExpiresGpsBeforePublication(t *testing.T) {
if err != nil || path.Len() != 0 || extOut.HasPosition() {
t.Fatal("expired road leaked into extended output")
}
// The next fresh fix matches on the kept tile without loading it again.
next := syntheticSample(t, cereal.GpsSourceExternal, boot+50_000_000, 7, false, true)
boot = next.FixMonoTime + 100_000_000
s.ProcessGps(next, true, func(m.Position) (maps.Offline, error) {
t.Fatal("expired fix discarded the verified tile; the next fix reloaded it")
return maps.Offline{}, nil
}, time.Unix(1, 0))
event, out = decodedOutput(t, &s)
if !event.Valid() || out.RoadStatus() != custom.MapdOut_SampleStatus_matchedLimit || out.GpsMonoTime() != next.FixMonoTime {
t.Fatalf("fresh fix after a slow load did not match: status=%v valid=%v", out.RoadStatus(), event.Valid())
}
}
func TestDelayedSerializationRechecksOriginalFixAge(t *testing.T) {
+3 -2
View File
@@ -192,9 +192,10 @@ func (s *MsgqSubscriber) Read() []byte {
if err != nil {
panic("Msgq Flush Error")
}
start := int64(readPointer.Position) + 8
result := make([]byte, size)
copy(result, s.Msgq.Data[start:start+size])
for i := range size {
result[i] = s.Msgq.Data[int64(readPointer.Position)+8+i]
}
err = s.Msgq.Mem.Flush()
if err != nil {
panic("Msgq Flush Error")
-3
View File
@@ -82,9 +82,6 @@ func plausibleLoadedWays(offlineMaps *maps.Offline, location log.GpsLocationData
if indexed {
i = int(candidates[c])
}
if !offlineMaps.WayMayMatch(i, location, 2) {
continue
}
way := offlineMaps.Ways.At(i)
if way.Id() <= 0 || way.Nodes.Len() < 2 {
continue
-54
View File
@@ -10,7 +10,6 @@ import (
"pfeifer.dev/mapd/cereal/log"
"pfeifer.dev/mapd/maps"
m "pfeifer.dev/mapd/math"
ms "pfeifer.dev/mapd/settings"
)
// denseStreetSpecs lays out a city grid of short blocks plus two long
@@ -185,56 +184,3 @@ func BenchmarkPlausibleWaysDenseTile(b *testing.B) {
}
})
}
// referenceOnWay is OnWay's decision without the match-distance prefilter:
// the 0.01 degree box, then the measured distance.
func referenceOnWay(way *maps.Way, location log.GpsLocationData, multiplier float32) bool {
pos := m.NewPosition(location.Latitude(), location.Longitude())
box := way.Box()
oversized := box.Overlap(0.01)
if !oversized.PosInside(pos) {
return false
}
d, err := way.DistanceFrom(pos)
if err != nil || !(d.Distance < maps.MatchDistance(location, way.Width(), multiplier)) {
return false
}
return maps.IsForward(d.LineStart, d.LineEnd, float64(location.BearingDeg())) || !way.OneWay()
}
// Fixes are scattered around way nodes out to a few match distances, with
// accuracies from 0 to 400 m, so many land just inside or outside the bound.
func TestMatchPrefilterIsExact(t *testing.T) {
tile := packedRoadTile(t, denseStreetSpecs(40))
random := rand.New(rand.NewSource(11))
near, accepted := 0, 0
for range 20000 {
index := random.Intn(tile.Ways.Len())
way := tile.Ways.At(index)
node := way.Nodes.At(random.Intn(way.Nodes.Len()))
accuracy := []float32{0, 3, 5, 12, 40, 400}[random.Intn(6)]
reach := float64(accuracy+30) * 3 / ms.R * ms.TO_DEGREES
lat := node.Lat() + (random.Float64()*2-1)*reach
lon := node.Lon() + (random.Float64()*2-1)*reach
location := gpsAt(t, lat, lon, random.Float64()*359)
location.SetHorizontalAccuracy(accuracy)
for _, multiplier := range []float32{2, way.DistanceMultiplier()} {
want := referenceOnWay(&way, location, multiplier)
got, err := way.OnWay(location, multiplier)
if (err == nil && got.OnWay) != want {
t.Fatalf("way %d at %f,%f accuracy %v x%v: prefiltered %v, reference %v",
way.Id(), lat, lon, accuracy, multiplier, got.OnWay, want)
}
if want && !tile.WayMayMatch(index, location, multiplier) {
t.Fatalf("index prefilter rejected way %d that matches at %f,%f", way.Id(), lat, lon)
}
if want {
accepted++
}
near++
}
}
if accepted == 0 || accepted == near {
t.Fatalf("samples did not straddle the bound: %d of %d matched", accepted, near)
}
}
-1
View File
@@ -1 +0,0 @@
* text=auto eol=lf
+6 -3
View File
@@ -6,6 +6,10 @@ on:
- master
pull_request:
concurrency:
group: ${{ github.workflow }}-${{ github.ref != 'refs/heads/master' && github.ref || github.run_id }}-${{ github.event_name }}
cancel-in-progress: true
jobs:
test:
name: ./test.sh
@@ -13,9 +17,8 @@ jobs:
strategy:
fail-fast: false
matrix:
os: ['macos-latest', 'ubuntu-latest', 'ubuntu-24.04-arm']
os: ['macos-latest', 'ubuntu-latest']
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v4
- run: ./test.sh
timeout-minutes: 1
-48
View File
@@ -1,48 +0,0 @@
name: wheels
on:
pull_request:
release:
types: [published]
workflow_dispatch:
permissions:
contents: read
jobs:
build:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, ubuntu-24.04-arm, macos-15-intel, macos-latest]
steps:
- uses: actions/checkout@v7
- uses: pypa/cibuildwheel@e090b81e30c4d855ea63bf4b6e59204c09a101ae # v4.2.1
- uses: actions/setup-python@v7
with:
python-version: '3.11'
- name: Build source distribution
run: |
python -m pip install build
python -m build --sdist --outdir wheelhouse
- uses: actions/upload-artifact@v7
with:
name: dist-${{ matrix.os }}
path: wheelhouse/*
if-no-files-found: error
publish:
if: github.event_name == 'release'
needs: build
runs-on: ubuntu-latest
environment: pypi
steps:
- uses: actions/download-artifact@v8
with:
pattern: dist-*
merge-multiple: true
path: dist
- uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
with:
password: ${{ secrets.PYPI_PAT }}
+2 -4
View File
@@ -10,13 +10,11 @@ __pycache__
*.a
uv.lock
catch2/
*.egg-info/
/build/
/dist/
msgq.egg-info/
test_runner
libmessaging.*
libmessaging_shared.*
.sconsign.dblite
.mypy_cache/
.mypy_cache/
-21
View File
@@ -1,21 +0,0 @@
MIT License
Copyright (c) comma.ai contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
-4
View File
@@ -1,4 +0,0 @@
include README.md pyproject.toml setup.py SConstruct SConscript
recursive-include site_scons *.py
recursive-include msgq *.py *.pyx *.pxd *.h *.cc
global-exclude *.so *.o *.os *.a *.pyc
+7 -122
View File
@@ -1,122 +1,9 @@
<div align="center" style="text-align: center;">
# MSGQ: A lock free single producer multi consumer message queue
<h1>MSGQ</h1>
<p><b>High-performance <a href="https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern">pub/sub</a> messaging, made simple.<br>For Python, C, and C++.</b></p>
## What is this library?
MSGQ is a generic high performance IPC pub sub system with a single publisher and multiple subscribers. It uses a ring buffer in shared memory to efficiently read and write data. Each read requires a copy. Writing can be done without a copy, as long as the size of the data is known in advance. This library also provides a spoofed implementation that can be used for deterministic testing, and visionipc, an IPC system specifically for large contiguous buffers (like images/video).
<h3>
<a href="#quickstart">Quickstart</a>
<span> · </span>
<a href="#cheatsheet">Cheatsheet</a>
<span> · </span>
<a href="https://github.com/commaai/msgq/tree/master/msgq/examples">Examples</a>
<span> · </span>
<a href="https://discord.comma.ai">Discord</a>
</h3>
[![Discord](https://img.shields.io/badge/Discord-Join-5865F2?logo=discord&logoColor=white)](https://discord.comma.ai)
[![PyPI](https://img.shields.io/pypi/v/msgq-ipc)](https://pypi.org/project/msgq-ipc/)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/commaai/msgq)
[![Tests](https://github.com/commaai/msgq/actions/workflows/tests.yml/badge.svg?branch=master)](https://github.com/commaai/msgq/actions/workflows/tests.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/commaai/msgq/blob/master/LICENSE)
</div>
---
MSGQ lets programs on the same machine exchange messages. A publisher sends messages to a named endpoint, and subscribers listen on that same endpoint. Each endpoint supports one publisher and multiple subscribers.
<p align="center">
<img src="https://github.com/user-attachments/assets/79bb91cf-c9ad-4fb4-97d9-33359a083f0f" alt="1 KiB cross-process ping-pong benchmark"><br>
<sub>1 KiB cross-process ping-pong on x86 Linux. <a href="https://github.com/commaai/msgq/blob/master/msgq/examples/benchmark.py">Benchmark script</a>.</sub>
</p>
## Quickstart
```sh
python -m pip install msgq-ipc
```
Run the included [publisher](https://github.com/commaai/msgq/blob/master/msgq/examples/publisher.py) and [subscriber](https://github.com/commaai/msgq/blob/master/msgq/examples/subscriber.py) examples in separate terminals:
```sh
python -m msgq.examples.publisher --endpoint demo # terminal 1
python -m msgq.examples.subscriber --endpoint demo # terminal 2
```
The subscriber prints `Hello from MSGQ!` once per second.
The core API sends and receives bytes:
```python
import msgq
publisher = msgq.pub_sock("hello")
subscriber = msgq.sub_sock("hello")
publisher.send(b"Hello from MSGQ!")
print(subscriber.receive()) # b'Hello from MSGQ!'
```
## Cheatsheet
```python
import msgq
# API reference; calls are not intended to run in sequence.
# === Create sockets ===
pub = msgq.pub_sock("demo") # One publisher per endpoint
sub = msgq.sub_sock("demo") # Receive messages from that endpoint
sub = msgq.sub_sock("demo", timeout=1000) # Wait up to 1000 milliseconds per receive
sub = msgq.sub_sock("demo", conflate=True) # Receive only the latest available message
# === Send & Receive ===
pub.send(b"hello") # Send nonempty bytes; returns None
sub.receive() # Return bytes; block by default
sub.receive(non_blocking=True) # Return bytes immediately, or None
sub.setTimeout(1000) # Set receive timeout in milliseconds
sub.setTimeout(-1) # Restore indefinite blocking
msgq.drain_sock_raw(sub) # Return a list of all available messages
msgq.drain_sock_raw(sub, wait_for_one=True) # Wait for the first message, then drain
# A receive timeout returns None; draining returns [] if no messages arrive.
# Slow subscribers can miss messages when the ring buffer wraps.
# === Poll multiple subscribers ===
poller = msgq.Poller() # Create a group of subscribers to watch
sub = msgq.sub_sock("demo", poller=poller) # Create and register a subscriber
poller.registerSocket(sub) # Alternatively, register an existing socket
poller.poll(1000) # Return readable sockets; timeout in milliseconds
poller.poll(0) # Check immediately; return [] if none are ready
poller.poll(-1) # Wait indefinitely for a readable socket
# Register each socket once. Call receive() on the sockets returned by poll().
# === Reader synchronization and errors ===
pub.all_readers_updated() # Check whether tracked readers have caught up
pub.wait_for_readers(timeout=1.0, interval=0.01) # Wait for that condition; times are in seconds
msgq.IpcError # Messaging failure exception
msgq.MultiplePublishersError # Publisher conflict; subclass of IpcError
# wait_for_readers() raises TimeoutError if its deadline expires.
# Synchronization checks queue positions, not application processing. It requires
# at least one tracked reader and ignores readers invalidated by an overwrite.
```
## Contributing
Issues and pull requests are welcome on [GitHub](https://github.com/commaai/msgq). Run `./test.sh` to build, lint, and test the package.
## License
MSGQ is available under the [MIT License](https://github.com/commaai/msgq/blob/master/LICENSE).
<details>
<summary>Under the hood</summary>
The message queue copies data on send and receive. A fake implementation is also available for deterministic testing.
### Storage
## Storage
The storage for the queue consists of an area of metadata, and the actual buffer. The metadata contains:
1. A counter to the number of readers that are active
@@ -131,7 +18,7 @@ The counter and the pointer are both 32 bit values, packed into 64 bit so they c
The data buffer is a ring buffer. All messages are prefixed by an 8 byte size field, followed by the data. A size of -1 indicates a wrap-around, and means the next message is stored at the beginning of the buffer.
### Writing
## Writing
Writing involves the following steps:
1. Check if the area that is to be written overlaps with any of the read pointers, mark those readers as invalid by clearing the validity flag.
@@ -142,14 +29,14 @@ In case there is not enough space at the end of the buffer, a special empty mess
There always needs to be 8 bytes of empty space at the end of the buffer. By doing this there is always space to write the -1.
### Reset reader
## Reset reader
When the reader is lagging too much behind the read pointer becomes invalid and no longer points to the beginning of a valid message. To reset a reader to the current write pointer, the following steps are performed:
1. Set valid flag
2. Set read cycle counter to that of the writer
3. Set read pointer to write pointer
### Reading
## Reading
Reading involves the following steps:
1. Read the size field at the current read pointer
@@ -165,5 +52,3 @@ If a writer overwrites the data while it's being copied out, the data will be in
If at steps 2 or 5 the validity flag is not set, the reader is reset. Any data that was already read is discarded. After the reader is reset, the reading starts from the beginning.
If a message with size -1 is encountered, step 3 and 4 are replaced by increasing the cycle counter and setting the read pointer to the beginning of the buffer. After that another read is performed.
</details>
+2
View File
@@ -7,6 +7,8 @@ visionipc_dir = Dir('msgq/visionipc')
msgq_objects = env.SharedObject([
'msgq/ipc.cc',
'msgq/event.cc',
'msgq/impl_msgq.cc',
'msgq/impl_fake.cc',
'msgq/msgq.cc',
])
msgq = env.Library('msgq', msgq_objects)
+1 -2
View File
@@ -7,8 +7,7 @@ arch = subprocess.check_output(["uname", "-m"], encoding='utf8').rstrip()
if platform.system() == "Darwin":
arch = "Darwin"
# shm_open/shm_unlink live in librt on older glibc versions (including manylinux).
common = [] if arch == "Darwin" else ["rt"]
common = []
cpppath = [
"#/",
Binary file not shown.
Binary file not shown.
+11 -5
View File
@@ -1,10 +1,12 @@
from msgq.ipc_pyx import Poller, SubSocket, PubSocket, SocketEventHandle, toggle_fake_events, \
# must be built with scons
from msgq.ipc_pyx import Context, Poller, SubSocket, PubSocket, SocketEventHandle, toggle_fake_events, \
set_fake_prefix, get_fake_prefix, delete_fake_prefix, wait_for_one_event
from msgq.ipc_pyx import MultiplePublishersError, IpcError
from typing import Optional, List, Union
__all__ = [
"Context",
"Poller",
"SubSocket",
"PubSocket",
@@ -17,6 +19,7 @@ __all__ = [
"delete_fake_prefix",
"wait_for_one_event",
"NO_TRAVERSAL_LIMIT",
"context",
"fake_event_handle",
"pub_sock",
"sub_sock",
@@ -25,6 +28,9 @@ __all__ = [
NO_TRAVERSAL_LIMIT = 2**64-1
context = Context()
def fake_event_handle(endpoint: str, identifier: Optional[Union[str, bytes]] = None, override: bool = True, enable: bool = False) -> SocketEventHandle:
ident = identifier if identifier is not None else get_fake_prefix()
handle = SocketEventHandle(endpoint, ident, override)
@@ -35,14 +41,14 @@ def fake_event_handle(endpoint: str, identifier: Optional[Union[str, bytes]] = N
def pub_sock(endpoint: str, segment_size: int = 0) -> PubSocket:
sock = PubSocket()
sock.connect(endpoint, segment_size)
sock.connect(context, endpoint, segment_size)
return sock
def sub_sock(endpoint: str, poller: Optional[Poller] = None, conflate: bool = False,
timeout: Optional[int] = None, segment_size: int = 0) -> SubSocket:
def sub_sock(endpoint: str, poller: Optional[Poller] = None, addr: str = "127.0.0.1",
conflate: bool = False, timeout: Optional[int] = None, segment_size: int = 0) -> SubSocket:
sock = SubSocket()
sock.connect(endpoint, conflate, segment_size)
sock.connect(context, endpoint, addr.encode('utf8'), conflate, segment_size)
if timeout is not None:
sock.setTimeout(timeout)
-292
View File
@@ -1,292 +0,0 @@
# /// script
# requires-python = ">=3.11"
# dependencies = ["msgq-ipc>=1.0", "pyzmq", "eclipse-zenoh>=1.10", "lcm", "matplotlib"]
# [tool.uv.sources]
# msgq-ipc = { path = "../.." }
# [tool.ty.rules]
# unresolved-import = "ignore"
# ///
import os
import lcm
import zmq
import msgq
import time
import zenoh
import random
import argparse
import platform
import tempfile
import matplotlib
import statistics
import multiprocessing
from pathlib import Path
from threading import Event
from collections import deque # codespell:ignore deque
from functools import partial
from contextlib import ExitStack
matplotlib.use("Agg")
import matplotlib.pyplot as plt
BACKENDS = ("msgq", "pyzmq IPC", "Pipe bytes", "zenoh IPC", "LCM UDP")
ROUNDS = 1000
SECONDS = 2
REPEATS = 5
TIMEOUT = 60
def setup_backend(name, role, directory, prefix, barrier, pipe, cleanup, size):
outgoing, incoming = ("request", "reply") if role == "sender" else ("reply", "request")
if name == "msgq":
os.environ["OPENPILOT_PREFIX"] = prefix
publisher = msgq.pub_sock(outgoing)
# Creating a publisher resets its queue, so both must exist before subscribing.
barrier.wait(10)
subscriber = msgq.sub_sock(incoming)
return publisher.send, subscriber.receive
if name == "pyzmq IPC":
context = cleanup.enter_context(zmq.Context())
publisher = context.socket(zmq.XPUB)
subscriber = context.socket(zmq.SUB)
cleanup.callback(publisher.close, linger=0)
cleanup.callback(subscriber.close, linger=0)
publisher.setsockopt(zmq.RCVTIMEO, 5000)
subscriber.setsockopt(zmq.RCVTIMEO, 5000)
subscriber.setsockopt(zmq.SUBSCRIBE, b"")
publisher.bind(f"ipc://{directory}/{outgoing}")
subscriber.connect(f"ipc://{directory}/{incoming}")
if publisher.recv() != b"\x01":
raise RuntimeError("Unexpected ZeroMQ subscription")
return publisher.send, subscriber.recv
if name == "Pipe bytes":
cleanup.enter_context(pipe)
return pipe.send_bytes, pipe.recv_bytes
if name == "zenoh IPC":
endpoint = f"unixsock-stream/{directory}/zenoh"
config = zenoh.Config()
config.insert_json5("listen/endpoints", repr([endpoint] if role == "sender" else []))
config.insert_json5("connect/endpoints", repr([] if role == "sender" else [endpoint]))
config.insert_json5("scouting/multicast/enabled", "false")
config.insert_json5("scouting/gossip/enabled", "false")
config.insert_json5("transport/unicast/qos/enabled", "false")
# Low-latency transport cannot fragment a 64 KiB payload.
config.insert_json5("transport/unicast/lowlatency", str(size <= 1024).lower())
session = cleanup.enter_context(zenoh.open(config))
zenoh_subscriber = session.declare_subscriber(incoming)
cleanup.callback(zenoh_subscriber.undeclare)
publisher = session.declare_publisher(outgoing)
cleanup.callback(publisher.undeclare)
ready = Event()
listener = publisher.declare_matching_listener(lambda status: ready.set() if status.matching else None)
cleanup.callback(listener.undeclare)
if publisher.matching_status.matching:
ready.set()
if not ready.wait(10):
raise RuntimeError("Zenoh subscriber did not become ready")
return publisher.put, lambda: zenoh_subscriber.recv().payload.to_bytes()
if name == "LCM UDP":
bus = lcm.LCM("udpm://239.255.76.67:7667?ttl=0")
messages = deque() # codespell:ignore deque
token = Path(directory).name
subscriber = bus.subscribe(token + incoming, lambda channel, data: messages.append(data))
cleanup.callback(bus.unsubscribe, subscriber)
def receive():
while not messages:
if bus.handle_timeout(5000) == 0:
raise TimeoutError("LCM receive timed out; check host multicast support")
return messages.popleft()
return partial(bus.publish, token + outgoing), receive
raise ValueError(name)
def payload(sequence, padding):
return sequence.to_bytes(8, "little") + padding
def sender(send, receive, size):
padding = b"x" * (size - 8)
sequence = 0
def exchange():
nonlocal sequence
message = payload(sequence, padding)
send(message)
reply = receive()
if reply != message:
raise RuntimeError(f"Invalid reply at sequence {sequence}: {None if reply is None else (len(reply), reply[:8])}")
sequence += 1
start = time.perf_counter()
while sequence < 100 or time.perf_counter() - start < 0.1:
exchange()
send(b"BEGIN")
if receive() != b"BEGIN":
raise RuntimeError("Invalid measurement handshake")
warm_up_count = sequence
start = time.perf_counter()
while True:
for _ in range(64):
exchange()
elapsed = time.perf_counter() - start
count = sequence - warm_up_count
if count >= ROUNDS and elapsed >= SECONDS:
break
# Stop timing before the completion handshake.
send(b"END")
if receive() != b"END":
raise RuntimeError("Invalid completion handshake")
return {"round_trips": count, "seconds": elapsed}
def echo(send, receive, size):
padding = b"x" * (size - 8)
sequence = 0
measured_start = None
while True:
message = receive()
if message == b"BEGIN" and measured_start is None:
measured_start = sequence
send(message)
elif message == b"END" and measured_start is not None:
send(message)
return {"round_trips": sequence - measured_start}
else:
if message != payload(sequence, padding):
raise RuntimeError(f"Invalid request at sequence {sequence}: {None if message is None else (len(message), message[:8])}")
sequence += 1
send(message)
def peer(name, role, directory, prefix, barrier, pipe, control, size):
try:
with ExitStack() as cleanup:
send, receive = setup_backend(name, role, directory, prefix, barrier, pipe, cleanup, size)
barrier.wait(10)
result = sender(send, receive, size) if role == "sender" else echo(send, receive, size)
control.send(("ok", result))
except BaseException as error:
control.send(("error", f"{name} {role}: {type(error).__name__}: {error}"))
finally:
pipe.close()
control.close()
def measure(name, size):
context = multiprocessing.get_context("spawn")
shared_root = "/tmp" if platform.system() == "Darwin" else "/dev/shm"
with tempfile.TemporaryDirectory(prefix="mq_", dir="/tmp") as directory, \
tempfile.TemporaryDirectory(prefix="msgq_", dir=shared_root) as shared:
# An owned prefix isolates and removes MSGQ files without touching existing queues.
prefix = Path(shared).name.removeprefix("msgq_")
barrier = context.Barrier(2)
pipes = context.Pipe()
processes, controls = [], []
deadline = time.monotonic() + TIMEOUT
try:
for role, pipe in zip(("sender", "echo"), pipes, strict=True):
parent, child = context.Pipe(duplex=False)
controls.append(parent)
process = context.Process(target=peer, args=(name, role, directory, prefix, barrier, pipe, child, size))
process.start()
processes.append(process)
child.close()
for pipe in pipes:
pipe.close()
results = {}
while len(results) < 2:
if time.monotonic() >= deadline:
raise TimeoutError(f"{name}: trial exceeded {TIMEOUT}s")
for index, control in enumerate(controls):
if index in results:
continue
if control.poll(0.05):
try:
status, result = control.recv()
except EOFError as error:
raise RuntimeError(f"{name}: peer {index} closed its result pipe unexpectedly") from error
if status != "ok":
raise RuntimeError(result)
results[index] = result
elif processes[index].exitcode is not None:
raise RuntimeError(f"{name}: peer exited without results ({processes[index].exitcode})")
for process in processes:
process.join(max(0, deadline - time.monotonic()))
if process.exitcode != 0:
raise RuntimeError(f"{name}: peer failed to exit cleanly")
result = results[0]
if result["round_trips"] != results[1]["round_trips"]:
raise RuntimeError(f"{name}: peers disagree on verified message count")
return result
finally:
for process in processes:
if process.is_alive():
process.terminate()
for process in processes:
process.join(2)
if process.is_alive():
process.kill()
process.join()
for connection in (*controls, *pipes):
connection.close()
def plot_results(results, path):
from matplotlib.ticker import EngFormatter, MaxNLocator
labels = {"msgq": "msgq", "pyzmq IPC": "pyzmq", "Pipe bytes": "multiprocessing.Pipe", "zenoh IPC": "zenoh", "LCM UDP": "LCM"}
rows = sorted(((name, rate) for name, size, rate in results if size == 1024), key=lambda row: row[1])
maximum = max(rate for _, rate in rows)
formatter = EngFormatter(sep="", places=2)
with plt.rc_context({"font.family": "sans-serif", "font.sans-serif": ["Helvetica Neue", "Arial", "DejaVu Sans"]}):
fig, ax = plt.subplots(figsize=(10, max(2.2, len(rows) * 0.48 + 0.7)))
fig.set_facecolor("white")
ax.set(xlim=(0, maximum * 1.2), ylim=(len(rows) - 0.5, -0.5), yticks=[])
# Keep grid lines out of the values on the right.
ticks = MaxNLocator(nbins=4).tick_values(0, maximum)
ax.set_xticks([tick for tick in ticks if 0 <= tick < maximum])
ax.xaxis.set_major_formatter(EngFormatter(sep=""))
ax.set_xlabel("messages/sec — higher is better", fontsize=14, labelpad=8, color="#172b3a")
ax.tick_params(length=0, pad=6, labelsize=11, colors="#425b6c")
ax.set_axisbelow(True)
ax.grid(axis="x", color="#b7b7b7", linewidth=1.5)
for side, spine in ax.spines.items():
spine.set(visible=side == "left", color="#192a32", linewidth=2)
for index, (name, rate) in enumerate(rows):
gray = str(0.88 - 0.28 * index / max(len(rows) - 2, 1))
ax.barh(index, rate, height=1, color="#31cf43" if name == "msgq" else gray, zorder=2)
ax.text(maximum * 0.029, index, labels[name], va="center", fontsize=16, fontweight="bold", color="#172b3a")
ax.text(maximum * 1.173, index, formatter(rate), ha="right", va="center", fontsize=16, fontweight="bold", color="#172b3a")
fig.subplots_adjust(left=0.055, right=0.98, top=0.98, bottom=0.21)
path.parent.mkdir(parents=True, exist_ok=True)
fig.savefig(path, dpi=180, facecolor=fig.get_facecolor())
plt.close(fig)
print(f"Chart saved to {path}")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Cross-process pub/sub ping-pong benchmark.")
parser.add_argument("--plot", type=Path, help="save a 1 KiB chart")
args = parser.parse_args()
if "CEREAL_FAKE" in os.environ:
parser.error("Unset CEREAL_FAKE to benchmark the real MSGQ backend")
print("Cross-process ping-pong; messages/sec counts requests and replies.", flush=True)
samples = {(name, size): [] for name in BACKENDS for size in (64, 1024, 65536)}
randomizer = random.Random(0)
for _ in range(REPEATS):
jobs = list(samples)
randomizer.shuffle(jobs)
for name, size in jobs:
sample = measure(name, size)
rate = 2 * sample["round_trips"] / sample["seconds"]
samples[name, size].append(rate)
print(f"{name:<12} {size:>6} bytes: {rate:>12,.0f} messages/sec", flush=True)
results = [(name, size, statistics.median(rates)) for (name, size), rates in samples.items()]
print("\nMedians:")
for name, size, rate in results:
print(f"{name:<12} {size:>6} bytes: {rate:>12,.0f} messages/sec")
if args.plot:
plot_results(results, args.plot)
-21
View File
@@ -1,21 +0,0 @@
import argparse
import time
import msgq
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Publish a greeting once per second.")
parser.add_argument("--endpoint", default="msgq_example", help="endpoint name (default: %(default)s)")
args = parser.parse_args()
publisher = msgq.pub_sock(args.endpoint)
print("Ctrl-C to exit")
try:
while True:
message = "Hello from MSGQ!"
publisher.send(message.encode("utf-8"))
print(f"Sent: {message}", flush=True)
time.sleep(1)
except KeyboardInterrupt:
pass
-19
View File
@@ -1,19 +0,0 @@
import argparse
import msgq
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Receive and print greetings from a publisher.")
parser.add_argument("--endpoint", default="msgq_example", help="endpoint name (default: %(default)s)")
args = parser.parse_args()
subscriber = msgq.sub_sock(args.endpoint, timeout=1000)
print("Ctrl-C to exit")
try:
while True:
message = subscriber.receive()
if message is not None:
print(f"Received: {message.decode('utf-8')}", flush=True)
except KeyboardInterrupt:
pass
+10
View File
@@ -0,0 +1,10 @@
#include <vector>
#include "msgq/impl_fake.h"
void FakePoller::registerSocket(SubSocket *socket) {
this->sockets.push_back(socket);
}
std::vector<SubSocket*> FakePoller::poll(int timeout) {
return this->sockets;
}
+75
View File
@@ -0,0 +1,75 @@
#pragma once
#include <cassert>
#include <iostream>
#include <string>
#include <vector>
#include <filesystem>
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include "msgq/ipc.h"
#include "msgq/event.h"
template<typename TSubSocket>
class FakeSubSocket: public TSubSocket {
private:
EventState *state = nullptr;
int fds[2] = {-1, -1};
void ensure_fifos_open() {
for (size_t i = 0; i < 2; i++) {
if (fds[i] < 0 && state->paths[i][0] != '\0') {
fds[i] = open(state->paths[i], O_RDWR | O_NONBLOCK);
}
}
}
public:
FakeSubSocket(): TSubSocket() {}
~FakeSubSocket() {
for (int fd : fds) {
if (fd >= 0) close(fd);
}
if (state != nullptr) {
munmap(state, sizeof(EventState));
}
}
int connect(Context *context, std::string endpoint, std::string address, bool conflate=false, bool check_endpoint=true, size_t segment_size=0) override {
const char* cereal_prefix = std::getenv("CEREAL_FAKE_PREFIX");
char* mem;
std::string identifier = cereal_prefix != nullptr ? std::string(cereal_prefix) : "";
event_state_shm_mmap(endpoint, identifier, &mem, nullptr);
this->state = (EventState*)mem;
ensure_fifos_open();
return TSubSocket::connect(context, endpoint, address, conflate, check_endpoint, segment_size);
}
Message *receive(bool non_blocking=false) override {
if (this->state->enabled) {
ensure_fifos_open();
Event(fds[EventPurpose::RECV_CALLED]).set();
Event(fds[EventPurpose::RECV_READY]).wait();
Event(fds[EventPurpose::RECV_READY]).clear();
}
return TSubSocket::receive(non_blocking);
}
};
class FakePoller: public Poller {
private:
std::vector<SubSocket*> sockets;
public:
void registerSocket(SubSocket *socket) override;
std::vector<SubSocket*> poll(int timeout) override;
~FakePoller() {}
};
+169
View File
@@ -0,0 +1,169 @@
#include <cassert>
#include <cstring>
#include <iostream>
#include <cstdlib>
#include <cerrno>
#include <string>
#include <vector>
#include "msgq/impl_msgq.h"
void Message::init(size_t sz) {
size = sz;
data = new char[size];
}
void Message::init(char * d, size_t sz) {
size = sz;
data = new char[size];
memcpy(data, d, size);
}
void Message::takeOwnership(char * d, size_t sz) {
size = sz;
data = d;
}
void Message::close() {
if (size > 0){
delete[] data;
}
size = 0;
}
Message::~Message() {
this->close();
}
int MSGQSubSocket::connect(Context *context, std::string endpoint, std::string address, bool conflate, bool check_endpoint, size_t segment_size){
assert(context);
assert(address == "127.0.0.1");
q = new msgq_queue_t;
size_t size = segment_size > 0 ? segment_size : DEFAULT_SEGMENT_SIZE;
int r = msgq_new_queue(q, endpoint.c_str(), size);
if (r != 0){
return r;
}
msgq_init_subscriber(q);
if (conflate){
q->read_conflate = true;
}
timeout = -1;
return 0;
}
Message * MSGQSubSocket::receive(bool non_blocking){
msgq_msg_t msg;
Message *r = NULL;
int rc = msgq_msg_recv(&msg, q);
// Hack to implement blocking read with a poller. Don't use this
while (!non_blocking && rc == 0){
msgq_pollitem_t items[1];
items[0].q = q;
int t = (timeout != -1) ? timeout : 100;
int n = msgq_poll(items, 1, t);
rc = msgq_msg_recv(&msg, q);
// The poll indicated a message was ready, but the receive failed. Try again
if (n == 1 && rc == 0){
continue;
}
if (timeout != -1){
break;
}
}
if (rc > 0){
r = new Message;
r->takeOwnership(msg.data, msg.size);
}
return r;
}
void MSGQSubSocket::setTimeout(int t){
timeout = t;
}
MSGQSubSocket::~MSGQSubSocket(){
if (q != NULL){
msgq_close_queue(q);
delete q;
}
}
int PubSocket::connect(Context *context, std::string endpoint, bool check_endpoint, size_t segment_size){
assert(context);
q = new msgq_queue_t;
size_t size = segment_size > 0 ? segment_size : DEFAULT_SEGMENT_SIZE;
int r = msgq_new_queue(q, endpoint.c_str(), size);
if (r != 0){
return r;
}
msgq_init_publisher(q);
return 0;
}
int PubSocket::sendMessage(Message *message){
msgq_msg_t msg;
msg.data = message->getData();
msg.size = message->getSize();
return msgq_msg_send(&msg, q);
}
int PubSocket::send(char *data, size_t size){
msgq_msg_t msg;
msg.data = data;
msg.size = size;
return msgq_msg_send(&msg, q);
}
bool PubSocket::all_readers_updated() {
return msgq_all_readers_updated(q);
}
PubSocket::~PubSocket(){
if (q != NULL){
msgq_close_queue(q);
delete q;
}
}
void MSGQPoller::registerSocket(SubSocket * socket){
assert(num_polls + 1 < MAX_POLLERS);
polls[num_polls].q = static_cast<MSGQSubSocket*>(socket)->getQueue();
sockets.push_back(socket);
num_polls++;
}
std::vector<SubSocket*> MSGQPoller::poll(int timeout){
std::vector<SubSocket*> r;
msgq_poll(polls, num_polls, timeout);
for (size_t i = 0; i < num_polls; i++){
if (polls[i].revents){
r.push_back(sockets[i]);
}
}
return r;
}
+33
View File
@@ -0,0 +1,33 @@
#pragma once
#include <string>
#include <vector>
#include "msgq/ipc.h"
#include "msgq/msgq.h"
#define MAX_POLLERS 128
class MSGQSubSocket : public SubSocket {
private:
msgq_queue_t * q = NULL;
int timeout;
public:
int connect(Context *context, std::string endpoint, std::string address, bool conflate=false, bool check_endpoint=true, size_t segment_size=0);
void setTimeout(int timeout);
msgq_queue_t * getQueue() {return q;}
Message *receive(bool non_blocking=false);
~MSGQSubSocket();
};
class MSGQPoller : public Poller {
private:
std::vector<SubSocket*> sockets;
msgq_pollitem_t polls[MAX_POLLERS];
size_t num_polls = 0;
public:
void registerSocket(SubSocket *socket);
std::vector<SubSocket*> poll(int timeout);
~MSGQPoller(){}
};
+26 -195
View File
@@ -1,134 +1,30 @@
#include <cassert>
#include <cerrno>
#include <cstdlib>
#include <cstring>
#include <iostream>
#include <string>
#include <vector>
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>
#include "msgq/ipc.h"
#include "msgq/event.h"
#include "msgq/impl_msgq.h"
#include "msgq/impl_fake.h"
static bool messaging_use_fake(){
return std::getenv("CEREAL_FAKE") != NULL;
bool messaging_use_fake(){
char* fake_enabled = std::getenv("CEREAL_FAKE");
return fake_enabled != NULL;
}
void Message::init(size_t sz) {
size = sz;
data = new char[size];
Context * Context::create(){
return new Context();
}
void Message::init(char * d, size_t sz) {
size = sz;
data = new char[size];
memcpy(data, d, size);
}
void Message::takeOwnership(char * d, size_t sz) {
size = sz;
data = d;
}
void Message::close() {
if (size > 0){
delete[] data;
}
size = 0;
}
Message::~Message() {
this->close();
}
void SubSocket::open_fake_fifos() {
for (size_t i = 0; i < 2; i++) {
if (fake_fds[i] < 0 && fake_state->paths[i][0] != '\0') {
fake_fds[i] = open(fake_state->paths[i], O_RDWR | O_NONBLOCK);
}
}
}
int SubSocket::connect(std::string endpoint, bool conflate, size_t segment_size){
SubSocket * SubSocket::create(){
if (messaging_use_fake()) {
const char* cereal_prefix = std::getenv("CEREAL_FAKE_PREFIX");
std::string identifier = cereal_prefix != nullptr ? std::string(cereal_prefix) : "";
char* mem;
event_state_shm_mmap(endpoint, identifier, &mem, nullptr);
fake_state = (EventState*)mem;
open_fake_fifos();
return new FakeSubSocket<MSGQSubSocket>();
}
q = new msgq_queue_t;
size_t size = segment_size > 0 ? segment_size : DEFAULT_SEGMENT_SIZE;
int r = msgq_new_queue(q, endpoint.c_str(), size);
if (r != 0){
return r;
}
msgq_init_subscriber(q);
if (conflate){
q->read_conflate = true;
}
return 0;
return new MSGQSubSocket();
}
Message * SubSocket::receive(bool non_blocking){
if (fake_state != nullptr && fake_state->enabled) {
open_fake_fifos();
Event(fake_fds[EventPurpose::RECV_CALLED]).set();
Event(fake_fds[EventPurpose::RECV_READY]).wait();
Event(fake_fds[EventPurpose::RECV_READY]).clear();
}
msgq_msg_t msg;
Message *r = NULL;
int rc = msgq_msg_recv(&msg, q);
// Hack to implement blocking read with a poller. Don't use this
while (!non_blocking && rc == 0){
msgq_pollitem_t items[1];
items[0].q = q;
int t = (timeout != -1) ? timeout : 100;
int n = msgq_poll(items, 1, t);
rc = msgq_msg_recv(&msg, q);
// The poll indicated a message was ready, but the receive failed. Try again
if (n == 1 && rc == 0){
continue;
}
if (timeout != -1){
break;
}
}
if (rc > 0){
r = new Message;
r->takeOwnership(msg.data, msg.size);
}
return r;
}
void SubSocket::setTimeout(int t){
timeout = t;
}
SubSocket * SubSocket::create(std::string endpoint, bool conflate, size_t segment_size){
SubSocket *s = new SubSocket();
int r = s->connect(endpoint, conflate, segment_size);
SubSocket * SubSocket::create(Context * context, std::string endpoint, std::string address, bool conflate, bool check_endpoint, size_t segment_size){
SubSocket *s = SubSocket::create();
int r = s->connect(context, endpoint, address, conflate, check_endpoint, segment_size);
if (r == 0) {
return s;
@@ -140,56 +36,13 @@ SubSocket * SubSocket::create(std::string endpoint, bool conflate, size_t segmen
}
}
SubSocket::~SubSocket(){
for (int fd : fake_fds) {
if (fd >= 0) ::close(fd);
}
if (fake_state != nullptr) {
munmap(fake_state, sizeof(EventState));
}
if (q != NULL){
msgq_close_queue(q);
delete q;
}
PubSocket * PubSocket::create(){
return new PubSocket();
}
int PubSocket::connect(std::string endpoint, size_t segment_size){
q = new msgq_queue_t;
size_t size = segment_size > 0 ? segment_size : DEFAULT_SEGMENT_SIZE;
int r = msgq_new_queue(q, endpoint.c_str(), size);
if (r != 0){
return r;
}
msgq_init_publisher(q);
return 0;
}
int PubSocket::sendMessage(Message *message){
msgq_msg_t msg;
msg.data = message->getData();
msg.size = message->getSize();
return msgq_msg_send(&msg, q);
}
int PubSocket::send(char *data, size_t size){
msgq_msg_t msg;
msg.data = data;
msg.size = size;
return msgq_msg_send(&msg, q);
}
bool PubSocket::all_readers_updated() {
return msgq_all_readers_updated(q);
}
PubSocket * PubSocket::create(std::string endpoint, size_t segment_size){
PubSocket *s = new PubSocket();
int r = s->connect(endpoint, segment_size);
PubSocket * PubSocket::create(Context * context, std::string endpoint, bool check_endpoint, size_t segment_size){
PubSocket *s = PubSocket::create();
int r = s->connect(context, endpoint, check_endpoint, segment_size);
if (r == 0) {
return s;
@@ -201,40 +54,18 @@ PubSocket * PubSocket::create(std::string endpoint, size_t segment_size){
}
}
PubSocket::~PubSocket(){
if (q != NULL){
msgq_close_queue(q);
delete q;
Poller * Poller::create(){
if (messaging_use_fake()) {
return new FakePoller();
}
return new MSGQPoller();
}
Poller::Poller() : fake(messaging_use_fake()) {}
Poller * Poller::create(std::vector<SubSocket*> sockets){
Poller * p = Poller::create();
Poller::Poller(std::vector<SubSocket*> sockets) : Poller() {
for (auto s : sockets){
registerSocket(s);
p->registerSocket(s);
}
}
void Poller::registerSocket(SubSocket * socket){
sockets.push_back(socket);
polls.push_back({socket->getQueue(), 0});
}
std::vector<SubSocket*> Poller::poll(int timeout){
// in fake mode, the sockets block in receive() until the test releases them
if (fake) {
return sockets;
}
std::vector<SubSocket*> r;
msgq_poll(polls.data(), polls.size(), timeout);
for (size_t i = 0; i < polls.size(); i++){
if (polls[i].revents){
r.push_back(sockets[i]);
}
}
return r;
return p;
}
+26 -29
View File
@@ -1,17 +1,25 @@
#pragma once
#include <cstddef>
#include <map>
#include <string>
#include <vector>
#include <utility>
#include <time.h>
#include "msgq/msgq.h"
#ifdef __APPLE__
#define CLOCK_BOOTTIME CLOCK_MONOTONIC
#endif
struct EventState;
#define MSG_MULTIPLE_PUBLISHERS 100
class Context {
public:
static Context * create();
~Context(){}
};
class Message {
private:
@@ -27,46 +35,35 @@ public:
~Message();
};
class SubSocket {
private:
msgq_queue_t * q = nullptr;
int timeout = -1;
// fake mode (CEREAL_FAKE)
EventState *fake_state = nullptr;
int fake_fds[2] = {-1, -1};
void open_fake_fifos();
public:
int connect(std::string endpoint, bool conflate=false, size_t segment_size=0);
void setTimeout(int timeout);
msgq_queue_t * getQueue() {return q;}
Message *receive(bool non_blocking=false);
static SubSocket * create(std::string endpoint, bool conflate=false, size_t segment_size=0);
~SubSocket();
virtual int connect(Context *context, std::string endpoint, std::string address, bool conflate=false, bool check_endpoint=true, size_t segment_size=0) = 0;
virtual void setTimeout(int timeout) = 0;
virtual Message *receive(bool non_blocking=false) = 0;
static SubSocket * create();
static SubSocket * create(Context * context, std::string endpoint, std::string address="127.0.0.1", bool conflate=false, bool check_endpoint=true, size_t segment_size=0);
virtual ~SubSocket(){}
};
class PubSocket {
private:
msgq_queue_t * q = nullptr;
struct msgq_queue_t * q = nullptr;
public:
int connect(std::string endpoint, size_t segment_size=0);
int connect(Context *context, std::string endpoint, bool check_endpoint=true, size_t segment_size=0);
int sendMessage(Message *message);
int send(char *data, size_t size);
bool all_readers_updated();
static PubSocket * create(std::string endpoint, size_t segment_size=0);
static PubSocket * create();
static PubSocket * create(Context * context, std::string endpoint, bool check_endpoint=true, size_t segment_size=0);
~PubSocket();
};
class Poller {
private:
bool fake;
std::vector<SubSocket*> sockets;
std::vector<msgq_pollitem_t> polls;
public:
Poller();
Poller(std::vector<SubSocket*> sockets);
void registerSocket(SubSocket *socket);
std::vector<SubSocket*> poll(int timeout);
virtual void registerSocket(SubSocket *socket) = 0;
virtual std::vector<SubSocket*> poll(int timeout) = 0;
static Poller * create();
static Poller * create(std::vector<SubSocket*> sockets);
virtual ~Poller(){}
};
+13 -6
View File
@@ -6,7 +6,7 @@ from libcpp.vector cimport vector
from libcpp cimport bool
cdef extern from "msgq/event.h":
cdef extern from "msgq/impl_fake.h":
cdef cppclass Event:
@staticmethod
int wait_for_one(vector[Event], int) except +
@@ -35,6 +35,10 @@ cdef extern from "msgq/event.h":
cdef extern from "msgq/ipc.h":
cdef cppclass Context:
@staticmethod
Context * create()
cdef cppclass Message:
void init(size_t)
void init(char *, size_t)
@@ -43,19 +47,22 @@ cdef extern from "msgq/ipc.h":
char *getData()
cdef cppclass SubSocket:
SubSocket() nogil
int connect(string, bool, size_t) nogil
@staticmethod
SubSocket * create() nogil
int connect(Context *, string, string, bool, bool, size_t) nogil
Message * receive(bool) nogil
void setTimeout(int) nogil
cdef cppclass PubSocket:
PubSocket()
int connect(string, size_t)
@staticmethod
PubSocket * create()
int connect(Context *, string, bool, size_t)
int sendMessage(Message *)
int send(char *, size_t)
bool all_readers_updated()
cdef cppclass Poller:
Poller()
@staticmethod
Poller * create()
void registerSocket(SubSocket *)
vector[SubSocket*] poll(int) nogil
+160 -7
View File
@@ -7,9 +7,50 @@ from libcpp.vector cimport vector
from libcpp cimport bool
from libc cimport errno
from libc.string cimport strerror
from libc.stdint cimport int64_t, uint64_t
from cython.operator import dereference
cdef extern from "Python.h":
unsigned long PyThread_get_thread_ident() noexcept
cdef extern from *:
"""
#include <errno.h>
#include <stdint.h>
#include <time.h>
// Diagnostics must neither raise nor disturb the backend's errno.
static inline void msgq_receive_clock_pair(int64_t *wall, int64_t *cpu) {
const int saved_errno = errno;
struct timespec ts;
*wall = clock_gettime(CLOCK_MONOTONIC, &ts) == 0
? (int64_t)ts.tv_sec * 1000000000LL + ts.tv_nsec : -1;
*cpu = clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts) == 0
? (int64_t)ts.tv_sec * 1000000000LL + ts.tv_nsec : -1;
errno = saved_errno;
}
"""
void msgq_receive_clock_pair(int64_t *wall, int64_t *cpu) noexcept nogil
cdef struct ReceiveTiming:
uint64_t sequence
bool non_blocking
bool returned_message
size_t byte_length
int64_t before_wall
int64_t before_cpu
int64_t backend_return_wall
int64_t backend_return_cpu
int64_t gil_reacquired_wall
int64_t gil_reacquired_cpu
int64_t python_return_wall
int64_t python_return_cpu
from .ipc cimport Context as cppContext
from .ipc cimport SubSocket as cppSubSocket
from .ipc cimport PubSocket as cppPubSocket
from .ipc cimport Poller as cppPoller
@@ -110,13 +151,30 @@ cdef class SocketEventHandle:
return e
cdef class Context:
cdef cppContext * context
def __cinit__(self):
self.context = cppContext.create()
def term(self):
del self.context
self.context = NULL
def __dealloc__(self):
pass
# Deleting the context will hang if sockets are still active
# TODO: Figure out a way to make sure the context is closed last
# del self.context
cdef class Poller:
cdef cppPoller * poller
cdef list sub_sockets
def __cinit__(self):
self.sub_sockets = []
self.poller = new cppPoller()
self.poller = cppPoller.create()
def __dealloc__(self):
del self.poller
@@ -143,11 +201,22 @@ cdef class Poller:
cdef class SubSocket:
cdef cppSubSocket * socket
cdef bool is_owner
cdef bool receive_timing_enabled
cdef unsigned long receive_timing_owner
cdef ReceiveTiming receive_timing_ring[64]
cdef unsigned int receive_timing_next
cdef unsigned int receive_timing_count
cdef uint64_t receive_timing_sequence
cdef uint64_t receive_timing_omitted
cdef uint64_t receive_timing_generation
def __cinit__(self):
with nogil:
self.socket = new cppSubSocket()
self.socket = cppSubSocket.create()
self.is_owner = True
if self.socket == NULL:
raise IpcError
def __dealloc__(self):
if self.is_owner:
@@ -162,10 +231,10 @@ cdef class SubSocket:
self.is_owner = False
self.socket = ptr
def connect(self, string endpoint, bool conflate=False, size_t segment_size=0):
def connect(self, Context context, string endpoint, string address=b"127.0.0.1", bool conflate=False, size_t segment_size=0):
cdef int r
with nogil:
r = self.socket.connect(endpoint, conflate, segment_size)
r = self.socket.connect(context.context, endpoint, address, conflate, True, segment_size)
if r != 0:
if errno.errno == errno.EADDRINUSE:
@@ -177,10 +246,90 @@ cdef class SubSocket:
with nogil:
self.socket.setTimeout(timeout)
cdef check_receive_timing_owner(self):
if self.receive_timing_owner != 0 and self.receive_timing_owner != PyThread_get_thread_ident():
raise RuntimeError("receive timing must be managed by the thread that enabled it")
def enable_receive_timing(self, bool enabled):
"""Opt in on the receiving thread; disabling preserves previous records."""
self.check_receive_timing_owner()
if enabled:
self.receive_timing_owner = PyThread_get_thread_ident()
self.receive_timing_enabled = enabled
def reset_receive_timing(self):
"""Explicitly clear records and counts, without changing enablement."""
self.check_receive_timing_owner()
self.receive_timing_next = 0
self.receive_timing_count = 0
self.receive_timing_sequence = 0
self.receive_timing_omitted = 0
self.receive_timing_generation += 1
def take_receive_timing(self):
"""Return (oldest-to-newest records, overwritten count), without clearing.
Each record is (sequence, non_blocking, returned_message, byte_length,
before_mono_ns, before_thread_cpu_ns, backend_return_mono_ns,
backend_return_thread_cpu_ns, gil_reacquired_mono_ns,
gil_reacquired_thread_cpu_ns, python_return_mono_ns,
python_return_thread_cpu_ns). Failed clocks are -1. The final interval
includes the bytes copy and message cleanup; gil_handoff includes scheduling.
"""
self.check_receive_timing_owner()
cdef unsigned int i
cdef unsigned int start = (self.receive_timing_next + 64 - self.receive_timing_count) % 64
cdef ReceiveTiming *record
records = []
for i in range(self.receive_timing_count):
record = &self.receive_timing_ring[(start + i) % 64]
records.append((record.sequence, record.non_blocking, record.returned_message, record.byte_length,
record.before_wall, record.before_cpu, record.backend_return_wall, record.backend_return_cpu,
record.gil_reacquired_wall, record.gil_reacquired_cpu, record.python_return_wall, record.python_return_cpu))
return tuple(records), self.receive_timing_omitted
cdef receive_timed(self, bool non_blocking):
cdef cppMessage *msg
cdef ReceiveTiming record
cdef uint64_t generation = self.receive_timing_generation
record.non_blocking = non_blocking
with nogil:
msgq_receive_clock_pair(&record.before_wall, &record.before_cpu)
msg = self.socket.receive(non_blocking)
msgq_receive_clock_pair(&record.backend_return_wall, &record.backend_return_cpu)
msgq_receive_clock_pair(&record.gil_reacquired_wall, &record.gil_reacquired_cpu)
record.returned_message = msg != NULL
record.byte_length = 0
if msg == NULL:
m = None
else:
record.byte_length = msg.getSize()
m = msg.getData()[:record.byte_length]
with nogil:
del msg
msgq_receive_clock_pair(&record.python_return_wall, &record.python_return_cpu)
# All shared ring access holds the GIL. Only this stack-local record is
# written while nogil; no Python object is created for the receipt here.
if generation == self.receive_timing_generation:
self.receive_timing_sequence += 1
record.sequence = self.receive_timing_sequence
self.receive_timing_ring[self.receive_timing_next] = record
self.receive_timing_next = (self.receive_timing_next + 1) % 64
if self.receive_timing_count < 64:
self.receive_timing_count += 1
else:
self.receive_timing_omitted += 1
return m
def receive(self, bool non_blocking=False):
cdef cppMessage *msg
if self.receive_timing_enabled and self.receive_timing_owner == PyThread_get_thread_ident():
return self.receive_timed(non_blocking)
with nogil:
msg = self.socket.receive(non_blocking)
if msg == NULL:
return None
else:
@@ -188,19 +337,23 @@ cdef class SubSocket:
m = msg.getData()[:sz]
with nogil:
del msg
return m
cdef class PubSocket:
cdef cppPubSocket * socket
def __cinit__(self):
self.socket = new cppPubSocket()
self.socket = cppPubSocket.create()
if self.socket == NULL:
raise IpcError
def __dealloc__(self):
del self.socket
def connect(self, string endpoint, size_t segment_size=0):
r = self.socket.connect(endpoint, segment_size)
def connect(self, Context context, string endpoint, size_t segment_size=0):
r = self.socket.connect(context.context, endpoint, True, segment_size)
if r != 0:
if errno.errno == errno.EADDRINUSE:
Binary file not shown.
+21
View File
@@ -0,0 +1,21 @@
#pragma once
#ifdef SWAGLOG
// cppcheck-suppress preprocessorErrorDirective
#include SWAGLOG
#else
#define CLOUDLOG_DEBUG 10
#define CLOUDLOG_INFO 20
#define CLOUDLOG_WARNING 30
#define CLOUDLOG_ERROR 40
#define CLOUDLOG_CRITICAL 50
#define cloudlog(lvl, fmt, ...) printf(fmt "\n", ## __VA_ARGS__)
#define LOGD(fmt, ...) cloudlog(CLOUDLOG_DEBUG, fmt, ## __VA_ARGS__)
#define LOG(fmt, ...) cloudlog(CLOUDLOG_INFO, fmt, ## __VA_ARGS__)
#define LOGW(fmt, ...) cloudlog(CLOUDLOG_WARNING, fmt, ## __VA_ARGS__)
#define LOGE(fmt, ...) cloudlog(CLOUDLOG_ERROR, fmt, ## __VA_ARGS__)
#endif
-2
View File
@@ -443,8 +443,6 @@ int msgq_poll(msgq_pollitem_t * items, size_t nitems, int timeout){
if (items[i].revents) num++;
}
if (timeout == 0) return num;
int ms = (timeout == -1) ? 100 : timeout;
#ifdef __APPLE__
+2 -91
View File
@@ -1,20 +1,14 @@
#include <array>
#include <chrono>
#include <thread>
#include <unistd.h>
#include "msgq/test_runner.h"
#include "msgq/msgq.h"
static void cleanup_test_queue() {
#ifdef __APPLE__
std::string file = "/tmp/msgq_";
remove("/tmp/msgq_test_queue");
#else
std::string file = "/dev/shm/msgq_";
remove("/dev/shm/msgq_test_queue");
#endif
const char *prefix = std::getenv("OPENPILOT_PREFIX");
if (prefix != nullptr) file += std::string(prefix) + "/";
file += "test_queue";
std::remove(file.c_str());
}
TEST_CASE("ALIGN")
@@ -26,89 +20,6 @@ TEST_CASE("ALIGN")
REQUIRE(ALIGN(99999) == 100000);
}
TEST_CASE("msgq_poll nonblocking readiness")
{
cleanup_test_queue();
msgq_queue_t writer, first, second;
msgq_new_queue(&writer, "test_queue", 1024);
msgq_init_publisher(&writer);
msgq_new_queue(&first, "test_queue", 1024);
msgq_init_subscriber(&first);
msgq_new_queue(&second, "test_queue", 1024);
msgq_init_subscriber(&second);
msgq_pollitem_t items[] = {{&first, 1}, {&second, 1}};
REQUIRE(msgq_poll(nullptr, 0, 0) == 0);
REQUIRE(msgq_poll(items, 2, 0) == 0);
REQUIRE(items[0].revents == 0);
REQUIRE(items[1].revents == 0);
char data[] = "poll payload";
msgq_msg_t outgoing, incoming;
msgq_msg_init_data(&outgoing, data, sizeof(data));
REQUIRE(msgq_msg_send(&outgoing, &writer) == sizeof(data));
REQUIRE(msgq_poll(items, 2, 0) == 2);
REQUIRE(items[0].revents == 1);
REQUIRE(items[1].revents == 1);
REQUIRE(msgq_msg_recv(&incoming, &first) == sizeof(data));
REQUIRE(memcmp(incoming.data, data, sizeof(data)) == 0);
msgq_msg_close(&incoming);
REQUIRE(msgq_poll(items, 2, 0) == 1);
REQUIRE(items[0].revents == 0);
REQUIRE(items[1].revents == 1);
REQUIRE(msgq_msg_recv(&incoming, &second) == sizeof(data));
REQUIRE(memcmp(incoming.data, data, sizeof(data)) == 0);
msgq_msg_close(&incoming);
REQUIRE(msgq_poll(items, 2, 0) == 0);
REQUIRE(items[0].revents == 0);
REQUIRE(items[1].revents == 0);
msgq_msg_close(&outgoing);
msgq_close_queue(&second);
msgq_close_queue(&first);
msgq_close_queue(&writer);
cleanup_test_queue();
}
TEST_CASE("msgq_poll waiting modes")
{
for (int timeout : {1000, -1}) {
cleanup_test_queue();
msgq_queue_t writer, reader;
msgq_new_queue(&writer, "test_queue", 1024);
msgq_init_publisher(&writer);
msgq_new_queue(&reader, "test_queue", 1024);
msgq_init_subscriber(&reader);
msgq_pollitem_t item{&reader, 1};
auto start = std::chrono::steady_clock::now();
REQUIRE(msgq_poll(&item, 1, 2) == 0);
REQUIRE(item.revents == 0);
REQUIRE(std::chrono::steady_clock::now() - start >= std::chrono::milliseconds(2));
msgq_msg_t outgoing;
msgq_msg_init_size(&outgoing, 1);
outgoing.data[0] = 'p';
std::thread sender([&]() {
std::this_thread::sleep_for(std::chrono::milliseconds(2));
msgq_msg_send(&outgoing, &writer);
});
int received = msgq_poll(&item, 1, timeout);
sender.join();
REQUIRE(received == 1);
REQUIRE(item.revents == 1);
msgq_msg_t incoming;
REQUIRE(msgq_msg_recv(&incoming, &reader) == 1);
REQUIRE(incoming.data[0] == 'p');
msgq_msg_close(&incoming);
msgq_msg_close(&outgoing);
msgq_close_queue(&reader);
msgq_close_queue(&writer);
cleanup_test_queue();
}
}
TEST_CASE("msgq_msg_init_size")
{
const size_t msg_size = 30;
+28 -10
View File
@@ -5,10 +5,12 @@ import concurrent.futures
SERVICE_NAME = 'myService'
def poller():
context = msgq.Context()
p = msgq.Poller()
sub = msgq.SubSocket()
sub.connect(SERVICE_NAME)
sub.connect(context, SERVICE_NAME)
p.registerSocket(sub)
socks = p.poll(10000)
@@ -19,8 +21,10 @@ def poller():
class TestPoller(unittest.TestCase):
def test_poll_once(self):
context = msgq.Context()
pub = msgq.PubSocket()
pub.connect(SERVICE_NAME)
pub.connect(context, SERVICE_NAME)
with concurrent.futures.ThreadPoolExecutor() as e:
poll = e.submit(poller)
@@ -34,19 +38,23 @@ class TestPoller(unittest.TestCase):
result = poll.result()
del pub
context.term()
assert result == [b"a"]
def test_poll_and_create_many_subscribers(self):
context = msgq.Context()
pub = msgq.PubSocket()
pub.connect(SERVICE_NAME)
pub.connect(context, SERVICE_NAME)
with concurrent.futures.ThreadPoolExecutor() as e:
poll = e.submit(poller)
pub.wait_for_readers()
c = msgq.Context()
for _ in range(10):
msgq.SubSocket().connect(SERVICE_NAME)
msgq.SubSocket().connect(c, SERVICE_NAME)
# Send message
pub.send(b"a")
@@ -55,28 +63,34 @@ class TestPoller(unittest.TestCase):
result = poll.result()
del pub
context.term()
assert result == [b"a"]
def test_multiple_publishers_exception(self):
context = msgq.Context()
with self.assertRaises(msgq.MultiplePublishersError):
pub1 = msgq.PubSocket()
pub1.connect(SERVICE_NAME)
pub1.connect(context, SERVICE_NAME)
pub2 = msgq.PubSocket()
pub2.connect(SERVICE_NAME)
pub2.connect(context, SERVICE_NAME)
pub1.send(b"a")
del pub1
del pub2
context.term()
def test_multiple_messages(self):
context = msgq.Context()
pub = msgq.PubSocket()
pub.connect(SERVICE_NAME)
pub.connect(context, SERVICE_NAME)
sub = msgq.SubSocket()
sub.connect(SERVICE_NAME)
sub.connect(context, SERVICE_NAME)
pub.wait_for_readers()
@@ -99,13 +113,16 @@ class TestPoller(unittest.TestCase):
del pub
del sub
context.term()
def test_conflate(self):
context = msgq.Context()
pub = msgq.PubSocket()
pub.connect(SERVICE_NAME)
pub.connect(context, SERVICE_NAME)
sub = msgq.SubSocket()
sub.connect(SERVICE_NAME, conflate=True)
sub.connect(context, SERVICE_NAME, conflate=True)
pub.wait_for_readers()
pub.send(b'a')
@@ -115,3 +132,4 @@ class TestPoller(unittest.TestCase):
del pub
del sub
context.term()
+131
View File
@@ -0,0 +1,131 @@
import ctypes
import threading
import time
import uuid
import msgq
import pytest
def sockets(timeout=10):
endpoint = f"receive_timing_{uuid.uuid4().hex}"
pub = msgq.pub_sock(endpoint)
sub = msgq.sub_sock(endpoint, timeout=timeout)
return pub, sub
def assert_clocks(record):
assert len(record) == 12
assert all(t >= 0 for t in record[4:])
assert list(record[4::2]) == sorted(record[4::2])
assert list(record[5::2]) == sorted(record[5::2])
def test_receive_timing_opt_in_and_preserves_messages():
pub, sub = sockets()
for payload in (b"x", b"unchanged\x00message"):
pub.send(payload)
assert sub.receive() == payload
assert sub.receive(non_blocking=True) is None
assert sub.take_receive_timing() == ((), 0)
sub.enable_receive_timing(True)
for payload in (b"x", b"unchanged\x00message"):
pub.send(payload)
assert sub.receive() == payload
assert sub.receive(non_blocking=True) is None
assert sub.receive() is None
records, omitted = sub.take_receive_timing()
assert omitted == 0
assert [r[:4] for r in records] == [(1, False, True, 1), (2, False, True, 17),
(3, True, False, 0), (4, False, False, 0)]
for record in records:
assert_clocks(record)
assert sub.take_receive_timing() == (records, omitted)
sub.enable_receive_timing(False)
pub.send(b"disabled again")
assert sub.receive() == b"disabled again"
assert sub.receive(non_blocking=True) is None
assert sub.take_receive_timing() == (records, omitted)
sub.reset_receive_timing()
assert sub.take_receive_timing() == ((), 0)
def test_receive_timing_bounded_ring_and_explicit_reset():
_, sub = sockets()
sub.enable_receive_timing(True)
for _ in range(150):
assert sub.receive(non_blocking=True) is None
records, omitted = sub.take_receive_timing()
assert len(records) == 64
assert omitted == 86
assert [r[0] for r in records] == list(range(87, 151))
sub.reset_receive_timing()
assert sub.take_receive_timing() == ((), 0)
assert sub.receive(non_blocking=True) is None
assert sub.take_receive_timing()[0][0][0] == 1
def test_receive_timing_thread_owner_and_socket_isolation():
_, sub = sockets()
_, other_sub = sockets()
sub.enable_receive_timing(True)
failures = []
def other_thread():
try:
# The receive itself remains valid and uninstrumented on other threads.
assert sub.receive(non_blocking=True) is None
for op in (sub.take_receive_timing, sub.reset_receive_timing,
lambda: sub.enable_receive_timing(False), lambda: sub.enable_receive_timing(True)):
with pytest.raises(RuntimeError, match="thread that enabled"):
op()
except BaseException as exc:
failures.append(exc)
worker = threading.Thread(target=other_thread)
worker.start()
worker.join(timeout=3)
assert not worker.is_alive()
assert not failures
assert sub.take_receive_timing() == ((), 0)
assert sub.receive(non_blocking=True) is None
assert len(sub.take_receive_timing()[0]) == 1
assert other_sub.receive(non_blocking=True) is None
assert other_sub.take_receive_timing() == ((), 0)
def test_receive_timing_exposes_gil_handoff_delay():
# PyDLL deliberately retains the main thread's GIL while libc sleeps.
# The receiving thread's native timeout can expire independently of it.
libc = ctypes.PyDLL(None)
libc.usleep.argtypes = [ctypes.c_uint]
libc.usleep.restype = ctypes.c_int
ready = threading.Event()
results = []
failures = []
def receiver():
try:
_, sub = sockets(timeout=40)
sub.enable_receive_timing(True)
ready.set()
assert sub.receive() is None
results.append(sub.take_receive_timing()[0][0])
except BaseException as exc:
failures.append(exc)
worker = threading.Thread(target=receiver)
worker.start()
assert ready.wait(timeout=3)
time.sleep(0.01)
assert libc.usleep(150_000) == 0
worker.join(timeout=3)
assert not worker.is_alive()
assert not failures
record = results[0]
assert_clocks(record)
assert record[6] - record[4] >= 30_000_000
assert record[8] - record[6] >= 50_000_000
assert record[9] - record[7] < 20_000_000
+8 -5
View File
@@ -9,6 +9,7 @@
#include "msgq/visionipc/visionipc.h"
#include "msgq/visionipc/visionipc_client.h"
#include "msgq/visionipc/visionipc_server.h"
#include "msgq/logger/logger.h"
static int connect_to_vipc_server(const std::string &name, bool blocking) {
const std::string ipc_path = get_ipc_path(name);
@@ -26,9 +27,10 @@ static int connect_to_vipc_server(const std::string &name, bool blocking) {
}
VisionIpcClient::VisionIpcClient(std::string name_, VisionStreamType type_, bool conflate) : name(name_), type(type_) {
sock = SubSocket::create(get_endpoint_name(name, type), conflate);
msg_ctx = Context::create();
sock = SubSocket::create(msg_ctx, get_endpoint_name(name, type), "127.0.0.1", conflate, false);
poller = new Poller();
poller = Poller::create();
poller->registerSocket(sock);
}
@@ -39,7 +41,7 @@ bool VisionIpcClient::connect(bool blocking) {
// Cleanup old buffers on reconnect
for (size_t i = 0; i < num_buffers; i++) {
if (buffers[i].free() != 0) {
std::cerr << "Failed to free buffer " << i << std::endl;
LOGE("Failed to free buffer %zu", i);
}
}
@@ -116,7 +118,7 @@ VisionBuf * VisionIpcClient::recv(VisionIpcBufExtra * extra, const int timeout_m
}
if (buf->sync(VISIONBUF_SYNC_TO_DEVICE) != 0) {
std::cerr << "Failed to sync buffer" << std::endl;
LOGE("Failed to sync buffer");
}
delete r;
@@ -150,10 +152,11 @@ std::set<VisionStreamType> VisionIpcClient::getAvailableStreams(const std::strin
VisionIpcClient::~VisionIpcClient() {
for (size_t i = 0; i < num_buffers; i++) {
if (buffers[i].free() != 0) {
std::cerr << "Failed to free buffer " << i << std::endl;
LOGE("Failed to free buffer %zu", i);
}
}
delete sock;
delete poller;
delete msg_ctx;
}
@@ -10,6 +10,7 @@
class VisionIpcClient {
private:
std::string name;
Context * msg_ctx;
SubSocket * sock;
Poller * poller;
Binary file not shown.
+11 -4
View File
@@ -12,6 +12,7 @@
#include "msgq/visionipc/visionipc.h"
#include "msgq/visionipc/visionipc_server.h"
#include "msgq/logger/logger.h"
std::string get_endpoint_name(std::string name, VisionStreamType type){
return "visionipc_" + name + "_" + std::to_string(type);
@@ -26,6 +27,8 @@ std::string get_ipc_path(const std::string& name) {
}
VisionIpcServer::VisionIpcServer(std::string name_) : name(name_) {
msg_ctx = Context::create();
std::random_device rd("/dev/urandom");
std::uniform_int_distribution<uint64_t> distribution(0, std::numeric_limits<uint64_t>::max());
server_id = distribution(rd);
@@ -58,7 +61,7 @@ void VisionIpcServer::create_buffers_with_sizes(VisionStreamType type, size_t nu
cur_idx[type] = 0;
// Create msgq publisher for each of the `name` + type combos
sockets[type] = PubSocket::create(get_endpoint_name(name, type));
sockets[type] = PubSocket::create(msg_ctx, get_endpoint_name(name, type), false);
}
@@ -68,6 +71,8 @@ void VisionIpcServer::start_listener(){
void VisionIpcServer::listener(){
LOGD("Starting listener for: %s", name.c_str());
const std::string ipc_path = get_ipc_path(name);
int sock = ipc_bind(ipc_path.c_str());
assert(sock >= 0);
@@ -139,6 +144,7 @@ void VisionIpcServer::listener(){
close(fd);
}
LOGD("Stopping listener for: %s", name.c_str());
close(sock);
unlink(ipc_path.c_str());
}
@@ -148,7 +154,7 @@ void VisionIpcServer::listener(){
VisionBuf * VisionIpcServer::get_buffer(VisionStreamType type, int idx){
// Do we want to keep track if the buffer has been sent out yet and warn user?
assert(buffers.count(type));
const auto &b = buffers[type];
auto b = buffers[type];
if (idx < 0) {
idx = cur_idx[type]++ % b.size();
} else {
@@ -161,7 +167,7 @@ VisionBuf * VisionIpcServer::get_buffer(VisionStreamType type, int idx){
void VisionIpcServer::send(VisionBuf * buf, VisionIpcBufExtra * extra, bool sync){
if (sync) {
if (buf->sync(VISIONBUF_SYNC_FROM_DEVICE) != 0) {
std::cerr << "Failed to sync buffer" << std::endl;
LOGE("Failed to sync buffer");
}
}
assert(buffers.count(buf->type));
@@ -190,7 +196,7 @@ VisionIpcServer::~VisionIpcServer(){
for (auto const& [type, buf] : buffers) {
for (VisionBuf* b : buf){
if (b->free() != 0) {
std::cerr << "Failed to free buffer" << std::endl;
LOGE("Failed to free buffer");
}
delete b;
}
@@ -200,4 +206,5 @@ VisionIpcServer::~VisionIpcServer(){
for (auto const& [type, sock] : sockets) {
delete sock;
}
delete msg_ctx;
}
@@ -22,6 +22,7 @@ class VisionIpcServer {
std::map<VisionStreamType, std::atomic<size_t> > cur_idx;
std::map<VisionStreamType, std::vector<VisionBuf*> > buffers;
Context * msg_ctx;
std::map<VisionStreamType, PubSocket*> sockets;
void listener(void);
+5 -24
View File
@@ -1,14 +1,10 @@
[build-system]
requires = ["setuptools>=77", "Cython>=3.1", "scons"]
build-backend = "setuptools.build_meta"
[project]
name = "msgq-ipc"
version = "1.1"
description = "High-performance pub/sub messaging, made simple. For Python, C, and C++."
name = "msgq"
version = "0.0.1"
description = "Lock-free IPC pub/sub message queue"
readme = "README.md"
requires-python = ">=3.11"
license = "MIT"
license = {text = "MIT"}
authors = [{name = "comma.ai"}]
classifiers = [
"Natural Language :: English",
@@ -17,9 +13,6 @@ classifiers = [
]
dependencies = []
[project.urls]
Repository = "https://github.com/commaai/msgq"
[project.optional-dependencies]
dev = [
"setuptools", # for distutils
@@ -33,20 +26,8 @@ dev = [
"lefthook",
]
[tool.setuptools]
include-package-data = false
[tool.setuptools.packages.find]
include = ["msgq", "msgq.*"]
namespaces = false
[tool.setuptools.package-data]
"msgq" = ["*.pxd", "*.h"]
"msgq.visionipc" = ["*.pxd", "*.h"]
[tool.cibuildwheel]
build = "cp311-* cp312-* cp313-* cp314-*"
test-command = "python -m unittest discover -s msgq"
"*" = ["*.so"]
# https://beta.ruff.rs/docs/configuration/#using-pyprojecttoml
[tool.ruff]
-26
View File
@@ -1,26 +0,0 @@
import os
from pathlib import Path
import subprocess
import sys
from setuptools import Extension, setup
from setuptools.command.build_ext import build_ext
class SConsBuildExt(build_ext):
def build_extensions(self):
subprocess.check_call([sys.executable, "-m", "SCons", "--minimal", "-j", str(self.parallel or os.cpu_count() or 1)])
for ext in self.extensions:
source = Path(*ext.name.split(".")).with_suffix(".so")
target = Path(self.get_ext_fullpath(ext.name))
target.parent.mkdir(parents=True, exist_ok=True)
self.copy_file(str(source), str(target))
setup(
ext_modules=[
Extension("msgq.ipc_pyx", sources=[]),
Extension("msgq.visionipc.visionipc_pyx", sources=[]),
],
cmdclass={"build_ext": SConsBuildExt},
)
-10
View File
@@ -12,16 +12,6 @@ scons -j8
# *** lint + test ***
lefthook run test
# *** test installed package outside the checkout ***
(
TEST_DIR=$(mktemp -d)
trap 'rm -rf "$TEST_DIR"' EXIT
uv venv --python "$DIR/.venv/bin/python" "$TEST_DIR/.venv"
uv pip install --python "$TEST_DIR/.venv/bin/python" "$DIR"
cd "$TEST_DIR"
"$TEST_DIR/.venv/bin/python" -m unittest msgq.tests.test_messaging
)
# *** all done ***
GREEN='\033[0;32m'
NC='\033[0m'
+1 -1
View File
@@ -116,7 +116,7 @@ jobs:
path: /tmp/comma_download_cache
key: car_models-${{ hashFiles('opendbc/car/tests/test_models.py', 'opendbc/car/tests/routes.py') }}-${{ matrix.job }}
- name: Test car models
timeout-minutes: ${{ contains(runner.name, 'nsc') && (steps.routes-cache.outputs.cache-hit == 'true') && 2 || 8 }}
timeout-minutes: ${{ contains(runner.name, 'nsc') && (steps.routes-cache.outputs.cache-hit == 'true') && 2 || 6 }}
run: source .venv/bin/activate && MAX_EXAMPLES=1 opendbc/car/tests/test_models.py
env:
NUM_JOBS: 4
+3 -45
View File
@@ -1,6 +1,6 @@
<!--- AUTOGENERATED FROM selfdrive/car/CARS_template.md, DO NOT EDIT. --->
# Support Information for 448 Known Cars
# Support Information for 406 Known Cars
|Make|Model|Package|Support Level|
|---|---|---|:---:|
@@ -34,50 +34,12 @@
|Audi|Q5 2017-24|All|[Not compatible](#flexray)|
|Audi|RS3 2018|Adaptive Cruise Control (ACC) & Lane Assist|[Upstream](#upstream)|
|Audi|S3 2015-17|Adaptive Cruise Control (ACC) & Lane Assist|[Upstream](#upstream)|
|Buick|Baby Enclave (SDGM Harness) 2020-23|Driver Assist Package|[Community](#community)|
|Buick|LaCrosse (ASCM Harness) 2017-19|Adaptive Cruise Control (ACC)|[Community](#community)|
|Buick|LaCrosse (OBD-II Harness) 2017-19|Driver Confidence Package 2|[Community](#community)|
|Buick|LaCrosse US (ASCM Harness) 2019|Adaptive Cruise Control (ACC)|[Community](#community)|
|Buick|Regal Essence (OBD-II Harness) 2018|Adaptive Cruise Control (ACC)|[Community](#community)|
|BYD|Atto 3 2022-25|All|[Dashcam mode](#dashcam)|
|Cadillac|ATS Premium Performance (OBD-II Harness) 2018|Adaptive Cruise Control (ACC)|[Community](#community)|
|Cadillac|CT6 (CC)|Cruise control; configuration-specific installation|[Community](#community)|
|Cadillac|Escalade (ASCM Harness) 2018|Adaptive Cruise Control (ACC)|[Community](#community)|
|Cadillac|Escalade (OBD-II Harness) 2017|Driver Assist Package|[Community](#community)|
|Cadillac|Escalade ESV (OBD-II Harness) 2016|Adaptive Cruise Control (ACC) & LKAS|[Community](#community)|
|Cadillac|Escalade ESV (OBD-II Harness) 2019|Adaptive Cruise Control (ACC) & LKAS|[Community](#community)|
|Cadillac|Escalade ESV Platinum (ASCM Harness) 2019|Adaptive Cruise Control (ACC)|[Community](#community)|
|Cadillac|XT4 (CC)|Cruise control; configuration-specific installation|[Community](#community)|
|Cadillac|XT4 (SDGM Harness) 2023|Driver Assist Package|[Community](#community)|
|Cadillac|XT5 (CC)|Cruise control; configuration-specific installation|[Community](#community)|
|Cadillac|XT5 (SDGM Harness) 2022|Driver Assist Package|[Community](#community)|
|Cadillac|XT6 (SDGM Harness) 2020|Driver Assist Package|[Community](#community)|
|Chevrolet|Blazer (SDGM Harness) 2019-25|Driver Assist Package|[Community](#community)|
|Chevrolet|Bolt (Camera Harness) 2022-23|ACC; recognized camera layout required|[Community](#community)|
|Chevrolet|Bolt EUV 2022-23|Premier or Premier Redline Trim, without Super Cruise Package|[Upstream](#upstream)|
|Chevrolet|Bolt EV 2022-23|2LT Trim with Adaptive Cruise Control Package|[Upstream](#upstream)|
|Chevrolet|Equinox 2019-22|Adaptive Cruise Control (ACC)|[Upstream](#upstream)|
|Chevrolet|Equinox (CC)|Cruise control; configuration-specific installation|[Community](#community)|
|Chevrolet|Malibu (ASCM Harness) 2017-19|Adaptive Cruise Control (ACC)|[Community](#community)|
|Chevrolet|Malibu (CC)|Cruise control; configuration-specific installation|[Community](#community)|
|Chevrolet|Malibu (SDGM Harness) 2019|SDGM Harness|[Community](#community)|
|Chevrolet|Malibu Hybrid (CC)|Cruise control; configuration-specific installation|[Community](#community)|
|Chevrolet|Malibu Premier (OBD-II Harness) 2017|Adaptive Cruise Control (ACC)|[Community](#community)|
|Chevrolet|Silverado 1500 2020-21|Safety Package II|[Upstream](#upstream)|
|Chevrolet|Silverado (CC)|Cruise control; configuration-specific installation|[Community](#community)|
|Chevrolet|Suburban (Camera Harness)|Manual camera configuration; recognized CAN layout required|[Community](#community)|
|Chevrolet|Suburban (CC)|Cruise control; configuration-specific installation|[Community](#community)|
|Chevrolet|Suburban (OBD-II Harness)|Configuration-specific install; observed CAN sources required|[Community](#community)|
|Chevrolet|Suburban Premier (ASCM Harness) 2016-20|Adaptive Cruise Control (ACC)|[Community](#community)|
|Chevrolet|Trailblazer 2021-22|Adaptive Cruise Control (ACC)|[Upstream](#upstream)|
|Chevrolet|Trailblazer (CC)|Cruise control; configuration-specific installation|[Community](#community)|
|Chevrolet|Traverse (SDGM Harness) 2022-23|RS, Premier, or High Country Trim|[Community](#community)|
|Chevrolet|Trax (Camera Harness)|Manual camera configuration; recognized CAN layout required|[Community](#community)|
|Chevrolet|Volt (ASCM Harness) 2017-18|Adaptive Cruise Control (ACC); custom ASCM harness install|[Community](#community)|
|Chevrolet|Volt (Camera Harness)|Manual configuration; recognized camera layout required|[Community](#community)|
|Chevrolet|Volt (CC)|Manual development configuration; installation must be verified|[Community](#community)|
|Chevrolet|Volt (OBD-II Harness) 2017-18|Adaptive Cruise Control (ACC)|[Community](#community)|
|Chevrolet|Volt (SDGM Harness) 2019|Adaptive Cruise Control (ACC) & LKAS|[Community](#community)|
|Chrysler|Pacifica 2017-18|Adaptive Cruise Control (ACC)|[Upstream](#upstream)|
|Chrysler|Pacifica 2019-20|Adaptive Cruise Control (ACC)|[Upstream](#upstream)|
|Chrysler|Pacifica 2021-23|All|[Upstream](#upstream)|
@@ -126,12 +88,8 @@
|Genesis|GV70 Electrified (Australia Only) 2022|All|[Upstream](#upstream)|
|Genesis|GV70 Electrified (with HDA II) 2023-24|Highway Driving Assist II|[Upstream](#upstream)|
|Genesis|GV80 2023|All|[Upstream](#upstream)|
|GMC|Acadia (ASCM Harness) 2018|Adaptive Cruise Control (ACC)|[Community](#community)|
|GMC|Acadia (OBD-II Harness) 2018|Adaptive Cruise Control (ACC)|[Community](#community)|
|GMC|Sierra 1500 2020-21|Driver Alert Package II|[Upstream](#upstream)|
|GMC|Yukon 2019-20|Adaptive Cruise Control (ACC) & LKAS|[Dashcam mode](#dashcam)|
|GMC|Yukon (CC)|Cruise control; configuration-specific installation|[Community](#community)|
|Holden|Astra (OBD-II Harness) 2017|Adaptive Cruise Control (ACC)|[Community](#community)|
|Honda|Accord 2016-17|Honda Sensing|[Community](#community)|
|Honda|Accord 2018-22|All|[Upstream](#upstream)|
|Honda|Accord 2023-25|All|[Upstream](#upstream)|
@@ -153,7 +111,7 @@
|Honda|CR-V 2017-22|Honda Sensing|[Upstream](#upstream)|
|Honda|CR-V 2023-26|All|[Upstream](#upstream)|
|Honda|CR-V Hybrid 2017-22|Honda Sensing|[Upstream](#upstream)|
|Honda|CR-V Hybrid 2023-27|All|[Upstream](#upstream)|
|Honda|CR-V Hybrid 2023-26|All|[Upstream](#upstream)|
|Honda|e 2020|All|[Upstream](#upstream)|
|Honda|Fit 2018-20|Honda Sensing|[Upstream](#upstream)|
|Honda|Freed 2020|Honda Sensing|[Upstream](#upstream)|
@@ -515,4 +473,4 @@ Toyota, and the GM Global B platform.
All the cars that openpilot supports use a [CAN bus](https://en.wikipedia.org/wiki/CAN_bus) for communication between all the car's computers, however a
CAN bus isn't the only way that the computers in your car can communicate. Most, if not all, vehicles from the following
manufacturers use [FlexRay](https://en.wikipedia.org/wiki/FlexRay) instead of a CAN bus: **BMW, Mercedes, Audi, Land Rover, and some Volvo**. These cars
may one day be supported, but we have no immediate plans to support FlexRay.
may one day be supported, but we have no immediate plans to support FlexRay.
-6
View File
@@ -1,6 +0,0 @@
Import('envCython')
raw_value = envCython.Clone(SHLIBPREFIX='', SHLIBSUFFIX='.so')
raw_value.Append(CFLAGS=['-std=c11'], CCFLAGS=['-O3', '-Werror'])
raw_value.Replace(SHLINKFLAGS=['$LINKFLAGS'])
raw_value.SharedLibrary('_raw_value', ['_raw_value.c'])
-135
View File
@@ -1,135 +0,0 @@
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include <stdint.h>
typedef struct {
PyObject *names[4];
} ExtractState;
static PyObject *get_raw_value(PyObject *self, PyObject *args) {
ExtractState *state = PyModule_GetState(self);
PyObject *signal, *payload, *expected;
PyObject *msb_obj = NULL, *lsb_obj = NULL, *size_obj = NULL, *little_obj = NULL;
const unsigned char *data;
Py_ssize_t length;
long msb, lsb, size, little, first, last;
if (!PyArg_UnpackTuple(args, "get_raw_value", 3, 3, &signal, &payload, &expected)) return NULL;
if (state == NULL || state->names[0] == NULL || state->names[1] == NULL ||
state->names[2] == NULL || state->names[3] == NULL) goto unsupported;
if (!PyType_Check(expected) || Py_TYPE(signal) != (PyTypeObject *)expected) goto unsupported;
if (!PyBytes_CheckExact(payload) && !PyByteArray_CheckExact(payload)) goto unsupported;
msb_obj = PyObject_GetAttr(signal, state->names[0]);
if (msb_obj == NULL) goto unsupported;
lsb_obj = PyObject_GetAttr(signal, state->names[1]);
if (lsb_obj == NULL) goto unsupported;
size_obj = PyObject_GetAttr(signal, state->names[2]);
if (size_obj == NULL) goto unsupported;
little_obj = PyObject_GetAttr(signal, state->names[3]);
if (little_obj == NULL) goto unsupported;
if (!PyLong_CheckExact(msb_obj) || !PyLong_CheckExact(lsb_obj) ||
!PyLong_CheckExact(size_obj) || !PyBool_Check(little_obj)) goto unsupported;
msb = PyLong_AsLong(msb_obj);
if (PyErr_Occurred()) goto unsupported;
lsb = PyLong_AsLong(lsb_obj);
if (PyErr_Occurred()) goto unsupported;
size = PyLong_AsLong(size_obj);
if (PyErr_Occurred()) goto unsupported;
little = little_obj == Py_True;
/* Attribute access may execute Python; acquire mutable storage only afterward. */
if (PyBytes_CheckExact(payload)) {
data = (const unsigned char *)PyBytes_AS_STRING(payload);
length = PyBytes_GET_SIZE(payload);
} else {
data = (const unsigned char *)PyByteArray_AS_STRING(payload);
length = PyByteArray_GET_SIZE(payload);
}
if (msb < 0 || msb >= 512 || lsb < 0 || lsb >= 512 || size <= 0 || size > 64) goto unsupported;
if (little) {
first = lsb / 8;
last = msb / 8;
if (size != msb - lsb + 1) goto unsupported;
} else {
first = msb / 8;
last = lsb / 8;
if (size != 8 * (last - first) + msb % 8 - lsb % 8 + 1) goto unsupported;
}
if (first < 0 || first > last || last >= length) goto unsupported;
uint64_t result = 0;
long remaining = size;
for (long byte = msb / 8; remaining > 0; byte += little ? -1 : 1) {
if (byte < 0 || byte >= length) goto unsupported;
const long low = lsb / 8 == byte ? lsb : byte * 8;
const long high = msb / 8 == byte ? msb : (byte + 1) * 8 - 1;
const long width = high - low + 1;
const long bit_shift = low - byte * 8;
const long result_shift = remaining - width;
if (width <= 0 || width > 8 || bit_shift < 0 || bit_shift > 7 ||
result_shift < 0 || result_shift > 63) goto unsupported;
const unsigned int fragment = (data[byte] >> bit_shift) & ((1U << width) - 1U);
result |= (uint64_t)fragment << result_shift;
remaining -= width;
}
Py_DECREF(msb_obj);
Py_DECREF(lsb_obj);
Py_DECREF(size_obj);
Py_DECREF(little_obj);
return PyLong_FromUnsignedLongLong((unsigned long long)result);
unsupported:
Py_XDECREF(msb_obj);
Py_XDECREF(lsb_obj);
Py_XDECREF(size_obj);
Py_XDECREF(little_obj);
PyErr_Clear();
Py_INCREF(Py_NotImplemented);
return Py_NotImplemented;
}
static PyMethodDef methods[] = {
{"get_raw_value", get_raw_value, METH_VARARGS, "Extract an exact complete <=64-bit CAN signal, or NotImplemented."},
{NULL, NULL, 0, NULL},
};
static int module_traverse(PyObject *self, visitproc visit, void *arg) {
ExtractState *state = PyModule_GetState(self);
if (state != NULL) {
for (int i = 0; i < 4; ++i) Py_VISIT(state->names[i]);
}
return 0;
}
static int module_clear(PyObject *self) {
ExtractState *state = PyModule_GetState(self);
if (state != NULL) {
for (int i = 0; i < 4; ++i) Py_CLEAR(state->names[i]);
}
return 0;
}
static void module_free(void *self) {
(void)module_clear((PyObject *)self);
}
static struct PyModuleDef module = {
PyModuleDef_HEAD_INIT,
.m_name = "_raw_value",
.m_doc = "Complete raw CAN signal bit extraction.",
.m_size = sizeof(ExtractState),
.m_methods = methods,
.m_traverse = module_traverse,
.m_clear = module_clear,
.m_free = module_free,
};
PyMODINIT_FUNC PyInit__raw_value(void) {
PyObject *created = PyModule_Create(&module);
if (created == NULL) return NULL;
ExtractState *state = PyModule_GetState(created);
static const char *names[4] = {"msb", "lsb", "size", "is_little_endian"};
for (int i = 0; i < 4; ++i) {
state->names[i] = PyUnicode_InternFromString(names[i]);
if (state->names[i] == NULL) {
Py_DECREF(created);
return NULL;
}
}
return created;
}
Binary file not shown.
-23
View File
@@ -6,11 +6,6 @@ from functools import cache
from opendbc import DBC_PATH, get_generated_dbcs
try:
from opendbc.can._raw_value import get_raw_value as _native_raw_value
except ImportError:
_native_raw_value = None
# TODO: these should just be passed in along with the DBC file
from opendbc.car.honda.hondacan import honda_checksum
from opendbc.car.toyota.toyotacan import toyota_checksum
@@ -66,24 +61,6 @@ class Signal:
checksum_fields: tuple['Signal', ...] = ()
def get_raw_value(self, dat: bytes | bytearray) -> int:
if _native_raw_value is not None:
value = _native_raw_value(self, dat, Signal)
if value is not NotImplemented:
return value
msb_byte = self.msb // 8
lsb_byte = self.lsb // 8
if self.is_little_endian:
first, last = lsb_byte, msb_byte
contiguous = self.size == self.msb - self.lsb + 1
byteorder = "little"
else:
first, last = msb_byte, lsb_byte
contiguous = self.size == 8 * (last - first) + self.msb % 8 - self.lsb % 8 + 1
byteorder = "big"
if contiguous and 0 < self.size and 0 <= first <= last < len(dat):
raw = dat[first] if first == last else int.from_bytes(dat[first:last + 1], byteorder)
return (raw >> (self.lsb % 8)) & ((1 << self.size) - 1)
ret = 0
i = self.msb // 8
bits = self.size
-12
View File
@@ -56,18 +56,6 @@ class CANPacker:
def set_value(msg: bytearray, sig: Signal, ival: int) -> None:
if type(msg) is bytearray and type(sig) is Signal and type(ival) is int:
lsb = getattr(sig, 'lsb', None)
size = getattr(sig, 'size', None)
little_endian = getattr(sig, 'is_little_endian', None)
if type(lsb) is int and type(size) is int and type(little_endian) is bool:
i, shift = lsb // 8, lsb % 8
if 0 < size <= 8 - shift and 0 <= i < len(msg):
field_mask = (1 << size) - 1
mask = field_mask << shift
msg[i] = (msg[i] & ~mask) | ((ival & field_mask) << shift)
return
i = sig.lsb // 8
bits = sig.size
if sig.size < 64:
+2 -2
View File
@@ -204,8 +204,8 @@ class CANParser:
strings = [strings]
for addr in self.addresses:
for values in self.vl_all[addr].values():
values.clear()
for k in self.vl_all[addr]:
self.vl_all[addr][k].clear()
updated_addrs: set[int] = set()
for entry in strings:
@@ -1,146 +0,0 @@
import builtins
import importlib.util
import sys
import pytest
from opendbc.can import dbc
from opendbc.can.tests.test_signal_raw_value import make_signal, raw_reference
@pytest.fixture(params=['python', 'native'])
def decoder(request, monkeypatch):
native = dbc._native_raw_value
if request.param == 'native' and native is None:
pytest.skip('optional raw CAN extension is not built')
monkeypatch.setattr(dbc, '_native_raw_value', native if request.param == 'native' else None)
return dbc.Signal
@pytest.fixture
def native():
if dbc._native_raw_value is None:
pytest.skip('optional raw CAN extension is not built')
return dbc._native_raw_value
@pytest.mark.parametrize('little', [False, True])
@pytest.mark.parametrize('container', [bytes, bytearray])
def test_raw_widths_offsets_and_unsigned_high_bit(decoder, little, container):
data = container(bytes(range(64)))
for base in (0, 16, 55):
for alignment in range(8):
start = base * 8 + alignment
available = 512 - start if little else alignment + 1 + 8 * (63 - base)
for size in range(1, min(64, available) + 1):
signal = make_signal(start, size, little)
assert signal.get_raw_value(data) == raw_reference(data, start, size, little)
signal = make_signal(7 if little else 0, 64, little)
assert signal.get_raw_value(container(b'\xff' * 9)) == 2**64 - 1
@pytest.mark.parametrize('msb,lsb,size,little,payload,expected', [
(23, 0, 24, True, b'\xab\xcd', 0),
(7, 16, 24, False, b'\xab\xcd', 0xABCD00),
(15, 0, 8, True, b'\xab\xcd', 0xCD),
(7, 16, 16, False, b'\xab\xcd\xef', 0xABCD),
(7, 0, 0, True, b'\xff', 0),
(7, 0, -1, False, b'\xff', 0),
(23, 24, 16, False, b'\xab\xcd', 0),
(7, 0, 8, True, b'', 0),
])
def test_raw_partial_and_noncanonical_fallback(decoder, msb, lsb, size, little, payload, expected):
signal = decoder('TEST', msb, msb, lsb, size, False, 1., 0., little)
assert signal.get_raw_value(payload) == expected
def test_raw_full_width_python_fallback(decoder):
for little in (False, True):
signal = make_signal(0 if little else 7, 512, little)
assert signal.get_raw_value(b'\xff' * 64) == 2**512 - 1
def test_raw_noncanonical_exception_order(decoder):
for msb, lsb, size, little in ((7, 4, 2, True), (3, 0, 2, False)):
signal = decoder('TEST', msb, msb, lsb, size, False, 1., 0., little)
with pytest.raises(ValueError, match='negative shift count'):
signal.get_raw_value(b'\xff')
def test_raw_geometry_updates_and_custom_values(decoder):
class EqualInt(int):
pass
signal = make_signal(0, 16, True)
assert signal.get_raw_value(b'\xab\xcd') == 0xCDAB
signal.size = 8
assert signal.get_raw_value(b'\xab\xcd') == 0xCD
signal.__dict__.update(msb=7, lsb=0, size=8, is_little_endian=True)
assert signal.get_raw_value(b'\xab\xcd') == 0xAB
signal.__dict__['size'] = EqualInt(8)
assert signal.get_raw_value(b'\xab\xcd') == 0xAB
signal.__dict__['msb'] = 2**100
assert signal.get_raw_value(b'\xab\xcd') == 0
@pytest.mark.parametrize('container', [memoryview, list])
def test_raw_custom_payload_falls_back(decoder, container):
signal = make_signal(0, 16, True)
assert signal.get_raw_value(container(b'\xab\xcd')) == 0xCDAB
def test_native_rejects_subclass_without_reading_fields(native):
class CustomSignal(dbc.Signal):
def __getattribute__(self, name):
if name in ('msb', 'lsb', 'size', 'is_little_endian'):
raise AssertionError('native must not read subclass fields')
return super().__getattribute__(name)
signal = CustomSignal('CUSTOM', 0, 7, 0, 8, False, 1., 0., True)
assert native(signal, b'\xab', dbc.Signal) is NotImplemented
def test_native_keeps_mutable_payload_after_attribute_callbacks(native):
class CallbackSignal:
lsb, size, is_little_endian = 0, 8, True
def __init__(self, payload, length):
self.payload, self.length = payload, length
@property
def msb(self):
self.payload[:] = b'\xcd' * self.length
return 7
for length in (0, 4096):
payload = bytearray(b'\xab' * 64)
result = native(CallbackSignal(payload, length), payload, CallbackSignal)
assert result is NotImplemented if length == 0 else result == 0xCD
def test_native_unsupported_keeps_attribute_references(native):
value = int('1000000000000000000000000000000000000000')
signal = dbc.Signal('REFS', 0, value, 0, 8, False, 1., 0., True)
before = sys.getrefcount(value)
for _ in range(1000):
assert native(signal, b'\xff', dbc.Signal) is NotImplemented
assert sys.getrefcount(value) == before
def test_python_import_without_native_extension(monkeypatch):
original_import = builtins.__import__
def without_native(name, module_globals=None, module_locals=None, fromlist=(), level=0):
if name == 'opendbc.can._raw_value':
raise ImportError('optional extension unavailable')
return original_import(name, module_globals, module_locals, fromlist, level)
name = '_dbc_python_fallback_test'
spec = importlib.util.spec_from_file_location(name, dbc.__file__)
module = importlib.util.module_from_spec(spec)
monkeypatch.setitem(sys.modules, name, module)
monkeypatch.setattr(builtins, '__import__', without_native)
spec.loader.exec_module(module)
assert module._native_raw_value is None
signal = module.Signal('FALLBACK', 0, 15, 0, 16, False, 1., 0., True)
assert signal.get_raw_value(b'\xab\xcd') == 0xCDAB

Some files were not shown because too many files have changed in this diff Show More