Compare commits

..

15 Commits

Author SHA1 Message Date
Isaac Barham aefef88446 modeld_v2: floor turn steering at 0.75x the model's plan
While a turn desire is active, the action head can under-commit on turn
entry even though the model's own plan already has the turn, so the car
swings wide into the far lane. Keep the curvature command from falling
below 0.75x the plan-derived curvature, in the turn direction only. The
plan-derived curvature reads ~15-20% tight against driven turns, hence
the scale.

Offline replay of 42 logged turns (Cinque Terre V3): a 0.7 floor halved
the median wide miss with no added inside drift on turns openpilot made
alone; 0.8 nearly closed it with more inside drift. 0.75 splits the two.
Not yet tested on the car.
2026-11-03 20:51:28 -05:00
Isaac Barham cff1a7c1ea Load navigation basemaps without a comma account
The imported Api wrapper has no dongle_id attribute, so the original tile
loader failed before its first HTTP request and mapsd supplied black frames.
Replace that account-dependent lookup with supported public Protomaps build
discovery and bounded HTTP byte ranges. Pin archive URL, size and strong ETag,
isolate cache identities and retain strict integrity/freshness checks.

Show the specific model-side waiting reason for the matching signal and route,
and display OpenStreetMap attribution over the preview without changing model
pixels. Keep the selected navigation model, steering, cooperative steering,
action conversion and all route ownership gates unchanged.

Validation includes real public index/range requests and a saved-route raster
accepted by the unchanged navigation input buffer, dedicated public transport
tests, navigation lifecycle and original routing regressions, and UI reason
tests. This is transport/input validation; target-device consumption and driven
turn quality still require verification.
2026-11-03 15:07:18 -05:00
Isaac Barham c2588f3958 Accept official wrapped model metadata without weakening protocol validation
The exact RL navigation ONNX stores output_slices as newline-wrapped Base64.
The installer used strict Base64 decoding without removing format whitespace,
so it rejected the successfully compiled model. Synthetic metadata fixtures
were unwrapped and missed the bug.

Normalize only ASCII whitespace before strict decoding, keeping the restricted
unpickler and all shape, dtype, device, state-pair and action-slice checks. Add
an exact-source metadata fixture and run it through the actual plain artifact
writer, reader and installed-bundle validator. Keep one rejected completed build
and diagnostic snapshot outside normal registration for future investigation.

Validation: 14 model/metadata tests, 18 preparation tests including real cereal,
3 real worker subprocess tests; Ruff and diff checks pass. Invalid encoding,
executable metadata and incompatible slices remain rejected. No steering,
cooperative steering, action conversion or compiled-model source changes.
2026-11-03 14:26:16 -05:00
Isaac Barham 6195f92398 Fix navigation preparation startup and report its actual stage
Preparation read the unregistered IsOnroad key before its error handler, so
native Params raised UnknownKeyName and left the UI at the initial checking
message. Require registered IsOffroad=True, catch startup errors, capture worker
stderr, and bound waiting for its first status. Report checking, verification
and building separately and remove the unmeasured duration claim.

Validation: exact exception confirmed on the user's offroad device; actual
compiler/libusb/LLVM preflight passes with 8.84 GiB free. Software regressions
use strict registered Params keys and real worker subprocesses for bootstrap,
import failure and startup timeout. No GPU compilation result claimed here.
2026-11-03 14:08:11 -05:00
Isaac Barham 18b7a83280 Prepare navigation model on device and release completed lane-change ownership
Add a parked Models action that downloads the exact official-SHA navigation
ONNX, checks target requirements, compiles and validates it, and exposes the
result through the normal Chestnut selector. Keep model selection explicit,
show preparation errors and support cancellation/onroad abort. Preserve the
active local model descriptor when clearing cache.

Keep a signal route prefetched during lane changes and allow navigation only
after the existing helper's normal completion condition. Recover failed initial
Mapbox discovery probes with one bounded endpoint-radius retry while preserving
actual turn validation. Leave steering, cooperative steering, action conversion
and the 1.6 m/s handling unchanged.

Validation: 240 software tests, including native cereal selector serialization,
cached activation without network, preparation lifecycle, route/model coherence,
lane-change behavior and original route-engine regressions; Ruff and diff checks.
No target GPU build/load or physical turn-clearance result is claimed.
2026-11-03 13:52:43 -05:00
Isaac Barham 2bb640ff21 Support experimental signal routes with learned navigation inputs
Install coherent transient Directions routes without replacing manual
destinations, and feed fresh generation-matched maps and maneuver vectors to
compatible navigation-trained models. Preserve existing action conversion,
the 1.6 m/s hold and Tesla/cooperative steering.

Keep signal intent pending during discovery. Preserve normal lane-change
eligibility, commit the first actual LC request or eligible mapped turn, and
prevent later same-signal role changes. Guard cancellation, retained exits,
manual handoff, stale input and delayed parameter writes.

Add explicit verified local model registration and an offroad target build
recipe. No navigation model or automatic model swap is bundled.

Validation: 223 software test methods, Ruff and syntax checks; separately,
800 Mac forward calls compare an available RL navigation model with and
without real short-route inputs. Automatic intent still misses recorded
requests; target compilation, device parity and physical clearance remain
unverified. Keep this opt-in and experimental.
2026-11-03 13:31:47 -05:00
Isaac Barham 3b1816ae6c navigation: add experimental signal-selected Mapbox turn cues
Use a physical turn signal to select the next router-validated street without
setting a destination. Track its incoming approach and time the existing model
turn cue from fresh localization; slow approaches also require a sustained
forecast of movement from the model's own trajectory.

Keep this opt-in and preserve legacy timing when disabled. Retain existing
speed, blindspot and one-cue rules; leave steering, minimum curvature handling
and Tesla cooperative steering unchanged. Show map availability on the MICI
navigation panel and record cue provenance.

Ship a reproducibly rebuilt ARM64 Params library for the prebuilt branch,
including stable buffers for filtered-key enumeration. Software validation:
167 tests, lint, compile, and 52,061-row disabled-mode equivalence. Actual CTV3
recorded-view pilots are mixed; the full saved-route sweep remains in progress.
This is an experimental test build, with no physical curb-clearance claim.
2026-11-03 12:11:07 -05:00
Isaac Barham f5d861d86d modeld: restore 1.6 m/s curvature hold threshold
Restore the original minimum lateral-control speed in both model runners. At or below 1.6 m/s, keep the previous desired curvature. This does not disable steering or replace the model speed input.

Validation: both files byte-match commit 32f16216977bceb42923eb7eeaa5dd926979ce65; Python syntax, scoped Ruff and diff checks pass. Turn-cue and cooperative-steering behavior are otherwise unchanged.
2026-11-03 08:02:15 -05:00
Isaac Barham 995e5d49e9 turns: prevent repeated cues under a held signal
Keep the consumed turn cue separate from temporary speed and blind-spot eligibility. Canceling the signal or changing sides permits the next cue; hazards and feature toggles cannot re-issue the old cue. Preserve existing first-release conditions.

Validation: 46 focused controller cases, 14 source-level DesireHelper/AutoLaneChange cases, and 7 host pulse scenarios pass. Actual CTV3 recorded-input replay removes the duplicate while preserving the first cue. Existing 27 turn cases and two traffic circles retain identical neural input pulses. Recorded-view tests do not establish physical curb clearance.
2026-11-03 08:02:07 -05:00
Isaac Barham 4943158ffc navigation: support privately provisioned Mapbox routing
Load a protected public Mapbox token on the device and refresh routing when it changes. Add an authenticated setMapboxToken method with atomic private storage, and remove raw RPC logging from Athena and Sunnylink. Keep route request credentials out of error logs.

Validation: 27 navigation tests plus 8 subtests, five token RPC tests, and Ruff pass.
2026-11-02 17:05:15 -05:00
Isaac Barham 93453194f3 turns: remove stopped trigger and restore low-speed curvature
Remove stopped speed as an independent turn-cue trigger. Restore the 0.3 m/s minimum curvature-update speed in both modeld paths, undoing the changes from 32f1621. Preserve model-prediction and directional-nudge behavior and update regression coverage.

Validation: 25 focused controller source tests and 16 action-boundary probes passed.
2026-11-02 16:28:31 -05:00
Claude 32f1621697 modeld: raise minimum lateral control speed to 1.6 m/s
Port of commaai/openpilot#39174. Below this speed modeld holds the
previous desired curvature instead of following the model, so the
wheel stops chasing noisy plans while creeping or stopped.
Applied to both selfdrive/modeld and sunnypilot modeld_v2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ATDgLMxRqNpP9H2Ndeg65v
2026-11-02 02:46:33 +00:00
Claude b12077a70c lane turn desire: fix modeld crash slicing the capnp desirePrediction
modeld passes modelV2.meta.desirePrediction, a capnp list, which can't be
sliced, so model_turn_probs raised every frame and modeld never ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ATDgLMxRqNpP9H2Ndeg65v
2026-11-02 00:28:06 +00:00
Claude a6f281d2f0 lane turn desire: wait for the turn before sending the desire
The model treats the turn desire as "turn now", so sending it as soon as the
blinker is on below the lane turn speed makes the car take the first opening.
Now the blinker only arms the turn, and the desire is sent once one of these
says the turn is here:
- the model's own desirePrediction for that turn reaches 0.35 within the two
  soonest horizons
- the driver nudges the wheel toward the turn
- the car is stopped (red light, stop sign)
Once sent it stays on until the blinker goes off, so the model still sees a
single pulse.

Built on dev-chestnut, which is prebuilt, so this is always on with Lane Turn
Desires (WAIT_FOR_TURN) instead of adding a param the compiled params library
would not know.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ATDgLMxRqNpP9H2Ndeg65v
2026-11-01 20:43:04 +00:00
github-actions[bot] 46e7ff91e4 sunnypilot v2026.11.01-4997
version: sunnypilot v2026.003.000 (dev)
date: 2026-11-01T17:10:35
master commit: a86979d2b2
2026-11-01 17:10:35 +00:00
81 changed files with 10673 additions and 3407 deletions
+1
View File
@@ -0,0 +1 @@
openpilot/selfdrive/modeld/models/big_*_tinygrad.pkl filter=lfs diff=lfs merge=lfs -text
+5
View File
@@ -0,0 +1,5 @@
[lfs]
url = https://huggingface.co/sunnypilot/sunnypilot-lfs.git/info/lfs
pushurl = https://huggingface.co/sunnypilot/sunnypilot-lfs.git/info/lfs
locksverify = false
Binary file not shown.
+11 -5
View File
@@ -162,12 +162,18 @@ ParamsBuffer params_key_at(ParamsHandle *handle, size_t index) noexcept {
size_t params_keys_by_flag(ParamsHandle *handle, uint32_t flag, ParamsBuffer *out, size_t out_size) noexcept {
return translate_exceptions(size_t{0}, [&]() {
auto filtered = handle->params.allKeys(static_cast<ParamKeyFlag>(flag));
size_t count = std::min(filtered.size(), out_size);
for (size_t i = 0; i < count; i++) {
out[i] = return_string(filtered[i]);
size_t count = 0;
// Keep each returned buffer stable and distinct until Python copies it.
// return_string() would alias one thread-local buffer for every entry.
for (const auto &key : handle->keys) {
if (flag == ALL || (handle->params.getKeyFlag(key) & flag)) {
if (count < out_size) {
out[count] = {key.data(), key.size()};
}
++count;
}
}
return filtered.size();
return count;
});
}
+6 -1
View File
@@ -149,7 +149,6 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"Version", {PERSISTENT, STRING}},
// --- sunnypilot params --- //
{"AccelBoostSP", {PERSISTENT | BACKUP, BOOL, "1"}},
{"ApiCache_DriveStats", {PERSISTENT, JSON}},
{"AutoLaneChangeBsmDelay", {PERSISTENT | BACKUP, BOOL, "0"}},
{"AutoLaneChangeTimer", {PERSISTENT | BACKUP, INT, "0"}},
@@ -258,6 +257,12 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"LagdValueCache", {PERSISTENT, FLOAT, "0.2"}},
{"LaneTurnDesire", {PERSISTENT | BACKUP, BOOL, "0"}},
{"LaneTurnValue", {PERSISTENT | BACKUP, FLOAT, "19.0"}},
{"MapboxNextTurnEnabled", {PERSISTENT | BACKUP, BOOL, "0"}},
{"MapboxTurnLeadDistance", {PERSISTENT | BACKUP, FLOAT, "20.0"}},
{"MapboxTurnIntent", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, JSON}},
{"MapboxTurnContext", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, JSON}},
{"MapboxTurnCue", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, JSON}},
{"NavModelState", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, JSON}},
{"Blendful", {PERSISTENT | BACKUP, BOOL, "0"}},
{"PlanplusControl", {PERSISTENT | BACKUP, FLOAT, "1.0"}},
@@ -0,0 +1,43 @@
"""Filtered registry names must remain distinct across the C/Python boundary."""
import tempfile
import unittest
from openpilot.common import params as native
class TestParamsFlagBuffers(unittest.TestCase):
def setUp(self):
self.directory = tempfile.TemporaryDirectory()
self.params = native.Params(self.directory.name)
def tearDown(self):
self.params._finalizer()
self.directory.cleanup()
def test_filtered_names_are_unique_registered_keys(self):
all_keys = set(self.params.all_keys())
for flag in (native.ParamKeyFlag.BACKUP, native.ParamKeyFlag.CLEAR_ON_MANAGER_START, native.ParamKeyFlag.CLEAR_ON_ONROAD_TRANSITION):
names = self.params.all_keys(flag)
self.assertEqual(len(names), len(set(names)))
self.assertTrue(set(names) <= all_keys)
backup = set(self.params.all_keys(native.ParamKeyFlag.BACKUP))
self.assertTrue({b'MapboxNextTurnEnabled', b'MapboxTurnLeadDistance'} <= backup)
self.assertTrue({b'MapboxTurnIntent', b'MapboxTurnContext', b'MapboxTurnCue'}.isdisjoint(backup))
def test_each_output_buffer_survives_another_c_api_string_result(self):
expected = self.params.all_keys(native.ParamKeyFlag.BACKUP)
output = (native.ParamsBuffer*1024)()
count = native.params_keys_by_flag(self.params.p, int(native.ParamKeyFlag.BACKUP), output, len(output))
native.params_get_default(self.params.p, b'LongitudinalPersonality')
self.assertEqual([native._copy_string(output[i]) for i in range(count)], expected)
def test_small_output_array_still_reports_total_matching_key_count(self):
expected = self.params.all_keys(native.ParamKeyFlag.BACKUP)
output = (native.ParamsBuffer*2)()
count = native.params_keys_by_flag(self.params.p, int(native.ParamKeyFlag.BACKUP), output, len(output))
self.assertEqual(count, len(expected))
self.assertEqual([native._copy_string(output[i]) for i in range(len(output))], expected[:len(output)])
if __name__ == '__main__':
unittest.main()
@@ -2,7 +2,7 @@ from openpilot.cereal import log, custom
from openpilot.common.constants import CV
from openpilot.common.realtime import DT_MDL
from openpilot.sunnypilot.selfdrive.controls.lib.auto_lane_change import AutoLaneChangeController, AutoLaneChangeMode
from openpilot.sunnypilot.selfdrive.controls.lib.lane_turn_desire import LaneTurnController
from openpilot.sunnypilot.selfdrive.controls.lib.lane_turn_desire import LaneTurnController, model_turn_probs
LaneChangeState = log.LaneChangeState
LaneChangeDirection = log.LaneChangeDirection
@@ -33,19 +33,28 @@ class DesireHelper:
def get_lane_change_direction(CS):
return LaneChangeDirection.left if CS.leftBlinker else LaneChangeDirection.right
def update(self, carstate, lateral_active, lane_change_prob, left_edge_detected=False, right_edge_detected=False):
def update(self, carstate, lateral_active, lane_change_prob, left_edge_detected=False, right_edge_detected=False, desire_prediction=None,
model_position=None, map_turn_supported=False):
self.alc.update_params()
self.lane_turn_controller.update_params()
v_ego = carstate.vEgo
one_blinker = carstate.leftBlinker != carstate.rightBlinker
below_lane_change_speed = v_ego < LANE_CHANGE_SPEED_MIN
lane_change_completed = False
# Lane turn controller update
self.lane_turn_controller.update_lane_turn(blindspot_left=carstate.leftBlindspot, blindspot_right=carstate.rightBlindspot,
left_blinker=carstate.leftBlinker, right_blinker=carstate.rightBlinker, v_ego=v_ego)
left_blinker=carstate.leftBlinker, right_blinker=carstate.rightBlinker, v_ego=v_ego,
steering_pressed=carstate.steeringPressed, steering_torque=carstate.steeringTorque,
turn_probs=model_turn_probs(desire_prediction), lateral_active=lateral_active,
model_position=model_position, map_turn_supported=map_turn_supported)
self.lane_turn_direction = self.lane_turn_controller.get_turn_direction()
if not lateral_active or self.lane_change_timer > LANE_CHANGE_TIME_MAX or self.alc.lane_change_set_timer == AutoLaneChangeMode.OFF:
# Only an already committed turn owns the signal. Pending map discovery
# leaves the existing lane-change eligibility and timing unchanged.
navigation_signal = self.lane_turn_controller.navigation_owns_signal
if (navigation_signal or not lateral_active or self.lane_change_timer > LANE_CHANGE_TIME_MAX or
self.alc.lane_change_set_timer == AutoLaneChangeMode.OFF):
self.lane_change_state = LaneChangeState.off
self.lane_change_direction = LaneChangeDirection.none
self.lane_change_timer = 0.0
@@ -84,6 +93,7 @@ class DesireHelper:
self.lane_change_timer += DT_MDL
if lane_change_prob < 0.02 and self.lane_change_timer >= LANE_CHANGE_START_TIME:
lane_change_completed = True
self.lane_change_timer = 0.0
if one_blinker:
self.lane_change_state = LaneChangeState.preLaneChange
@@ -104,4 +114,9 @@ class DesireHelper:
elif self.lane_change_direction == LaneChangeDirection.right:
self.desire = log.Desire.laneChangeRight
# LC wins while active. Only its normal completion releases its ownership;
# speed changes, timeout and disengagement do not indicate completion.
self.lane_turn_controller.finalize_navigation_intent(
self.desire in (log.Desire.laneChangeLeft, log.Desire.laneChangeRight), v_ego, lateral_active, map_turn_supported,
lane_change_completed=lane_change_completed)
self.alc.update_state()
@@ -41,326 +41,326 @@ const static double MAHA_THRESH_31 = 3.8414588206941227;
* *
* This file is part of 'ekf' *
******************************************************************************/
void err_fun(double *nom_x, double *delta_x, double *out_7776651228618240194) {
out_7776651228618240194[0] = delta_x[0] + nom_x[0];
out_7776651228618240194[1] = delta_x[1] + nom_x[1];
out_7776651228618240194[2] = delta_x[2] + nom_x[2];
out_7776651228618240194[3] = delta_x[3] + nom_x[3];
out_7776651228618240194[4] = delta_x[4] + nom_x[4];
out_7776651228618240194[5] = delta_x[5] + nom_x[5];
out_7776651228618240194[6] = delta_x[6] + nom_x[6];
out_7776651228618240194[7] = delta_x[7] + nom_x[7];
out_7776651228618240194[8] = delta_x[8] + nom_x[8];
void err_fun(double *nom_x, double *delta_x, double *out_3585265450871354510) {
out_3585265450871354510[0] = delta_x[0] + nom_x[0];
out_3585265450871354510[1] = delta_x[1] + nom_x[1];
out_3585265450871354510[2] = delta_x[2] + nom_x[2];
out_3585265450871354510[3] = delta_x[3] + nom_x[3];
out_3585265450871354510[4] = delta_x[4] + nom_x[4];
out_3585265450871354510[5] = delta_x[5] + nom_x[5];
out_3585265450871354510[6] = delta_x[6] + nom_x[6];
out_3585265450871354510[7] = delta_x[7] + nom_x[7];
out_3585265450871354510[8] = delta_x[8] + nom_x[8];
}
void inv_err_fun(double *nom_x, double *true_x, double *out_5273421026182551058) {
out_5273421026182551058[0] = -nom_x[0] + true_x[0];
out_5273421026182551058[1] = -nom_x[1] + true_x[1];
out_5273421026182551058[2] = -nom_x[2] + true_x[2];
out_5273421026182551058[3] = -nom_x[3] + true_x[3];
out_5273421026182551058[4] = -nom_x[4] + true_x[4];
out_5273421026182551058[5] = -nom_x[5] + true_x[5];
out_5273421026182551058[6] = -nom_x[6] + true_x[6];
out_5273421026182551058[7] = -nom_x[7] + true_x[7];
out_5273421026182551058[8] = -nom_x[8] + true_x[8];
void inv_err_fun(double *nom_x, double *true_x, double *out_4534440969468340130) {
out_4534440969468340130[0] = -nom_x[0] + true_x[0];
out_4534440969468340130[1] = -nom_x[1] + true_x[1];
out_4534440969468340130[2] = -nom_x[2] + true_x[2];
out_4534440969468340130[3] = -nom_x[3] + true_x[3];
out_4534440969468340130[4] = -nom_x[4] + true_x[4];
out_4534440969468340130[5] = -nom_x[5] + true_x[5];
out_4534440969468340130[6] = -nom_x[6] + true_x[6];
out_4534440969468340130[7] = -nom_x[7] + true_x[7];
out_4534440969468340130[8] = -nom_x[8] + true_x[8];
}
void H_mod_fun(double *state, double *out_2181953899905443363) {
out_2181953899905443363[0] = 1.0;
out_2181953899905443363[1] = 0.0;
out_2181953899905443363[2] = 0.0;
out_2181953899905443363[3] = 0.0;
out_2181953899905443363[4] = 0.0;
out_2181953899905443363[5] = 0.0;
out_2181953899905443363[6] = 0.0;
out_2181953899905443363[7] = 0.0;
out_2181953899905443363[8] = 0.0;
out_2181953899905443363[9] = 0.0;
out_2181953899905443363[10] = 1.0;
out_2181953899905443363[11] = 0.0;
out_2181953899905443363[12] = 0.0;
out_2181953899905443363[13] = 0.0;
out_2181953899905443363[14] = 0.0;
out_2181953899905443363[15] = 0.0;
out_2181953899905443363[16] = 0.0;
out_2181953899905443363[17] = 0.0;
out_2181953899905443363[18] = 0.0;
out_2181953899905443363[19] = 0.0;
out_2181953899905443363[20] = 1.0;
out_2181953899905443363[21] = 0.0;
out_2181953899905443363[22] = 0.0;
out_2181953899905443363[23] = 0.0;
out_2181953899905443363[24] = 0.0;
out_2181953899905443363[25] = 0.0;
out_2181953899905443363[26] = 0.0;
out_2181953899905443363[27] = 0.0;
out_2181953899905443363[28] = 0.0;
out_2181953899905443363[29] = 0.0;
out_2181953899905443363[30] = 1.0;
out_2181953899905443363[31] = 0.0;
out_2181953899905443363[32] = 0.0;
out_2181953899905443363[33] = 0.0;
out_2181953899905443363[34] = 0.0;
out_2181953899905443363[35] = 0.0;
out_2181953899905443363[36] = 0.0;
out_2181953899905443363[37] = 0.0;
out_2181953899905443363[38] = 0.0;
out_2181953899905443363[39] = 0.0;
out_2181953899905443363[40] = 1.0;
out_2181953899905443363[41] = 0.0;
out_2181953899905443363[42] = 0.0;
out_2181953899905443363[43] = 0.0;
out_2181953899905443363[44] = 0.0;
out_2181953899905443363[45] = 0.0;
out_2181953899905443363[46] = 0.0;
out_2181953899905443363[47] = 0.0;
out_2181953899905443363[48] = 0.0;
out_2181953899905443363[49] = 0.0;
out_2181953899905443363[50] = 1.0;
out_2181953899905443363[51] = 0.0;
out_2181953899905443363[52] = 0.0;
out_2181953899905443363[53] = 0.0;
out_2181953899905443363[54] = 0.0;
out_2181953899905443363[55] = 0.0;
out_2181953899905443363[56] = 0.0;
out_2181953899905443363[57] = 0.0;
out_2181953899905443363[58] = 0.0;
out_2181953899905443363[59] = 0.0;
out_2181953899905443363[60] = 1.0;
out_2181953899905443363[61] = 0.0;
out_2181953899905443363[62] = 0.0;
out_2181953899905443363[63] = 0.0;
out_2181953899905443363[64] = 0.0;
out_2181953899905443363[65] = 0.0;
out_2181953899905443363[66] = 0.0;
out_2181953899905443363[67] = 0.0;
out_2181953899905443363[68] = 0.0;
out_2181953899905443363[69] = 0.0;
out_2181953899905443363[70] = 1.0;
out_2181953899905443363[71] = 0.0;
out_2181953899905443363[72] = 0.0;
out_2181953899905443363[73] = 0.0;
out_2181953899905443363[74] = 0.0;
out_2181953899905443363[75] = 0.0;
out_2181953899905443363[76] = 0.0;
out_2181953899905443363[77] = 0.0;
out_2181953899905443363[78] = 0.0;
out_2181953899905443363[79] = 0.0;
out_2181953899905443363[80] = 1.0;
void H_mod_fun(double *state, double *out_7815971298073658297) {
out_7815971298073658297[0] = 1.0;
out_7815971298073658297[1] = 0.0;
out_7815971298073658297[2] = 0.0;
out_7815971298073658297[3] = 0.0;
out_7815971298073658297[4] = 0.0;
out_7815971298073658297[5] = 0.0;
out_7815971298073658297[6] = 0.0;
out_7815971298073658297[7] = 0.0;
out_7815971298073658297[8] = 0.0;
out_7815971298073658297[9] = 0.0;
out_7815971298073658297[10] = 1.0;
out_7815971298073658297[11] = 0.0;
out_7815971298073658297[12] = 0.0;
out_7815971298073658297[13] = 0.0;
out_7815971298073658297[14] = 0.0;
out_7815971298073658297[15] = 0.0;
out_7815971298073658297[16] = 0.0;
out_7815971298073658297[17] = 0.0;
out_7815971298073658297[18] = 0.0;
out_7815971298073658297[19] = 0.0;
out_7815971298073658297[20] = 1.0;
out_7815971298073658297[21] = 0.0;
out_7815971298073658297[22] = 0.0;
out_7815971298073658297[23] = 0.0;
out_7815971298073658297[24] = 0.0;
out_7815971298073658297[25] = 0.0;
out_7815971298073658297[26] = 0.0;
out_7815971298073658297[27] = 0.0;
out_7815971298073658297[28] = 0.0;
out_7815971298073658297[29] = 0.0;
out_7815971298073658297[30] = 1.0;
out_7815971298073658297[31] = 0.0;
out_7815971298073658297[32] = 0.0;
out_7815971298073658297[33] = 0.0;
out_7815971298073658297[34] = 0.0;
out_7815971298073658297[35] = 0.0;
out_7815971298073658297[36] = 0.0;
out_7815971298073658297[37] = 0.0;
out_7815971298073658297[38] = 0.0;
out_7815971298073658297[39] = 0.0;
out_7815971298073658297[40] = 1.0;
out_7815971298073658297[41] = 0.0;
out_7815971298073658297[42] = 0.0;
out_7815971298073658297[43] = 0.0;
out_7815971298073658297[44] = 0.0;
out_7815971298073658297[45] = 0.0;
out_7815971298073658297[46] = 0.0;
out_7815971298073658297[47] = 0.0;
out_7815971298073658297[48] = 0.0;
out_7815971298073658297[49] = 0.0;
out_7815971298073658297[50] = 1.0;
out_7815971298073658297[51] = 0.0;
out_7815971298073658297[52] = 0.0;
out_7815971298073658297[53] = 0.0;
out_7815971298073658297[54] = 0.0;
out_7815971298073658297[55] = 0.0;
out_7815971298073658297[56] = 0.0;
out_7815971298073658297[57] = 0.0;
out_7815971298073658297[58] = 0.0;
out_7815971298073658297[59] = 0.0;
out_7815971298073658297[60] = 1.0;
out_7815971298073658297[61] = 0.0;
out_7815971298073658297[62] = 0.0;
out_7815971298073658297[63] = 0.0;
out_7815971298073658297[64] = 0.0;
out_7815971298073658297[65] = 0.0;
out_7815971298073658297[66] = 0.0;
out_7815971298073658297[67] = 0.0;
out_7815971298073658297[68] = 0.0;
out_7815971298073658297[69] = 0.0;
out_7815971298073658297[70] = 1.0;
out_7815971298073658297[71] = 0.0;
out_7815971298073658297[72] = 0.0;
out_7815971298073658297[73] = 0.0;
out_7815971298073658297[74] = 0.0;
out_7815971298073658297[75] = 0.0;
out_7815971298073658297[76] = 0.0;
out_7815971298073658297[77] = 0.0;
out_7815971298073658297[78] = 0.0;
out_7815971298073658297[79] = 0.0;
out_7815971298073658297[80] = 1.0;
}
void f_fun(double *state, double dt, double *out_2503069281528488208) {
out_2503069281528488208[0] = state[0];
out_2503069281528488208[1] = state[1];
out_2503069281528488208[2] = state[2];
out_2503069281528488208[3] = state[3];
out_2503069281528488208[4] = state[4];
out_2503069281528488208[5] = dt*(stiffness_front*(-state[2] - state[3] + state[7])*state[0]/state[1] + (-stiffness_front*state[0] - stiffness_rear*state[0])*state[5]/state[4] + ((-center_to_front*stiffness_front*state[0] + center_to_rear*stiffness_rear*state[0])/state[4] - state[4])*state[6] - 9.8100000000000005*state[8]) + state[5];
out_2503069281528488208[6] = dt*(center_to_front*stiffness_front*(-state[2] - state[3] + state[7])*state[0]/(rotational_inertia*state[1]) + (-center_to_front*stiffness_front*state[0] + center_to_rear*stiffness_rear*state[0])*state[5]/(rotational_inertia*state[4]) + (-pow(center_to_front, 2)*stiffness_front*state[0] - pow(center_to_rear, 2)*stiffness_rear*state[0])*state[6]/(rotational_inertia*state[4])) + state[6];
out_2503069281528488208[7] = state[7];
out_2503069281528488208[8] = state[8];
void f_fun(double *state, double dt, double *out_3011916394083426809) {
out_3011916394083426809[0] = state[0];
out_3011916394083426809[1] = state[1];
out_3011916394083426809[2] = state[2];
out_3011916394083426809[3] = state[3];
out_3011916394083426809[4] = state[4];
out_3011916394083426809[5] = dt*(stiffness_front*(-state[2] - state[3] + state[7])*state[0]/state[1] + (-stiffness_front*state[0] - stiffness_rear*state[0])*state[5]/state[4] + ((-center_to_front*stiffness_front*state[0] + center_to_rear*stiffness_rear*state[0])/state[4] - state[4])*state[6] - 9.8100000000000005*state[8]) + state[5];
out_3011916394083426809[6] = dt*(center_to_front*stiffness_front*(-state[2] - state[3] + state[7])*state[0]/(rotational_inertia*state[1]) + (-center_to_front*stiffness_front*state[0] + center_to_rear*stiffness_rear*state[0])*state[5]/(rotational_inertia*state[4]) + (-pow(center_to_front, 2)*stiffness_front*state[0] - pow(center_to_rear, 2)*stiffness_rear*state[0])*state[6]/(rotational_inertia*state[4])) + state[6];
out_3011916394083426809[7] = state[7];
out_3011916394083426809[8] = state[8];
}
void F_fun(double *state, double dt, double *out_1406798686887127281) {
out_1406798686887127281[0] = 1;
out_1406798686887127281[1] = 0;
out_1406798686887127281[2] = 0;
out_1406798686887127281[3] = 0;
out_1406798686887127281[4] = 0;
out_1406798686887127281[5] = 0;
out_1406798686887127281[6] = 0;
out_1406798686887127281[7] = 0;
out_1406798686887127281[8] = 0;
out_1406798686887127281[9] = 0;
out_1406798686887127281[10] = 1;
out_1406798686887127281[11] = 0;
out_1406798686887127281[12] = 0;
out_1406798686887127281[13] = 0;
out_1406798686887127281[14] = 0;
out_1406798686887127281[15] = 0;
out_1406798686887127281[16] = 0;
out_1406798686887127281[17] = 0;
out_1406798686887127281[18] = 0;
out_1406798686887127281[19] = 0;
out_1406798686887127281[20] = 1;
out_1406798686887127281[21] = 0;
out_1406798686887127281[22] = 0;
out_1406798686887127281[23] = 0;
out_1406798686887127281[24] = 0;
out_1406798686887127281[25] = 0;
out_1406798686887127281[26] = 0;
out_1406798686887127281[27] = 0;
out_1406798686887127281[28] = 0;
out_1406798686887127281[29] = 0;
out_1406798686887127281[30] = 1;
out_1406798686887127281[31] = 0;
out_1406798686887127281[32] = 0;
out_1406798686887127281[33] = 0;
out_1406798686887127281[34] = 0;
out_1406798686887127281[35] = 0;
out_1406798686887127281[36] = 0;
out_1406798686887127281[37] = 0;
out_1406798686887127281[38] = 0;
out_1406798686887127281[39] = 0;
out_1406798686887127281[40] = 1;
out_1406798686887127281[41] = 0;
out_1406798686887127281[42] = 0;
out_1406798686887127281[43] = 0;
out_1406798686887127281[44] = 0;
out_1406798686887127281[45] = dt*(stiffness_front*(-state[2] - state[3] + state[7])/state[1] + (-stiffness_front - stiffness_rear)*state[5]/state[4] + (-center_to_front*stiffness_front + center_to_rear*stiffness_rear)*state[6]/state[4]);
out_1406798686887127281[46] = -dt*stiffness_front*(-state[2] - state[3] + state[7])*state[0]/pow(state[1], 2);
out_1406798686887127281[47] = -dt*stiffness_front*state[0]/state[1];
out_1406798686887127281[48] = -dt*stiffness_front*state[0]/state[1];
out_1406798686887127281[49] = dt*(-(-stiffness_front*state[0] - stiffness_rear*state[0])*state[5]/pow(state[4], 2) + (-(-center_to_front*stiffness_front*state[0] + center_to_rear*stiffness_rear*state[0])/pow(state[4], 2) - 1)*state[6]);
out_1406798686887127281[50] = dt*(-stiffness_front*state[0] - stiffness_rear*state[0])/state[4] + 1;
out_1406798686887127281[51] = dt*((-center_to_front*stiffness_front*state[0] + center_to_rear*stiffness_rear*state[0])/state[4] - state[4]);
out_1406798686887127281[52] = dt*stiffness_front*state[0]/state[1];
out_1406798686887127281[53] = -9.8100000000000005*dt;
out_1406798686887127281[54] = dt*(center_to_front*stiffness_front*(-state[2] - state[3] + state[7])/(rotational_inertia*state[1]) + (-center_to_front*stiffness_front + center_to_rear*stiffness_rear)*state[5]/(rotational_inertia*state[4]) + (-pow(center_to_front, 2)*stiffness_front - pow(center_to_rear, 2)*stiffness_rear)*state[6]/(rotational_inertia*state[4]));
out_1406798686887127281[55] = -center_to_front*dt*stiffness_front*(-state[2] - state[3] + state[7])*state[0]/(rotational_inertia*pow(state[1], 2));
out_1406798686887127281[56] = -center_to_front*dt*stiffness_front*state[0]/(rotational_inertia*state[1]);
out_1406798686887127281[57] = -center_to_front*dt*stiffness_front*state[0]/(rotational_inertia*state[1]);
out_1406798686887127281[58] = dt*(-(-center_to_front*stiffness_front*state[0] + center_to_rear*stiffness_rear*state[0])*state[5]/(rotational_inertia*pow(state[4], 2)) - (-pow(center_to_front, 2)*stiffness_front*state[0] - pow(center_to_rear, 2)*stiffness_rear*state[0])*state[6]/(rotational_inertia*pow(state[4], 2)));
out_1406798686887127281[59] = dt*(-center_to_front*stiffness_front*state[0] + center_to_rear*stiffness_rear*state[0])/(rotational_inertia*state[4]);
out_1406798686887127281[60] = dt*(-pow(center_to_front, 2)*stiffness_front*state[0] - pow(center_to_rear, 2)*stiffness_rear*state[0])/(rotational_inertia*state[4]) + 1;
out_1406798686887127281[61] = center_to_front*dt*stiffness_front*state[0]/(rotational_inertia*state[1]);
out_1406798686887127281[62] = 0;
out_1406798686887127281[63] = 0;
out_1406798686887127281[64] = 0;
out_1406798686887127281[65] = 0;
out_1406798686887127281[66] = 0;
out_1406798686887127281[67] = 0;
out_1406798686887127281[68] = 0;
out_1406798686887127281[69] = 0;
out_1406798686887127281[70] = 1;
out_1406798686887127281[71] = 0;
out_1406798686887127281[72] = 0;
out_1406798686887127281[73] = 0;
out_1406798686887127281[74] = 0;
out_1406798686887127281[75] = 0;
out_1406798686887127281[76] = 0;
out_1406798686887127281[77] = 0;
out_1406798686887127281[78] = 0;
out_1406798686887127281[79] = 0;
out_1406798686887127281[80] = 1;
void F_fun(double *state, double dt, double *out_4768916904116545459) {
out_4768916904116545459[0] = 1;
out_4768916904116545459[1] = 0;
out_4768916904116545459[2] = 0;
out_4768916904116545459[3] = 0;
out_4768916904116545459[4] = 0;
out_4768916904116545459[5] = 0;
out_4768916904116545459[6] = 0;
out_4768916904116545459[7] = 0;
out_4768916904116545459[8] = 0;
out_4768916904116545459[9] = 0;
out_4768916904116545459[10] = 1;
out_4768916904116545459[11] = 0;
out_4768916904116545459[12] = 0;
out_4768916904116545459[13] = 0;
out_4768916904116545459[14] = 0;
out_4768916904116545459[15] = 0;
out_4768916904116545459[16] = 0;
out_4768916904116545459[17] = 0;
out_4768916904116545459[18] = 0;
out_4768916904116545459[19] = 0;
out_4768916904116545459[20] = 1;
out_4768916904116545459[21] = 0;
out_4768916904116545459[22] = 0;
out_4768916904116545459[23] = 0;
out_4768916904116545459[24] = 0;
out_4768916904116545459[25] = 0;
out_4768916904116545459[26] = 0;
out_4768916904116545459[27] = 0;
out_4768916904116545459[28] = 0;
out_4768916904116545459[29] = 0;
out_4768916904116545459[30] = 1;
out_4768916904116545459[31] = 0;
out_4768916904116545459[32] = 0;
out_4768916904116545459[33] = 0;
out_4768916904116545459[34] = 0;
out_4768916904116545459[35] = 0;
out_4768916904116545459[36] = 0;
out_4768916904116545459[37] = 0;
out_4768916904116545459[38] = 0;
out_4768916904116545459[39] = 0;
out_4768916904116545459[40] = 1;
out_4768916904116545459[41] = 0;
out_4768916904116545459[42] = 0;
out_4768916904116545459[43] = 0;
out_4768916904116545459[44] = 0;
out_4768916904116545459[45] = dt*(stiffness_front*(-state[2] - state[3] + state[7])/state[1] + (-stiffness_front - stiffness_rear)*state[5]/state[4] + (-center_to_front*stiffness_front + center_to_rear*stiffness_rear)*state[6]/state[4]);
out_4768916904116545459[46] = -dt*stiffness_front*(-state[2] - state[3] + state[7])*state[0]/pow(state[1], 2);
out_4768916904116545459[47] = -dt*stiffness_front*state[0]/state[1];
out_4768916904116545459[48] = -dt*stiffness_front*state[0]/state[1];
out_4768916904116545459[49] = dt*(-(-stiffness_front*state[0] - stiffness_rear*state[0])*state[5]/pow(state[4], 2) + (-(-center_to_front*stiffness_front*state[0] + center_to_rear*stiffness_rear*state[0])/pow(state[4], 2) - 1)*state[6]);
out_4768916904116545459[50] = dt*(-stiffness_front*state[0] - stiffness_rear*state[0])/state[4] + 1;
out_4768916904116545459[51] = dt*((-center_to_front*stiffness_front*state[0] + center_to_rear*stiffness_rear*state[0])/state[4] - state[4]);
out_4768916904116545459[52] = dt*stiffness_front*state[0]/state[1];
out_4768916904116545459[53] = -9.8100000000000005*dt;
out_4768916904116545459[54] = dt*(center_to_front*stiffness_front*(-state[2] - state[3] + state[7])/(rotational_inertia*state[1]) + (-center_to_front*stiffness_front + center_to_rear*stiffness_rear)*state[5]/(rotational_inertia*state[4]) + (-pow(center_to_front, 2)*stiffness_front - pow(center_to_rear, 2)*stiffness_rear)*state[6]/(rotational_inertia*state[4]));
out_4768916904116545459[55] = -center_to_front*dt*stiffness_front*(-state[2] - state[3] + state[7])*state[0]/(rotational_inertia*pow(state[1], 2));
out_4768916904116545459[56] = -center_to_front*dt*stiffness_front*state[0]/(rotational_inertia*state[1]);
out_4768916904116545459[57] = -center_to_front*dt*stiffness_front*state[0]/(rotational_inertia*state[1]);
out_4768916904116545459[58] = dt*(-(-center_to_front*stiffness_front*state[0] + center_to_rear*stiffness_rear*state[0])*state[5]/(rotational_inertia*pow(state[4], 2)) - (-pow(center_to_front, 2)*stiffness_front*state[0] - pow(center_to_rear, 2)*stiffness_rear*state[0])*state[6]/(rotational_inertia*pow(state[4], 2)));
out_4768916904116545459[59] = dt*(-center_to_front*stiffness_front*state[0] + center_to_rear*stiffness_rear*state[0])/(rotational_inertia*state[4]);
out_4768916904116545459[60] = dt*(-pow(center_to_front, 2)*stiffness_front*state[0] - pow(center_to_rear, 2)*stiffness_rear*state[0])/(rotational_inertia*state[4]) + 1;
out_4768916904116545459[61] = center_to_front*dt*stiffness_front*state[0]/(rotational_inertia*state[1]);
out_4768916904116545459[62] = 0;
out_4768916904116545459[63] = 0;
out_4768916904116545459[64] = 0;
out_4768916904116545459[65] = 0;
out_4768916904116545459[66] = 0;
out_4768916904116545459[67] = 0;
out_4768916904116545459[68] = 0;
out_4768916904116545459[69] = 0;
out_4768916904116545459[70] = 1;
out_4768916904116545459[71] = 0;
out_4768916904116545459[72] = 0;
out_4768916904116545459[73] = 0;
out_4768916904116545459[74] = 0;
out_4768916904116545459[75] = 0;
out_4768916904116545459[76] = 0;
out_4768916904116545459[77] = 0;
out_4768916904116545459[78] = 0;
out_4768916904116545459[79] = 0;
out_4768916904116545459[80] = 1;
}
void h_25(double *state, double *unused, double *out_3759466007672649026) {
out_3759466007672649026[0] = state[6];
void h_25(double *state, double *unused, double *out_8627167620031889565) {
out_8627167620031889565[0] = state[6];
}
void H_25(double *state, double *unused, double *out_8343452140830248448) {
out_8343452140830248448[0] = 0;
out_8343452140830248448[1] = 0;
out_8343452140830248448[2] = 0;
out_8343452140830248448[3] = 0;
out_8343452140830248448[4] = 0;
out_8343452140830248448[5] = 0;
out_8343452140830248448[6] = 1;
out_8343452140830248448[7] = 0;
out_8343452140830248448[8] = 0;
void H_25(double *state, double *unused, double *out_1719062931467283681) {
out_1719062931467283681[0] = 0;
out_1719062931467283681[1] = 0;
out_1719062931467283681[2] = 0;
out_1719062931467283681[3] = 0;
out_1719062931467283681[4] = 0;
out_1719062931467283681[5] = 0;
out_1719062931467283681[6] = 1;
out_1719062931467283681[7] = 0;
out_1719062931467283681[8] = 0;
}
void h_24(double *state, double *unused, double *out_3758799493215832204) {
out_3758799493215832204[0] = state[4];
out_3758799493215832204[1] = state[5];
void h_24(double *state, double *unused, double *out_5179238137807796494) {
out_5179238137807796494[0] = state[4];
out_5179238137807796494[1] = state[5];
}
void H_24(double *state, double *unused, double *out_4785526542027925312) {
out_4785526542027925312[0] = 0;
out_4785526542027925312[1] = 0;
out_4785526542027925312[2] = 0;
out_4785526542027925312[3] = 0;
out_4785526542027925312[4] = 1;
out_4785526542027925312[5] = 0;
out_4785526542027925312[6] = 0;
out_4785526542027925312[7] = 0;
out_4785526542027925312[8] = 0;
out_4785526542027925312[9] = 0;
out_4785526542027925312[10] = 0;
out_4785526542027925312[11] = 0;
out_4785526542027925312[12] = 0;
out_4785526542027925312[13] = 0;
out_4785526542027925312[14] = 1;
out_4785526542027925312[15] = 0;
out_4785526542027925312[16] = 0;
out_4785526542027925312[17] = 0;
void H_24(double *state, double *unused, double *out_453586667538215885) {
out_453586667538215885[0] = 0;
out_453586667538215885[1] = 0;
out_453586667538215885[2] = 0;
out_453586667538215885[3] = 0;
out_453586667538215885[4] = 1;
out_453586667538215885[5] = 0;
out_453586667538215885[6] = 0;
out_453586667538215885[7] = 0;
out_453586667538215885[8] = 0;
out_453586667538215885[9] = 0;
out_453586667538215885[10] = 0;
out_453586667538215885[11] = 0;
out_453586667538215885[12] = 0;
out_453586667538215885[13] = 0;
out_453586667538215885[14] = 1;
out_453586667538215885[15] = 0;
out_453586667538215885[16] = 0;
out_453586667538215885[17] = 0;
}
void h_30(double *state, double *unused, double *out_3207849157008383468) {
out_3207849157008383468[0] = state[4];
void h_30(double *state, double *unused, double *out_4528864321752517701) {
out_4528864321752517701[0] = state[4];
}
void H_30(double *state, double *unused, double *out_5575595602751694970) {
out_5575595602751694970[0] = 0;
out_5575595602751694970[1] = 0;
out_5575595602751694970[2] = 0;
out_5575595602751694970[3] = 0;
out_5575595602751694970[4] = 1;
out_5575595602751694970[5] = 0;
out_5575595602751694970[6] = 0;
out_5575595602751694970[7] = 0;
out_5575595602751694970[8] = 0;
void H_30(double *state, double *unused, double *out_4237395889974532308) {
out_4237395889974532308[0] = 0;
out_4237395889974532308[1] = 0;
out_4237395889974532308[2] = 0;
out_4237395889974532308[3] = 0;
out_4237395889974532308[4] = 1;
out_4237395889974532308[5] = 0;
out_4237395889974532308[6] = 0;
out_4237395889974532308[7] = 0;
out_4237395889974532308[8] = 0;
}
void h_26(double *state, double *unused, double *out_9031737129220547093) {
out_9031737129220547093[0] = state[7];
void h_26(double *state, double *unused, double *out_5472374174654180136) {
out_5472374174654180136[0] = state[7];
}
void H_26(double *state, double *unused, double *out_6361788614005246944) {
out_6361788614005246944[0] = 0;
out_6361788614005246944[1] = 0;
out_6361788614005246944[2] = 0;
out_6361788614005246944[3] = 0;
out_6361788614005246944[4] = 0;
out_6361788614005246944[5] = 0;
out_6361788614005246944[6] = 0;
out_6361788614005246944[7] = 1;
out_6361788614005246944[8] = 0;
void H_26(double *state, double *unused, double *out_2022440387406772543) {
out_2022440387406772543[0] = 0;
out_2022440387406772543[1] = 0;
out_2022440387406772543[2] = 0;
out_2022440387406772543[3] = 0;
out_2022440387406772543[4] = 0;
out_2022440387406772543[5] = 0;
out_2022440387406772543[6] = 0;
out_2022440387406772543[7] = 1;
out_2022440387406772543[8] = 0;
}
void h_27(double *state, double *unused, double *out_2486985049787524442) {
out_2486985049787524442[0] = state[3];
void h_27(double *state, double *unused, double *out_2749092444325053967) {
out_2749092444325053967[0] = state[3];
}
void H_27(double *state, double *unused, double *out_3400832290951270059) {
out_3400832290951270059[0] = 0;
out_3400832290951270059[1] = 0;
out_3400832290951270059[2] = 0;
out_3400832290951270059[3] = 1;
out_3400832290951270059[4] = 0;
out_3400832290951270059[5] = 0;
out_3400832290951270059[6] = 0;
out_3400832290951270059[7] = 0;
out_3400832290951270059[8] = 0;
void H_27(double *state, double *unused, double *out_6460989961158475525) {
out_6460989961158475525[0] = 0;
out_6460989961158475525[1] = 0;
out_6460989961158475525[2] = 0;
out_6460989961158475525[3] = 1;
out_6460989961158475525[4] = 0;
out_6460989961158475525[5] = 0;
out_6460989961158475525[6] = 0;
out_6460989961158475525[7] = 0;
out_6460989961158475525[8] = 0;
}
void h_29(double *state, double *unused, double *out_2762179112072030331) {
out_2762179112072030331[0] = state[1];
void h_29(double *state, double *unused, double *out_5132180206428995218) {
out_5132180206428995218[0] = state[1];
}
void H_29(double *state, double *unused, double *out_6085826947066087154) {
out_6085826947066087154[0] = 0;
out_6085826947066087154[1] = 1;
out_6085826947066087154[2] = 0;
out_6085826947066087154[3] = 0;
out_6085826947066087154[4] = 0;
out_6085826947066087154[5] = 0;
out_6085826947066087154[6] = 0;
out_6085826947066087154[7] = 0;
out_6085826947066087154[8] = 0;
void H_29(double *state, double *unused, double *out_4747627234288924492) {
out_4747627234288924492[0] = 0;
out_4747627234288924492[1] = 1;
out_4747627234288924492[2] = 0;
out_4747627234288924492[3] = 0;
out_4747627234288924492[4] = 0;
out_4747627234288924492[5] = 0;
out_4747627234288924492[6] = 0;
out_4747627234288924492[7] = 0;
out_4747627234288924492[8] = 0;
}
void h_28(double *state, double *unused, double *out_4515777654005541337) {
out_4515777654005541337[0] = state[0];
void h_28(double *state, double *unused, double *out_9148133961738903113) {
out_9148133961738903113[0] = state[0];
}
void H_28(double *state, double *unused, double *out_1003427929996556580) {
out_1003427929996556580[0] = 1;
out_1003427929996556580[1] = 0;
out_1003427929996556580[2] = 0;
out_1003427929996556580[3] = 0;
out_1003427929996556580[4] = 0;
out_1003427929996556580[5] = 0;
out_1003427929996556580[6] = 0;
out_1003427929996556580[7] = 0;
out_1003427929996556580[8] = 0;
void H_28(double *state, double *unused, double *out_334771782780606082) {
out_334771782780606082[0] = 1;
out_334771782780606082[1] = 0;
out_334771782780606082[2] = 0;
out_334771782780606082[3] = 0;
out_334771782780606082[4] = 0;
out_334771782780606082[5] = 0;
out_334771782780606082[6] = 0;
out_334771782780606082[7] = 0;
out_334771782780606082[8] = 0;
}
void h_31(double *state, double *unused, double *out_6376371694301822855) {
out_6376371694301822855[0] = state[8];
void h_31(double *state, double *unused, double *out_745894969277840380) {
out_745894969277840380[0] = state[8];
}
void H_31(double *state, double *unused, double *out_5735580511771895468) {
out_5735580511771895468[0] = 0;
out_5735580511771895468[1] = 0;
out_5735580511771895468[2] = 0;
out_5735580511771895468[3] = 0;
out_5735580511771895468[4] = 0;
out_5735580511771895468[5] = 0;
out_5735580511771895468[6] = 0;
out_5735580511771895468[7] = 0;
out_5735580511771895468[8] = 1;
void H_31(double *state, double *unused, double *out_2648648489640124019) {
out_2648648489640124019[0] = 0;
out_2648648489640124019[1] = 0;
out_2648648489640124019[2] = 0;
out_2648648489640124019[3] = 0;
out_2648648489640124019[4] = 0;
out_2648648489640124019[5] = 0;
out_2648648489640124019[6] = 0;
out_2648648489640124019[7] = 0;
out_2648648489640124019[8] = 1;
}
#include <eigen3/Eigen/Dense>
#include <iostream>
@@ -514,68 +514,68 @@ void car_update_28(double *in_x, double *in_P, double *in_z, double *in_R, doubl
void car_update_31(double *in_x, double *in_P, double *in_z, double *in_R, double *in_ea) {
update<1, 3, 0>(in_x, in_P, h_31, H_31, NULL, in_z, in_R, in_ea, MAHA_THRESH_31);
}
void car_err_fun(double *nom_x, double *delta_x, double *out_7776651228618240194) {
err_fun(nom_x, delta_x, out_7776651228618240194);
void car_err_fun(double *nom_x, double *delta_x, double *out_3585265450871354510) {
err_fun(nom_x, delta_x, out_3585265450871354510);
}
void car_inv_err_fun(double *nom_x, double *true_x, double *out_5273421026182551058) {
inv_err_fun(nom_x, true_x, out_5273421026182551058);
void car_inv_err_fun(double *nom_x, double *true_x, double *out_4534440969468340130) {
inv_err_fun(nom_x, true_x, out_4534440969468340130);
}
void car_H_mod_fun(double *state, double *out_2181953899905443363) {
H_mod_fun(state, out_2181953899905443363);
void car_H_mod_fun(double *state, double *out_7815971298073658297) {
H_mod_fun(state, out_7815971298073658297);
}
void car_f_fun(double *state, double dt, double *out_2503069281528488208) {
f_fun(state, dt, out_2503069281528488208);
void car_f_fun(double *state, double dt, double *out_3011916394083426809) {
f_fun(state, dt, out_3011916394083426809);
}
void car_F_fun(double *state, double dt, double *out_1406798686887127281) {
F_fun(state, dt, out_1406798686887127281);
void car_F_fun(double *state, double dt, double *out_4768916904116545459) {
F_fun(state, dt, out_4768916904116545459);
}
void car_h_25(double *state, double *unused, double *out_3759466007672649026) {
h_25(state, unused, out_3759466007672649026);
void car_h_25(double *state, double *unused, double *out_8627167620031889565) {
h_25(state, unused, out_8627167620031889565);
}
void car_H_25(double *state, double *unused, double *out_8343452140830248448) {
H_25(state, unused, out_8343452140830248448);
void car_H_25(double *state, double *unused, double *out_1719062931467283681) {
H_25(state, unused, out_1719062931467283681);
}
void car_h_24(double *state, double *unused, double *out_3758799493215832204) {
h_24(state, unused, out_3758799493215832204);
void car_h_24(double *state, double *unused, double *out_5179238137807796494) {
h_24(state, unused, out_5179238137807796494);
}
void car_H_24(double *state, double *unused, double *out_4785526542027925312) {
H_24(state, unused, out_4785526542027925312);
void car_H_24(double *state, double *unused, double *out_453586667538215885) {
H_24(state, unused, out_453586667538215885);
}
void car_h_30(double *state, double *unused, double *out_3207849157008383468) {
h_30(state, unused, out_3207849157008383468);
void car_h_30(double *state, double *unused, double *out_4528864321752517701) {
h_30(state, unused, out_4528864321752517701);
}
void car_H_30(double *state, double *unused, double *out_5575595602751694970) {
H_30(state, unused, out_5575595602751694970);
void car_H_30(double *state, double *unused, double *out_4237395889974532308) {
H_30(state, unused, out_4237395889974532308);
}
void car_h_26(double *state, double *unused, double *out_9031737129220547093) {
h_26(state, unused, out_9031737129220547093);
void car_h_26(double *state, double *unused, double *out_5472374174654180136) {
h_26(state, unused, out_5472374174654180136);
}
void car_H_26(double *state, double *unused, double *out_6361788614005246944) {
H_26(state, unused, out_6361788614005246944);
void car_H_26(double *state, double *unused, double *out_2022440387406772543) {
H_26(state, unused, out_2022440387406772543);
}
void car_h_27(double *state, double *unused, double *out_2486985049787524442) {
h_27(state, unused, out_2486985049787524442);
void car_h_27(double *state, double *unused, double *out_2749092444325053967) {
h_27(state, unused, out_2749092444325053967);
}
void car_H_27(double *state, double *unused, double *out_3400832290951270059) {
H_27(state, unused, out_3400832290951270059);
void car_H_27(double *state, double *unused, double *out_6460989961158475525) {
H_27(state, unused, out_6460989961158475525);
}
void car_h_29(double *state, double *unused, double *out_2762179112072030331) {
h_29(state, unused, out_2762179112072030331);
void car_h_29(double *state, double *unused, double *out_5132180206428995218) {
h_29(state, unused, out_5132180206428995218);
}
void car_H_29(double *state, double *unused, double *out_6085826947066087154) {
H_29(state, unused, out_6085826947066087154);
void car_H_29(double *state, double *unused, double *out_4747627234288924492) {
H_29(state, unused, out_4747627234288924492);
}
void car_h_28(double *state, double *unused, double *out_4515777654005541337) {
h_28(state, unused, out_4515777654005541337);
void car_h_28(double *state, double *unused, double *out_9148133961738903113) {
h_28(state, unused, out_9148133961738903113);
}
void car_H_28(double *state, double *unused, double *out_1003427929996556580) {
H_28(state, unused, out_1003427929996556580);
void car_H_28(double *state, double *unused, double *out_334771782780606082) {
H_28(state, unused, out_334771782780606082);
}
void car_h_31(double *state, double *unused, double *out_6376371694301822855) {
h_31(state, unused, out_6376371694301822855);
void car_h_31(double *state, double *unused, double *out_745894969277840380) {
h_31(state, unused, out_745894969277840380);
}
void car_H_31(double *state, double *unused, double *out_5735580511771895468) {
H_31(state, unused, out_5735580511771895468);
void car_H_31(double *state, double *unused, double *out_2648648489640124019) {
H_31(state, unused, out_2648648489640124019);
}
void car_predict(double *in_x, double *in_P, double *in_Q, double dt) {
predict(in_x, in_P, in_Q, dt);
@@ -9,27 +9,27 @@ void car_update_27(double *in_x, double *in_P, double *in_z, double *in_R, doubl
void car_update_29(double *in_x, double *in_P, double *in_z, double *in_R, double *in_ea);
void car_update_28(double *in_x, double *in_P, double *in_z, double *in_R, double *in_ea);
void car_update_31(double *in_x, double *in_P, double *in_z, double *in_R, double *in_ea);
void car_err_fun(double *nom_x, double *delta_x, double *out_7776651228618240194);
void car_inv_err_fun(double *nom_x, double *true_x, double *out_5273421026182551058);
void car_H_mod_fun(double *state, double *out_2181953899905443363);
void car_f_fun(double *state, double dt, double *out_2503069281528488208);
void car_F_fun(double *state, double dt, double *out_1406798686887127281);
void car_h_25(double *state, double *unused, double *out_3759466007672649026);
void car_H_25(double *state, double *unused, double *out_8343452140830248448);
void car_h_24(double *state, double *unused, double *out_3758799493215832204);
void car_H_24(double *state, double *unused, double *out_4785526542027925312);
void car_h_30(double *state, double *unused, double *out_3207849157008383468);
void car_H_30(double *state, double *unused, double *out_5575595602751694970);
void car_h_26(double *state, double *unused, double *out_9031737129220547093);
void car_H_26(double *state, double *unused, double *out_6361788614005246944);
void car_h_27(double *state, double *unused, double *out_2486985049787524442);
void car_H_27(double *state, double *unused, double *out_3400832290951270059);
void car_h_29(double *state, double *unused, double *out_2762179112072030331);
void car_H_29(double *state, double *unused, double *out_6085826947066087154);
void car_h_28(double *state, double *unused, double *out_4515777654005541337);
void car_H_28(double *state, double *unused, double *out_1003427929996556580);
void car_h_31(double *state, double *unused, double *out_6376371694301822855);
void car_H_31(double *state, double *unused, double *out_5735580511771895468);
void car_err_fun(double *nom_x, double *delta_x, double *out_3585265450871354510);
void car_inv_err_fun(double *nom_x, double *true_x, double *out_4534440969468340130);
void car_H_mod_fun(double *state, double *out_7815971298073658297);
void car_f_fun(double *state, double dt, double *out_3011916394083426809);
void car_F_fun(double *state, double dt, double *out_4768916904116545459);
void car_h_25(double *state, double *unused, double *out_8627167620031889565);
void car_H_25(double *state, double *unused, double *out_1719062931467283681);
void car_h_24(double *state, double *unused, double *out_5179238137807796494);
void car_H_24(double *state, double *unused, double *out_453586667538215885);
void car_h_30(double *state, double *unused, double *out_4528864321752517701);
void car_H_30(double *state, double *unused, double *out_4237395889974532308);
void car_h_26(double *state, double *unused, double *out_5472374174654180136);
void car_H_26(double *state, double *unused, double *out_2022440387406772543);
void car_h_27(double *state, double *unused, double *out_2749092444325053967);
void car_H_27(double *state, double *unused, double *out_6460989961158475525);
void car_h_29(double *state, double *unused, double *out_5132180206428995218);
void car_H_29(double *state, double *unused, double *out_4747627234288924492);
void car_h_28(double *state, double *unused, double *out_9148133961738903113);
void car_H_28(double *state, double *unused, double *out_334771782780606082);
void car_h_31(double *state, double *unused, double *out_745894969277840380);
void car_H_31(double *state, double *unused, double *out_2648648489640124019);
void car_predict(double *in_x, double *in_P, double *in_Q, double dt);
void car_set_rotational_inertia(double x);
void car_set_center_to_front(double x);
File diff suppressed because it is too large Load Diff
@@ -5,18 +5,18 @@ void pose_update_4(double *in_x, double *in_P, double *in_z, double *in_R, doubl
void pose_update_10(double *in_x, double *in_P, double *in_z, double *in_R, double *in_ea);
void pose_update_13(double *in_x, double *in_P, double *in_z, double *in_R, double *in_ea);
void pose_update_14(double *in_x, double *in_P, double *in_z, double *in_R, double *in_ea);
void pose_err_fun(double *nom_x, double *delta_x, double *out_8239082397058567475);
void pose_inv_err_fun(double *nom_x, double *true_x, double *out_7476243289318583052);
void pose_H_mod_fun(double *state, double *out_6682901757221528583);
void pose_f_fun(double *state, double dt, double *out_8926187974092623704);
void pose_F_fun(double *state, double dt, double *out_7762434416297877994);
void pose_h_4(double *state, double *unused, double *out_4305306830373346392);
void pose_H_4(double *state, double *unused, double *out_1517403126751696184);
void pose_h_10(double *state, double *unused, double *out_6976768425782679545);
void pose_H_10(double *state, double *unused, double *out_3773372038423704930);
void pose_h_13(double *state, double *unused, double *out_6361379321214522320);
void pose_H_13(double *state, double *unused, double *out_9128034335068397113);
void pose_h_14(double *state, double *unused, double *out_4542222400622544050);
void pose_H_14(double *state, double *unused, double *out_5480643983091180713);
void pose_err_fun(double *nom_x, double *delta_x, double *out_1372154176746733435);
void pose_inv_err_fun(double *nom_x, double *true_x, double *out_1523580565963320994);
void pose_H_mod_fun(double *state, double *out_7104221342799173532);
void pose_f_fun(double *state, double dt, double *out_7349931890006772642);
void pose_F_fun(double *state, double dt, double *out_1064691538495726265);
void pose_h_4(double *state, double *unused, double *out_5786601062018184200);
void pose_H_4(double *state, double *unused, double *out_8923920068705470393);
void pose_h_10(double *state, double *unused, double *out_8200446056116706374);
void pose_H_10(double *state, double *unused, double *out_8156688557227160880);
void pose_h_13(double *state, double *unused, double *out_9145870174400750931);
void pose_H_13(double *state, double *unused, double *out_5711646243373137592);
void pose_h_14(double *state, double *unused, double *out_5989511771500844071);
void pose_H_14(double *state, double *unused, double *out_6440035572708708927);
void pose_predict(double *in_x, double *in_P, double *in_Q, double dt);
}
+1 -1
View File
@@ -49,7 +49,7 @@ SEND_RAW_PRED = os.getenv('SEND_RAW_PRED')
LAT_SMOOTH_SECONDS = 0.0
LONG_SMOOTH_SECONDS = 0.3
MIN_LAT_CONTROL_SPEED = 0.3
MIN_LAT_CONTROL_SPEED = 1.6
BIG_MODEL_TIMEOUT = 60
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d4c7a3fcfc95813826ca871a80bea18df833aacb2eb6f5333e25a01e65b1a422
size 799941965
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4b1709b9f6b05f94b1165afe0d9763ab4bf1d59102417708e4b10aaa519bd771
size 858796
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a17a2f2ae70decfca855c5626178aefe8dac0d0d303c72edd44dfa582e98d73e
size 858531
@@ -0,0 +1,98 @@
# Signal navigation experiment
The physical turn signal selects a short real route through the next supported
street. The route and maneuver instructions condition a navigation-trained
driving network continuously. This replaces the earlier map-timed turn-cue
experiment when a compatible model is selected; it does not add steering angles.
The automatic candidate begins with a pending signal and may prefetch a route
at any speed. Pending discovery does not send navigation inputs or suppress the
existing lane-change checks. An emitted host lane-change request holds
lane-change mode while preserving the candidate route. Otherwise, a fresh validated
street approach below the configured turn speed (19 mph by default) may commit
turn mode. An early fast signal can therefore prepare a route and later become
a turn. LC wins a same-frame tie. A lane-change request blocks navigation until
the existing helper reports normal completion (its probability/timer condition),
after which the held signal may become a turn at a fresh eligible approach.
Slowing down, timeout, disengagement and disabling lane changes do not themselves
release this block. A committed turn remains a turn until cancel/opposite.
Only a committed turn suppresses competing lane-change pulses. Map conditioning
requires a validated connected street, matching position and heading, and a
coherent fresh route. Pending and lane-change intent cannot feed navigation.
This arbitration still cannot reliably identify every early fast turn signal or
slow lane change. A nearby legal street does not establish the driver's intended street.
Predicted missing lane lines must not be treated as proof that no adjacent lane
exists. These ambiguity cases remain validation limitations, not solved intent.
A causal host audit using captured map contexts and recorded helper/model heads
finds eventual navigation eligibility on 29 of 39 street-turn requests, compared
with 23 under the earlier permanent lane-change lock. Only two of those six
additional opportunities precede the recorded intervention cutoff; later
eligibility must not be counted as recovering the original turn. All four
recorded lane-change controls retain their lane-change request and none gains
navigation eligibility. These are selection proxies, not new navigation-model
inference or neural turn successes. Map-target identity is a separate check and
can still select the wrong street. The two newly recovered Mapbox probe responses
were verified separately and are not silently included in this frozen-context audit.
An earlier speed-at-signal classifier was rejected because 18 street-turn
requests began above 20 mph. The automatic selector is not dependable enough to
claim road-test readiness from these results.
## Model requirement
CTV3 and the small fallback have no navigation input sockets. A route on the
screen cannot make those models follow navigation. The runtime checks actual
`nav_map` and `navigation` tensors, and publishes `NavModelState` separately from
route availability. The UI distinguishes **Route ready** from **Navigation active**.
The basis is commaai/openpilot `nav-model-integration`, pinned at
`afbf2e547e9319e2eb9f515e1e4330af63705cec`. The available BIG ONNX is the earlier
RL navigation checkpoint `80eedcdd/12864`, SHA256
`1d18c42ea4cd95c8bffab4635844b78b8e2d4e25b9de44a9c5ac2e0961371a2e`.
The pinned commit's newer N16 compiled PKL is a different artifact; running the
ONNX is not evidence of running those N16 weights. Official LFS downloads for
the latest three compiled navigation artifacts returned object-not-found.
The ONNX was downloaded through a third-party mirror and independently matched
against comma's exact LFS pointer. A Chestnut-compatible plain artifact still
requires the target compilation and verification step. Use **Settings → Models →
Prepare Navigation Model** while parked, then explicitly select the prepared
model. See [model preparation](../../sunnypilot/models/NAVIGATION_MODEL.md).
A Mac CPU ONNX smoke is not a verified device build. No model is automatically
swapped when signaling.
## Route ownership and input coherence
Manual destinations take priority. Automatic routes do not overwrite
`NavDestination`. Each physical signal has an identifier; a held indicator does
not repeatedly select turns. Cancellation before entry retires the route;
already-entered turns may retain guidance through the receiving road, after
which it is cleared. A missed turn is retired rather than rerouted back to the
short endpoint.
Map frames carry the exact rendered `navRoute.logMonoTime` as their stream EOF
generation; SOF remains the GPS publication time. The driving process requires
a matching route generation, fresh map/instructions and the current signal
session (or explicitly retained exit guidance). Stale/opposite/canceled route
inputs are zeroed. A route preview alone never establishes active model input.
## Scope and validation
The feature is opt-in under **Signal Navigation (Experimental)**. It needs
online Directions requests, a Mapbox public token, and automatically fetched
road-map tiles. Directions use the configured Mapbox token. The basemap is fetched
from public Protomaps builds, without comma authentication or a Prime subscription.
Only the required archive byte ranges are downloaded and cached; the entire
planet archive is not downloaded. The map data is derived from
[OpenStreetMap](https://www.openstreetmap.org/copyright); attribution is displayed
over the map preview without changing the model raster. The public basemap is not
claimed to be byte-identical to comma's training archive. See
[Protomaps downloads](https://docs.protomaps.com/basemaps/downloads) for distribution
and attribution information; a wider deployment should serve its own archive mirror.
Existing steering, cooperative steering, action conversion and
the restored 1.6 m/s low-speed handling are preserved.
Recorded-camera model replay can test whether real route inputs change the
model's plan. It cannot establish physical curb clearance or a successful driven
turn, because the camera views remain the recorded ones. Model runtime checks,
software fixture checks and recorded-sequence results must be reported separately.
@@ -0,0 +1,452 @@
"""Discover a router-validated next street turn; never issue a driving command.
The ahead/exit probe points are API queries only. A target requires a separate
route proving its actual first turn. Map data is not a curb, signal-phase or
physical-clearance oracle. Callers own localization freshness and target expiry.
"""
from dataclasses import dataclass, fields
import hashlib
import json
import math
import re
import time
from types import MappingProxyType
import requests
EARTH_RADIUS_M = 6371008.8
PROBE_DISTANCE_M = 500.
PROBE_ENDPOINT_RADIUS_M = 100.
PROBE_FALLBACK_ENDPOINT_RADIUS_M = 300.
MAX_TARGET_DISTANCE_M = 300.
EXIT_QUERY_DISTANCE_M = 45.
MIN_EXIT_DISTANCE_M = 20.
SOURCE_RADIUS_M = 20.
EXIT_RADIUS_M = 15.
JUNCTION_TOLERANCE_M = 2.
BEARING_TOLERANCE_DEG = 25.
MIN_TURN_DEG = 35.
MAX_TURN_DEG = 145.
TIMEOUT = (2., 3.)
FORBIDDEN_CLASSES = {'restricted', 'motorway', 'ferry'}
STREET_CLASSES = {'trunk', 'primary', 'secondary', 'tertiary', 'street', 'residential', 'unclassified'}
UNSUPPORTED_MANEUVERS = {'roundabout', 'rotary', 'roundabout turn', 'exit roundabout', 'exit rotary', 'on ramp', 'off ramp', 'merge', 'fork'}
class NextTurnError(Exception):
"""A fixed reason code, containing no request URL, token or response body."""
def __init__(self, reason):
self.reason = reason
super().__init__(reason)
def _freeze(value):
if isinstance(value, dict):
return MappingProxyType({key: _freeze(item) for key, item in value.items()})
if isinstance(value, list):
return tuple(_freeze(item) for item in value)
return value
def _thaw(value):
if isinstance(value, (dict, MappingProxyType)):
return {key: _thaw(item) for key, item in value.items()}
if isinstance(value, tuple):
return [_thaw(item) for item in value]
return value
@dataclass(frozen=True)
class NextTurnTarget:
junction: tuple[float, float]
distance_m: float
incoming_bearing_deg: float
exit_bearing_deg: float
direction: str
name: str
road_class: str
geometry: tuple
steps: tuple
approach_geometry: tuple
approach_distance_m: float
turn_step_index: int
junction_geometry_index: int
origin: tuple[float, float]
discovered_at: float
probe_uuid: str
validation_uuid: str
classification_uuid: str | None
provenance: MappingProxyType
def to_dict(self):
return {field.name: _thaw(getattr(self, field.name)) for field in fields(self)}
def _point(value):
if not isinstance(value, (list, tuple)) or len(value) != 2:
raise NextTurnError('invalid_coordinates')
if any(isinstance(v, bool) or not isinstance(v, (int, float)) or not math.isfinite(v) for v in value):
raise NextTurnError('invalid_coordinates')
lon, lat = map(float, value)
if not (-180 <= lon <= 180 and -85 < lat < 85):
raise NextTurnError('invalid_coordinates')
return lon, lat
def distance(a, b):
lon0, lat0, lon1, lat1 = map(math.radians, (*a, *b))
h = math.sin((lat1 - lat0) / 2)**2 + math.cos(lat0) * math.cos(lat1) * math.sin((lon1 - lon0) / 2)**2
return 2 * EARTH_RADIUS_M * math.asin(math.sqrt(min(1., max(0., h))))
def bearing(a, b):
lon0, lat0, lon1, lat1 = map(math.radians, (*a, *b))
return math.degrees(math.atan2(math.sin(lon1-lon0) * math.cos(lat1),
math.cos(lat0) * math.sin(lat1) - math.sin(lat0) * math.cos(lat1) * math.cos(lon1-lon0))) % 360
def offset(point, heading, meters):
lon, lat = map(math.radians, point)
heading, arc = math.radians(heading), meters / EARTH_RADIUS_M
lat1 = math.asin(math.sin(lat)*math.cos(arc) + math.cos(lat)*math.sin(arc)*math.cos(heading))
lon1 = lon + math.atan2(math.sin(heading)*math.sin(arc)*math.cos(lat), math.cos(arc)-math.sin(lat)*math.sin(lat1))
return (math.degrees(lon1)+180) % 360 - 180, math.degrees(lat1)
def _delta(before, after):
return (after - before + 180) % 360 - 180
def _length(points):
return sum(distance(a, b) for a, b in zip(points[:-1], points[1:], strict=True))
def _initial_bearing(points):
for point in points[1:]:
if distance(points[0], point) >= .5:
return bearing(points[0], point)
raise NextTurnError('insufficient_approach')
def _number(value):
if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value):
raise NextTurnError('missing_metadata')
return float(value)
def _parse_route(body, position, heading, endpoint, endpoint_radius):
if not isinstance(body, dict) or body.get('code') != 'Ok':
raise NextTurnError('no_route')
if not isinstance(body.get('uuid'), str) or not body['uuid']:
raise NextTurnError('missing_request_id')
routes = body.get('routes')
if not isinstance(routes, list) or not routes or not isinstance(routes[0], dict):
raise NextTurnError('invalid_response')
route = routes[0]
waypoints = body.get('waypoints', route.get('waypoints'))
if not isinstance(waypoints, list) or len(waypoints) != 2:
raise NextTurnError('missing_metadata')
snapped_start, snapped_end = (_point(value['location']) for value in waypoints)
if distance(position, snapped_start) > SOURCE_RADIUS_M or distance(endpoint, snapped_end) > endpoint_radius:
raise NextTurnError('snapping_mismatch')
geometry = route.get('geometry')
if not isinstance(geometry, dict) or geometry.get('type') != 'LineString':
raise NextTurnError('missing_metadata')
points = tuple(_point(point) for point in geometry.get('coordinates', []))
if not 2 <= len(points) <= 10000 or distance(points[0], snapped_start) > 3 or distance(points[-1], snapped_end) > 3:
raise NextTurnError('invalid_geometry')
if abs(_delta(heading, _initial_bearing(points))) > BEARING_TOLERANCE_DEG:
raise NextTurnError('incoming_heading_mismatch')
legs = route.get('legs')
if not isinstance(legs, list) or len(legs) != 1 or not isinstance(legs[0], dict):
raise NextTurnError('missing_metadata')
leg = legs[0]
if leg.get('annotation', {}).get('closure') or any(item.get('type') == 'closure' for item in leg.get('incidents', [])):
raise NextTurnError('closed_route')
if any(item.get('type') == 'violation' for item in route.get('notifications', [])):
raise NextTurnError('route_violation')
steps = leg.get('steps')
if not isinstance(steps, list) or not steps:
raise NextTurnError('missing_metadata')
cumulative = [0.]
for a, b in zip(points[:-1], points[1:], strict=True):
cumulative.append(cumulative[-1] + distance(a, b))
intersections = []
for step_index, step in enumerate(steps):
if not isinstance(step, dict) or not isinstance(step.get('maneuver'), dict) or not isinstance(step.get('intersections'), list):
raise NextTurnError('missing_metadata')
kind = step['maneuver'].get('type')
if not isinstance(kind, str) or not step.get('geometry', {}).get('coordinates'):
raise NextTurnError('missing_metadata')
if step.get('mode') != 'driving':
raise NextTurnError('unsupported_road')
for intersection in step['intersections']:
if not isinstance(intersection, dict):
raise NextTurnError('missing_metadata')
if 'in' not in intersection or 'out' not in intersection:
location = _point(intersection.get('location'))
if kind == 'depart' and 'in' not in intersection and distance(location, points[0]) <= 3:
continue
if kind == 'arrive' and 'out' not in intersection and distance(location, points[-1]) <= 3:
continue
raise NextTurnError('missing_metadata')
bearings, entry = intersection.get('bearings'), intersection.get('entry')
if not isinstance(bearings, list) or not isinstance(entry, list) or len(bearings) != len(entry) or not 2 <= len(entry) <= 16:
raise NextTurnError('missing_metadata')
if any(type(flag) is not bool for flag in entry):
raise NextTurnError('missing_metadata')
bearings = [_number(value) for value in bearings]
if any(not 0 <= value <= 360 for value in bearings):
raise NextTurnError('missing_metadata')
# Actual Directions responses sometimes encode north as 360 rather than 0.
bearings = [value % 360 for value in bearings]
incoming, outgoing, index = intersection['in'], intersection['out'], intersection.get('geometry_index')
if any(type(value) is not int for value in (incoming, outgoing, index)):
raise NextTurnError('missing_metadata')
if not (0 <= incoming < len(entry) and 0 <= outgoing < len(entry) and 0 <= index < len(points)):
raise NextTurnError('missing_metadata')
location = _point(intersection.get('location'))
if distance(location, points[index]) > JUNCTION_TOLERANCE_M or not entry[outgoing]:
raise NextTurnError('inconsistent_intersection')
before = (bearings[incoming]+180) % 360
# A fixed long chord can cross the preceding corner at close junctions.
prior = next((p for p in reversed(points[:index]) if distance(p, location) >= .5), None)
if prior is None or abs(_delta(bearing(prior, location), before)) > BEARING_TOLERANCE_DEG:
raise NextTurnError('incoming_heading_mismatch')
intersections.append({'raw': intersection, 'location': location, 'bearings': bearings, 'entry': entry,
'incoming': before, 'outgoing': bearings[outgoing], 'out': outgoing,
'index': index, 'distance': cumulative[index], 'step': step_index, 'kind': kind})
return {'body': body, 'route': route, 'steps': steps, 'points': points, 'intersections': sorted(intersections, key=lambda i:i['index']),
'uuid': body['uuid'], 'responseSha256': hashlib.sha256(json.dumps(body, sort_keys=True, allow_nan=False).encode()).hexdigest()}
def _choose_branch(route, direction):
for item in route['intersections']:
if item['distance'] > MAX_TARGET_DISTANCE_M:
break
if item['kind'] in UNSUPPORTED_MANEUVERS:
raise NextTurnError('unsupported_junction')
maneuver = route['steps'][item['step']]['maneuver']
explicit_turn = ((item['kind'] in ('turn', 'end of road') or maneuver.get('modifier') == 'uturn')
and distance(_point(maneuver.get('location')), item['location']) <= JUNCTION_TOLERANCE_M)
has_branch = len(item['bearings']) >= 3
candidates = []
for heading, allowed in zip(item['bearings'], item['entry'], strict=True):
delta = _delta(item['incoming'], heading)
side = 'right' if delta > 0 else 'left'
continuing_bend = heading == item['outgoing'] and not explicit_turn
if allowed and not continuing_bend and side == direction and MIN_TURN_DEG <= abs(delta) <= MAX_TURN_DEG:
candidates.append(heading)
if len(candidates) > 1:
raise NextTurnError('ambiguous_branch')
if candidates and item['distance'] >= 2:
return {**item, 'exit': candidates[0]}
if not has_branch and not explicit_turn:
continue
if explicit_turn:
# Only a maneuver at this intersection proves the probe left the approach;
# a continuing main road can bend even where a minor branch joins it.
break
raise NextTurnError('no_matching_turn')
def _validate_turn(route, candidate, direction):
matched = None
truncate = False
for item in route['intersections']:
if matched is not None and item['distance'] >= matched['distance'] + MIN_EXIT_DISTANCE_M:
# A query endpoint can require a further turn beyond the local exit. Its
# later route is irrelevant only after this first turn is proven below.
if abs(_delta(item['incoming'], item['outgoing'])) >= MIN_TURN_DEG or item['kind'] in UNSUPPORTED_MANEUVERS:
truncate = True
break
near = distance(item['location'], candidate['location']) <= JUNCTION_TOLERANCE_M
turn = abs(_delta(item['incoming'], item['outgoing'])) >= MIN_TURN_DEG
if item['kind'] in UNSUPPORTED_MANEUVERS:
raise NextTurnError('unsupported_junction')
if near and turn:
if (abs(_delta(candidate['incoming'], item['incoming'])) > BEARING_TOLERANCE_DEG
or abs(_delta(candidate['exit'], item['outgoing'])) > BEARING_TOLERANCE_DEG
or ('right' if _delta(item['incoming'], item['outgoing']) > 0 else 'left') != direction
or abs(_delta(item['incoming'], item['outgoing'])) > MAX_TURN_DEG):
raise NextTurnError('target_mismatch')
if abs(item['distance'] - candidate['distance']) > max(12., candidate['distance'] * .15):
raise NextTurnError('approach_detour')
if matched is None:
matched = item
elif turn:
raise NextTurnError('earlier_or_additional_turn')
if matched is None:
raise NextTurnError('target_not_reached')
for index, step in enumerate(route['steps']):
if index != matched['step'] and step['maneuver']['type'] not in ('depart', 'arrive', 'continue', 'new name'):
location = _point(step['maneuver'].get('location'))
point_index = min(range(len(route['points'])), key=lambda i: distance(route['points'][i], location))
along = _length(route['points'][:point_index+1])
if (index <= matched['step'] or distance(route['points'][point_index], location) > JUNCTION_TOLERANCE_M
or along < matched['distance'] + MIN_EXIT_DISTANCE_M):
raise NextTurnError('earlier_or_additional_turn')
truncate = True
if truncate:
route = _local_exit_scope(route, matched)
points = route['points']
if (_length(points[matched['index']:]) < MIN_EXIT_DISTANCE_M - 1e-6
or (not route.get('targetScopeTruncated') and distance(matched['location'], points[-1]) < MIN_EXIT_DISTANCE_M - 1e-6)
or abs(_delta(candidate['exit'], bearing(matched['location'], points[-1]))) > BEARING_TOLERANCE_DEG):
raise NextTurnError('insufficient_exit')
chosen_class = matched['raw'].get('mapbox_streets_v8', {}).get('class')
for item in route['intersections']:
if item['index'] >= matched['index']:
if set(item['raw'].get('classes', [])) & FORBIDDEN_CLASSES:
raise NextTurnError('unsupported_road')
known_class = item['raw'].get('mapbox_streets_v8', {}).get('class')
if known_class is not None and known_class not in STREET_CLASSES:
raise NextTurnError('unsupported_road')
if chosen_class is not None and chosen_class not in STREET_CLASSES:
raise NextTurnError('unsupported_road')
return matched, chosen_class, route
def _local_exit_scope(route, matched):
"""Return an explicit prefix of real geometry, ending before any later turn."""
points = list(route['points'][:matched['index']+1])
remaining = MIN_EXIT_DISTANCE_M
for point in route['points'][matched['index']+1:]:
length = distance(points[-1], point)
if length >= remaining:
points.append(offset(points[-1], bearing(points[-1], point), remaining))
break
points.append(point)
remaining -= length
else:
raise NextTurnError('insufficient_exit')
cutoff = matched['distance'] + MIN_EXIT_DISTANCE_M
steps = json.loads(json.dumps(route['steps'][:matched['step']+1]))
steps[-1]['geometry'] = {'type': 'LineString', 'coordinates': points[matched['index']:]}
steps[-1]['intersections'] = [item['raw'] for item in route['intersections']
if item['step'] == matched['step'] and item['distance'] <= cutoff]
return {**route, 'points': tuple(points), 'steps': steps,
'intersections': [item for item in route['intersections'] if item['distance'] <= cutoff],
'targetScopeTruncated': True}
def coherent_route_steps(route):
"""Require a complete actual Directions leg, including its arrival step.
A clipped discovery prefix is not a new route: its old distances/banners no
longer describe its geometry. It must be rerouted to the real exit endpoint.
"""
if route.get('targetScopeTruncated'):
raise NextTurnError('incomplete_navigation_route')
steps = json.loads(json.dumps(route['steps']))
if not steps or steps[-1]['maneuver']['type'] != 'arrive':
raise NextTurnError('incomplete_navigation_route')
for step in steps:
points = [_point(point) for point in step['geometry']['coordinates']]
distance_m, duration = _number(step.get('distance')), _number(step.get('duration'))
if not points or distance_m < 0 or duration < 0:
raise NextTurnError('incomplete_navigation_route')
if abs(_length(points)-distance_m) > max(5., distance_m*.2):
raise NextTurnError('inconsistent_navigation_distance')
step.setdefault('bannerInstructions', [])
for before, after in zip(steps[:-1], steps[1:], strict=True):
if distance(_point(before['geometry']['coordinates'][-1]), _point(after['geometry']['coordinates'][0])) > 3.:
raise NextTurnError('disconnected_navigation_route')
return steps
class MapboxNextTurnClient:
def __init__(self, token):
if not isinstance(token, str) or not re.fullmatch(r'[ps]k\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+', token):
raise NextTurnError('invalid_token')
self._token = token
def _request(self, position, heading, endpoint, *, profile, endpoint_radius):
coords = ';'.join(f'{lon:.7f},{lat:.7f}' for lon,lat in (position,endpoint))
params = {'access_token': self._token, 'geometries': 'geojson', 'overview': 'full', 'steps': 'true',
'alternatives': 'false', 'banner_instructions': 'true', 'bearings': f'{heading:.0f},30;',
'radiuses': f'{SOURCE_RADIUS_M:.0f};{endpoint_radius:.0f}', 'depart_at': 'now',
'exclude': 'ferry,motorway'}
if profile == 'driving-traffic':
params.update(annotations='closure', snapping_include_closures='false;false', snapping_include_static_closures='false;false')
try:
response = requests.get(f'https://api.mapbox.com/directions/v5/mapbox/{profile}/{coords}', params=params, timeout=TIMEOUT)
response.raise_for_status()
if len(response.content) > 2 * 1024 * 1024:
raise NextTurnError('oversized_response')
body = response.json()
except requests.RequestException:
raise NextTurnError('request_failed') from None
except (ValueError, TypeError):
raise NextTurnError('invalid_response') from None
return _parse_route(body, position, heading, endpoint, endpoint_radius)
def discover(self, position, bearing_deg, direction):
started = time.monotonic()
position = _point(position)
heading = _number(bearing_deg) % 360
if direction not in ('left','right'):
raise NextTurnError('invalid_direction')
try:
probe_endpoint = offset(position, heading, PROBE_DISTANCE_M)
probe_radius = PROBE_ENDPOINT_RADIUS_M
probe_fallback_used = False
try:
probe = self._request(position, heading, probe_endpoint, profile='driving-traffic', endpoint_radius=probe_radius)
except NextTurnError as error:
if error.reason != 'no_route':
raise
# This point is only a discovery probe, not the turn destination. A
# straight500m projection can land over100m from any road. Retry once
# with the same origin, heading and endpoint; actual exit validation
# below keeps its own strict radius and first-turn/legality checks.
probe_radius = PROBE_FALLBACK_ENDPOINT_RADIUS_M
probe_fallback_used = True
probe = self._request(position, heading, probe_endpoint, profile='driving-traffic', endpoint_radius=probe_radius)
candidate = _choose_branch(probe, direction)
query_endpoint = offset(candidate['location'], candidate['exit'], EXIT_QUERY_DISTANCE_M)
validated = self._request(position, heading, query_endpoint, profile='driving-traffic', endpoint_radius=EXIT_RADIUS_M)
matched, road_class, validated = _validate_turn(validated, candidate, direction)
request_count = 2+int(probe_fallback_used)
if validated.get('targetScopeTruncated'):
# Query an actual endpoint on the validated receiving road. Keep the
# complete new leg/arrival rather than turning a clipped preview into
# model navigation data with stale step distances and instructions.
query_endpoint = validated['points'][-1]
validated = self._request(position, heading, query_endpoint, profile='driving-traffic', endpoint_radius=3.)
matched, road_class, validated = _validate_turn(validated, candidate, direction)
request_count += 1
classification = None
if road_class is None:
request_count += 1
classification = self._request(position, heading, query_endpoint, profile='driving', endpoint_radius=EXIT_RADIUS_M)
_, road_class, classification = _validate_turn(classification, candidate, direction)
if road_class is None:
raise NextTurnError('missing_road_class')
complete_steps = coherent_route_steps(validated)
approach = tuple(validated['points'][:matched['index']]) + (matched['location'],)
turn_step = validated['steps'][matched['step']]
provenance = {'method': 'router intersection discovery plus separately validated first-turn route',
'requestedPosition': position, 'requestedBearingDeg': heading, 'requestedDirection': direction,
'probeResponseSha256': probe['responseSha256'], 'validationResponseSha256': validated['responseSha256'],
'classificationResponseSha256': classification['responseSha256'] if classification else None,
'requestCount': request_count,
'probeEndpointRadiusM': probe_radius, 'probeFallbackUsed': probe_fallback_used,
'probeFallbackOriginalReason': 'no_route' if probe_fallback_used else None,
'targetScopeTruncated': validated.get('targetScopeTruncated', False),
'targetExitDistanceM': _length(validated['points'][matched['index']:]),
'fullValidationResponsePreservedByHash': True,
'requestStartedAt': started, 'requestCompletedAt': time.monotonic(),
'probeEndpointIsTarget': False, 'steeringCommandIssued': False}
provenance['coherentNavigationRoute'] = True
return NextTurnTarget(junction=matched['location'], distance_m=matched['distance'],
incoming_bearing_deg=matched['incoming'], exit_bearing_deg=matched['outgoing'], direction=direction,
name=str(turn_step.get('name') or 'Unnamed road'), road_class=road_class,
geometry=tuple(validated['points']), steps=_freeze(complete_steps), approach_geometry=approach,
approach_distance_m=_length(approach), turn_step_index=matched['step'], junction_geometry_index=matched['index'],
origin=position, discovered_at=started, probe_uuid=probe['uuid'], validation_uuid=validated['uuid'],
classification_uuid=classification['uuid'] if classification else None, provenance=_freeze(provenance))
except NextTurnError:
raise
except (KeyError, TypeError, ValueError, OverflowError, IndexError, AttributeError):
raise NextTurnError('invalid_response') from None
+6 -1
View File
@@ -24,10 +24,12 @@ class MapsD:
self.route = np.empty((0, 2))
self.prefetched = False
self.retry_at = 0.0
self.route_generation = 0
def update(self, sm) -> tuple[np.ndarray, int]:
"""The map frame for the latest messages: black with a zero timestamp without a usable map, never stale."""
if sm.updated['navRoute']:
self.route_generation = int(sm.logMonoTime['navRoute'])
route = np.array([(c.longitude, c.latitude) for c in sm['navRoute'].coordinates], dtype=np.float64).reshape((-1, 2))
if not np.array_equal(route, self.route):
self.route = route
@@ -74,7 +76,10 @@ def main():
while True:
sm.update(0)
image, timestamp = mapsd.update(sm)
server.send(VisionStreamType.VISION_STREAM_MAP, image.tobytes(), rk.frame, timestamp, time.monotonic_ns())
# EOF is the exact route generation rendered, not completion time. An
# old-route render finishing after a new route cannot impersonate it.
generation = mapsd.route_generation if timestamp else 0
server.send(VisionStreamType.VISION_STREAM_MAP, image.tobytes(), rk.frame, timestamp, generation)
rk.keep_time()
finally:
mapsd.close()
+549 -27
View File
@@ -1,5 +1,10 @@
#!/usr/bin/env python3
import copy
import os
import re
import time
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
import numpy as np
import requests
@@ -10,13 +15,389 @@ from openpilot.common.params import Params
from openpilot.common.realtime import Ratekeeper
from openpilot.common.swaglog import cloudlog
from openpilot.selfdrive.navigation.helpers import latest_gps_source
from openpilot.selfdrive.navigation.route import along_path, local_meters
from openpilot.selfdrive.navigation.mapbox_next_turn import MapboxNextTurnClient, NextTurnError
from openpilot.selfdrive.navigation.route import RouteTracker, along_path, cumulative_distance, local_meters, nearest_segment
REROUTE_DISTANCE = 25
MANEUVER_TRANSITION_THRESHOLD = 10
REROUTE_COUNTER_MIN = 3
ROUTE_RESEND_PERIOD = 10 # updates (seconds); a restarted mapsd gets the route within this
RECOMPUTE_HOLD = 10 # updates (seconds) between route requests
MAPBOX_TOKEN_PATH = Path('/data/navigation/mapbox_token')
TURN_TARGET_TTL = 300. # seconds; context and GPS freshness are checked independently
TURN_GPS_MAX_AGE = .5
TURN_GPS_MAX_ACCURACY = 3.
TURN_MAX_CROSS_TRACK = 6.
TURN_MAX_HEADING_ERROR = 35.
TURN_MAX_HEADING_UNCERTAINTY = 15.
TURN_RETRY_SECONDS = 5.
TURN_PASSED_DISTANCE = 10.
TURN_ENTRY_HEADING_DEG = 25.
TURN_EXIT_COMPLETION_M = 12.
def chosen_heading(now, llk, gps):
"""Select a past, fresh calibrated heading; GPS course is a bounded fallback.
llk: valid/status/orientationValid/orientationNed/orientationStd, optional
baseOrientationValid/baseOrientationStd, and monotonic receivedAt/publishedAt.
gps: valid/bearingDeg/bearingAccuracyDeg/fixAgeSeconds and those timestamps.
The f5d producer leaves calibrated std NaN; an explicit base-orientation std
fallback describes filter uncertainty only, not unknown calibration error.
"""
invalid = {'headingDeg': None, 'headingValid': False, 'headingSource': None,
'headingAgeSeconds': None, 'headingUncertaintyDeg': None,
'headingUncertaintyKind': None, 'calibrationUncertaintyKnown': False}
def age(sample):
values = [now-sample['receivedAt'], now-sample['publishedAt']]
return max(values) if np.isfinite(values).all() and all(0. <= value <= TURN_GPS_MAX_AGE for value in values) else None
try:
values, deviations = np.asarray(llk['orientationNed'], dtype=float), np.asarray(llk['orientationStd'], dtype=float)
uncertainty_kind, calibration_known = 'calibrated_orientation', True
if deviations.shape != (3,) or not np.isfinite(deviations).all() or not (deviations >= 0.).all():
if llk.get('baseOrientationValid') is not True:
raise ValueError('No available orientation uncertainty')
deviations = np.asarray(llk['baseOrientationStd'], dtype=float)
uncertainty_kind, calibration_known = 'base_orientation', False
sample_age = age(llk)
uncertainty = float(np.degrees(deviations[2]))
if (llk['valid'] is True and llk['status'] == 'valid' and llk['orientationValid'] is True and
values.shape == (3,) and deviations.shape == (3,) and np.isfinite(values).all() and
np.isfinite(deviations).all() and (deviations >= 0.).all() and sample_age is not None and
uncertainty <= TURN_MAX_HEADING_UNCERTAINTY):
return {'headingDeg': float(np.degrees(values[2]) % 360.), 'headingValid': True,
'headingSource': 'liveLocationKalman', 'headingAgeSeconds': sample_age, 'headingUncertaintyDeg': uncertainty,
'headingUncertaintyKind': uncertainty_kind, 'calibrationUncertaintyKnown': calibration_known}
except (KeyError, TypeError, ValueError, IndexError):
pass
try:
sample_age = age(gps)
course, uncertainty, fix_age = gps['bearingDeg'], gps['bearingAccuracyDeg'], gps['fixAgeSeconds']
if (gps['valid'] is True and sample_age is not None and np.isfinite([course, uncertainty, fix_age]).all() and
0. <= fix_age <= TURN_GPS_MAX_AGE and 0. < uncertainty <= TURN_MAX_HEADING_UNCERTAINTY and 0. <= course <= 360.):
return {'headingDeg': float(course % 360.), 'headingValid': True, 'headingSource': 'gps',
'headingAgeSeconds': max(sample_age, fix_age), 'headingUncertaintyDeg': float(uncertainty),
'headingUncertaintyKind': 'gps_course', 'calibrationUncertaintyKnown': False}
except (KeyError, TypeError, ValueError):
pass
return invalid
def approach_context(target, position, bearing_deg, horizontal_accuracy):
"""Project the raw GPS fix onto the incoming route only, without extrapolation.
Endpoint extensions preserve signed distance before/after this junction. The
outgoing road is deliberately excluded, so a crossroad cannot capture the
projection. This map/GPS check does not measure physical corner clearance.
"""
invalid = {'distanceM': None, 'crossTrackM': None, 'approachBearingDeg': None,
'headingValid': False, 'locationValid': False, 'reason': 'invalid_target_geometry'}
try:
coordinates = np.asarray(target.approach_geometry, dtype=np.float64).reshape((-1, 2)).copy()
position = np.asarray(position, dtype=np.float64).reshape(2).copy()
if len(coordinates) < 2 or not np.isfinite(coordinates).all() or not np.isfinite(position).all():
return invalid
# Keep the local longitude difference short across the antimeridian.
origin = coordinates[0].copy()
coordinates[:, 0] = origin[0] + (coordinates[:, 0]-origin[0]+180.) % 360.-180.
position[0] = origin[0] + (position[0]-origin[0]+180.) % 360.-180.
points, point = local_meters(coordinates, origin), local_meters(position, origin)
vectors = np.diff(points, axis=0)
lengths = np.linalg.norm(vectors, axis=1)
keep = lengths > .001
starts, vectors, lengths = points[:-1][keep], vectors[keep], lengths[keep]
if not len(lengths) or not np.isfinite([bearing_deg, horizontal_accuracy]).all():
return invalid
fractions = ((point-starts)*vectors).sum(axis=1)/lengths**2
clipped = np.clip(fractions, 0., 1.)
clipped[0] = min(clipped[0], fractions[0])
clipped[-1] = max(clipped[-1], fractions[-1])
distances = np.linalg.norm(point-starts-clipped[:, None]*vectors, axis=1)
index = int(np.argmin(distances))
cumulative = np.r_[0., np.cumsum(lengths)]
progress = cumulative[:-1]+clipped*lengths
heading = float(np.degrees(np.arctan2(vectors[index, 0], vectors[index, 1])) % 360.)
heading_error = abs((bearing_deg-heading+180.) % 360.-180.)
ambiguous = np.any((distances <= distances[index]+1.) & (np.abs(progress-progress[index]) > 15.))
location_valid = 0. < horizontal_accuracy <= TURN_GPS_MAX_ACCURACY and distances[index] <= TURN_MAX_CROSS_TRACK and not ambiguous
reason = ('ambiguous_approach' if ambiguous else 'approach_location_mismatch' if not location_valid else
'approach_heading_mismatch' if heading_error > TURN_MAX_HEADING_ERROR else 'approach_match')
return {'distanceM': float(cumulative[-1]-progress[index]), 'crossTrackM': float(distances[index]),
'approachBearingDeg': heading, 'headingErrorDeg': float(heading_error),
'headingValid': bool(heading_error <= TURN_MAX_HEADING_ERROR), 'locationValid': bool(location_valid),
'reason': reason}
except (AttributeError, TypeError, ValueError, IndexError):
return invalid
def turn_route_progress(target, position, heading, entered=False):
"""Observed route progress only; no future pose or steering command."""
points = np.asarray(target.geometry, dtype=float)
local, point = local_meters(points, points[0]), local_meters(position, points[0])
index, fraction, cross = nearest_segment(local[:-1], local[1:], point)
distance = cumulative_distance(points)
progress = float(distance[index]+fraction*(distance[index+1]-distance[index]))
junction_distance = float(distance[target.junction_geometry_index])
sign = 1. if target.direction == 'right' else -1.
change = ((heading-target.incoming_bearing_deg+180.) % 360.-180.)*sign
exit_error = abs((heading-target.exit_bearing_deg+180.) % 360.-180.)
near = np.linalg.norm(point-local[target.junction_geometry_index]) <= 25.
entered = bool(entered or near and change >= TURN_ENTRY_HEADING_DEG and exit_error <= 65. and cross <= TURN_MAX_CROSS_TRACK)
exit_distance = min(TURN_EXIT_COMPLETION_M, .6*max(0., distance[-1]-junction_distance))
completed = bool(entered and progress >= junction_distance+exit_distance and exit_error <= TURN_MAX_HEADING_ERROR and cross <= TURN_MAX_CROSS_TRACK)
approach = approach_context(target, position, heading, 1.)
missed = bool(not entered and approach['distanceM'] is not None and approach['distanceM'] < -TURN_PASSED_DISTANCE)
return {'entered': entered, 'completed': completed, 'missed': missed,
'routeProgressM': progress, 'crossTrackM': cross, 'exitHeadingErrorDeg': exit_error}
class SignalTurnEngine:
"""Session-tagged, token-only discovery with one off-thread request at a time."""
def __init__(self, params, client_factory=None, executor=None, clock=None, wall_clock=None):
self.params = params
self.client_factory = client_factory if client_factory is not None else MapboxNextTurnClient
self.executor = executor
self.clock = clock if clock is not None else time.monotonic
# GNSS fix time is UTC; only this comparison needs the wall clock.
self.wall_clock = wall_clock if wall_clock is not None else time.time # noqa: TID251
self.key = None
self.generation = 0
self.pending = None
self.target = None
self.context = None
self.client = None
self.token_loaded = False
self.failure = None
self.retry_at = 0.
self.blocked = False
self.passed = False
self.signal_blocked = False
self.entered = False
self.retained = False
self.intent_mode = None
self.retired_key = None
def _invalidate(self):
self.generation += 1
if self.pending is not None:
self.pending[0].cancel()
self.target = None
self.client = None
self.token_loaded = False
self.failure = None
self.retry_at = 0.
self.passed = False
self.signal_blocked = False
self.entered = False
self.retained = False
self.intent_mode = None
def _clear(self):
if self.key is not None:
self._invalidate()
self.key = None
self.blocked = False
if self.context is not None:
self.params.remove('MapboxTurnContext')
self.context = None
def _publish(self, now, status, **values):
self.context = {'session': self.key[0], 'direction': self.key[1], 'status': status, 'updatedAt': now,
'distanceM': None, 'locationValid': False, 'headingValid': False, **values}
# route_pending is an internal proposal. Exposing it between every ready
# update would intermittently clear valid model inputs. The RouteEngine
# publishes ready only after the route and instruction are coherent.
if status != 'route_pending':
# Preserve ordering with synchronous retirement/removal of this key.
self.params.put('MapboxTurnContext', self.context, block=True)
return self.context
def _consume(self):
if self.pending is None or not self.pending[0].done():
return
future, generation, key = self.pending
self.pending = None
if generation != self.generation or key != self.key:
return # A canceled/opposite signal, disable or destination change retires the result.
try:
self.target = future.result()
if self.target.direction != self.key[1]:
self.target = None
self.failure = 'target_direction_mismatch'
except NextTurnError as error:
self.target = None
self.failure = error.reason
except Exception:
# Never log exception text/tracebacks: request URLs can contain tokens.
self.target = None
self.failure = 'discovery_failed'
self.retry_at = self.clock()+TURN_RETRY_SECONDS
def update(self, sm, source, destination_active=False):
now = self.clock()
if self.params.get_bool('MapboxNextTurnEnabled') is not True:
self._clear()
self._consume()
return None
intent = self.params.get('MapboxTurnIntent')
valid_intent = (isinstance(intent, dict) and isinstance(intent.get('session'), str) and
1 <= len(intent['session']) <= 128 and intent.get('direction') in ('left', 'right') and
intent.get('mode') in ('pending', 'lane_change', 'turn'))
model = self.params.get('NavModelState')
supported = (isinstance(model, dict) and model.get('supported') is True and
isinstance(model.get('updatedAt'), (int, float)) and 0. <= now-model['updatedAt'] <= 1.)
try:
control_age = now-sm.logMonoTime['carControl']*1e-9
lateral = bool(sm.valid['carControl'] and sm['carControl'].latActive and 0. <= control_age <= TURN_GPS_MAX_AGE)
car = sm['carState']
canceled_after_entry = self.target is not None and self.key is not None and not car.leftBlinker and not car.rightBlinker
except (AttributeError, KeyError, TypeError):
lateral, canceled_after_entry = False, False
retained = bool(canceled_after_entry and self.intent_mode == 'turn' and supported and lateral and
(not valid_intent or (intent['session'], intent['direction']) == self.key))
if not valid_intent and not retained:
self._clear()
self._consume()
return None
if retained:
intent = {'session': self.key[0], 'direction': self.key[1], 'mode': 'turn', 'modelSupportsNavigation': True, 'lateralActive': True}
key = intent['session'], intent['direction']
if key != self.key:
self._invalidate()
self.key = key
self.retired_key = None
if destination_active != self.blocked:
self._invalidate()
self.blocked = destination_active
self.intent_mode = intent['mode']
self._consume()
if destination_active:
self.retired_key = key
return self._publish(now, 'blocked', reason='explicit_destination')
if self.retired_key == key:
return self._publish(now, 'blocked', reason='manual_destination_preempted')
if not supported or intent.get('modelSupportsNavigation') is not True:
self.target = None
return self._publish(now, 'unavailable', reason='navigation_model_required')
if not lateral or intent.get('lateralActive') is not True:
self.target = None
return self._publish(now, 'unavailable', reason='lateral_inactive')
self.retained = retained
if self.passed:
return self._publish(now, 'passed', reason='route_completed_or_missed')
if source is None:
return self._publish(now, 'unavailable', reason='gps_unavailable')
try:
gps, car = sm[source], sm['carState']
receipt_age, publication_age = now-sm.recv_time[source], now-sm.logMonoTime[source]*1e-9
message_age = max(receipt_age, publication_age)
fix_age = self.wall_clock()-gps.unixTimestampMillis*.001
gps_age = max(message_age, fix_age)
accuracy = float(gps.horizontalAccuracy)
bearing = float(gps.bearingDeg)
position = float(gps.longitude), float(gps.latitude)
car_receipt_age, car_publication_age = now-sm.recv_time['carState'], now-sm.logMonoTime['carState']*1e-9
gps_valid = bool(sm.valid[source] and gps.hasFix and gps.unixTimestampMillis > 0 and
0. <= receipt_age <= TURN_GPS_MAX_AGE and 0. <= publication_age <= TURN_GPS_MAX_AGE and
0. <= fix_age <= TURN_GPS_MAX_AGE and
0. < accuracy <= TURN_GPS_MAX_ACCURACY and
np.isfinite([*position, accuracy, gps_age, car.vEgo]).all() and
-180. <= position[0] <= 180. and -85. < position[1] < 85. and
sm.valid['carState'] and 0. <= car_receipt_age <= TURN_GPS_MAX_AGE and
0. <= car_publication_age <= TURN_GPS_MAX_AGE and str(car.gearShifter) != 'reverse')
except (AttributeError, KeyError, TypeError, ValueError):
return self._publish(now, 'unavailable', reason='gps_or_car_state_invalid')
quality = {'gpsAgeSeconds': float(gps_age), 'gpsMessageAgeSeconds': float(message_age),
'gpsFixAgeSeconds': float(fix_age), 'gpsHorizontalAccuracyM': accuracy,
'gpsSource': source, 'gpsLogMonoTime': int(sm.logMonoTime[source]), 'speedMps': float(car.vEgo)}
if not gps_valid:
# Do not serialize NaN/inf into a shared JSON context.
quality = {key: value for key, value in quality.items() if not isinstance(value, float) or np.isfinite(value)}
return self._publish(now, 'unavailable', reason='gps_or_car_state_invalid', **quality)
try:
location = sm['liveLocationKalman']
llk = {'valid': sm.valid['liveLocationKalman'], 'status': str(location.status),
'orientationValid': location.calibratedOrientationNED.valid,
'orientationNed': list(location.calibratedOrientationNED.value), 'orientationStd': list(location.calibratedOrientationNED.std),
'baseOrientationValid': location.orientationNED.valid, 'baseOrientationStd': list(location.orientationNED.std),
'receivedAt': sm.recv_time['liveLocationKalman'], 'publishedAt': sm.logMonoTime['liveLocationKalman']*1e-9}
except (AttributeError, KeyError, TypeError):
llk = None
heading = chosen_heading(now, llk, {'valid': gps_valid, 'bearingDeg': bearing, 'bearingAccuracyDeg': float(gps.bearingAccuracyDeg),
'receivedAt': sm.recv_time[source], 'publishedAt': sm.logMonoTime[source]*1e-9, 'fixAgeSeconds': fix_age})
quality.update({key: value for key, value in heading.items() if key != 'headingValid'})
if not heading['headingValid']:
return self._publish(now, 'unavailable', reason='heading_unavailable', **quality)
bearing = heading['headingDeg']
signal = 'left' if car.leftBlinker and not car.rightBlinker else 'right' if car.rightBlinker and not car.leftBlinker else None
if signal != key[1] and not retained:
if not self.signal_blocked:
self._invalidate()
self.signal_blocked = True
return self._publish(now, 'blocked', reason='physical_signal_mismatch', **quality)
self.signal_blocked = False
if self.target is not None:
age = now-self.target.discovered_at
if not 0. <= age <= TURN_TARGET_TTL:
return self._publish(now, 'unavailable', reason='target_expired', **quality)
match = approach_context(self.target, position, bearing, accuracy)
progress = turn_route_progress(self.target, position, bearing, self.entered)
if retained and not progress['entered']:
self._clear()
return None
self.entered = progress['entered']
if progress['completed'] or progress['missed']:
self.passed = True
if self.entered:
match.update(locationValid=progress['crossTrackM'] <= TURN_MAX_CROSS_TRACK,
headingValid=progress['exitHeadingErrorDeg'] <= 65.)
status = 'passed' if self.passed else 'route_pending' if match['locationValid'] and match['headingValid'] else 'unavailable'
return self._publish(now, status, **quality, **match, **{k:v for k,v in progress.items() if k != 'crossTrackM'},
retainedAfterSignalCancel=retained, routeOwner='signal', junction=list(self.target.junction),
incomingBearingDeg=self.target.incoming_bearing_deg, exitBearingDeg=self.target.exit_bearing_deg,
requestId=self.target.validation_uuid, probeRequestId=self.target.probe_uuid,
targetAgeSeconds=age, name=self.target.name)
if not self.token_loaded:
token = get_mapbox_token() # Once per session, never at 10 Hz.
self.token_loaded = True
if token is None:
self.failure = 'missing_token'
else:
try:
self.client = self.client_factory(token)
except Exception:
self.failure = 'invalid_token'
if self.client is not None and self.pending is None and now >= self.retry_at:
if self.executor is None:
self.executor = ThreadPoolExecutor(max_workers=1, thread_name_prefix='navd-signal-turn')
self.pending = (self.executor.submit(self.client.discover, position=position, bearing_deg=bearing, direction=key[1]), self.generation, key)
self.failure = None
return self._publish(now, 'searching' if self.pending is not None else 'unavailable',
reason='discovering' if self.pending is not None else self.failure or 'retry_wait', **quality)
def close(self):
self._clear()
if self.executor is not None:
self.executor.shutdown(wait=False, cancel_futures=True)
def get_mapbox_token():
token = os.environ.get('MAPBOX_TOKEN')
if token is not None:
return token
try:
token = MAPBOX_TOKEN_PATH.read_text().strip()
except FileNotFoundError:
return None
except (OSError, UnicodeError) as error:
cloudlog.event('navd.mapbox_token_unavailable', error_type=type(error).__name__, error=True)
return None
if not token:
return None
if re.fullmatch(r'pk\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+', token) is None:
cloudlog.event('navd.mapbox_token_invalid', error=True)
return None
return token
def set_banner(instruction, banners, distance_to_maneuver):
@@ -34,10 +415,21 @@ def set_banner(instruction, banners, distance_to_maneuver):
class RouteEngine:
def __init__(self, sm, pm, params=None):
def __init__(self, sm, pm, params=None, update_frequency=1):
self.sm = sm
self.pm = pm
self.params = params if params is not None else Params()
if type(update_frequency) is not int or update_frequency < 1:
raise ValueError('Navigation update frequency must be a positive integer')
self.update_frequency = update_frequency
self.update_counter = 0
self.signal_turn = SignalTurnEngine(self.params)
self.signal_preview_key = None
self.route_owner = None
self.route_key = None
self.route_generation = 0
self.route_tracker = None
self.maneuver_distances = None
self.last_position = None # (longitude, latitude)
self.last_bearing = None
@@ -50,27 +442,98 @@ class RouteEngine:
self.recompute_hold = 0
self.reroute_counter = 0
self.mapbox_token = os.environ.get("MAPBOX_TOKEN")
self.configure_routing(get_mapbox_token())
def configure_routing(self, token):
self.mapbox_token = token
self.api = Api(self.params.get("DongleId")) if self.mapbox_token is None else None
self.mapbox_host = "https://api.mapbox.com" if self.mapbox_token is not None else API_HOST.rstrip('/') + '/v1/nav'
def update(self):
self.sm.update(0)
try:
self._update_route()
except Exception as error:
# API URLs can contain credentials: report only a fixed event and type.
cloudlog.event('navd.route_processing_failed', error_type=type(error).__name__, error=True)
self.clear_route()
self.send_route()
self.params.put('MapboxTurnContext', {'status': 'unavailable', 'updatedAt': time.monotonic(),
'reason': 'route_processing_failed', 'routeGeneration': 0}, block=True)
self.send_instruction()
def _update_route(self):
source = latest_gps_source(self.sm)
if source is not None:
gps = self.sm[source]
self.last_position, self.last_bearing = (gps.longitude, gps.latitude), gps.bearingDeg
route = self.route
try:
self.recompute_route(gps_ok=source is not None)
destination = self.params.get('NavDestination')
destination_active = destination is not None
context = self.signal_turn.update(self.sm, source, destination_active=destination_active)
legacy_tick = self.update_counter % self.update_frequency == 0
self.update_counter += 1
previous = self.route
if destination_active:
requested_destination = destination['longitude'], destination['latitude']
if self.route_owner == 'signal' or (self.route is not None and self.nav_destination != requested_destination):
self.clear_route()
self.send_route()
if legacy_tick:
self.recompute_route(gps_ok=source is not None)
if self.route is not previous:
self.send_route()
if legacy_tick or self.route is not previous:
self.send_instruction()
if self.route is not None and self.nav_destination == requested_destination:
self.params.put('MapboxTurnContext', {'routeOwner': 'destination', 'status': 'ready',
'updatedAt': time.monotonic(), 'routeGeneration': self.route_generation,
'routePublishedAt': self.route_generation*1e-9,
'destination': {'longitude': self.nav_destination[0], 'latitude': self.nav_destination[1]},
'automaticStatus': 'blocked' if context is not None else None}, block=True)
else:
self.params.put('MapboxTurnContext', {'routeOwner': 'destination', 'status': 'searching',
'updatedAt': time.monotonic(), 'reason': 'destination_route_pending'}, block=True)
elif context is not None and context['status'] == 'route_pending':
target = self.signal_turn.target
key = context['session'], target.validation_uuid
if self.route_owner != 'signal' or self.route_key != key:
self.install_route([copy.deepcopy(dict(step)) for step in target.to_dict()['steps']],
tuple(target.geometry[-1]), owner='signal', key=key)
self.send_route()
self.send_instruction()
except Exception:
cloudlog.exception("navd.failed_to_compute")
# mapsd dedupes repeats, so a periodic resend is how a restarted mapsd gets the route
if self.route is not route or self.sm.frame % ROUTE_RESEND_PERIOD == 0:
self.signal_turn._publish(context['updatedAt'], 'ready', **{k:v for k,v in context.items()
if k not in ('session', 'direction', 'status', 'updatedAt')}, routeGeneration=self.route_generation,
routePublishedAt=self.route_generation*1e-9)
else:
target = self.signal_turn.target
same_signal_route = (self.route_owner == 'signal' and context is not None and target is not None and
self.route_key == (context['session'], target.validation_uuid))
transient_unavailable = (same_signal_route and context['status'] == 'unavailable' and
context.get('reason') not in ('target_expired', 'navigation_model_required', 'lateral_inactive', 'route_processing_failed'))
# Keep the generation through a brief localization outage. Context and
# instructions remain invalid, so no stale conditioning is authorized.
if self.route_owner is not None and not transient_unavailable:
self.clear_route()
self.send_route()
if context is not None:
self.send_signal_instruction(context)
else:
self.params.remove('MapboxTurnContext')
self.send_instruction()
resend_period = self.update_frequency if self.route_owner == 'signal' else ROUTE_RESEND_PERIOD*self.update_frequency
if self.sm.frame % resend_period == 0:
self.send_route()
def install_route(self, steps, destination, owner='destination', key=None):
self.route = steps
self.route_geometry = [np.asarray(step['geometry']['coordinates'], dtype=float).reshape((-1, 2)) for step in steps]
self.route_tracker = RouteTracker(np.concatenate(self.route_geometry))
starts = np.cumsum([0]+[len(path) for path in self.route_geometry[:-1]])
self.maneuver_distances = self.route_tracker.distance[starts]
self.step_idx, self.nav_destination = 0, destination
self.route_owner, self.route_key = owner, key
self.route_generation = 0
def recompute_route(self, gps_ok: bool):
if self.last_position is None:
return
@@ -86,8 +549,14 @@ class RouteEngine:
if not gps_ok and self.step_idx is not None:
return
token = get_mapbox_token()
routing_changed = token != self.mapbox_token
if routing_changed:
self.configure_routing(token)
self.recompute_hold = 0
self.recompute_hold -= 1
if (self.should_recompute() or new_destination != self.nav_destination) and self.recompute_hold <= 0:
if (routing_changed or self.should_recompute() or new_destination != self.nav_destination) and self.recompute_hold <= 0:
self.recompute_hold = RECOMPUTE_HOLD
self.calculate_route(new_destination)
self.reroute_counter = 0
@@ -119,23 +588,24 @@ class RouteEngine:
coords_str = ';'.join(f'{lon},{lat}' for lon, lat in (self.last_position, destination))
url = self.mapbox_host + '/directions/v5/mapbox/driving-traffic/' + coords_str
status_code = None
try:
resp = requests.get(url, params=params, timeout=10)
if resp.status_code != 200:
cloudlog.event("API request failed", status_code=resp.status_code, text=resp.text, error=True)
status_code = resp.status_code
resp.raise_for_status()
r = resp.json()
if len(r['routes']):
self.route = r['routes'][0]['legs'][0]['steps']
self.route_geometry = [np.array(step['geometry']['coordinates'], dtype=np.float64).reshape((-1, 2)) for step in self.route]
self.step_idx = 0
self.install_route(r['routes'][0]['legs'][0]['steps'], destination)
else:
cloudlog.warning("Got empty route response")
self.clear_route()
except requests.exceptions.RequestException:
cloudlog.exception("failed to get route")
except requests.exceptions.RequestException as error:
# HTTP errors and response bodies can contain the URL, including access_token.
if error.response is not None:
status_code = error.response.status_code
cloudlog.event('navd.route_request_failed', status_code=status_code, error_type=type(error).__name__, error=True)
self.clear_route()
def send_instruction(self):
@@ -148,17 +618,26 @@ class RouteEngine:
assert self.route is not None and self.route_geometry is not None and self.last_position is not None
step = self.route[self.step_idx]
along_geometry, _ = along_path(self.route_geometry[self.step_idx], self.last_position)
geometry = self.route_geometry[self.step_idx]
along_geometry = (along_path(geometry, self.last_position)[0] if len(geometry) >= 2 else
float(np.linalg.norm(local_meters(self.last_position, geometry[0]))))
distance_to_maneuver_along_geometry = step['distance'] - along_geometry
# Banner instructions are for the following maneuver step, don't use empty last step
banner_step = step
if not len(banner_step['bannerInstructions']) and self.step_idx == len(self.route) - 1:
if not len(banner_step.get('bannerInstructions', [])) and self.step_idx == len(self.route) - 1:
banner_step = self.route[max(self.step_idx - 1, 0)]
# Current instruction
msg.navInstruction.maneuverDistance = distance_to_maneuver_along_geometry
set_banner(msg.navInstruction, banner_step['bannerInstructions'], distance_to_maneuver_along_geometry)
set_banner(msg.navInstruction, banner_step.get('bannerInstructions', []), distance_to_maneuver_along_geometry)
# Exact afbf2e5 upstream allManeuvers semantics: each step starts at its
# maneuver, relative to monotonic measured progress along the full route.
progress = self.route_tracker.project(*self.last_position)
msg.navInstruction.allManeuvers = [
{'distance': float(distance-progress), 'type': item['maneuver']['type'], 'modifier': item['maneuver'].get('modifier', '')}
for item, distance in zip(self.route, self.maneuver_distances, strict=True) if distance >= progress]
# Total remaining distance and time: the rest of this step, then every step after it
remaining = 1.0 - along_geometry / max(step['distance'], 1)
@@ -168,6 +647,12 @@ class RouteEngine:
self.pm.send('navInstruction', msg)
if self.route_owner == 'signal':
# The signal engine owns bounded completion/miss; never chase its outlet
# with destination rerouting or remove a user-owned NavDestination.
self.step_idx = min(int(np.searchsorted(self.maneuver_distances, progress, side='right'))-1, len(self.route)-1)
return
# Transition to next route segment
if distance_to_maneuver_along_geometry < -MANEUVER_TRANSITION_THRESHOLD:
if self.step_idx + 1 < len(self.route):
@@ -184,14 +669,48 @@ class RouteEngine:
def send_route(self):
coords = np.concatenate(self.route_geometry) if self.route_geometry is not None else np.empty((0, 2))
msg = messaging.new_message('navRoute', valid=True)
if not self.route_generation:
self.route_generation = int(msg.logMonoTime)
msg.logMonoTime = self.route_generation
msg.navRoute.coordinates = [{'longitude': lon, 'latitude': lat} for lon, lat in coords.tolist()]
self.pm.send('navRoute', msg)
def send_signal_instruction(self, context):
msg = messaging.new_message('navInstruction', valid=False)
if context['status'] == 'ready':
instruction = msg.navInstruction
instruction.maneuverPrimaryText = 'Signal turn'
name = context.get('name', '')
instruction.maneuverSecondaryText = f"{context['direction'].capitalize()} onto {name}" if name else context['direction'].capitalize()
instruction.maneuverType = 'turn'
instruction.maneuverModifier = context['direction']
instruction.maneuverDistance = max(0., context['distanceM'])
instruction.distanceRemaining = max(0., context['distanceM'])
instruction.timeRemaining = max(0., context['distanceM'])/max(context['speedMps'], 1.)
instruction.showFull = True
else:
texts = {'searching': (f"Finding next {context['direction']}", 'Creating a short navigation route'),
'unavailable': ('Navigation unavailable', 'Route conditioning withheld'),
'blocked': ('Signal turn blocked', 'Destination navigation active'),
'passed': ('Turn passed', 'Signal again for another turn')}
primary, secondary = texts.get(context['status'], texts['unavailable'])
msg.navInstruction.maneuverPrimaryText = primary
msg.navInstruction.maneuverSecondaryText = secondary
msg.navInstruction.maneuverType = 'notification'
msg.navInstruction.maneuverModifier = ''
self.pm.send('navInstruction', msg)
def close(self):
self.signal_turn.close()
def clear_route(self):
self.route = None
self.route_geometry = None
self.step_idx = None
self.nav_destination = None
self.route_owner, self.route_key = None, None
self.route_generation = 0
self.route_tracker, self.maneuver_distances = None, None
def should_recompute(self):
if self.step_idx is None:
@@ -212,13 +731,16 @@ class RouteEngine:
def main():
pm = messaging.PubMaster(['navInstruction', 'navRoute'])
sm = messaging.SubMaster(['gpsLocation', 'gpsLocationExternal'])
sm = messaging.SubMaster(['gpsLocation', 'gpsLocationExternal', 'carState', 'carControl', 'liveLocationKalman'])
rk = Ratekeeper(1.0)
route_engine = RouteEngine(sm, pm)
while True:
route_engine.update()
rk.keep_time()
rk = Ratekeeper(10.)
route_engine = RouteEngine(sm, pm, update_frequency=10)
try:
while True:
route_engine.update()
rk.keep_time()
finally:
route_engine.close()
if __name__ == "__main__":
@@ -0,0 +1,447 @@
"""Standalone recorded-schema fixtures: no credentials, network or driving code."""
from dataclasses import FrozenInstanceError
import importlib.util
import json
from pathlib import Path
import sys
import traceback
import unittest
from unittest.mock import Mock, patch
MODULE = Path(__file__).resolve().parents[1] / 'mapbox_next_turn.py'
spec = importlib.util.spec_from_file_location('mapbox_next_turn_under_test', MODULE)
next_turn = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = next_turn
spec.loader.exec_module(next_turn)
ORIGIN = (0., 0.)
JUNCTION = next_turn.offset(ORIGIN, 0., 100.)
PROBE = next_turn.offset(ORIGIN, 0., next_turn.PROBE_DISTANCE_M)
EXIT = next_turn.offset(JUNCTION, 90., next_turn.EXIT_QUERY_DISTANCE_M)
TOKEN = 'pk.synthetic.signature'
def junction(location=JUNCTION, index=1, out=0):
return {'location': list(location), 'geometry_index': index, 'bearings': [0, 90, 180, 270],
'entry': [True, True, False, True], 'in': 2, 'out': out}
def body(turn=False, road_class='street'):
endpoint = EXIT if turn else PROBE
points = [list(ORIGIN), list(JUNCTION), list(endpoint)]
source = {'location': list(ORIGIN), 'bearings': [0], 'entry': [True], 'out': 0, 'geometry_index': 0}
arrival = {'location': list(endpoint), 'bearings': [270 if turn else 180], 'entry': [True], 'in': 0, 'geometry_index': 2}
at = junction(out=1 if turn else 0)
if road_class is not None:
at['mapbox_streets_v8'] = {'class': road_class}
steps = [
{'mode': 'driving', 'name': 'Approach', 'maneuver': {'type': 'depart', 'location': list(ORIGIN)},
'geometry': {'type': 'LineString', 'coordinates': points[:2] if turn else points},
'intersections': [source] if turn else [source, at]},
]
if turn:
steps.append({'mode': 'driving', 'name': 'Side Street', 'maneuver': {'type': 'turn', 'modifier': 'right', 'location': list(JUNCTION)},
'geometry': {'type': 'LineString', 'coordinates': points[1:]}, 'intersections': [at], 'bannerInstructions': []})
steps.append({'mode': 'driving', 'name': 'End', 'maneuver': {'type': 'arrive', 'location': list(endpoint)},
'geometry': {'type': 'LineString', 'coordinates': [points[-1]]}, 'intersections': [arrival]})
for step in steps:
step['distance'] = next_turn._length(step['geometry']['coordinates'])
step['duration'] = step['distance']/4.
step.setdefault('bannerInstructions', [])
return {'code': 'Ok', 'uuid': 'validation-id' if turn else 'probe-id',
'waypoints': [{'location': list(ORIGIN)}, {'location': list(endpoint)}],
'routes': [{'geometry': {'type': 'LineString', 'coordinates': points}, 'legs': [{'steps': steps}]}]}
def response(value):
return Mock(content=json.dumps(value).encode(), json=Mock(return_value=value), raise_for_status=Mock())
class NextTurnClientTests(unittest.TestCase):
def discover(self, probe=None, validated=None, third=None, direction='right', bearing=0.):
values = [body() if probe is None else probe, body(True) if validated is None else validated]
if third is not None:
values.append(third)
with patch.object(next_turn.requests, 'get', side_effect=[response(value) for value in values]) as request:
target = next_turn.MapboxNextTurnClient(TOKEN).discover(ORIGIN, bearing, direction)
return target, request
def test_validated_target_excludes_synthetic_ahead_probe(self):
target, request = self.discover()
self.assertEqual(request.call_count, 2)
self.assertEqual(target.junction, JUNCTION)
self.assertAlmostEqual(target.distance_m, 100.)
self.assertEqual(target.incoming_bearing_deg, 0.)
self.assertEqual(target.exit_bearing_deg, 90.)
self.assertEqual(target.geometry[-1], EXIT)
self.assertNotEqual(target.geometry[-1], PROBE)
self.assertEqual(target.approach_geometry, (ORIGIN, JUNCTION))
self.assertEqual(target.turn_step_index, 1)
self.assertEqual(target.junction_geometry_index, 1)
self.assertFalse(target.provenance['probeEndpointIsTarget'])
self.assertFalse(target.provenance['steeringCommandIssued'])
self.assertFalse(target.provenance['probeFallbackUsed'])
self.assertEqual(target.provenance['probeEndpointRadiusM'], 100.)
for call in request.call_args_list:
self.assertEqual(call.kwargs['timeout'], (2., 3.))
self.assertEqual(call.kwargs['params']['snapping_include_closures'], 'false;false')
self.assertEqual(call.kwargs['params']['snapping_include_static_closures'], 'false;false')
self.assertNotIn(TOKEN, json.dumps(target.to_dict()))
def test_target_and_recursive_steps_are_immutable(self):
target, _ = self.discover()
with self.assertRaises(FrozenInstanceError):
target.name = 'changed'
with self.assertRaises(TypeError):
target.steps[1]['name'] = 'changed'
mutable = target.to_dict()
mutable['steps'][1]['name'] = 'changed'
self.assertEqual(target.steps[1]['name'], 'Side Street')
def test_restricted_entry_is_not_a_candidate(self):
probe = body()
probe['routes'][0]['legs'][0]['steps'][0]['intersections'][1]['entry'][1] = False
with self.assertRaisesRegex(next_turn.NextTurnError, 'no_matching_turn'):
self.discover(probe=probe)
def test_incoming_heading_is_rotated_180_degrees(self):
target, _ = self.discover()
self.assertEqual(next_turn._delta(target.incoming_bearing_deg, target.exit_bearing_deg), 90.)
with self.assertRaisesRegex(next_turn.NextTurnError, 'incoming_heading_mismatch'):
next_turn._parse_route(body(), ORIGIN, 180., PROBE, 100.)
def test_service_exit_is_rejected_even_when_named(self):
for road_class in ('service', 'street_limited'):
with self.subTest(road_class=road_class), self.assertRaisesRegex(next_turn.NextTurnError, 'unsupported_road'):
self.discover(validated=body(True, road_class))
def test_restricted_selected_exit_is_rejected(self):
value = body(True)
value['routes'][0]['legs'][0]['steps'][1]['intersections'][0]['classes'] = ['restricted']
with self.assertRaisesRegex(next_turn.NextTurnError, 'unsupported_road'):
self.discover(validated=value)
def test_city_trunk_street_is_allowed_but_motorway_and_ramp_are_not(self):
target, _ = self.discover(validated=body(True, 'trunk'))
self.assertEqual(target.road_class, 'trunk')
for kind in ('motorway', 'restricted'):
value = body(True, 'trunk')
value['routes'][0]['legs'][0]['steps'][1]['intersections'][0]['classes'] = [kind]
with self.subTest(kind=kind), self.assertRaisesRegex(next_turn.NextTurnError, 'unsupported_road'):
self.discover(validated=value)
value = body(True, 'trunk')
value['routes'][0]['legs'][0]['steps'][1]['maneuver']['type'] = 'off ramp'
with self.assertRaisesRegex(next_turn.NextTurnError, 'unsupported_junction'):
self.discover(validated=value)
def test_native_north_bearing_360_is_normalized_without_accepting_out_of_range(self):
# Real 5e-15 Directions intersections contain north encoded as 360.
probe = body()
at = probe['routes'][0]['legs'][0]['steps'][0]['intersections'][1]
at['bearings'][0] = 360
target, _ = self.discover(probe=probe)
self.assertEqual(target.incoming_bearing_deg, 0.)
at['bearings'][0] = 361
with self.assertRaisesRegex(next_turn.NextTurnError, 'missing_metadata'):
self.discover(probe=probe)
def test_missing_road_class_requires_separate_same_turn_validation(self):
third = body(True)
third['uuid'] = 'classification-id'
target, request = self.discover(validated=body(True, None), third=third)
self.assertEqual(request.call_count, 3)
self.assertIn('/mapbox/driving/', request.call_args_list[2].args[0])
self.assertEqual(target.classification_uuid, 'classification-id')
with self.assertRaisesRegex(next_turn.NextTurnError, 'missing_road_class'):
self.discover(validated=body(True, None), third=body(True, None))
def test_missing_intermediate_legality_metadata_fails_closed(self):
probe = body()
del probe['routes'][0]['legs'][0]['steps'][0]['intersections'][1]['in']
with self.assertRaisesRegex(next_turn.NextTurnError, 'missing_metadata'):
self.discover(probe=probe)
def test_non_boolean_entry_flags_are_not_treated_as_truth(self):
probe = body()
probe['routes'][0]['legs'][0]['steps'][0]['intersections'][1]['entry'][1] = 1
with self.assertRaisesRegex(next_turn.NextTurnError, 'missing_metadata'):
self.discover(probe=probe)
def test_validation_cannot_turn_at_an_earlier_junction(self):
value = body(True)
earlier = next_turn.offset(ORIGIN, 0, 50.)
value['routes'][0]['geometry']['coordinates'].insert(1, list(earlier))
turn = value['routes'][0]['legs'][0]['steps'][1]['intersections'][0]
turn.update(location=list(earlier), geometry_index=1)
with self.assertRaisesRegex(next_turn.NextTurnError, 'earlier_or_additional_turn'):
self.discover(validated=value)
def test_nearby_different_junction_is_not_the_selected_target(self):
value = body(True)
different = next_turn.offset(JUNCTION, 0., 5.)
value['routes'][0]['geometry']['coordinates'][1] = list(different)
step = value['routes'][0]['legs'][0]['steps'][1]
step['geometry']['coordinates'][0] = list(different)
step['intersections'][0]['location'] = list(different)
with self.assertRaisesRegex(next_turn.NextTurnError, 'earlier_or_additional_turn'):
self.discover(validated=value)
def test_two_edge_road_bend_does_not_consume_turn_intent(self):
item = {'location': JUNCTION, 'distance': 100., 'kind': 'depart', 'step': 0,
'incoming': 0., 'outgoing': 90., 'bearings': [90., 180.], 'entry': [True, False]}
route = {'intersections': [item], 'steps': [{'maneuver': {'type': 'depart', 'location': ORIGIN}}]}
with self.assertRaisesRegex(next_turn.NextTurnError, 'no_matching_turn'):
next_turn._choose_branch(route, 'right')
# The same two edges are eligible if the router explicitly labels a turn here.
item['kind'] = 'turn'
route['steps'][0]['maneuver'] = {'type': 'turn', 'location': JUNCTION}
self.assertEqual(next_turn._choose_branch(route, 'right')['exit'], 90.)
def test_continuing_main_road_bend_with_minor_branch_does_not_hide_next_turn(self):
bend = {'location': JUNCTION, 'distance': 100., 'kind': 'continue', 'step': 0,
'incoming': 0., 'outgoing': 90., 'out': 0, 'bearings': [90., 180., 270.], 'entry': [True, False, True]}
following = {'location': next_turn.offset(JUNCTION, 90., 50.), 'distance': 150., 'kind': 'continue', 'step': 0,
'incoming': 90., 'outgoing': 90., 'out': 0, 'bearings': [90., 180., 270.], 'entry': [True, True, False]}
route = {'intersections': [bend, following], 'steps': [{'maneuver': {'type': 'continue', 'location': ORIGIN}}]}
selected = next_turn._choose_branch(route, 'right')
self.assertEqual(selected['location'], following['location'])
self.assertEqual(selected['exit'], 180.)
def test_probe_uturn_can_expose_first_divided_road_branch_but_is_not_target_authority(self):
# Native route60's ahead query U-turns at Thornbrook. Its first left edge
# can be queried, but the second route must prove an ordinary left turn.
item = {'location': JUNCTION, 'distance': 100., 'kind': 'continue', 'step': 0,
'incoming': 123., 'outgoing': 37., 'out': 0,
'bearings': [37., 122., 220., 303.], 'entry': [True, True, True, False]}
route = {'intersections': [item],
'steps': [{'maneuver': {'type': 'continue', 'modifier': 'uturn', 'location': JUNCTION}}]}
selected = next_turn._choose_branch(route, 'left')
self.assertEqual(selected['exit'], 37.)
def test_later_turn_beyond_proven_exit_is_excluded_from_local_target(self):
value = body(True)
later = next_turn.offset(JUNCTION, 90., 35.)
endpoint = next_turn.offset(later, 0., 5.)
value['waypoints'][-1]['location'] = list(endpoint)
points = [list(ORIGIN), list(JUNCTION), list(later), list(endpoint)]
route = value['routes'][0]
route['geometry']['coordinates'] = points
steps = route['legs'][0]['steps']
steps[1]['geometry']['coordinates'] = points[1:3]
extra = junction(location=later, index=2, out=0)
extra['in'] = 3
extra['mapbox_streets_v8'] = {'class': 'service'}
steps.insert(2, {'mode': 'driving', 'name': 'Later road',
'maneuver': {'type': 'turn', 'location': list(later)},
'geometry': {'type': 'LineString', 'coordinates': points[2:]}, 'intersections': [extra]})
steps[-1]['maneuver']['location'] = list(endpoint)
steps[-1]['geometry']['coordinates'] = [list(endpoint)]
steps[-1]['intersections'][0].update(location=list(endpoint), geometry_index=3)
parsed = next_turn._parse_route(value, ORIGIN, 0., endpoint, 15.)
probe = next_turn._parse_route(body(), ORIGIN, 0., PROBE, 100.)
candidate = next_turn._choose_branch(probe, 'right')
_, _, scoped = next_turn._validate_turn(parsed, candidate, 'right')
self.assertTrue(scoped['targetScopeTruncated'])
with self.assertRaisesRegex(next_turn.NextTurnError, 'incomplete_navigation_route'):
next_turn.coherent_route_steps(scoped)
def test_additional_turn_within_first_20m_of_exit_is_rejected(self):
value = body(True)
near = next_turn.offset(JUNCTION, 90., 10.)
value['routes'][0]['geometry']['coordinates'].insert(2, list(near))
step = value['routes'][0]['legs'][0]['steps'][1]
extra = junction(location=near, index=2, out=0)
extra['in'] = 3
step['intersections'].append(extra)
with self.assertRaisesRegex(next_turn.NextTurnError, 'earlier_or_additional_turn'):
self.discover(validated=value)
def test_scoped_exit_uses_along_road_length_for_gently_curved_street(self):
value = body(True)
first = next_turn.offset(JUNCTION, 90., 10.)
later = next_turn.offset(first, 100., 25.)
endpoint = next_turn.offset(later, 0., 5.)
value['waypoints'][-1]['location'] = list(endpoint)
points = [list(ORIGIN), list(JUNCTION), list(first), list(later), list(endpoint)]
value['routes'][0]['geometry']['coordinates'] = points
steps = value['routes'][0]['legs'][0]['steps']
steps[1]['geometry']['coordinates'] = points[1:4]
extra = {'location': list(later), 'geometry_index': 3, 'bearings': [0, 180, 280],
'entry': [True, True, False], 'in': 2, 'out': 0, 'mapbox_streets_v8': {'class': 'street'}}
steps.insert(2, {'mode': 'driving', 'name': 'Later street',
'maneuver': {'type': 'turn', 'location': list(later)},
'geometry': {'type': 'LineString', 'coordinates': points[3:]}, 'intersections': [extra]})
steps[-1]['maneuver']['location'] = list(endpoint)
steps[-1]['geometry']['coordinates'] = [list(endpoint)]
steps[-1]['intersections'][0].update(location=list(endpoint), geometry_index=4)
parsed = next_turn._parse_route(value, ORIGIN, 0., endpoint, 15.)
probe = next_turn._parse_route(body(), ORIGIN, 0., PROBE, 100.)
_, _, scoped = next_turn._validate_turn(parsed, next_turn._choose_branch(probe, 'right'), 'right')
self.assertTrue(scoped['targetScopeTruncated'])
self.assertAlmostEqual(next_turn._length(scoped['points'][1:]), 20., places=5)
with self.assertRaisesRegex(next_turn.NextTurnError, 'incomplete_navigation_route'):
next_turn.coherent_route_steps(scoped)
def test_validation_cannot_choose_opposite_exit(self):
value = body(True)
value['routes'][0]['legs'][0]['steps'][1]['intersections'][0]['out'] = 3
with self.assertRaisesRegex(next_turn.NextTurnError, 'target_mismatch'):
self.discover(validated=value)
def test_clipped_discovery_is_requeried_as_a_complete_actual_exit_route(self):
value = body(True)
later = next_turn.offset(JUNCTION, 90., 35.)
endpoint = next_turn.offset(later, 0., 5.)
value['waypoints'][-1]['location'] = list(endpoint)
points = [list(ORIGIN), list(JUNCTION), list(later), list(endpoint)]
route = value['routes'][0]
route['geometry']['coordinates'] = points
steps = route['legs'][0]['steps']
steps[1]['geometry']['coordinates'] = points[1:3]
extra = junction(location=later, index=2, out=0)
extra['in'] = 3
steps.insert(2, {'mode': 'driving', 'name': 'Later', 'maneuver': {'type': 'turn', 'location': list(later)},
'geometry': {'coordinates': points[2:]}, 'intersections': [extra]})
steps[-1]['maneuver']['location'] = list(endpoint)
steps[-1]['geometry']['coordinates'] = [list(endpoint)]
steps[-1]['intersections'][0].update(location=list(endpoint), geometry_index=3)
final = body(True)
actual_endpoint = next_turn.offset(JUNCTION, 90., 20.)
final['waypoints'][-1]['location'] = list(actual_endpoint)
final_route = final['routes'][0]
final_route['geometry']['coordinates'][-1] = list(actual_endpoint)
final_steps = final_route['legs'][0]['steps']
final_steps[1]['geometry']['coordinates'][-1] = list(actual_endpoint)
final_steps[1]['distance'], final_steps[1]['duration'] = 20., 5.
final_steps[-1]['geometry']['coordinates'] = [list(actual_endpoint)]
final_steps[-1]['maneuver']['location'] = list(actual_endpoint)
final_steps[-1]['intersections'][0]['location'] = list(actual_endpoint)
target, request = self.discover(validated=value, third=final)
self.assertEqual(request.call_count, 3)
self.assertEqual(target.provenance['requestCount'], 3)
self.assertTrue(target.provenance['coherentNavigationRoute'])
self.assertFalse(target.provenance['targetScopeTruncated'])
self.assertEqual(target.steps[-1]['maneuver']['type'], 'arrive')
self.assertAlmostEqual(target.steps[1]['distance'], 20.)
def test_navigation_route_rejects_missing_arrival_or_disconnected_steps(self):
route = body(True)['routes'][0]
steps = route['legs'][0]['steps']
with self.assertRaisesRegex(next_turn.NextTurnError, 'incomplete_navigation_route'):
next_turn.coherent_route_steps({'steps': steps[:-1]})
steps[1]['geometry']['coordinates'][0] = list(next_turn.offset(JUNCTION, 0., 10.))
steps[1]['distance'] = next_turn._length(steps[1]['geometry']['coordinates'])
with self.assertRaisesRegex(next_turn.NextTurnError, 'disconnected_navigation_route'):
next_turn.coherent_route_steps({'steps': steps})
def test_close_intersection_after_corner_uses_its_local_incoming_segment(self):
value = body(True)
close = next_turn.offset(JUNCTION, 90., 3.)
value['routes'][0]['geometry']['coordinates'].insert(2, list(close))
step = value['routes'][0]['legs'][0]['steps'][1]
step['geometry']['coordinates'].insert(1, list(close))
step['intersections'].append({'location': list(close), 'geometry_index': 2,
'bearings': [90, 270], 'entry': [True, False], 'in': 1, 'out': 0,
'mapbox_streets_v8': {'class': 'street'}})
target, _ = self.discover(validated=value)
self.assertEqual(target.junction, JUNCTION)
def test_unrelated_small_turn_step_cannot_be_hidden_by_bearing_threshold(self):
value = body(True)
extra = {'mode': 'driving', 'name': 'Detour',
'maneuver': {'type': 'turn', 'modifier': 'slight right', 'location': list(ORIGIN)},
'geometry': {'type': 'LineString', 'coordinates': [list(ORIGIN), list(JUNCTION)]}, 'intersections': []}
value['routes'][0]['legs'][0]['steps'].insert(1, extra)
with self.assertRaisesRegex(next_turn.NextTurnError, 'earlier_or_additional_turn'):
self.discover(validated=value)
def test_validation_rejects_uturn(self):
value = body(True)
at = value['routes'][0]['legs'][0]['steps'][1]['intersections'][0]
at['out'], at['entry'][2] = 2, True
with self.assertRaisesRegex(next_turn.NextTurnError, 'target_mismatch'):
self.discover(validated=value)
def test_bad_origin_snap_is_rejected(self):
value = body()
value['waypoints'][0]['location'] = list(next_turn.offset(ORIGIN, 90, 30.))
with self.assertRaisesRegex(next_turn.NextTurnError, 'snapping_mismatch'):
self.discover(probe=value)
def test_missing_request_id_and_no_route_are_explicit(self):
with self.assertRaisesRegex(next_turn.NextTurnError, 'missing_request_id'):
self.discover(probe={**body(), 'uuid': ''})
with patch.object(next_turn.requests, 'get', return_value=response({'code': 'NoRoute'})) as request:
with self.assertRaisesRegex(next_turn.NextTurnError, 'no_route'):
next_turn.MapboxNextTurnClient(TOKEN).discover(ORIGIN, 0., 'right')
self.assertEqual(request.call_count, 2)
def test_failed_probe_retries_once_with_same_coordinates_heading_and_strict_exit(self):
with patch.object(next_turn.requests, 'get', side_effect=[response({'code': 'NoSegment'}), response(body()), response(body(True))]) as request:
target = next_turn.MapboxNextTurnClient(TOKEN).discover(ORIGIN, 0., 'right')
calls = request.call_args_list
self.assertEqual(len(calls), 3)
self.assertEqual(calls[0].args[0], calls[1].args[0])
self.assertEqual([call.kwargs['params']['radiuses'] for call in calls], ['20;100', '20;300', '20;15'])
self.assertEqual([call.kwargs['params']['bearings'] for call in calls], ['0,30;']*3)
self.assertEqual(target.provenance['requestCount'], 3)
self.assertTrue(target.provenance['probeFallbackUsed'])
self.assertEqual(target.provenance['probeFallbackOriginalReason'], 'no_route')
self.assertTrue(target.provenance['coherentNavigationRoute'])
self.assertEqual(target.steps[-1]['maneuver']['type'], 'arrive')
def test_fallback_does_not_relax_source_snap(self):
invalid = body()
invalid['waypoints'][0]['location'] = list(next_turn.offset(ORIGIN, 90., 30.))
with patch.object(next_turn.requests, 'get', side_effect=[response({'code': 'NoSegment'}), response(invalid)]) as request:
with self.assertRaisesRegex(next_turn.NextTurnError, 'snapping_mismatch'):
next_turn.MapboxNextTurnClient(TOKEN).discover(ORIGIN, 0., 'right')
self.assertEqual(request.call_count, 2)
def test_exit_failure_never_uses_probe_fallback(self):
with patch.object(next_turn.requests, 'get', side_effect=[response(body()), response({'code': 'NoSegment'})]) as request:
with self.assertRaisesRegex(next_turn.NextTurnError, 'no_route'):
next_turn.MapboxNextTurnClient(TOKEN).discover(ORIGIN, 0., 'right')
self.assertEqual(request.call_count, 2)
self.assertEqual(request.call_args_list[-1].kwargs['params']['radiuses'], '20;15')
def test_heading_mismatch_never_uses_probe_fallback(self):
value = body()
value['routes'][0]['geometry']['coordinates'][1] = list(next_turn.offset(ORIGIN, 30., 100.))
with patch.object(next_turn.requests, 'get', return_value=response(value)) as request:
with self.assertRaisesRegex(next_turn.NextTurnError, 'incoming_heading_mismatch'):
next_turn.MapboxNextTurnClient(TOKEN).discover(ORIGIN, 0., 'right')
self.assertEqual(request.call_count, 1)
def test_closed_route_and_exclusion_violation_are_rejected(self):
closed = body(True)
closed['routes'][0]['legs'][0]['annotation'] = {'closure': [{'geometry_index_start': 1}]}
violation = body(True)
violation['routes'][0]['notifications'] = [{'type': 'violation'}]
for value, reason in ((closed, 'closed_route'), (violation, 'route_violation')):
with self.subTest(reason=reason), self.assertRaisesRegex(next_turn.NextTurnError, reason):
self.discover(validated=value)
def test_network_exceptions_do_not_expose_token_or_urls(self):
with patch.object(next_turn.requests, 'get', side_effect=next_turn.requests.Timeout('https://example/?access_token='+TOKEN)):
try:
next_turn.MapboxNextTurnClient(TOKEN).discover(ORIGIN, 0., 'right')
except next_turn.NextTurnError as error:
self.assertEqual(error.reason, 'request_failed')
self.assertNotIn(TOKEN, traceback.format_exc())
self.assertNotIn('https://example', traceback.format_exc())
else:
self.fail('Expected fixed redacted failure')
def test_invalid_direction_and_invalid_token_never_query(self):
with patch.object(next_turn.requests, 'get') as request:
with self.assertRaisesRegex(next_turn.NextTurnError, 'invalid_direction'):
next_turn.MapboxNextTurnClient(TOKEN).discover(ORIGIN, 0., 'straight')
with self.assertRaisesRegex(next_turn.NextTurnError, 'invalid_token'):
next_turn.MapboxNextTurnClient('not a token')
request.assert_not_called()
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,370 @@
"""Execute navd integration with mocked messaging/HTTP and controllable futures.
Openpilot imports are supplied as test doubles so these tests also run on a Mac
without the vehicle messaging/Params native libraries. Production function and
class bodies are executed unchanged; no Mapbox/network request is made.
"""
import ast
from concurrent.futures import Future
from pathlib import Path
import subprocess
from types import SimpleNamespace
import unittest
from unittest.mock import Mock, patch
import numpy as np
NAVIGATION = Path(__file__).resolve().parents[1]
WORKTREE = NAVIGATION.parents[2]
def source(name):
path = NAVIGATION/name
if path.exists():
return path.read_text()
return subprocess.check_output(['git', 'show', f'HEAD:openpilot/selfdrive/navigation/{name}'], cwd=WORKTREE, text=True)
class Params:
def __init__(self, enabled=True):
self.values = {'MapboxNextTurnEnabled': enabled, 'MapboxTurnIntent':
{'session': 'one', 'direction': 'right', 'mode': 'turn', 'modelSupportsNavigation': True, 'lateralActive': True}}
self.values['NavModelState'] = {'updatedAt': 100., 'supported': True}
self.writes = []
def get(self, key, **kwargs):
return self.values.get(key)
def get_bool(self, key):
return self.values.get(key) is True
def put(self, key, value, block=False):
self.values[key] = value
self.writes.append((key, value, block))
def remove(self, key):
self.values.pop(key, None)
class NextTurnError(Exception):
def __init__(self, reason):
self.reason = reason
class Executor:
def __init__(self):
self.jobs = []
def submit(self, function, **kwargs):
future = Future()
future.set_running_or_notify_cancel() # Simulate an HTTP request already in flight.
self.jobs.append((future, function, kwargs))
return future
def shutdown(self, **kwargs):
pass
def message(service, valid=True):
return SimpleNamespace(valid=valid, logMonoTime=100_000_000_001, **{service: SimpleNamespace(coordinates=[])})
def load_navd():
route = {'__name__': 'route_test_fixture'}
exec(compile(source('route.py'), 'route.py', 'exec'), route)
namespace = {'__name__': 'navd_test_fixture', 'messaging': SimpleNamespace(new_message=message),
'API_HOST': 'https://api.example.test', 'Api': Mock(), 'Params': Params, 'Ratekeeper': Mock(),
'cloudlog': Mock(), 'MapboxNextTurnClient': Mock(), 'NextTurnError': NextTurnError,
'latest_gps_source': lambda sm: 'gpsLocation' if sm.valid['gpsLocation'] else None,
'along_path': route['along_path'], 'local_meters': route['local_meters'],
'RouteTracker': route['RouteTracker'], 'cumulative_distance': route['cumulative_distance'],
'nearest_segment': route['nearest_segment']}
tree = ast.parse(source('navd.py'))
tree.body = [node for node in tree.body
if not (isinstance(node, ast.ImportFrom) and node.module.startswith('openpilot.'))
and not (isinstance(node, ast.Import) and any(item.name.startswith('openpilot.') for item in node.names))]
exec(compile(tree, str(NAVIGATION/'navd.py'), 'exec'), namespace)
return SimpleNamespace(**namespace)
class SM:
def __init__(self):
self.frame = 1
self.valid = dict.fromkeys(('gpsLocation', 'carState', 'carControl'), True)
self.recv_time = dict.fromkeys(self.valid, 100.)
self.logMonoTime = dict.fromkeys(self.valid, 100_000_000_000)
self.gps = SimpleNamespace(hasFix=True, longitude=0., latitude=0., bearingDeg=0., bearingAccuracyDeg=3.,
horizontalAccuracy=1., unixTimestampMillis=1_000_000_000)
self.car = SimpleNamespace(vEgo=2., gearShifter='drive', leftBlinker=False, rightBlinker=True)
def __getitem__(self, key):
return SimpleNamespace(latActive=True) if key == 'carControl' else self.car if key == 'carState' else self.gps
def update(self, timeout):
pass
def target(direction='right', request='validation-one', discovered=100.):
result = SimpleNamespace(direction=direction, discovered_at=discovered, approach_geometry=((0., 0.), (0., .001)),
geometry=((0., 0.), (0., .001), (.0004, .001)), junction=(0., .001),
incoming_bearing_deg=0., exit_bearing_deg=90. if direction == 'right' else 270.,
validation_uuid=request, probe_uuid='probe-one', name='Example Street', junction_geometry_index=1)
points = list(result.geometry)
result.steps = [
{'geometry': {'coordinates': points[:2]}, 'distance': 111., 'duration': 20.,
'maneuver': {'type': 'depart'}, 'bannerInstructions': [{'distanceAlongGeometry': 111.,
'primary': {'text': 'Turn right', 'type': 'turn', 'modifier': direction}}]},
{'geometry': {'coordinates': points[1:]}, 'distance': 44., 'duration': 8.,
'maneuver': {'type': 'turn', 'modifier': direction}, 'bannerInstructions': []},
{'geometry': {'coordinates': [points[-1]]}, 'distance': 0., 'duration': 0.,
'maneuver': {'type': 'arrive'}, 'bannerInstructions': []}]
result.to_dict = lambda: {'steps': result.steps}
return result
class TestSignalTurnEngine(unittest.TestCase):
def setUp(self):
self.navd = load_navd()
self.params, self.sm, self.executor = Params(), SM(), Executor()
self.now = 100.
self.wall = 1_000_000.
self.client_factory = Mock(return_value=SimpleNamespace(discover=Mock()))
self.engine = self.navd.SignalTurnEngine(self.params, self.client_factory, self.executor,
clock=lambda: self.now, wall_clock=lambda: self.wall)
self.token = patch.dict(self.engine.update.__globals__, get_mapbox_token=Mock(return_value='pk.example.signature'))
self.token.start()
self.addCleanup(self.token.stop)
def update(self, destination=False):
return self.engine.update(self.sm, 'gpsLocation', destination_active=destination)
def ready(self):
self.assertEqual(self.update()['status'], 'searching')
self.executor.jobs[-1][0].set_result(target())
self.assertEqual(self.update()['status'], 'route_pending')
def test_disabled_makes_no_token_client_worker_or_account_request(self):
self.params.values['MapboxNextTurnEnabled'] = False
with patch.dict(self.engine.update.__globals__, get_mapbox_token=Mock(side_effect=AssertionError('unexpected credential read'))):
self.assertIsNone(self.update())
self.client_factory.assert_not_called()
self.assertEqual(self.executor.jobs, [])
self.assertNotIn('MapboxTurnContext', self.params.values)
def test_missing_token_is_unavailable_and_read_once_per_session(self):
getter = Mock(return_value=None)
with patch.dict(self.engine.update.__globals__, get_mapbox_token=getter):
for _ in range(3):
self.assertEqual(self.update()['reason'], 'missing_token')
getter.assert_called_once_with()
self.assertEqual(self.executor.jobs, [])
def test_raw_fix_distance_not_extrapolated_and_context_has_true_ages(self):
self.ready()
distance = self.engine.context['distanceM']
self.now += .2
self.wall += .2
context = self.update()
self.assertAlmostEqual(context['distanceM'], distance)
self.assertAlmostEqual(context['gpsAgeSeconds'], .2)
self.assertEqual(context['updatedAt'], self.now)
self.assertTrue(context['headingValid'] and context['locationValid'])
self.assertEqual(context['requestId'], 'validation-one')
def test_opposite_signal_waits_for_single_worker_and_discards_old_result(self):
self.update()
self.params.values['MapboxTurnIntent'] = {'session': 'two', 'direction': 'left', 'mode': 'turn', 'modelSupportsNavigation': True, 'lateralActive': True}
self.sm.car.leftBlinker, self.sm.car.rightBlinker = True, False
self.assertEqual(self.update()['status'], 'searching')
self.assertEqual(len(self.executor.jobs), 1)
self.executor.jobs[0][0].set_result(target())
self.assertEqual(self.update()['session'], 'two')
self.assertIsNone(self.engine.target)
self.assertEqual(len(self.executor.jobs), 2)
self.executor.jobs[1][0].set_result(target('left', 'validation-two'))
context = self.update()
self.assertEqual((context['session'], context['direction'], context['status']), ('two', 'left', 'route_pending'))
def test_cancel_or_disable_retires_inflight_result(self):
for cancel in ('intent', 'disable'):
with self.subTest(cancel=cancel):
self.setUp()
self.update()
if cancel == 'intent':
self.params.values.pop('MapboxTurnIntent')
else:
self.params.values['MapboxNextTurnEnabled'] = False
self.assertIsNone(self.update())
self.executor.jobs[0][0].set_result(target())
self.assertIsNone(self.update())
self.assertIsNone(self.engine.target)
self.assertNotIn('MapboxTurnContext', self.params.values)
def test_explicit_destination_blocks_requests_and_retires_prior_work(self):
self.assertEqual(self.update(destination=True)['status'], 'blocked')
self.assertEqual(self.executor.jobs, [])
self.client_factory.assert_not_called()
self.assertEqual(self.update(destination=False)['reason'], 'manual_destination_preempted')
self.assertEqual(self.executor.jobs, [])
self.assertIsNone(self.engine.target)
def test_stale_fix_poor_accuracy_future_publication_and_reverse_block_discovery(self):
changes = (lambda: setattr(self.sm.gps, 'unixTimestampMillis', self.sm.gps.unixTimestampMillis-1000),
lambda: setattr(self.sm.gps, 'horizontalAccuracy', 4.),
lambda: self.sm.logMonoTime.update(gpsLocation=101_000_000_000),
lambda: setattr(self.sm.car, 'gearShifter', 'reverse'))
for change in changes:
with self.subTest(change=change):
self.sm = SM()
change()
self.assertEqual(self.update()['status'], 'unavailable')
self.assertEqual(self.executor.jobs, [])
def test_physical_signal_cancellation_blocks_stale_intent(self):
self.ready()
self.sm.car.rightBlinker = False
context = self.update()
self.assertIsNone(context)
self.assertIsNone(self.engine.target)
self.assertEqual(len(self.executor.jobs), 1)
def test_wrong_heading_or_crossroad_position_has_no_ready_context(self):
self.ready()
self.sm.gps.bearingDeg = 90.
self.assertEqual(self.update()['status'], 'unavailable')
self.sm.gps.bearingDeg = 0.
self.sm.gps.longitude = .0002
context = self.update()
self.assertEqual(context['status'], 'unavailable')
self.assertGreater(context['crossTrackM'], 20.)
def test_expired_target_and_passed_junction_do_not_retarget_held_signal(self):
self.ready()
self.engine.target = target(discovered=-201.)
self.assertEqual(self.update()['reason'], 'target_expired')
self.engine.target = target()
self.sm.gps.latitude = .0012
self.assertEqual(self.update()['status'], 'passed')
self.sm.gps.latitude = 0.
self.assertEqual(self.update()['status'], 'passed')
self.assertEqual(len(self.executor.jobs), 1)
def test_discovery_exception_does_not_publish_or_log_url_or_token(self):
self.update()
secret = 'https://example.test/?access_token=pk.secret.signature'
self.executor.jobs[0][0].set_exception(RuntimeError(secret))
context = self.update()
self.assertEqual(context['reason'], 'discovery_failed')
self.assertNotIn(secret, str(self.params.writes))
self.assertEqual(self.navd.cloudlog.mock_calls, [])
class TestApproachProjection(unittest.TestCase):
def setUp(self):
self.navd = load_navd()
def test_signed_distance_uses_incoming_geometry_not_outgoing_leg(self):
before = self.navd.approach_context(target(), (0., .0009), 0., 1.)
after = self.navd.approach_context(target(), (0., .0011), 0., 1.)
self.assertAlmostEqual(before['distanceM'], 11.1195, delta=.01)
self.assertAlmostEqual(after['distanceM'], -11.1195, delta=.01)
on_outgoing = self.navd.approach_context(target(), (.0002, .001), 90., 1.)
self.assertFalse(on_outgoing['locationValid'])
self.assertGreater(on_outgoing['crossTrackM'], 20.)
def test_invalid_geometry_and_zero_accuracy_are_not_ready_and_inputs_unchanged(self):
pose = np.array([0., 0.])
self.assertFalse(self.navd.approach_context(target(), pose, 0., 0.)['locationValid'])
np.testing.assert_array_equal(pose, [0., 0.])
malformed = target()
malformed.approach_geometry = ((float('nan'), 0.), (0., .001))
self.assertFalse(self.navd.approach_context(malformed, pose, 0., 1.)['locationValid'])
class TestChosenHeading(unittest.TestCase):
def setUp(self):
self.navd = load_navd()
self.llk = {'valid': True, 'status': 'valid', 'orientationValid': True,
'orientationNed': [0., 0., np.pi/2], 'orientationStd': [.01, .01, .02],
'receivedAt': 99.9, 'publishedAt': 99.8}
self.gps = {'valid': True, 'bearingDeg': 0., 'bearingAccuracyDeg': 4.,
'fixAgeSeconds': .1, 'receivedAt': 99.9, 'publishedAt': 99.9}
def test_fresh_calibrated_localizer_heading_precedes_gps_course(self):
result = self.navd.chosen_heading(100., self.llk, self.gps)
self.assertEqual((result['headingSource'], result['headingDeg']), ('liveLocationKalman', 90.))
self.assertAlmostEqual(result['headingAgeSeconds'], .2)
def test_invalid_or_stale_localizer_falls_back_only_to_fresh_accurate_gps(self):
for update in ({'status': 'uncalibrated'}, {'publishedAt': 99.}, {'publishedAt': 100.1},
{'orientationStd': [.01, .01, np.radians(16.)]}):
result = self.navd.chosen_heading(100., {**self.llk, **update}, self.gps)
self.assertEqual(result['headingSource'], 'gps')
self.gps['bearingAccuracyDeg'] = 16.
self.assertFalse(self.navd.chosen_heading(100., None, self.gps)['headingValid'])
self.gps['bearingAccuracyDeg'] = 0.
self.assertFalse(self.navd.chosen_heading(100., None, self.gps)['headingValid'])
def test_future_nan_or_missing_samples_are_not_valid_headings(self):
self.gps['publishedAt'] = 100.1
for orientation in (None, [float('nan'), 0., 0.], [0., 0.], [0., 0., 0.]):
invalid = {**self.llk, 'orientationNed': orientation, 'receivedAt': 100.1}
self.assertFalse(self.navd.chosen_heading(100., invalid, self.gps)['headingValid'])
def test_producer_nan_calibrated_std_uses_explicit_base_covariance_without_claiming_calibration(self):
self.llk.update(orientationStd=[float('nan')]*3, baseOrientationValid=True, baseOrientationStd=[.01, .01, .02])
result = self.navd.chosen_heading(100., self.llk, self.gps)
self.assertEqual((result['headingSource'], result['headingDeg']), ('liveLocationKalman', 90.))
self.assertEqual(result['headingUncertaintyKind'], 'base_orientation')
self.assertFalse(result['calibrationUncertaintyKnown'])
self.llk['baseOrientationValid'] = False
self.assertEqual(self.navd.chosen_heading(100., self.llk, self.gps)['headingSource'], 'gps')
def test_finite_high_calibrated_uncertainty_cannot_select_smaller_base_covariance(self):
self.llk.update(orientationStd=[.01, .01, np.radians(16.)], baseOrientationValid=True, baseOrientationStd=[.01]*3)
self.assertEqual(self.navd.chosen_heading(100., self.llk, self.gps)['headingSource'], 'gps')
class TestNavigationIntegration(unittest.TestCase):
def test_10hz_updates_keep_destination_recompute_and_instruction_at1hz(self):
navd, sm, params, pm = load_navd(), SM(), Params(enabled=False), Mock()
params.values['NavDestination'] = {'longitude': 0., 'latitude': .002}
with patch.dict(navd.RouteEngine.__init__.__globals__, get_mapbox_token=Mock(return_value=None)):
engine = navd.RouteEngine(sm, pm, params, update_frequency=10)
engine.recompute_route, engine.send_instruction, engine.send_route = Mock(), Mock(), Mock()
for frame in range(1, 201):
sm.frame = frame
engine.update()
self.assertEqual(engine.recompute_route.call_count, 20)
self.assertEqual(engine.send_instruction.call_count, 20)
self.assertEqual(engine.send_route.call_count, 2)
engine.api.get_token.assert_not_called()
def test_signal_preview_is_labeled_and_clears_immediately_without_destination_write(self):
navd, sm, params, pm, executor = load_navd(), SM(), Params(), Mock(), Executor()
with patch.dict(navd.RouteEngine.__init__.__globals__, get_mapbox_token=Mock(return_value='pk.example.signature')):
engine = navd.RouteEngine(sm, pm, params, update_frequency=10)
engine.signal_turn = navd.SignalTurnEngine(params, Mock(return_value=SimpleNamespace(discover=Mock())), executor,
clock=lambda: 100., wall_clock=lambda: 1_000_000.)
engine.update()
executor.jobs[0][0].set_result(target())
sm.frame += 1
engine.update()
instructions = [call.args[1] for call in pm.send.call_args_list if call.args[0] == 'navInstruction']
self.assertEqual(instructions[-1].navInstruction.maneuverPrimaryText, 'Turn right')
self.assertTrue(instructions[-1].navInstruction.allManeuvers)
self.assertEqual(engine.route_owner, 'signal')
self.assertTrue(instructions[-1].valid)
self.assertNotIn('NavDestination', params.values)
params.values.pop('MapboxTurnIntent')
sm.frame += 1
engine.update()
instructions = [call.args[1] for call in pm.send.call_args_list if call.args[0] == 'navInstruction']
routes = [call.args[1] for call in pm.send.call_args_list if call.args[0] == 'navRoute']
self.assertFalse(instructions[-1].valid)
self.assertEqual(routes[-1].navRoute.coordinates, [])
if __name__ == '__main__':
unittest.main()
@@ -1,4 +1,6 @@
import gzip
import json
import os
import re
import struct
import tempfile
@@ -17,7 +19,7 @@ import requests
import openpilot.cereal.messaging as messaging
from openpilot.common.prefix import OpenpilotPrefix
from openpilot.selfdrive.navigation.mapsd import MapsD
from openpilot.selfdrive.navigation.navd import REROUTE_COUNTER_MIN, ROUTE_RESEND_PERIOD, RouteEngine, set_banner
from openpilot.selfdrive.navigation.navd import RECOMPUTE_HOLD, REROUTE_COUNTER_MIN, ROUTE_RESEND_PERIOD, RouteEngine, set_banner
from openpilot.selfdrive.navigation.pmtiles import PMTiles, decode_directory, tile_id
from openpilot.selfdrive.navigation.renderer import MAJOR, MEDIUM, MINOR, SIZE, STREET, TRACK, MapRenderer, draw_segments, mercator, project, stroke_widths
from openpilot.selfdrive.navigation.route import RouteTracker
@@ -100,6 +102,16 @@ def route_engine(sm, pm, values):
class RangeHandler(BaseHTTPRequestHandler):
def do_GET(self):
if self.path == '/builds.json':
self.server.index_requests += 1
entries = {} if self.server.invalid_index else [
{'key': '20261103.pmtiles', 'version': '4.15.2', 'size': len(self.server.archive)}]
data = json.dumps(entries).encode()
self.send_response(200)
self.send_header('Content-Length', str(len(data)))
self.end_headers()
self.wfile.write(data)
return
self.server.requests.append(dict(self.headers))
if self.headers.get('If-Match', self.server.etag) != self.server.etag:
self.send_error(412)
@@ -122,7 +134,7 @@ class RangeHandler(BaseHTTPRequestHandler):
class TileServerCase(unittest.TestCase):
"""Serves a leaf-directory archive over HTTP range requests, with the comma API and tile cache stubbed."""
"""Serves a real public-index/leaf-directory archive protocol over local HTTP."""
def setUp(self):
self.directory = tempfile.TemporaryDirectory()
@@ -136,11 +148,10 @@ class TileServerCase(unittest.TestCase):
threading.Thread(target=self.server.serve_forever, kwargs={'poll_interval': 0.01}, daemon=True).start()
self.addCleanup(self.server.server_close)
self.addCleanup(self.server.shutdown)
self.api = MagicMock(dongle_id='test-dongle')
self.api.get_token.return_value = 'device-token'
self.url = f'http://127.0.0.1:{self.server.server_port}/planet.pmtiles?sig=secret'
self.api.get.return_value.json.return_value = {'url': self.url, 'expires': time.time() + 86400} # noqa: TID251
for name, value in [('Api', MagicMock(return_value=self.api)), ('Params', MagicMock()), ('PC', True)]:
self.server.index_requests = 0
self.server.invalid_index = False
self.url = f'http://127.0.0.1:{self.server.server_port}'
for name, value in [('PC', True), ('BUILD_INDEX_URL', self.url + '/builds.json'), ('BUILD_BASE_URL', self.url + '/')]:
self.enterContext(patch(f'openpilot.selfdrive.navigation.tiles.{name}', value))
self.enterContext(patch('openpilot.selfdrive.navigation.tiles.Paths.comma_home', return_value=self.directory.name))
@@ -167,6 +178,15 @@ class TestEndToEnd(TileServerCase):
sm.update(0)
return mapsd.update(sm)
def restored_route_frame():
# Destination cancellation retains the existing recompute cooldown.
# Advance its bounded update count before testing tile-cache reuse.
for _ in range((RECOMPUTE_HOLD + 1) * navd.update_frequency):
result = frame()
if navd.route is not None:
return result
self.fail('Destination route was not restored within the existing recompute cooldown')
image, timestamp = frame()
# The route is centered over the white road, with the ego arrow three quarters down.
self.assertEqual((image[128, 128], image[128, 131], image[188, 128], image[195, 128]), (128, 255, 255, 0))
@@ -179,39 +199,39 @@ class TestEndToEnd(TileServerCase):
values.pop('NavDestination')
self.assertFalse(frame()[0].any()) # black without a route
values['NavDestination'] = dict(destination)
self.assertEqual(frame()[0][128, 128], 128)
self.assertEqual(restored_route_frame()[0][128, 128], 128)
self.assertEqual(len(self.server.requests), fetched) # the second route is served from the cache
image, timestamp = frame(fix=False)
self.assertFalse(image.any() or timestamp) # black, not stale, without a fix
self.api.get.assert_called_once()
self.assertEqual(self.server.index_requests, 1)
self.assertTrue(all('Range' in headers for headers in self.server.requests))
self.assertTrue(all(headers['If-Match'] == self.server.etag for headers in self.server.requests[1:]))
self.assertTrue(all(b'secret' not in p.read_bytes() for p in self.cache.iterdir()))
self.assertTrue(all(b'secret' not in p.read_bytes() for p in self.cache.rglob('*') if p.is_file()))
# The archive is replaced on the server and the cache evicted: the next fetch fails, the map goes black,
# and after a delay a fresh reader starts over on the new version.
self.server.etag = '"new-version"'
for cached in self.cache.iterdir():
cached.unlink()
for cached in self.cache.rglob('*'):
if cached.is_file() and re.fullmatch(r'\d+-\d+', cached.name):
cached.unlink()
values.pop('NavDestination')
frame()
values['NavDestination'] = dict(destination)
image, timestamp = frame()
image, timestamp = restored_route_frame()
self.assertFalse(image.any() or timestamp)
fetched = len(self.server.requests)
self.assertFalse(frame()[1]) # not retried yet
self.assertEqual(len(self.server.requests), fetched)
mapsd.retry_at = 0
self.assertEqual(frame()[0][128, 128], 128)
self.assertEqual((self.cache / 'version').read_text().split()[0], '"new-version"')
self.assertEqual(json.loads((self.cache / 'archive.json').read_text())['etag'], '"new-version"')
class TestRemoteTiles(TileServerCase):
def test_shared_cache_and_refresh(self):
with closing(PMTiles(RemoteTiles())) as archive:
archive.source.expires = 0 # the signed URL is refreshed before the next fetch
self.assertEqual(archive.get(0, 0, 0), road_tile())
self.assertEqual(archive.get(0, 0, 0), road_tile())
self.assertEqual(self.api.get.call_count, 2)
self.assertEqual(self.server.index_requests, 1)
count = len(self.server.requests)
with closing(PMTiles(RemoteTiles())) as archive:
self.assertEqual(archive.get(0, 0, 0), road_tile())
@@ -219,7 +239,9 @@ class TestRemoteTiles(TileServerCase):
self.assertEqual(len(self.server.requests), count + 1)
with patch('openpilot.selfdrive.navigation.tiles.CACHE_LIMIT', 1), closing(RemoteTiles()) as source:
source.read(1, 2) # an uncached range is downloaded, then the cache is trimmed to the limit
self.assertEqual([p.name for p in self.cache.iterdir()], ['version'])
retained = [p.name for p in self.cache.rglob('*') if p.is_file() and re.fullmatch(r'\d+-\d+', p.name)]
self.assertEqual(retained, ['0-127']) # current header is retained for offline startup
self.assertTrue((self.cache / 'archive.json').exists())
def test_prefetch_merges_nearby_ranges(self):
with patch('openpilot.selfdrive.navigation.tiles.MERGE_GAP', 16), closing(RemoteTiles()) as source:
@@ -234,20 +256,25 @@ class TestRemoteTiles(TileServerCase):
def test_failures(self):
with closing(PMTiles(RemoteTiles())) as archive:
self.server.etag = '"new-version"'
with self.assertRaisesRegex(OSError, '412'): # the archive changed under an open reader
with self.assertRaises(OSError): # the archive changed under an open reader
archive.get(0, 0, 0)
with closing(PMTiles(RemoteTiles())) as archive: # a new reader starts over on the new version
self.assertEqual(archive.get(0, 0, 0), road_tile())
self.server.ignore_range = True
with self.assertRaisesRegex(OSError, '200'):
with self.assertRaises(OSError):
RemoteTiles()
self.api.get.return_value.raise_for_status.side_effect = requests.HTTPError(self.url)
with self.assertRaisesRegex(OSError, '^Unable to fetch map tiles$'): # the signed URL stays out of the error
self.server.ignore_range = False
self.server.invalid_index = True
with self.assertRaisesRegex(OSError, '^Unable to load public Protomaps basemap$'):
RemoteTiles()
class TestRouteEngine(unittest.TestCase):
def setUp(self):
self.enterContext(patch.dict(os.environ))
os.environ.pop('MAPBOX_TOKEN', None)
self.token_path = Path(self.enterContext(tempfile.TemporaryDirectory())) / 'mapbox_token'
self.enterContext(patch('openpilot.selfdrive.navigation.navd.MAPBOX_TOKEN_PATH', self.token_path))
self.pm = MagicMock()
self.values = {'NavDestination': {'latitude': 0, 'longitude': 0.002}, 'LanguageSetting': 'main_en'}
self.sm = MagicMock()
@@ -259,9 +286,10 @@ class TestRouteEngine(unittest.TestCase):
self.engine.last_position, self.engine.last_bearing = (0, 0), 90
distance = 111.195 # Mapbox's length of 0.001 degrees at the equator
self.response = mapbox_response([
{'geometry': {'coordinates': [[0, 0], [0.0005, 0], [0.001, 0]]}, 'distance': distance, 'duration': 10,
{'maneuver': {'type': 'depart'}, 'geometry': {'coordinates': [[0, 0], [0.0005, 0], [0.001, 0]]}, 'distance': distance, 'duration': 10,
'bannerInstructions': [{'distanceAlongGeometry': 100, 'primary': {'text': 'Turn right', 'type': 'turn', 'modifier': 'right'}}]},
{'geometry': {'coordinates': [[0.001, 0], [0.002, 0]]}, 'distance': distance, 'duration': 10, 'bannerInstructions': []},
{'maneuver': {'type': 'continue', 'modifier': 'straight'}, 'geometry': {'coordinates': [[0.001, 0], [0.002, 0]]},
'distance': distance, 'duration': 10, 'bannerInstructions': []},
])
def last_message(self, service):
@@ -291,6 +319,97 @@ class TestRouteEngine(unittest.TestCase):
self.engine.send_instruction()
self.assertAlmostEqual(self.last_message('navInstruction').navInstruction.maneuverDistance, 0, delta=0.01)
def test_file_token_routes_directly_to_mapbox(self):
token = 'pk.test_payload.test_signature'
self.token_path.write_text(token + '\n')
self.engine = route_engine(self.sm, self.pm, self.values)
self.engine.last_position, self.engine.last_bearing = (0, 0), 90
get = self.load_route()
self.assertEqual(get.call_args.args[0], 'https://api.mapbox.com/directions/v5/mapbox/driving-traffic/0,0;0.002,0')
self.assertEqual(get.call_args.kwargs['params']['access_token'], token)
self.assertIsNone(self.engine.api)
self.engine.send_instruction()
self.assertTrue(self.last_message('navInstruction').valid)
def test_environment_token_does_not_read_file(self):
token = 'pk.environment.signature'
with patch.dict(os.environ, {'MAPBOX_TOKEN': token}), patch.object(Path, 'read_text') as read, \
patch('requests.get', return_value=self.response):
engine = route_engine(self.sm, self.pm, self.values)
engine.last_position = (0, 0)
engine.recompute_route(gps_ok=True)
read.assert_not_called()
self.assertEqual(engine.mapbox_token, token)
self.assertEqual(engine.mapbox_host, 'https://api.mapbox.com')
def test_token_provisioning_reloads_without_restart(self):
self.load_route()
self.engine.recompute_hold = 10
token = 'pk.provisioned.signature'
self.token_path.write_text(token)
with patch('requests.get', return_value=self.response) as get:
self.engine.recompute_route(gps_ok=True)
self.assertTrue(get.call_args.args[0].startswith('https://api.mapbox.com/directions/'))
self.assertEqual(get.call_args.kwargs['params']['access_token'], token)
self.assertIsNone(self.engine.api)
def test_token_rotation_recomputes_route_once(self):
self.token_path.write_text('pk.first.signature')
with patch('requests.get', return_value=self.response) as get:
self.engine.recompute_route(gps_ok=True)
self.assertEqual(get.call_args.kwargs['params']['access_token'], 'pk.first.signature')
self.token_path.write_text('pk.renewed.signature')
self.engine.recompute_route(gps_ok=True)
self.assertEqual(get.call_args.kwargs['params']['access_token'], 'pk.renewed.signature')
self.engine.recompute_route(gps_ok=True)
self.assertEqual(get.call_count, 2)
def test_missing_or_empty_file_uses_comma(self):
self.assertFalse(self.token_path.exists())
self.assertIsNone(self.engine.mapbox_token)
for contents in ('', ' \n\t'):
with self.subTest(contents=contents):
self.token_path.write_text(contents)
engine = route_engine(self.sm, self.pm, self.values)
self.assertIsNone(engine.mapbox_token)
self.assertEqual(engine.mapbox_host, 'https://api.commadotai.com/v1/nav')
def test_invalid_token_file_is_redacted(self):
for token in ('sk.secret.signature', 'pk.secret withspace.signature', 'pk.', 'pk.secret.\x00signature'):
with self.subTest(token=token), patch('openpilot.selfdrive.navigation.navd.cloudlog') as logger:
self.token_path.write_text(token)
engine = route_engine(self.sm, self.pm, self.values)
self.assertIsNone(engine.mapbox_token)
self.assertEqual(logger.mock_calls[0].args, ('navd.mapbox_token_invalid',))
self.assertEqual(logger.mock_calls[0].kwargs, {'error': True})
self.assertNotIn(token, str(logger.mock_calls))
def test_unreadable_token_file_is_redacted(self):
for error in (PermissionError('pk.secret.signature'), UnicodeError('pk.secret.signature')):
with self.subTest(error_type=type(error).__name__), \
patch.object(Path, 'read_text', side_effect=error), patch('openpilot.selfdrive.navigation.navd.cloudlog') as logger:
engine = route_engine(self.sm, self.pm, self.values)
self.assertIsNone(engine.mapbox_token)
logger.event.assert_called_once_with('navd.mapbox_token_unavailable', error_type=type(error).__name__, error=True)
self.assertNotIn('pk.secret.signature', str(logger.mock_calls))
def test_http_error_does_not_log_token_or_response(self):
token = 'pk.secret.signature'
self.engine.mapbox_token = token
self.engine.mapbox_host = 'https://api.mapbox.com'
self.load_route()
response = requests.Response()
response.status_code = 401
response.url = 'https://api.mapbox.com/directions?access_token=' + token
response._content = ('Invalid token: ' + token).encode()
with patch('requests.get', return_value=response), patch('openpilot.selfdrive.navigation.navd.cloudlog') as logger:
self.engine.calculate_route((0.002, 0))
logger.event.assert_called_once_with('navd.route_request_failed', status_code=401, error_type='HTTPError', error=True)
logger.exception.assert_not_called()
self.assertNotIn(token, str(logger.mock_calls))
self.assertNotIn(response.url, str(logger.mock_calls))
self.assertIsNone(self.engine.route)
def test_banner_selection(self):
banners = [{'distanceAlongGeometry': 100, 'primary': {'text': 'Turn right', 'type': 'turn', 'modifier': 'right'}},
{'distanceAlongGeometry': 50, 'primary': {'text': 'Turn right'}, 'secondary': {'text': 'onto Main St'}}]
@@ -0,0 +1,235 @@
"""Actual public tile client requests and cache behavior, without accounts/network."""
import ast
import io
import importlib.util
import json
from pathlib import Path
import re
import sys
import tempfile
from types import SimpleNamespace as NS
import unittest
from unittest.mock import patch
import requests
from requests.adapters import BaseAdapter
from requests.structures import CaseInsensitiveDict
SOURCE = Path(__file__).resolve().parents[1]
REAL_SESSION = requests.Session
class PublicServer(BaseAdapter):
def __init__(self):
self.archive = b'PMTiles\x03' + bytes(range(256)) * 64
self.entries = [{'key': '20261103.pmtiles', 'version': '4.15.2', 'size': len(self.archive)}]
self.etag = '"public-build"'
self.calls = []
self.offline = False
self.index_bytes = None
self.index_headers = {}
self.range_headers = {}
self.range_status = 206
self.range_body = None
def send(self, request, **_kwargs):
self.calls.append(request)
if self.offline:
raise requests.ConnectionError('fixture network unavailable')
response = requests.Response()
response.request, response.url = request, request.url
if request.url == 'https://build-metadata.protomaps.dev/builds.json':
body = self.index_bytes if self.index_bytes is not None else json.dumps(self.entries).encode()
response.status_code = 200
headers = {'Content-Length': str(len(body))} | self.index_headers
else:
if not re.fullmatch(r'https://build\.protomaps\.com/\d{8}\.pmtiles', request.url):
raise AssertionError('Client must never contact an account or arbitrary endpoint')
match = re.fullmatch(r'bytes=(\d+)-(\d+)', request.headers['Range'])
start, end = map(int, match.groups())
body = self.archive[start:end + 1] if self.range_body is None else self.range_body
response.status_code = self.range_status
headers = {'Content-Range': f'bytes {start}-{end}/{len(self.archive)}', 'ETag': self.etag,
'Content-Length': str(len(body))} | self.range_headers
response.headers = CaseInsensitiveDict(headers)
response.raw = io.BytesIO(body)
return response
def close(self):
pass
class PublicTilesTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.home = Path(self.temp.name)
dependencies = {'openpilot.common.hardware': NS(PC=True),
'openpilot.common.hardware.hw': NS(Paths=NS(comma_home=lambda: self.temp.name)),
'openpilot.selfdrive.navigation.pmtiles': NS(HEADER_LENGTH=127)}
spec = importlib.util.spec_from_file_location('actual_public_tiles', SOURCE / 'tiles.py')
self.tiles = importlib.util.module_from_spec(spec)
with patch.dict(sys.modules, dependencies):
spec.loader.exec_module(self.tiles)
self.server = PublicServer()
def session():
value = REAL_SESSION()
value.trust_env = False
value.mount('https://', self.server)
return value
self.enterContext(patch.object(self.tiles.requests, 'Session', side_effect=session))
def client(self):
value = self.tiles.RemoteTiles()
self.addCleanup(value.close)
return value
def ranges(self):
return [call for call in self.server.calls if 'Range' in call.headers]
def test_latest_compatible_public_build_has_no_account_or_token_dependency(self):
self.server.entries += [{'key': '20261104.pmtiles', 'version': '5.0.0', 'size': len(self.server.archive)},
{'key': '20261102.pmtiles', 'version': '4.99.1', 'size': len(self.server.archive)}]
client = self.client()
self.assertEqual(client.url, 'https://build.protomaps.com/20261103.pmtiles')
self.assertEqual(client.read(300, 100), self.server.archive[300:400])
self.assertEqual(self.ranges()[-1].headers['If-Match'], self.server.etag)
for call in self.server.calls:
self.assertNotIn('Authorization', call.headers)
self.assertNotIn('access_token', call.url)
self.assertNotIn('comma', call.url)
imported = [ast.unparse(node) for node in ast.parse((SOURCE / 'tiles.py').read_text()).body if isinstance(node, (ast.Import, ast.ImportFrom))]
self.assertFalse(any('api' in value.lower() or 'params' in value.lower() for value in imported))
def test_invalid_index_or_unsupported_schema_fails_before_archive_request(self):
invalid = ({'url': 'https://untrusted.invalid/archive'}, [],
[{'key': '../20261103.pmtiles', 'version': '4.15.2', 'size': 200}],
[{'key': '20260230.pmtiles', 'version': '4.15.2', 'size': 200}],
[{'key': '20261103.pmtiles', 'version': '5.0.0', 'size': 200}],
[{'key': '20261103.pmtiles', 'version': '4.15.2', 'size': True}])
for value in invalid:
with self.subTest(value=value):
self.server.entries = value
with self.assertRaises(OSError):
self.client()
self.assertEqual(self.ranges(), [])
self.server.index_bytes = b'not JSON'
with self.assertRaises(OSError):
self.client()
def test_index_body_and_declared_length_are_bounded(self):
self.server.index_headers = {'Content-Length': str(self.tiles.INDEX_LIMIT + 1)}
with self.assertRaises(OSError):
self.client()
self.server.index_headers = {}
self.server.index_bytes = b' ' * (self.tiles.INDEX_LIMIT + 1)
with self.assertRaises(OSError):
self.client()
self.assertEqual(self.ranges(), [])
def test_range_status_identity_length_and_content_range_fail_closed(self):
cases = [({'status': 200}), ({'headers': {'Content-Range': 'bytes 1-127/999'}}),
({'headers': {'ETag': ''}}), ({'headers': {'ETag': 'W/"weak"'}}),
({'headers': {'Content-Length': '99999999999999'}}),
({'headers': {'Content-Encoding': 'gzip'}}),
({'body': b'x' * 126, 'headers': {'Content-Length': '127'}}),
({'body': b'x' * 128, 'headers': {'Content-Length': '127'}})]
for case in cases:
with self.subTest(case=case):
self.server.range_status = case.get('status', 206)
self.server.range_headers = case.get('headers', {})
self.server.range_body = case.get('body')
with self.assertRaises(OSError):
self.client()
def test_archive_cannot_change_etag_or_total_size_mid_reader(self):
client = self.client()
self.server.etag = '"changed"'
with self.assertRaisesRegex(OSError, 'changed'):
client.read(200, 100)
self.assertFalse((client.cache / '200-100').exists())
self.server.etag = client.etag
self.server.range_headers = {'Content-Range': f'bytes 200-299/{client.size + 1}'}
with self.assertRaises(OSError):
client.read(200, 100)
def test_restart_revalidates_header_and_reuses_only_same_archive_ranges(self):
first = self.client()
first.read(200, 100)
count = len(self.server.calls)
second = self.client()
self.assertEqual(second.cache, first.cache)
self.assertEqual(second.read(200, 100), self.server.archive[200:300])
self.assertEqual(len(self.server.calls), count + 2) # build index and pinned header only
self.server.entries[0]['key'] = '20261104.pmtiles'
third = self.client()
self.assertNotEqual(third.cache, first.cache)
count = len(self.server.calls)
third.read(200, 100)
self.assertEqual(len(self.server.calls), count + 1)
def test_offline_restart_reuses_pinned_header_and_cached_ranges_only(self):
first = self.client()
first.read(200, 100)
self.server.offline = True
second = self.client()
self.assertEqual(second.url, first.url)
self.assertEqual(second.etag, first.etag)
self.assertEqual(second.read(200, 100), self.server.archive[200:300])
with self.assertRaises(OSError):
second.read(500, 100)
def test_malformed_fresh_index_does_not_fall_back_or_relabel_old_bytes(self):
first = self.client()
first.read(200, 100)
self.server.entries = []
with self.assertRaises(OSError):
self.client()
self.assertEqual(first.read(200, 100), self.server.archive[200:300])
def test_prefetch_merges_nearby_ranges_but_caps_every_request(self):
client = self.client()
with patch.object(self.tiles, 'MAX_RANGE', 64):
count = len(self.ranges())
client.prefetch([(200, 32), (244, 32), (400, 16), (400, 16)])
added = self.ranges()[count:]
self.assertEqual(len(added), 3)
for call in added:
start, end = map(int, call.headers['Range'][6:].split('-'))
self.assertLessEqual(end - start + 1, 64)
count = len(self.server.calls)
self.assertEqual(client.read(244, 32), self.server.archive[244:276])
self.assertEqual(len(self.server.calls), count)
count = len(self.ranges())
client.prefetch([(600, 10), (620, 10)])
self.assertEqual(len(self.ranges()), count + 1)
def test_cache_limit_counts_ranges_from_all_archive_identities(self):
client = self.client()
with patch.object(self.tiles, 'CACHE_LIMIT', 160):
client.read(200, 100)
total = sum(path.stat().st_size for path in client.base.rglob('*') if path.is_file() and re.fullmatch(r'\d+-\d+', path.name))
self.assertLessEqual(total, 160)
def test_cache_trim_retains_current_header_for_offline_restart(self):
first = self.client()
with patch.object(self.tiles, 'CACHE_LIMIT', 228):
first.read(200, 100)
first.read(400, 100)
self.assertTrue((first.cache / '0-127').exists())
self.assertFalse((first.cache / '200-100').exists())
self.server.offline = True
second = self.client()
self.assertEqual(second.read(400, 100), self.server.archive[400:500])
def test_invalid_or_excessive_ranges_do_not_make_requests(self):
client = self.client()
count = len(self.server.calls)
for offset, length in ((-1, 1), (0, 0), (0, self.tiles.MAX_RANGE + 1), (client.size, 1), (True, 1)):
with self.subTest(offset=offset, length=length), self.assertRaises(ValueError):
client.read(offset, length)
self.assertEqual(len(self.server.calls), count)
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,250 @@
"""Real route ownership/generation lifecycle; no HTTP or vehicle execution."""
from pathlib import Path
import ast
import importlib.util
from types import SimpleNamespace
import unittest
from unittest.mock import Mock, patch
import numpy as np
from test_navd_next_turn import Executor, Params, SM, load_navd, target
class SignalMiniRouteTests(unittest.TestCase):
def setUp(self):
self.navd, self.sm, self.params, self.pm, self.executor = load_navd(), SM(), Params(), Mock(), Executor()
self.now, self.wall = 100., 1_000_000.
self.token = patch.dict(self.navd.RouteEngine.__init__.__globals__, get_mapbox_token=Mock(return_value='pk.fixture.signature'))
self.token.start()
self.addCleanup(self.token.stop)
self.engine = self.navd.RouteEngine(self.sm, self.pm, self.params, update_frequency=10)
self.engine.signal_turn = self.navd.SignalTurnEngine(self.params, Mock(return_value=SimpleNamespace(discover=Mock())), self.executor,
clock=lambda: self.now, wall_clock=lambda: self.wall)
def fresh(self, longitude=0., latitude=0., heading=0.):
self.sm.gps.longitude, self.sm.gps.latitude, self.sm.gps.bearingDeg = longitude, latitude, heading
self.sm.gps.unixTimestampMillis = (self.wall-.001)*1000.
self.sm.logMonoTime.update({name: int(self.now*1e9)-1000 for name in self.sm.logMonoTime})
self.sm.recv_time.update(dict.fromkeys(self.sm.recv_time, self.now))
self.params.values['NavModelState'] = {'updatedAt': self.now, 'supported': True}
def ready(self):
self.engine.update()
self.executor.jobs[-1][0].set_result(target())
self.engine.update()
self.assertEqual(self.engine.route_owner, 'signal')
self.assertEqual(self.params.values['MapboxTurnContext']['status'], 'ready')
def enter_and_cancel(self):
self.params.values.pop('MapboxTurnIntent', None)
self.sm.car.leftBlinker = self.sm.car.rightBlinker = False
self.fresh(longitude=.000025, latitude=.001, heading=90.)
self.engine.update()
def test_unsupported_model_never_discovers_or_installs(self):
self.params.values['NavModelState']['supported'] = False
self.engine.update()
self.assertEqual(self.params.values['MapboxTurnContext']['reason'], 'navigation_model_required')
self.assertEqual(self.executor.jobs, [])
self.assertIsNone(self.engine.route)
def test_delayed_context_write_cannot_resurrect_retired_route(self):
pending, persist = [], self.params.put
def deferred(key, value, block=False):
if block:
persist(key, value, block=True)
else:
pending.append((key, value))
self.params.put = deferred
engine = self.engine.signal_turn
engine.key = ('one', 'right')
engine._publish(self.now, 'ready', routeOwner='signal', routeGeneration=123, retainedAfterSignalCancel=True)
engine._clear()
for key, value in pending:
self.params.values[key] = value
self.assertNotIn('MapboxTurnContext', self.params.values)
def test_coherent_route_arrival_all_maneuvers_and_publish_order(self):
self.ready()
self.assertEqual(self.engine.route[-1]['maneuver']['type'], 'arrive')
self.assertNotIn('NavDestination', self.params.values)
calls = self.pm.send.call_args_list
route_index = max(i for i, call in enumerate(calls) if call.args[0] == 'navRoute')
instruction_index = max(i for i, call in enumerate(calls) if call.args[0] == 'navInstruction')
self.assertLess(route_index, instruction_index)
instruction = calls[instruction_index].args[1].navInstruction
self.assertTrue(any(item['type'] == 'turn' and item['modifier'] == 'right' for item in instruction.allManeuvers))
self.assertEqual(self.params.values['MapboxTurnContext']['routeGeneration'], self.engine.route_generation)
self.assertFalse(any(key == 'MapboxTurnContext' and value.get('status') == 'route_pending'
for key, value, _block in self.params.writes))
def test_pending_route_can_prepare_without_authorizing_navigation_inputs(self):
self.params.values['MapboxTurnIntent']['mode'] = 'pending'
self.ready()
path = Path(__file__).resolve().parents[3]/'sunnypilot/modeld_v2/navigation_input.py'
spec = importlib.util.spec_from_file_location('mini_route_model_guard', path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
guard = module.SignalRouteGuard()
arguments = {'now': self.now, 'enabled': True, 'lateral_active': True, 'left_blinker': False,
'right_blinker': True, 'intent': self.params.values['MapboxTurnIntent'],
'context': self.params.values['MapboxTurnContext']}
self.assertFalse(guard.check(**arguments)['ready'])
self.params.values['MapboxTurnIntent']['mode'] = 'turn'
self.assertTrue(guard.check(**arguments)['ready'])
def test_lane_change_keeps_prefetch_generation_for_later_turn(self):
self.params.values['MapboxTurnIntent']['mode'] = 'lane_change'
self.ready()
generation = self.engine.route_generation
session = self.params.values['MapboxTurnIntent']['session']
self.assertEqual(len(self.executor.jobs), 1)
self.params.values['MapboxTurnIntent']['mode'] = 'pending'
self.engine.update()
self.params.values['MapboxTurnIntent']['mode'] = 'turn'
self.engine.update()
self.assertEqual(self.engine.route_generation, generation)
self.assertEqual(self.params.values['MapboxTurnContext']['session'], session)
self.assertEqual(len(self.executor.jobs), 1, 'Completing LC must not force another routing request')
def test_lane_change_only_route_is_cleared_on_cancel_even_at_junction(self):
self.params.values['MapboxTurnIntent']['mode'] = 'lane_change'
self.ready()
self.enter_and_cancel()
self.assertIsNone(self.engine.route)
self.assertNotIn('MapboxTurnContext', self.params.values)
def test_cancel_before_entry_clears_route_and_context(self):
self.ready()
self.params.values.pop('MapboxTurnIntent')
self.sm.car.rightBlinker = False
self.engine.update()
self.assertIsNone(self.engine.route)
self.assertNotIn('MapboxTurnContext', self.params.values)
def test_same_tick_first_entry_and_auto_cancel_retains_exact_generation(self):
self.ready()
generation = self.engine.route_generation
self.assertFalse(self.engine.signal_turn.entered)
self.enter_and_cancel()
context = self.params.values['MapboxTurnContext']
self.assertEqual(context['status'], 'ready')
self.assertTrue(context['retainedAfterSignalCancel'])
self.assertEqual(context['routeGeneration'], generation)
self.assertEqual(len(self.executor.jobs), 1)
def test_receiving_road_completion_retires_without_retarget(self):
self.ready()
self.enter_and_cancel()
self.fresh(longitude=.00015, latitude=.001, heading=90.)
self.engine.update()
self.assertEqual(self.params.values['MapboxTurnContext']['status'], 'passed')
self.assertIsNone(self.engine.route)
self.engine.update()
self.assertEqual(len(self.executor.jobs), 1)
def test_missed_junction_retires_instead_of_rerouting_to_endpoint(self):
self.ready()
self.fresh(latitude=.00115, heading=0.)
self.engine.update()
self.assertEqual(self.params.values['MapboxTurnContext']['status'], 'passed')
self.assertIsNone(self.engine.route)
self.engine.update()
self.assertEqual(len(self.executor.jobs), 1)
def test_transient_gps_loss_retains_geometry_generation_but_invalidates_inputs(self):
self.ready()
self.enter_and_cancel()
generation = self.engine.route_generation
self.now += .6
self.wall += .6
self.params.values['NavModelState']['updatedAt'] = self.now
self.sm.logMonoTime['carControl'] = int(self.now*1e9)-1000
self.engine.update()
self.assertEqual(self.params.values['MapboxTurnContext']['status'], 'unavailable')
self.assertEqual(self.engine.route_generation, generation)
self.assertIsNotNone(self.engine.route)
self.fresh(longitude=.00003, latitude=.001, heading=90.)
self.engine.update()
self.assertEqual(self.params.values['MapboxTurnContext']['status'], 'ready')
self.assertEqual(self.engine.route_generation, generation)
def test_manual_destination_switch_cannot_label_old_route_as_new_destination(self):
a, b = {'longitude': .002, 'latitude': 0.}, {'longitude': .003, 'latitude': 0.}
self.params.values['NavDestination'] = a
self.engine.install_route(target().steps, (.002, 0.))
self.engine.send_route()
self.engine.recompute_route = Mock()
self.engine.update()
self.assertEqual(self.params.values['MapboxTurnContext']['destination'], a)
self.params.values['NavDestination'] = b
self.engine.update() # not a1Hz legacy recompute tick
context = self.params.values['MapboxTurnContext']
self.assertEqual(context['status'], 'searching')
self.assertIsNone(self.engine.route)
self.assertEqual(self.params.values['NavDestination'], b)
def test_signal_route_resends_at_one_hz_without_changing_generation(self):
self.ready()
generation = self.engine.route_generation
before = sum(call.args[0] == 'navRoute' for call in self.pm.send.call_args_list)
self.sm.frame = 10
self.engine.update()
routes = [call.args[1] for call in self.pm.send.call_args_list if call.args[0] == 'navRoute']
self.assertEqual(len(routes), before+1)
self.assertEqual(routes[-1].logMonoTime, generation)
def test_manual_destination_cancel_removes_its_ready_generation_context(self):
self.params.values['NavDestination'] = {'longitude': .002, 'latitude': 0.}
self.engine.install_route(target().steps, (.002, 0.))
self.engine.send_route()
self.engine.recompute_route = Mock()
self.engine.update()
self.params.values.pop('NavDestination')
self.params.values.pop('MapboxTurnIntent')
self.sm.car.rightBlinker = False
self.engine.update()
self.assertIsNone(self.engine.route)
self.assertNotIn('MapboxTurnContext', self.params.values)
def test_malformed_route_is_contained_without_exception_text_logging(self):
self.engine._update_route = Mock(side_effect=ValueError('secret-url-must-not-be-logged'))
self.engine.update()
self.assertEqual(self.params.values['MapboxTurnContext']['reason'], 'route_processing_failed')
text = str(self.navd.cloudlog.event.call_args_list)
self.assertNotIn('secret-url', text)
class MapGenerationTests(unittest.TestCase):
def test_map_eof_source_is_route_generation_not_render_completion(self):
path = Path(__file__).resolve().parents[1]/'mapsd.py'
tree = ast.parse(path.read_text())
server_send = [node for node in ast.walk(tree) if isinstance(node, ast.Call) and
isinstance(node.func, ast.Attribute) and node.func.attr == 'send'][0]
self.assertEqual(ast.unparse(server_send.args[-1]), 'generation')
# Also execute the actual map class: an unchanging route resend may not
# relabel a render with wall-clock completion time.
namespace = {'np': np, 'time': SimpleNamespace(monotonic=lambda: 100.), 'RETRY_SECONDS': 10,
'RouteTracker': Mock(), 'latest_gps_source': lambda sm: 'gps', 'MapRenderer': Mock(), 'PMTiles': Mock(),
'RemoteTiles': Mock(), 'SIZE': 256, 'cloudlog': Mock()}
cls = next(node for node in tree.body if isinstance(node, ast.ClassDef) and node.name == 'MapsD')
exec(compile(ast.Module(body=[cls], type_ignores=[]), str(path), 'exec'), namespace)
maps = namespace['MapsD']()
maps.renderer = Mock()
maps.renderer.render.return_value = np.zeros((256, 256), np.uint8), True
sm = SimpleNamespace(updated={'navRoute': True}, logMonoTime={'navRoute': 123, 'gps': 99_900_000_000})
values = {'navRoute': SimpleNamespace(coordinates=[SimpleNamespace(longitude=0., latitude=0.),
SimpleNamespace(longitude=0., latitude=.001)]), 'gps': SimpleNamespace(longitude=0., latitude=0.)}
class Messages:
updated, logMonoTime = sm.updated, sm.logMonoTime
def __getitem__(self, key):
return values[key]
maps.update(Messages())
self.assertEqual(maps.route_generation, 123)
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,216 @@
"""Display-path tests without a window/GPU; execute the actual widget methods."""
import ast
from pathlib import Path
from types import SimpleNamespace
import unittest
from unittest.mock import Mock
import numpy as np
WORKTREE = Path(__file__).resolve().parents[4]
UI = WORKTREE/'openpilot/selfdrive/ui/mici/onroad'
class Label:
def __init__(self, *args, **kwargs):
self.text = ''
def set_text(self, text):
self.text = text
def render(self, rect):
pass
def load_widget(state):
def rectangle(x, y, width, height):
return SimpleNamespace(x=x, y=y, width=width, height=height)
rl = SimpleNamespace(Color=lambda *args: args, WHITE='white', BLACK='black', Rectangle=rectangle,
Vector2=lambda x, y: (x, y), draw_rectangle_rec=Mock(), draw_line_ex=Mock(), draw_circle_v=Mock())
namespace = {'__name__': 'navigation_ui_test', 'np': np, 'rl': rl, 'time': SimpleNamespace(monotonic=lambda: 100.),
'Widget': object, 'UnifiedLabel': Label, 'gui_label': Mock(), 'ui_state': state,
'VisionIpcClient': Mock(), 'VisionStreamType': SimpleNamespace(VISION_STREAM_MAP=1),
'FontWeight': SimpleNamespace(BOLD=1, MEDIUM=2), 'TextAlignmentVertical': SimpleNamespace(TOP=1),
'ROUTE_INTENSITY': 128, 'SIZE': 256, 'latest_gps_source': lambda sm: 'gps',
'CONNECTION_RETRY_INTERVAL': 1., 'EGO': [128, 192], 'CV': SimpleNamespace()}
tree = ast.parse((UI/'navigation.py').read_text())
tree.body = [node for node in tree.body if not isinstance(node, (ast.Import, ast.ImportFrom))]
exec(compile(tree, str(UI/'navigation.py'), 'exec'), namespace)
return SimpleNamespace(**namespace)
class SM:
def __init__(self):
self.valid, self.alive = {'navInstruction': True}, {'navInstruction': True}
self.car = SimpleNamespace(leftBlinker=False, rightBlinker=True)
self.instruction = SimpleNamespace(maneuverModifier='right', maneuverDistance=8., timeRemaining=20., distanceRemaining=100.)
def __getitem__(self, key):
return self.car if key == 'carState' else self.instruction
def state(status='unavailable', destination=False):
values = {'MapboxNextTurnEnabled': True, 'MapboxTurnIntent': {'session': 'one', 'direction': 'right'},
'MapboxTurnContext': {'session': 'one', 'direction': 'right', 'status': status, 'updatedAt': 100.,
'reason': 'explicit_destination' if status == 'blocked' else 'gps_unavailable',
'locationValid': status == 'ready', 'headingValid': status == 'ready',
'distanceM': 8., 'name': 'Example Street', 'routeGeneration': 10}}
params = SimpleNamespace(get=values.get, get_bool=lambda key: values.get(key) is True)
return SimpleNamespace(sm=SM(), params=params, nav_destination_set=destination, navigation_enabled=True, is_metric=True), values
class TestSignalTurnStatus(unittest.TestCase):
def test_all_pending_states_are_visible_and_ready_is_separate(self):
for status, text in [('searching', 'Finding next right'), ('unavailable', 'Map turn unavailable'),
('passed', 'Turn passed'), ('blocked', 'Signal turn blocked')]:
ui_state, _ = state(status)
module = load_widget(ui_state)
widget = module.NavigationPanel()
self.assertTrue(widget.signal_turn_active)
self.assertEqual(widget.signal_status['primary'], text)
widget._render(module.rl.Rectangle(0, 0, 261, 400))
module.rl.draw_line_ex.assert_not_called()
self.assertTrue(any(call.args[1] == text for call in module.gui_label.call_args_list))
def test_ready_has_direction_arrow_and_explicit_signal_label(self):
ui_state, _ = state('ready')
module = load_widget(ui_state)
widget = module.NavigationPanel()
widget._render(module.rl.Rectangle(0, 0, 261, 400))
self.assertGreater(module.rl.draw_line_ex.call_count, 0)
self.assertIn('Route ready', widget._eta_label.text)
def test_route_preview_is_not_navigation_active_without_fresh_model_consumption(self):
ui_state, values = state('ready')
for value in (None, {'updatedAt': 99., 'supported': True, 'active': True, 'session': 'one'},
{'updatedAt': 100., 'supported': True, 'active': True, 'session': 'old'},
{'updatedAt': 100., 'supported': False, 'active': False, 'session': 'one'}):
values['NavModelState'] = value
widget = load_widget(ui_state).NavigationPanel()
self.assertEqual(widget.signal_status['primary'], 'Route ready')
values['NavModelState'] = {'updatedAt': 100., 'supported': True, 'active': True, 'session': 'one', 'routeGeneration': 10}
widget = load_widget(ui_state).NavigationPanel()
self.assertEqual(widget.signal_status['primary'], 'Navigation active')
def test_ready_route_explains_fresh_matched_model_map_wait(self):
ui_state, values = state('ready')
values['NavModelState'] = {'updatedAt': 100., 'supported': True, 'active': False,
'session': 'one', 'routeGeneration': 10, 'reason': 'map_missing'}
module = load_widget(ui_state)
widget = module.NavigationPanel()
self.assertEqual(widget.signal_status['primary'], 'Route ready')
self.assertEqual(widget.signal_status['secondary'], 'Waiting for model map')
widget._render(module.rl.Rectangle(0, 0, 261, 400))
self.assertIn('Waiting for model map', widget._eta_label.text)
def test_fresh_matched_wait_reasons_are_specific_without_claiming_active(self):
expected = {'map_stale': 'Model map is stale', 'map_invalid': 'Model map is invalid',
'instruction_stale': 'Route instructions are stale', 'route_generation_mismatch': 'Waiting for current route map',
'disabled': 'Navigation inputs disabled', 'invalid_time_or_speed': 'Vehicle data unavailable'}
for reason, text in expected.items():
with self.subTest(reason=reason):
ui_state, values = state('ready')
values['NavModelState'] = {'updatedAt': 100., 'supported': True, 'active': False,
'session': 'one', 'routeGeneration': 10, 'reason': reason}
result = load_widget(ui_state).NavigationPanel().signal_status
self.assertEqual(result['primary'], 'Route ready')
self.assertEqual(result['secondary'], text)
def test_stale_wrong_route_or_unsupported_model_reason_is_not_reused(self):
for change in ({'updatedAt': 99.}, {'updatedAt': 101.}, {'updatedAt': float('nan')}, {'updatedAt': True},
{'session': 'other'}, {'routeGeneration': 11}, {'supported': False}, {'active': 0}):
with self.subTest(change=change):
ui_state, values = state('ready')
values['NavModelState'] = {'updatedAt': 100., 'supported': True, 'active': False,
'session': 'one', 'routeGeneration': 10, 'reason': 'map_invalid', **change}
result = load_widget(ui_state).NavigationPanel().signal_status
self.assertEqual(result['primary'], 'Route ready')
self.assertEqual(result['secondary'], 'Waiting for map and model')
def test_active_and_pending_route_labels_ignore_old_wait_reason(self):
for mode, active, primary in (('turn', True, 'Navigation active'), ('pending', False, 'Turn candidate')):
ui_state, values = state('ready')
values['MapboxTurnIntent']['mode'] = mode
values['NavModelState'] = {'updatedAt': 100., 'supported': True, 'active': active,
'session': 'one', 'routeGeneration': 10, 'reason': 'map_missing'}
result = load_widget(ui_state).NavigationPanel().signal_status
self.assertEqual(result['primary'], primary)
self.assertEqual(result['secondary'], 'Right onto Example Street')
def test_unknown_or_malformed_wait_reason_keeps_generic_message(self):
for reason in (None, 'unknown_future_reason', [], {}):
ui_state, values = state('ready')
values['NavModelState'] = {'updatedAt': 100., 'supported': True, 'active': False,
'session': 'one', 'routeGeneration': 10, 'reason': reason}
self.assertEqual(load_widget(ui_state).NavigationPanel().signal_status['secondary'], 'Waiting for map and model')
def test_explicit_destination_keeps_normal_arrow_and_shows_blocked_footer(self):
ui_state, _ = state('blocked', destination=True)
module = load_widget(ui_state)
widget = module.NavigationPanel()
widget._render(module.rl.Rectangle(0, 0, 261, 400))
self.assertGreater(module.rl.draw_line_ex.call_count, 0)
self.assertIn('Destination navigation active', widget._eta_label.text)
def test_pending_candidate_does_not_claim_navigation_is_active(self):
ui_state, values = state('ready')
values['MapboxTurnIntent']['mode'] = 'pending'
self.assertEqual(load_widget(ui_state).NavigationPanel().signal_status['primary'], 'Turn candidate')
def test_lane_change_holds_prepared_route_without_claiming_navigation(self):
ui_state, values = state('ready')
values['MapboxTurnIntent']['mode'] = 'lane_change'
status = load_widget(ui_state).NavigationPanel().signal_status
self.assertEqual(status['primary'], 'Lane change')
self.assertEqual(status['state'], 'lane_change')
def test_actual_destination_owned_context_has_priority_without_a_signal_session(self):
ui_state, values = state('ready', destination=True)
values['MapboxTurnContext'] = {'status': 'ready', 'updatedAt': 100., 'routeOwner': 'destination', 'routeGeneration': 20}
widget = load_widget(ui_state).NavigationPanel()
self.assertEqual(widget.signal_status['secondary'], 'Destination navigation active')
def test_retained_exit_guidance_remains_visible_after_auto_cancel(self):
ui_state, values = state('ready')
ui_state.sm.car.rightBlinker = False
values['MapboxTurnIntent'] = None
values['MapboxTurnContext']['retainedAfterSignalCancel'] = True
widget = load_widget(ui_state).NavigationPanel()
self.assertTrue(widget.signal_turn_active)
ui_state.sm.car.leftBlinker = True
self.assertFalse(load_widget(ui_state).NavigationPanel().signal_turn_active)
def test_cancel_disable_hazards_stale_or_wrong_session_does_not_claim_ready(self):
for update in ('off', 'cancel', 'hazards', 'stale', 'different_session'):
ui_state, values = state('ready')
if update == 'off':
values['MapboxNextTurnEnabled'] = False
elif update == 'cancel':
ui_state.sm.car.rightBlinker = False
elif update == 'hazards':
ui_state.sm.car.leftBlinker = True
elif update == 'stale':
values['MapboxTurnContext']['updatedAt'] = 99.
else:
values['MapboxTurnContext']['session'] = 'old'
widget = load_widget(ui_state).NavigationPanel()
if update in ('off', 'cancel', 'hazards'):
self.assertFalse(widget.signal_turn_active)
else:
self.assertEqual(widget.signal_status['state'], 'unavailable')
def test_augmented_view_opens_for_signal_without_destination_and_yields_to_alert(self):
tree = ast.parse((UI/'augmented_road_view.py').read_text())
assignment = next(node for node in ast.walk(tree) if isinstance(node, ast.Assign) and
any(isinstance(target, ast.Name) and target.id == 'show_navigation' for target in node.targets))
code = compile(ast.Expression(assignment.value), 'actual_navigation_visibility', 'eval')
for enabled, destination, signal, alert, expected in [(True, False, True, None, True),
(True, False, False, None, False), (False, False, True, None, False),
(True, True, False, None, True), (True, False, True, object(), False)]:
result = eval(code, {'ui_state': SimpleNamespace(navigation_enabled=enabled, nav_destination_set=destination),
'self': SimpleNamespace(_navigation=SimpleNamespace(signal_turn_active=signal)), 'alert_to_render': alert})
self.assertEqual(result, expected)
if __name__ == '__main__':
unittest.main()
+160 -59
View File
@@ -1,114 +1,215 @@
"""Byte ranges of the hosted PMTiles basemap, cached on disk."""
"""Bounded byte ranges of a public Protomaps v4 basemap; no device account.
Experimental single-device transport. Public daily-build hosting is not a fleet
backend: deployments should mirror the archive while preserving its identity.
"""
import hashlib
import json
import re
import time
from datetime import date
from pathlib import Path
import requests
from openpilot.common.api import Api
from openpilot.common.hardware import PC
from openpilot.common.hardware.hw import Paths
from openpilot.common.params import Params
from openpilot.selfdrive.navigation.pmtiles import HEADER_LENGTH
BUILD_INDEX_URL = 'https://build-metadata.protomaps.dev/builds.json'
BUILD_BASE_URL = 'https://build.protomaps.com/'
INDEX_LIMIT = 512 * 1024
MAX_RANGE = 4 * 1024 * 1024
CACHE_LIMIT = 64 * 1024 * 1024
MERGE_GAP = 256 * 1024 # tiles this close together in the archive are fetched in one range request
MERGE_GAP = 256 * 1024
TIMEOUT = (5, 10)
def _build(entry):
"""Only supported official dated archives, never a URL supplied by metadata."""
if not isinstance(entry, dict):
return None
key, size, version = (entry.get(name) for name in ('key', 'size', 'version'))
if (not isinstance(key, str) or not re.fullmatch(r'\d{8}\.pmtiles', key) or
isinstance(size, bool) or not isinstance(size, int) or not HEADER_LENGTH <= size <= 1024 ** 4 or
not isinstance(version, str) or not re.fullmatch(r'4\.\d+\.\d+', version)):
return None
try:
date(int(key[:4]), int(key[4:6]), int(key[6:8]))
except ValueError:
return None
return {'key': key, 'size': size, 'version': version, 'url': BUILD_BASE_URL + key}
def _strong_etag(value):
return isinstance(value, str) and re.fullmatch(r'"[\x21\x23-\x7e]{1,510}"', value) is not None
class RemoteTiles:
def __init__(self):
self.cache = Path(Paths.comma_home() if PC else '/data') / 'navigation' / 'tiles'
self.cache.mkdir(parents=True, exist_ok=True, mode=0o700)
self.base = Path(Paths.comma_home() if PC else '/data') / 'navigation' / 'tiles'
self.base.mkdir(parents=True, exist_ok=True, mode=0o700)
self.session = requests.Session()
self.api = Api(Params().get('DongleId'))
self.url = ''
self.expires = 0
self.etag = None
self.size = 0
version = self.cache / 'version'
self.url, self.etag, self.size = '', None, 0
self.cache = None
saved = self._saved_identity()
try:
header = self._fetch(0, HEADER_LENGTH) # learns the archive's ETag and size
if not version.exists() or version.read_text() != f'{self.etag} {self.size}':
for cached in self.cache.iterdir(): # a new archive has new offsets
cached.unlink()
version.write_text(f'{self.etag} {self.size}')
try:
build = self._latest_build()
self.url, self.size = build['url'], build['size']
header = self._fetch(0, HEADER_LENGTH)
identity = build | {'etag': self.etag}
except requests.RequestException:
# An offline reader may reuse only a previously pinned archive with its
# exact cached header. Missing ranges still fail; no route-only raster.
if saved is None:
raise
identity = saved
self.url, self.etag, self.size = identity['url'], identity['etag'], identity['size']
self.cache = self._cache_for(identity)
header = (self.cache / f'0-{HEADER_LENGTH}').read_bytes()
if len(header) != HEADER_LENGTH or header[:8] != b'PMTiles\x03':
raise ValueError('Invalid public PMTiles header')
self.cache = self._cache_for(identity)
self.cache.mkdir(parents=True, exist_ok=True)
self._store(0, header)
except Exception:
temporary = self.base / 'archive.tmp'
temporary.write_text(json.dumps(identity))
temporary.replace(self.base / 'archive.json')
self._trim()
except (requests.RequestException, OSError, ValueError, TypeError, KeyError):
self.close()
raise
raise OSError('Unable to load public Protomaps basemap') from None
def _cache_for(self, identity):
key = hashlib.sha256(json.dumps(identity, sort_keys=True).encode()).hexdigest()[:32]
return self.base / 'archives' / key
def _saved_identity(self):
try:
path = self.base / 'archive.json'
if path.stat().st_size > 4096:
return None
saved = json.loads(path.read_text())
build = _build(saved)
if build is not None and saved.get('url') == build['url'] and _strong_etag(saved.get('etag')):
return build | {'etag': saved['etag']}
except (OSError, ValueError, TypeError):
pass
return None
def _latest_build(self):
with self.session.get(BUILD_INDEX_URL, headers={'Accept-Encoding': 'identity'}, timeout=TIMEOUT, stream=True,
allow_redirects=False) as response:
response.raise_for_status()
if response.status_code != 200:
raise ValueError('Unexpected public build index response')
declared = response.headers.get('Content-Length')
if declared is not None and not 0 <= int(declared) <= INDEX_LIMIT:
raise ValueError('Oversized public build index')
data = bytearray()
for chunk in response.iter_content(chunk_size=65536):
if len(data) + len(chunk) > INDEX_LIMIT:
raise ValueError('Oversized public build index')
data.extend(chunk)
entries = json.loads(data)
if not isinstance(entries, list) or len(entries) > 4096:
raise ValueError('Invalid public build index')
builds = [build for entry in entries if (build := _build(entry)) is not None]
if not builds:
raise ValueError('No compatible public Protomaps v4 build')
return max(builds, key=lambda build: build['key'])
def _range(self, offset, length):
if (isinstance(offset, bool) or isinstance(length, bool) or not isinstance(offset, int) or not isinstance(length, int) or
offset < 0 or not 0 < length <= MAX_RANGE or offset + length > self.size):
raise ValueError('Invalid or oversized public map range')
def _fetch(self, offset: int, length: int) -> bytes:
try:
if time.time() >= self.expires - 60: # noqa: TID251 - SAS expiry is a Unix timestamp
response = self.api.get(f'v1/{self.api.dongle_id}/nav/tiles', access_token=self.api.get_token(), timeout=10)
response.raise_for_status()
metadata = response.json()
self.url, self.expires = metadata['url'], metadata['expires']
headers = {'Range': f'bytes={offset}-{offset + length - 1}', 'Accept-Encoding': 'identity'}
if self.etag is not None:
headers['If-Match'] = self.etag
with self.session.get(self.url, headers=headers, timeout=10, stream=True) as response:
if response.status_code != 206:
raise OSError(f'Map tile range request failed ({response.status_code})')
match = re.fullmatch(r'bytes (\d+)-(\d+)/(\d+)', response.headers.get('Content-Range', ''))
if match is None or tuple(map(int, match.groups()[:2])) != (offset, offset + length - 1):
raise OSError('Invalid map tile range response')
size = int(match[3])
etag = response.headers.get('ETag')
if not etag or (self.etag is not None and (etag != self.etag or size != self.size)):
raise OSError('Map archive changed during read')
data = bytearray()
for chunk in response.iter_content(chunk_size=min(length + 1, 65536)):
data.extend(chunk)
if len(data) > length:
raise OSError('Oversized map tile range response')
if len(data) != length:
raise OSError('Truncated map tile range response')
self.size, self.etag = size, etag
return bytes(data)
except (requests.RequestException, ValueError, KeyError, TypeError, RuntimeError):
# requests exceptions contain the signed URL; keep it out of logs.
raise OSError('Unable to fetch map tiles') from None
self._range(offset, length)
headers = {'Range': f'bytes={offset}-{offset + length - 1}', 'Accept-Encoding': 'identity'}
if self.etag is not None:
headers['If-Match'] = self.etag
with self.session.get(self.url, headers=headers, timeout=TIMEOUT, stream=True, allow_redirects=False) as response:
response.raise_for_status()
if response.status_code != 206:
raise OSError(f'Public map range request failed ({response.status_code})')
match = re.fullmatch(r'bytes (\d+)-(\d+)/(\d+)', response.headers.get('Content-Range', ''))
if match is None or tuple(map(int, match.groups())) != (offset, offset + length - 1, self.size):
raise OSError('Invalid public map range response')
etag = response.headers.get('ETag')
if not _strong_etag(etag) or (self.etag is not None and etag != self.etag):
raise OSError('Public map archive changed during read')
declared = response.headers.get('Content-Length')
if declared is not None and int(declared) != length:
raise OSError('Invalid public map range length')
if response.headers.get('Content-Encoding', 'identity') != 'identity':
raise OSError('Encoded public map range response')
data = bytearray()
for chunk in response.iter_content(chunk_size=min(length + 1, 65536)):
if len(data) + len(chunk) > length:
raise OSError('Oversized public map range response')
data.extend(chunk)
if len(data) != length:
raise OSError('Truncated public map range response')
self.etag = etag
return bytes(data)
def _store(self, offset: int, data: bytes):
(self.cache / f'{offset}-{len(data)}').write_bytes(data)
path = self.cache / f'{offset}-{len(data)}'
temporary = path.with_suffix('.tmp')
temporary.write_bytes(data)
temporary.replace(path)
def _trim(self):
files = [(cached.stat().st_mtime, cached.stat().st_size, cached) for cached in self.cache.iterdir() if re.fullmatch(r'\d+-\d+', cached.name)]
files = [(cached.stat().st_mtime, cached.stat().st_size, cached) for cached in self.base.rglob('*')
if cached.is_file() and re.fullmatch(r'\d+-\d+', cached.name)]
size = sum(cached_size for _, cached_size, _ in files)
for _, cached_size, cached in sorted(files):
if size <= CACHE_LIMIT:
break
if cached == self.cache / f'0-{HEADER_LENGTH}':
continue # Required for an offline restart of the current pinned archive.
cached.unlink(missing_ok=True)
size -= cached_size
def read(self, offset: int, length: int) -> bytes:
self._range(offset, length)
try:
data = (self.cache / f'{offset}-{length}').read_bytes()
if len(data) == length:
return data
except FileNotFoundError:
pass
data = self._fetch(offset, length)
try:
data = self._fetch(offset, length)
except (requests.RequestException, ValueError):
raise OSError('Unable to fetch public map tiles') from None
self._store(offset, data)
self._trim()
return data
def prefetch(self, ranges):
"""Cache (offset, length) ranges ahead of reads, fetching neighboring ones in a single request."""
missing = sorted((offset, length) for offset, length in set(ranges) if not (self.cache / f'{offset}-{length}').exists())
missing = []
for offset, length in sorted(set(ranges)):
self._range(offset, length)
if not (self.cache / f'{offset}-{length}').exists():
missing.append((offset, length))
if not missing:
return
groups = [[missing[0]]]
for offset, length in missing[1:]:
start = groups[-1][0][0]
last_offset, last_length = groups[-1][-1]
if offset - (last_offset + last_length) <= MERGE_GAP:
if offset - (last_offset + last_length) <= MERGE_GAP and offset + length - start <= MAX_RANGE:
groups[-1].append((offset, length))
else:
groups.append([(offset, length)])
for group in groups:
start, end = group[0][0], group[-1][0] + group[-1][1]
data = self._fetch(start, end - start)
start, end = group[0][0], max(offset + length for offset, length in group)
try:
data = self._fetch(start, end - start)
except (requests.RequestException, ValueError):
raise OSError('Unable to prefetch public map tiles') from None
for offset, length in group:
self._store(offset, data[offset - start:offset - start + length])
self._trim()
Binary file not shown.
@@ -221,7 +221,8 @@ class AugmentedRoadView(CameraView):
self._update_calibration()
alert_to_render, not_animating_out = self._alert_renderer.will_render()
show_navigation = ui_state.navigation_enabled and ui_state.nav_destination_set and alert_to_render is None
show_navigation = (ui_state.navigation_enabled and (ui_state.nav_destination_set or self._navigation.signal_turn_active)
and alert_to_render is None)
# Create inner content area with border padding
self._content_rect = rl.Rectangle(
@@ -1,5 +1,6 @@
import numpy as np
import pyray as rl
import time
from msgq.visionipc import VisionIpcClient
from openpilot.cereal.visionipc import VisionStreamType
@@ -22,6 +23,75 @@ MAP_COLORS = np.repeat(np.round(np.arange(256) * 0.45).astype(np.uint8)[:, None]
MAP_COLORS[ROUTE_INTENSITY - 1:ROUTE_INTENSITY + 2] = (72, 235, 132)
def signal_turn_status(enabled, intent, context, now, left_blinker, right_blinker, model_state=None):
"""Distinguish a route lookup from actual navigation inputs consumed by the model."""
direction = 'left' if left_blinker and not right_blinker else 'right' if right_blinker and not left_blinker else None
if (direction is None and not left_blinker and not right_blinker and isinstance(context, dict) and
context.get('retainedAfterSignalCancel') is True and context.get('status') == 'ready'):
# Auto-cancel during an entered turn does not end its short exit guidance.
direction = context.get('direction')
intent = {'session': context.get('session'), 'direction': direction}
if (enabled is not True or not isinstance(intent, dict) or not intent.get('session') or
direction not in ('left', 'right') or intent.get('direction') != direction):
return None
status, reason = 'unavailable', 'navigation_unavailable'
if isinstance(context, dict) and context.get('routeOwner') == 'destination':
return {'state': 'blocked', 'primary': 'Signal turn blocked', 'secondary': 'Destination navigation active'}
if intent.get('mode') == 'lane_change':
return {'state': 'lane_change', 'primary': 'Lane change', 'secondary': 'Street route on hold'}
if isinstance(context, dict) and context.get('session') == intent['session'] and context.get('direction') == direction:
updated = context.get('updatedAt')
if isinstance(updated, (int, float)) and not isinstance(updated, bool) and np.isfinite(updated) and 0. <= now-updated <= .35:
status, reason = context.get('status'), context.get('reason')
if status == 'ready' and context.get('locationValid') is True and context.get('headingValid') is True:
distance = context.get('distanceM')
if isinstance(distance, (int, float)) and not isinstance(distance, bool) and np.isfinite(distance):
model_updated = model_state.get('updatedAt') if isinstance(model_state, dict) else None
active = (isinstance(model_updated, (int, float)) and not isinstance(model_updated, bool) and
np.isfinite(model_updated) and 0. <= now-model_updated <= .5 and
model_state.get('supported') is True and model_state.get('active') is True and
model_state.get('session') == intent['session'] and
isinstance(context.get('routeGeneration'), int) and context['routeGeneration'] > 0 and
model_state.get('routeGeneration') == context['routeGeneration'])
waiting = 'Waiting for map and model'
matched_wait = (isinstance(model_updated, (int, float)) and not isinstance(model_updated, bool) and
np.isfinite(model_updated) and 0. <= now-model_updated <= .5 and
model_state.get('supported') is True and model_state.get('active') is False and
model_state.get('session') == intent['session'] and
isinstance(context.get('routeGeneration'), int) and not isinstance(context['routeGeneration'], bool) and
context['routeGeneration'] > 0 and model_state.get('routeGeneration') == context['routeGeneration'])
model_reason = model_state.get('reason') if matched_wait else None
if isinstance(model_reason, str):
waiting = {
'map_missing': 'Waiting for model map', 'map_stale': 'Model map is stale', 'map_invalid': 'Model map is invalid',
'instruction_stale': 'Route instructions are stale', 'route_generation_mismatch': 'Waiting for current route map',
'disabled': 'Navigation inputs disabled', 'invalid_time_or_speed': 'Vehicle data unavailable',
'signal_navigation_disabled': 'Signal navigation is inactive', 'route_context_missing': 'Waiting for route context',
'route_context_unavailable': 'Route context unavailable', 'signal_session_mismatch': 'Waiting for current signal',
'destination_route_unavailable': 'Waiting for destination route', 'waiting_for_navigation': 'Waiting for navigation inputs',
'ready': 'Waiting for model response',
}.get(model_reason, waiting)
pending = intent.get('mode') == 'pending'
return {'state': 'ready', 'primary': 'Turn candidate' if pending else 'Navigation active' if active else 'Route ready',
'direction': direction, 'distanceM': distance,
'secondary': (f"{direction.capitalize()} onto {context.get('name', '')}".strip() if active or pending else waiting)}
if status == 'searching':
return {'state': status, 'primary': f'Finding next {direction}', 'secondary': 'Checking mapped streets'}
if status == 'passed':
return {'state': status, 'primary': 'Turn passed', 'secondary': 'Signal again for another turn'}
if status == 'blocked':
return {'state': status, 'primary': 'Signal turn blocked',
'secondary': 'Destination navigation active' if reason == 'explicit_destination' else 'Signal changed'}
explanations = {'missing_token': 'Map token missing', 'invalid_token': 'Map token unavailable',
'navigation_model_required': 'Select a navigation-capable model',
'gps_unavailable': 'GPS position unavailable', 'gps_or_car_state_invalid': 'GPS position unavailable',
'heading_unavailable': 'Vehicle heading unavailable', 'navigation_unavailable': 'Waiting for navigation',
'target_expired': 'Signal again to refresh target', 'no_matching_turn': 'No supported street turn ahead',
'ambiguous_approach': 'Approach is uncertain', 'approach_location_mismatch': 'Position does not match approach',
'approach_heading_mismatch': 'Heading does not match approach'}
return {'state': 'unavailable', 'primary': 'Map turn unavailable', 'secondary': explanations.get(reason, 'Map lookup unavailable')}
def format_distance(meters: float) -> str:
"""m / ft under 1000 m / 300 m, km / mi beyond."""
if ui_state.is_metric:
@@ -41,6 +111,24 @@ class NavigationPanel(Widget):
self._eta_label = UnifiedLabel('', 34, FontWeight.BOLD, rl.WHITE, scroll=True)
self._accent = rl.Color(155, 230, 185, 255)
self._muted = rl.Color(162, 175, 185, 255)
self._signal_status = None
self._last_signal_poll = -np.inf
@property
def signal_status(self):
now = time.monotonic()
if now-self._last_signal_poll >= .1:
self._last_signal_poll = now
car = ui_state.sm['carState']
self._signal_status = signal_turn_status(
ui_state.params.get_bool('MapboxNextTurnEnabled'), ui_state.params.get('MapboxTurnIntent'),
ui_state.params.get('MapboxTurnContext'), now, car.leftBlinker, car.rightBlinker,
ui_state.params.get('NavModelState'))
return self._signal_status
@property
def signal_turn_active(self):
return self.signal_status is not None
def _update_state(self):
if not self._client.is_connected():
@@ -81,8 +169,14 @@ class NavigationPanel(Widget):
x, y = rect.x, rect.y
rl.draw_rectangle_rec(rect, rl.Color(7, 13, 18, 255))
instruction = ui_state.sm['navInstruction']
if self.routing:
direction = instruction.maneuverModifier
signal = self.signal_status
signal_ready = signal is not None and signal['state'] == 'ready' and not ui_state.nav_destination_set
notification = signal is not None and signal['state'] != 'ready' and not ui_state.nav_destination_set
if notification:
gui_label(rl.Rectangle(x+12, y+18, rect.width-24, 30), signal['primary'], 22, rl.WHITE, FontWeight.BOLD,
alignment_vertical=TextAlignmentVertical.TOP)
elif self.routing or signal_ready:
direction = signal['direction'] if signal_ready else instruction.maneuverModifier
flip = -1 if 'left' in direction else 1
# The maneuver arrow on a 1.4x grid, centered in the header's icon box, with round joins for the thick stroke.
if direction in ('straight', ''):
@@ -94,25 +188,38 @@ class NavigationPanel(Widget):
rl.draw_line_ex(a, b, 7, self._accent)
for point in points:
rl.draw_circle_v(point, 3.5, self._accent)
gui_label(rl.Rectangle(x+64, y+11, rect.width-72, 40), format_distance(max(0, instruction.maneuverDistance)), 34, rl.WHITE, FontWeight.BOLD,
distance = signal['distanceM'] if signal_ready else instruction.maneuverDistance
gui_label(rl.Rectangle(x+64, y+11, rect.width-72, 40), format_distance(max(0, distance)), 34, rl.WHITE, FontWeight.BOLD,
alignment_vertical=TextAlignmentVertical.TOP)
else:
gui_label(rl.Rectangle(x+12, y+18, rect.width-24, 30), 'Waiting for route', 26, rl.WHITE, FontWeight.BOLD,
alignment_vertical=TextAlignmentVertical.TOP)
map_rect = rl.Rectangle(x+8, y+61, rect.width-16, rect.height-111)
rl.draw_rectangle_rec(map_rect, rl.BLACK)
if self.map_ready:
if notification:
gui_label(rl.Rectangle(map_rect.x+10, map_rect.y+16, map_rect.width-20, 80), signal['secondary'], 24, self._muted,
FontWeight.MEDIUM, alignment_vertical=TextAlignmentVertical.TOP)
elif self.map_ready:
# A 1:1 crop of the model's frame, centered horizontally with the ego near the bottom.
width, height = min(map_rect.width, SIZE), min(map_rect.height, SIZE)
source = rl.Rectangle((SIZE - width) // 2, min(SIZE - height, max(0, EGO[1] + MARKER_MARGIN - height)), width, height)
target = rl.Rectangle(map_rect.x + (map_rect.width - width) / 2, map_rect.y + map_rect.height - height, width, height)
rl.draw_texture_pro(self._texture, source, target, rl.Vector2(0, 0), 0, rl.WHITE)
# Attribution is a display overlay; the model receives the original raster.
credit = rl.Rectangle(map_rect.x+2, map_rect.y+map_rect.height-17, map_rect.width-4, 16)
rl.draw_rectangle_rec(credit, rl.Color(7, 13, 18, 220))
gui_label(credit, '© OpenStreetMap contributors', 12, self._muted, FontWeight.MEDIUM,
alignment_vertical=TextAlignmentVertical.TOP)
else:
self._draw_status(map_rect)
if self.routing:
if signal is not None and (notification or ui_state.nav_destination_set):
self._eta_label.set_text(f"{signal['primary']}: {signal['secondary']}")
self._eta_label.render(rl.Rectangle(x+9, y+rect.height-45, rect.width-18, 40))
elif self.routing or signal_ready:
seconds = max(0, instruction.timeRemaining)
remaining = max(0, instruction.distanceRemaining)
self._eta_label.set_text(f'{max(1, round(seconds / 60))} min / {format_distance(remaining)}')
self._eta_label.set_text(f"{signal['primary']}: {signal['secondary']}" if signal is not None else
f'{max(1, round(seconds / 60))} min / {format_distance(remaining)}')
self._eta_label.render(rl.Rectangle(x+9, y+rect.height-45, rect.width-18, 40))
def _unload_texture(self):
@@ -57,12 +57,6 @@ class CruiseLayout(Widget):
description=tr("Use map data to estimate the appropriate speed to drive through turns ahead."),
param="SmartCruiseControlMap")
self.accel_boost_toggle = toggle_item_sp(
title=tr("Acceleration Boost"),
description=tr("Adds a small push on top of sunnypilot's own acceleration, stronger in Aggressive and gentler in Relaxed. " +
"Eases off in curves and when sunnypilot slows or stops. Never goes past your set speed, and lead-car following still comes first."),
param="AccelBoostSP")
self.custom_acc_toggle = toggle_item_sp(
title=tr("Custom ACC Speed Increments"),
description="",
@@ -92,7 +86,6 @@ class CruiseLayout(Widget):
self.icbm_toggle,
self.scc_v_toggle,
self.scc_m_toggle,
self.accel_boost_toggle,
self.custom_acc_toggle,
self.custom_acc_short_increment,
self.custom_acc_long_increment,
@@ -143,8 +136,6 @@ class CruiseLayout(Widget):
self.icbm_toggle.set_description(new_desc)
self.icbm_toggle.show_description(True)
self.accel_boost_toggle.action_item.set_enabled(has_long)
if has_long or has_icbm:
self.custom_acc_toggle.action_item.set_enabled(((has_long and not ui_state.CP.pcmCruise) or has_icbm) and ui_state.is_offroad())
self.scc_v_toggle.action_item.set_enabled(True)
@@ -160,7 +151,6 @@ class CruiseLayout(Widget):
else:
has_icbm = has_long = False
self.icbm_toggle.action_item.set_enabled(False)
self.accel_boost_toggle.action_item.set_enabled(False)
self.icbm_toggle.set_description(tr(ONROAD_ONLY_DESCRIPTION))
show_custom_acc_desc = False
@@ -4,12 +4,14 @@ Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
from html import escape
import re
import time
import pyray as rl
from openpilot.cereal import custom
from openpilot.sunnypilot.models.helpers import ACTIVE_BUNDLE_KEYS, get_selected_bundle, resolve_bundle_by_ref
from openpilot.sunnypilot.models.navigation_model import BUILD_REF
from openpilot.common.constants import CV
from openpilot.selfdrive.ui.ui_state import device, ui_state
from openpilot.selfdrive.ui.sunnypilot.model_info import (big_model_state, bundles_for_source, carrying_model, default_model_name,
@@ -67,6 +69,12 @@ class ModelsLayout(Widget):
callback=lambda: self._open_source_dialog("chestnut")
)
self._navigation_preparation_description = tr(
"Preparing the experimental navigation model requires a large download and an on-device build. Keep the device parked. " +
"When ready, select it from Big Model; preparation does not change the active model.")
self.prepare_navigation_item = button_item(tr("Prepare Navigation Model"), tr("PREPARE"),
self._navigation_preparation_description, self._prepare_navigation_model)
self.download_item = download_status_item(lambda: tr("Download") if self._downloading else tr("Model Status"))
self.refresh_item = button_item(tr("Refresh Model List"),
@@ -96,6 +104,14 @@ class ModelsLayout(Widget):
"This prevents situations (like at red lights) where the car might plan the wrong turn direction."),
param="LaneTurnDesire")
self.map_turn_toggle = toggle_item_sp(tr("Signal Navigation (Experimental)"),
tr("Signals can prepare a short street route while normal lane-change checks remain available. " +
"Navigation waits for an active lane change to complete. " +
"Requires a navigation-capable model, internet and a Mapbox token. " +
"Early fast turn signals and slow lane changes can be ambiguous; this is experimental."),
callback=self._enable_map_turns, enabled=lambda: ui_state.is_offroad() and ui_state.active_runner_tinygrad,
param="MapboxNextTurnEnabled")
self.delay_control = option_item_sp(tr("Adjust Software Delay"), "LagdToggleDelay", 5, 50,
tr("Adjust the software delay when Live Learning Steer Delay is toggled off. The default software delay value is 0.2"),
1, None, True, "", style.BUTTON_ACTION_WIDTH, None, True, lambda v: f"{v / 100:.2f}s")
@@ -111,10 +127,47 @@ class ModelsLayout(Widget):
1, None, True, "", style.BUTTON_ACTION_WIDTH, None, True,
lambda v: f"{v / 100:.2f} m")
self.items = [self.small_model_item, self.big_model_item, self.cancel_download_item, self.download_item, self.refresh_item, self.clear_cache_item,
self.blendful_toggle, self.lane_turn_desire_toggle, self.lane_turn_value_control, self.lagd_toggle, self.delay_control,
self.items = [self.small_model_item, self.big_model_item, self.prepare_navigation_item, self.cancel_download_item, self.download_item,
self.refresh_item, self.clear_cache_item,
self.blendful_toggle, self.lane_turn_desire_toggle, self.map_turn_toggle, self.lane_turn_value_control, self.lagd_toggle, self.delay_control,
self.camera_offset]
def _prepare_navigation_model(self):
if (ui_state.is_offroad() and ui_state.params.get("ModelManager_DownloadRef") is None and
not ui_state.params.get_bool("ModelManager_ClearCache")):
ui_state.params.put("ModelManager_DownloadRef", BUILD_REF)
def _handle_navigation_preparation(self, bundle, progresses):
if bundle is None or bundle.ref != BUILD_REF:
if getattr(self, '_navigation_preparation_message', None) is not None:
self._navigation_preparation_message = None
self.prepare_navigation_item.set_description(self._navigation_preparation_description)
self.prepare_navigation_item.show_description(False)
return False
if getattr(self, '_navigation_preparation_message', None) != bundle.displayName:
self._navigation_preparation_message = bundle.displayName
self._set_item_note(self.prepare_navigation_item, escape(bundle.displayName))
ds = custom.ModelManagerSP.DownloadStatus
self._downloading = bundle.status == ds.downloading
stage = bundle.internalName
self._verifying = stage in ('building', 'verifying')
self.cancel_download_item.set_visible(ui_state.params.get("ModelManager_DownloadRef") is not None)
state = self._download_row_state(progresses, bundle.displayName) if progresses else {"name": bundle.displayName}
state["status_text"] = {
'checking': tr("checking requirements"), 'downloading': tr("downloading"), 'building': tr("building"),
'verifying': tr("verifying"), 'canceling': tr("canceling..."),
}.get(stage, tr("preparing"))
if bundle.status in (ds.failed, ds.downloaded):
state["status_text"] = ""
self.download_item.action_item.update(**state)
if self._downloading:
device._reset_interactive_timeout()
return True
def _enable_map_turns(self, enabled):
if enabled:
ui_state.params.put_bool("NavigationEnabled", True)
def _update_lagd_description(self, lagd_toggle: bool):
desc = tr("Enable this for the car to learn and adapt its steering response time. Disable to use a fixed steering response time. " +
"Keeping this on provides the stock openpilot experience.")
@@ -158,6 +211,8 @@ class ModelsLayout(Widget):
bundle = self.model_manager.selectedBundle if self.model_manager else None
progresses = [model.artifact.downloadProgress for model in bundle.models if model.artifact.fileName] if bundle else []
if self._handle_navigation_preparation(bundle, progresses):
return
if not progresses or bundle.status not in (custom.ModelManagerSP.DownloadStatus.downloading,
custom.ModelManagerSP.DownloadStatus.failed):
self.download_item.action_item.update(name="", segments=self._slot_segments())
@@ -367,6 +422,8 @@ class ModelsLayout(Widget):
offroad = ui_state.is_offroad()
self.small_model_item.action_item.set_enabled(offroad)
self.big_model_item.action_item.set_enabled(offroad)
self.prepare_navigation_item.action_item.set_enabled(
offroad and not self._downloading and not self._clearing and ui_state.params.get("ModelManager_DownloadRef") is None)
self.small_model_item.set_description("" if offroad else tr("Only available when vehicle is off, or always offroad mode is on"))
# manager is offroad-only, so an onroad clear would never be serviced
@@ -9,8 +9,9 @@ import time
import pyray as rl
from openpilot.cereal import custom
from openpilot.selfdrive.ui.mici.widgets.dialog import BigConfirmationDialog, BigDialog
from openpilot.selfdrive.ui.mici.widgets.dialog import BigConfirmationDialog, BigDialog, SettingDescriptionDialog
from openpilot.sunnypilot.models.helpers import ACTIVE_BUNDLE_KEYS, get_selected_bundle
from openpilot.sunnypilot.models.navigation_model import BUILD_REF
from openpilot.selfdrive.ui.mici.widgets.button import BigButton, BigParamControl
from openpilot.selfdrive.ui.ui_state import ui_state, device
from openpilot.selfdrive.ui.sunnypilot.model_info import (active_source, big_model_state, bundles_for_source, carrying_model,
@@ -84,6 +85,15 @@ class ModelsLayoutMici(NavScroller):
self.select_model_btn = BigButton(tr("select model"))
self.select_model_btn.set_click_callback(self._show_folders)
self.prepare_navigation_btn = BigButton(tr("prepare navigation model"),
description=tr("Preparing the experimental navigation model requires a large download and an on-device build. Keep the device parked. " +
"When ready, select it from Big Model; preparation does not change the active model."))
self.prepare_navigation_btn.set_click_callback(self._prepare_navigation_model)
self.preparation_details_btn = BigButton(tr("preparation details"))
self.preparation_details_btn.set_click_callback(self._show_navigation_preparation_details)
self.preparation_details_btn.set_visible(False)
self._navigation_preparation_message = ""
self.refresh_btn = BigButton(tr("refresh models"))
self.refresh_btn.set_click_callback(self._refresh_models)
self._refresh_start: float | None = None
@@ -99,10 +109,58 @@ class ModelsLayoutMici(NavScroller):
description=tr("Adds a gentle nudge toward lane center on top of the model's own steering, stronger when " +
"lane lines are clear. Yields around obstacles. Not available with the Default model."))
self.main_items = [self.current_model_info, self.select_model_btn, self.cancel_download_btn, self.refresh_btn, self.clear_cache_btn,
self.blendful_toggle]
self.map_turn_toggle = BigParamControl(tr("signal navigation (experimental)"), "MapboxNextTurnEnabled",
toggle_callback=self._enable_map_turns,
description=tr("Signals can prepare a short street route while normal lane-change checks remain available. " +
"Navigation waits for an active lane change to complete. " +
"Requires a navigation-capable model, internet and a Mapbox token. " +
"Early fast turn signals and slow lane changes can be ambiguous; this is experimental."))
self.main_items = [self.current_model_info, self.preparation_details_btn, self.select_model_btn, self.prepare_navigation_btn, self.cancel_download_btn,
self.refresh_btn, self.clear_cache_btn,
self.blendful_toggle, self.map_turn_toggle]
self._scroller.add_widgets(self.main_items)
def _prepare_navigation_model(self):
if (ui_state.is_offroad() and ui_state.params.get("ModelManager_DownloadRef") is None and
not ui_state.params.get_bool("ModelManager_ClearCache")):
ui_state.params.put("ModelManager_DownloadRef", BUILD_REF)
def _show_navigation_preparation_details(self):
if self._navigation_preparation_message:
gui_app.push_widget(SettingDescriptionDialog(tr("Navigation model preparation"), self._navigation_preparation_message))
def _handle_navigation_preparation(self, bundle):
self.preparation_details_btn.set_visible(bundle is not None and bundle.ref == BUILD_REF)
if bundle is None or bundle.ref != BUILD_REF:
self._navigation_preparation_message = ""
return False
self._navigation_preparation_message = bundle.displayName
ds = custom.ModelManagerSP.DownloadStatus
downloading = bundle.status == ds.downloading
progresses = [model.artifact.downloadProgress for model in bundle.models if model.artifact.fileName]
stage = bundle.internalName
canceling = any(p.canceling for p in progresses)
self.cancel_download_btn.set_visible(downloading and ui_state.params.get("ModelManager_DownloadRef") is not None)
self.cancel_download_btn.set_text(tr("cancel preparation"))
self.current_model_info.current_model_header.set_text(tr("navigation model preparation"))
self.current_model_info.current_model_text.set_text(bundle.displayName)
self.current_model_info.current_model_header._shimmer = downloading
self.current_model_info.info_header.set_text(tr("progress") if downloading else tr("navigation model"))
progress = sum(p.progress for p in progresses) / len(progresses) if progresses else 0.
detail = {
'checking': tr("checking requirements"), 'downloading': f"{progress:.1f}%", 'building': tr("building; keep the device parked"),
'verifying': tr("verifying"), 'canceling': tr("canceling..."),
}.get(stage, tr("preparing"))
if canceling:
detail = tr("canceling...")
self.current_model_info.info_text.set_text(detail if downloading else bundle.displayName)
return True
def _enable_map_turns(self, enabled):
if enabled:
ui_state.params.put_bool("NavigationEnabled", True)
@property
def model_manager(self):
return ui_state.sm["modelManagerSP"]
@@ -218,6 +276,8 @@ class ModelsLayoutMici(NavScroller):
super()._update_state()
self.select_model_btn.set_enabled(ui_state.is_offroad())
self.map_turn_toggle.set_enabled(ui_state.is_offroad() and ui_state.active_runner_tinygrad)
self.map_turn_toggle.set_checked(ui_state.params.get_bool("MapboxNextTurnEnabled"))
self.cancel_download_btn.set_visible(False)
self.current_model_info.current_model_header._shimmer = False
self.current_model_info.info_header._shimmer = False
@@ -248,6 +308,8 @@ class ModelsLayoutMici(NavScroller):
# manager is offroad-only, so an onroad clear would never be serviced
clearing = ui_state.params.get_bool("ModelManager_ClearCache")
self.clear_cache_btn.set_enabled(ui_state.is_offroad() and not is_downloading and not clearing)
self.prepare_navigation_btn.set_enabled(
ui_state.is_offroad() and not is_downloading and not clearing and ui_state.params.get("ModelManager_DownloadRef") is None)
if clearing:
self.clear_cache_btn.set_value(tr("clearing..."))
self._cache_size_time = 0.0 # refresh the size as soon as clearing finishes
@@ -266,6 +328,9 @@ class ModelsLayoutMici(NavScroller):
self.current_model_info.info_header.set_text(info_header)
self.current_model_info.info_text.set_text(info_text)
if self._handle_navigation_preparation(manager.selectedBundle):
return
if manager.selectedBundle and manager.selectedBundle.status == custom.ModelManagerSP.DownloadStatus.failed:
self.current_model_info.info_header.set_text(tr("error") + self._download_progress)
self.current_model_info.info_text.set_text(tr("download failed"))
@@ -0,0 +1,181 @@
"""Execute actual UI action/status methods headlessly; no renderer or download."""
import ast
from html import escape
from pathlib import Path
from types import SimpleNamespace as NS
import unittest
from unittest.mock import Mock
ROOT = Path(__file__).resolve().parents[1]
BUILD_REF = 'local-prepare-rl-navigation-80eedcdd'
DS = NS(downloading=1, verifying=2, failed=3, downloaded=4, cached=5)
SOURCES = ((ROOT/'layouts/settings/models.py', 'ModelsLayout'), (ROOT/'mici/layouts/models.py', 'ModelsLayoutMici'))
def load_methods(path, name):
tree = ast.parse(path.read_text())
cls = next(n for n in tree.body if isinstance(n, ast.ClassDef) and n.name == name)
names = {'_prepare_navigation_model', '_handle_navigation_preparation', '_download_row_state', '_set_item_note', '_show_navigation_preparation_details'}
cls.body = [n for n in cls.body if isinstance(n, ast.FunctionDef) and n.name in names]
cls.bases = []
ui = NS(is_offroad=Mock(return_value=True), params=Mock())
ui.params.get.return_value = None
ui.params.get_bool.return_value = False
env = {'BUILD_REF': BUILD_REF, 'ui_state': ui, 'custom': NS(ModelManagerSP=NS(DownloadStatus=DS)),
'escape': escape, 'gui_app': Mock(), 'SettingDescriptionDialog': lambda title, message: NS(title=title, message=message),
'device': Mock(), 'tr': lambda s: s, 'ON_COLOR': 'green', 'rl': NS(RED='red', GRAY='gray', WHITE='white')}
exec(compile(ast.Module(body=[cls], type_ignores=[]), str(path), 'exec'), env)
obj = env[name]()
obj.prepare_navigation_item = Mock()
obj.preparation_details_btn = Mock()
return obj, ui
def bundle(status, *, ref=BUILD_REF, text='Preparing Navigation Model: Downloading…', progress_status=None, stage='building'):
p = NS(status=progress_status or status, progress=37., canceling=False, speed=2048.)
return NS(ref=ref, status=status, displayName=text, internalName=stage,
models=[NS(artifact=NS(fileName='navigation-model', downloadProgress=p))])
class NavigationPreparationUiTests(unittest.TestCase):
def test_action_queues_special_ref_without_selecting_model(self):
for path, name in SOURCES:
with self.subTest(layout=name):
obj, ui = load_methods(path, name)
obj._prepare_navigation_model()
ui.params.put.assert_called_once_with('ModelManager_DownloadRef', BUILD_REF)
ui.params.put_bool.assert_not_called()
ui.params.remove.assert_not_called()
def test_action_rejects_onroad_busy_or_cache_clear(self):
for path, name in SOURCES:
for offroad, queued, clearing in ((False, None, False), (True, 'other', False), (True, None, True)):
with self.subTest(layout=name, offroad=offroad, queued=queued, clearing=clearing):
obj, ui = load_methods(path, name)
ui.is_offroad.return_value = offroad
ui.params.get.return_value = queued
ui.params.get_bool.return_value = clearing
obj._prepare_navigation_model()
ui.params.put.assert_not_called()
def test_tici_keeps_full_failure_and_success_status(self):
for status, text in ((DS.failed, 'Preparation failed: LLVM 18+ required'),
(DS.downloaded, 'Navigation Model ready — select it below')):
obj, _ = load_methods(*SOURCES[0])
obj.cancel_download_item, obj.download_item = Mock(), Mock()
b = bundle(status, text=text)
self.assertTrue(obj._handle_navigation_preparation(b, [b.models[0].artifact.downloadProgress]))
state = obj.download_item.action_item.update.call_args.kwargs
self.assertEqual(state['name'], text)
self.assertEqual(state['status_text'], '')
self.assertFalse(obj._downloading)
def test_tici_compile_progress_and_cancel(self):
obj, ui = load_methods(*SOURCES[0])
ui.params.get.return_value = BUILD_REF
obj.cancel_download_item, obj.download_item = Mock(), Mock()
b = bundle(DS.downloading, progress_status=DS.verifying, text='Building Navigation Model…')
obj._handle_navigation_preparation(b, [b.models[0].artifact.downloadProgress])
state = obj.download_item.action_item.update.call_args.kwargs
self.assertEqual(state['name'], b.displayName)
self.assertEqual(state['status_text'], 'building')
self.assertTrue(obj._downloading and obj._verifying)
obj.cancel_download_item.set_visible.assert_called_with(True)
def test_mici_keeps_full_status_and_offers_cancel_during_build(self):
for status in (DS.downloading, DS.failed, DS.downloaded):
obj, ui = load_methods(*SOURCES[1])
ui.params.get.return_value = BUILD_REF if status == DS.downloading else None
obj.cancel_download_btn, obj.current_model_info = Mock(), Mock()
b = bundle(status, text='Full preparation stage or error', progress_status=DS.verifying)
self.assertTrue(obj._handle_navigation_preparation(b))
obj.current_model_info.current_model_text.set_text.assert_called_with(b.displayName)
obj.cancel_download_btn.set_visible.assert_called_with(status == DS.downloading)
if status != DS.downloading:
obj.current_model_info.info_text.set_text.assert_called_with(b.displayName)
else:
self.assertIn('keep the device parked', obj.current_model_info.info_text.set_text.call_args.args[0])
def test_checking_is_not_reported_as_building(self):
for source in SOURCES:
obj, _ = load_methods(*source)
obj.cancel_download_item, obj.download_item = Mock(), Mock()
obj.cancel_download_btn, obj.current_model_info = Mock(), Mock()
b = bundle(DS.downloading, progress_status=DS.verifying, stage='checking',
text='Checking navigation model preparation requirements')
if source == SOURCES[0]:
obj._handle_navigation_preparation(b, [b.models[0].artifact.downloadProgress])
self.assertEqual(obj.download_item.action_item.update.call_args.kwargs['status_text'], 'checking requirements')
self.assertFalse(obj._verifying)
else:
obj._handle_navigation_preparation(b)
self.assertEqual(obj.current_model_info.info_text.set_text.call_args.args[0], 'checking requirements')
def test_full_long_error_remains_readable(self):
message = 'Preparation failed: missing LLVM <18> dependency. ' + 'Detailed diagnostic text. ' * 35
for source in SOURCES:
obj, _ = load_methods(*source)
obj.cancel_download_item, obj.download_item = Mock(), Mock()
obj.cancel_download_btn, obj.current_model_info = Mock(), Mock()
b = bundle(DS.failed, stage='failed', text=message)
if source == SOURCES[0]:
obj._handle_navigation_preparation(b, [b.models[0].artifact.downloadProgress])
obj.prepare_navigation_item.set_description.assert_called_with(escape(message))
obj.prepare_navigation_item.show_description.assert_called_with(True)
else:
obj._handle_navigation_preparation(b)
obj.preparation_details_btn.set_visible.assert_called_with(True)
obj._show_navigation_preparation_details()
dialog = obj._show_navigation_preparation_details.__globals__['gui_app'].push_widget.call_args.args[0]
self.assertEqual(dialog.message, message)
def test_verification_and_unknown_stage_never_claim_building(self):
for stage, expected in (('verifying', 'verifying'), ('unexpected', 'preparing')):
for source in SOURCES:
obj, _ = load_methods(*source)
obj.cancel_download_item, obj.download_item = Mock(), Mock()
obj.cancel_download_btn, obj.current_model_info = Mock(), Mock()
b = bundle(DS.downloading, progress_status=DS.verifying, stage=stage)
if source == SOURCES[0]:
obj._handle_navigation_preparation(b, [b.models[0].artifact.downloadProgress])
self.assertEqual(obj.download_item.action_item.update.call_args.kwargs['status_text'], expected)
else:
obj._handle_navigation_preparation(b)
self.assertEqual(obj.current_model_info.info_text.set_text.call_args.args[0], expected)
def test_unmeasured_eta_removed_and_details_clear_with_normal_bundle(self):
for path, _ in SOURCES:
self.assertNotIn('several minutes', path.read_text())
obj, _ = load_methods(*SOURCES[1])
obj.cancel_download_btn, obj.current_model_info = Mock(), Mock()
obj._handle_navigation_preparation(bundle(DS.failed, stage='failed'))
obj._handle_navigation_preparation(bundle(DS.downloading, ref='normal-model'))
obj.preparation_details_btn.set_visible.assert_called_with(False)
self.assertEqual(obj._navigation_preparation_message, '')
def test_normal_bundle_status_follows_original_path(self):
for path, name in SOURCES:
obj, _ = load_methods(path, name)
args = (bundle(DS.downloading, ref='normal-model'), []) if name == 'ModelsLayout' else (bundle(DS.downloading, ref='normal-model'),)
self.assertFalse(obj._handle_navigation_preparation(*args))
def test_both_visible_actions_use_exported_ref_and_onroad_disable(self):
for path, _ in SOURCES:
source = path.read_text()
self.assertIn('from openpilot.sunnypilot.models.navigation_model import BUILD_REF', source)
self.assertIn('self._prepare_navigation_model)', source)
tree = ast.parse(source)
attributes = {n.attr for n in ast.walk(tree) if isinstance(n, ast.Attribute)}
self.assertTrue('prepare_navigation_item' in attributes or 'prepare_navigation_btn' in attributes)
self.assertIn('offroad and not self._downloading', source) if 'settings/models' in str(path) else self.assertIn(
'ui_state.is_offroad() and not is_downloading', source)
def test_cancel_buttons_remove_existing_request_key(self):
for path, _ in SOURCES:
source = path.read_text()
self.assertIn('ui_state.params.remove("ModelManager_DownloadRef")', source)
self.assertNotIn('params.put("ModelManager_ActiveBundle', source)
if __name__ == '__main__':
unittest.main()
+1 -1
View File
@@ -1 +1 @@
#define SUNNYPILOT_VERSION "2026.11.04-5002"
#define SUNNYPILOT_VERSION "2026.11.01-4997"
+71 -6
View File
@@ -35,7 +35,7 @@ from openpilot.common.realtime import config_realtime_process, DT_MDL
from openpilot.common.transformations.camera import DEVICE_CAMERAS
from openpilot.common.transformations.model import get_warp_matrix
from openpilot.selfdrive.controls.lib.desire_helper import DesireHelper
from openpilot.selfdrive.controls.lib.drive_helpers import get_accel_from_plan, smooth_value
from openpilot.selfdrive.controls.lib.drive_helpers import get_accel_from_plan, get_curvature_from_plan, smooth_value
from openpilot.selfdrive.modeld.modeld import ChestnutGpuState
from openpilot.sunnypilot.modeld_v2.fill_model_msg import fill_model_msg, fill_pose_msg, PublishState, get_curvature_from_output
@@ -49,6 +49,9 @@ from openpilot.sunnypilot.modeld_v2.modeld_base import ModelStateBase
from openpilot.sunnypilot.modeld_v2.helpers import load_model
from openpilot.sunnypilot.modeld_v2.frame_fusion import fused_with_warp
from openpilot.sunnypilot.modeld_v2.model_adapters import get_model_adapter
from openpilot.sunnypilot.modeld_v2.turn_plan_floor import apply_turn_plan_floor
from openpilot.sunnypilot.modeld_v2.navigation_input import (NavigationInputBuffer, NavigationStatePublisher, SignalRouteGuard,
destination_route_context, navigation_capability)
from openpilot.sunnypilot.models.helpers import get_active_bundle
from openpilot.sunnypilot.selfdrive.controls.lib.relc import RoadEdgeLaneChangeController
@@ -98,11 +101,12 @@ class ModelState(ModelStateBase):
else:
model_bundle = get_active_bundle(chestnut=chestnut)
self.generation = model_bundle.generation if model_bundle is not None else None
self.model_name = model_bundle.displayName if model_bundle is not None else ''
overrides = {override.key: override.value for override in model_bundle.overrides} if model_bundle else {}
self.LAT_SMOOTH_SECONDS = float(overrides.get('lat', ".0"))
self.LONG_SMOOTH_SECONDS = float(overrides.get('long', ".0"))
self.MIN_LAT_CONTROL_SPEED = 0.3
self.MIN_LAT_CONTROL_SPEED = 1.6
self.PLANPLUS_CONTROL: float = 1.0
self.chestnut = chestnut
@@ -125,6 +129,7 @@ class ModelState(ModelStateBase):
self._combined_model_type = self.adapter._combined_model_type
self._vision_input_names = self.adapter._vision_input_names
self.numpy_inputs = self.adapter.numpy_inputs
self.navigation_supported, self.navigation_capability_reason = navigation_capability(self.numpy_inputs, bundle, native=self.adapter.is_native)
self._policy_keys = self.adapter._policy_keys
self._has_on_policy = self.adapter._has_on_policy
self._desire_key = self.adapter._desire_key
@@ -174,6 +179,9 @@ class ModelState(ModelStateBase):
for key in ('traffic_convention', 'lateral_control_params', 'action_t'):
if key in self.numpy_inputs and key in inputs:
self.numpy_inputs[key][:] = inputs[key]
for key in ('nav_map', 'navigation'):
if key in self.numpy_inputs:
self.numpy_inputs[key][:] = inputs.get(key, 0)
if self.adapter.is_native:
for i, key in enumerate(self._vision_input_names):
@@ -231,7 +239,7 @@ class ModelState(ModelStateBase):
def get_action_from_model(self, model_output: dict[str, np.ndarray], prev_action: log.ModelDataV2.Action,
lat_action_t: float, long_action_t: float, v_ego: float,
blendful: Callable[[float], float] | None = None) -> log.ModelDataV2.Action:
blendful: Callable[[float], float] | None = None, turn_sign: float = 0.) -> log.ModelDataV2.Action:
if 'action' not in model_output:
plan = model_output['plan'][0]
desired_accel = get_accel_from_plan(plan[:, Plan.VELOCITY][:, 0], plan[:, Plan.ACCELERATION][:, 0], self.constants.T_IDXS,
@@ -243,6 +251,12 @@ class ModelState(ModelStateBase):
else:
desired_accel = model_output['action'][0, 1]
desired_curvature = model_output['action'][0, 0] / (max(1.0, v_ego))**2
if turn_sign:
# the action head can under-commit on turn entry while the model's own plan has the turn; use the plan as a floor
plan = model_output['plan'][0]
plan_curvature = get_curvature_from_plan(plan[:, Plan.T_FROM_CURRENT_EULER][:, 2], plan[:, Plan.ORIENTATION_RATE][:, 2],
self.constants.T_IDXS, v_ego, lat_action_t)
desired_curvature = apply_turn_plan_floor(desired_curvature, plan_curvature, turn_sign)
if blendful is not None:
desired_curvature += blendful(desired_curvature)
@@ -270,6 +284,8 @@ def main(demo=False):
AMDDevice.wait_timeout_ms = 3000
params = Params()
navigation_state = NavigationStatePublisher(params)
navigation_state.publish(now=time.monotonic(), supported=False, active=False, reason='model_loading', force=True)
params.put_bool("ChestnutLoading", CHESTNUT)
params.remove("ChestnutActive")
@@ -331,6 +347,9 @@ def main(demo=False):
params.put_bool("ChestnutLoading", False)
assert model is not None
cloudlog.warning(f"models loaded in {time.monotonic() - st:.1f}s, modeld starting")
navigation_state.publish(now=time.monotonic(), supported=model.navigation_supported, active=False,
reason='waiting_for_navigation' if model.navigation_supported else model.navigation_capability_reason,
model_name=model.model_name, force=True)
config_realtime_process(7, 54)
@@ -338,7 +357,11 @@ def main(demo=False):
pub_socks = ["modelV2", "drivingModelData", "cameraOdometry", "modelDataV2SP"] + (["chestnutGpuState"] if CHESTNUT else [])
pm = PubMaster(pub_socks)
sm = SubMaster(["deviceState", "carState", "narrowRoadCameraState", "extrinsicsCalibration", "driverMonitoringState",
"carControl", "controlsState", "lateralDelay"])
"carControl", "controlsState", "lateralDelay", "navInstruction"])
vipc_client_map = VisionIpcClient("mapsd", VisionStreamType.VISION_STREAM_MAP, True)
navigation_buffer = NavigationInputBuffer()
signal_route_guard = SignalRouteGuard()
publish_state = PublishState()
chestnut_state = ChestnutGpuState(pm, model.chestnut) if CHESTNUT else None
@@ -463,6 +486,33 @@ def main(demo=False):
if 'action_t' in model.numpy_inputs:
inputs['action_t'] = np.array([lat_action_t, long_action_t], dtype=np.float32)
navigation_route = {}
navigation_reason = model.navigation_capability_reason
if model.navigation_supported:
destination, context = params.get('NavDestination'), params.get('MapboxTurnContext')
intent = params.get('MapboxTurnIntent')
if vipc_client_map.is_connected() or vipc_client_map.connect(False):
if (buf_map := vipc_client_map.recv(0)) is not None:
navigation_buffer.observe_map(buf_map.data, buf_map.width, buf_map.height,
timestamp_ns=vipc_client_map.timestamp_sof, received_at=time.monotonic(),
route_generation=vipc_client_map.timestamp_eof)
now = time.monotonic()
if destination is not None:
signal_route_guard.reset()
navigation_route = destination_route_context(destination, context, now)
else:
navigation_route = signal_route_guard.check(
now=now, enabled=params.get_bool('MapboxNextTurnEnabled'), lateral_active=sm['carControl'].latActive,
left_blinker=sm['carState'].leftBlinker, right_blinker=sm['carState'].rightBlinker,
intent=intent, context=context,
)
inputs.update(navigation_buffer.prepare(
now=now, enabled=params.get_bool('NavigationEnabled') and navigation_route['ready'], instruction=sm['navInstruction'],
instruction_valid=sm.valid['navInstruction'], instruction_received_at=sm.recv_time['navInstruction'], speed=v_ego,
instruction_published_ns=sm.logMonoTime['navInstruction'], expected_route_generation=navigation_route.get('routeGeneration'),
))
navigation_reason = navigation_buffer.reason if navigation_route['ready'] else navigation_route['reason']
mt1 = time.perf_counter()
try:
send_chestnut = (chestnut_state is not None and
@@ -475,6 +525,9 @@ def main(demo=False):
params.put_bool("ChestnutActive", False)
assert small_model is not None
model = small_model
signal_route_guard.reset()
navigation_state.publish(now=time.monotonic(), supported=model.navigation_supported, active=False,
reason=model.navigation_capability_reason, model_name=model.model_name, force=True)
if chestnut_state is not None:
chestnut_state.big = False
run_count = 0
@@ -482,6 +535,14 @@ def main(demo=False):
mt2 = time.perf_counter()
model_execution_time = mt2 - mt1
navigation_active = model.navigation_supported and model_output is not None and navigation_reason == 'ready'
if navigation_active and navigation_route.get('session') is not None:
signal_route_guard.consumed(navigation_route['session'], navigation_route['routeGeneration'])
navigation_state.publish(now=time.monotonic(), supported=model.navigation_supported, active=navigation_active,
reason=navigation_reason if model.navigation_supported else model.navigation_capability_reason,
model_name=model.model_name, session=navigation_route.get('session'),
route_generation=navigation_route.get('routeGeneration'))
if model_output is not None:
modelv2_send = messaging.new_message('modelV2')
drivingdata_send = messaging.new_message('drivingModelData')
@@ -491,8 +552,10 @@ def main(demo=False):
if not sm['carControl'].latActive:
prev_action.desiredCurvature = sm['controlsState'].curvature
blendful_active = desire == log.Desire.none and DH.lane_change_state == log.LaneChangeState.off
turn_sign = 1. if desire == log.Desire.turnRight else -1. if desire == log.Desire.turnLeft else 0. # curvature + is right
action = model.get_action_from_model(model_output, prev_action, lat_action_t, long_action_t, v_ego,
partial(blendful.update, model_output, v_ego, blendful_active, enabled=blendful_enabled))
partial(blendful.update, model_output, v_ego, blendful_active, enabled=blendful_enabled),
turn_sign=turn_sign)
prev_action = action
fill_model_msg(drivingdata_send, modelv2_send, model_output, action,
publish_state, meta_main.frame_id, meta_extra.frame_id, frame_id,
@@ -504,7 +567,9 @@ def main(demo=False):
r_lane_change_prob = desire_state[log.Desire.laneChangeRight]
lane_change_prob = l_lane_change_prob + r_lane_change_prob
left_edge, right_edge = RELC.update_and_fill(modelv2_send.modelV2, mdv2sp_send.modelDataV2SP, v_ego)
DH.update(sm['carState'], sm['carControl'].latActive, lane_change_prob, left_edge, right_edge)
DH.update(sm['carState'], sm['carControl'].latActive, lane_change_prob, left_edge, right_edge,
desire_prediction=modelv2_send.modelV2.meta.desirePrediction, model_position=modelv2_send.modelV2.position,
map_turn_supported=model.navigation_supported)
modelv2_send.modelV2.meta.laneChangeState = DH.lane_change_state
modelv2_send.modelV2.meta.laneChangeDirection = DH.lane_change_direction
mdv2sp_send.valid = modelv2_send.valid
@@ -0,0 +1,236 @@
"""Inputs for comma's pinned nav-model-integration driving model.
The map raster and maneuver encoding follow afbf2e547e9319e2eb9f515e1e4330af63705cec.
They condition the trained driving network; they do not issue a steering command
or fabricate a turn desire. Missing/stale context clears both conditioning inputs.
"""
import math
from collections.abc import Mapping
import numpy as np
NAV_MAP_SIZE = 256
NAVIGATION_SIZE = 47
MAP_MAX_AGE = 0.5
INSTRUCTION_MAX_AGE = 2.0
VALHALLA_SIDED_TYPES = {
'depart': (1, 2, 3), 'arrive': (4, 5, 6),
'on ramp': (17, 18, 19), 'off ramp': (20, 20, 21), 'fork': (22, 23, 24),
'roundabout': (26, 26, 26), 'rotary': (26, 26, 26), 'roundabout turn': (26, 26, 26),
'exit roundabout': (27, 27, 27), 'exit rotary': (27, 27, 27), 'notification': (28, 28, 28),
}
VALHALLA_TURN_TYPES = {'straight': 8, 'slight right': 9, 'right': 10, 'sharp right': 11, 'uturn': 13,
'sharp left': 14, 'left': 15, 'slight left': 16}
def _field(value, name, default=None):
return value.get(name, default) if isinstance(value, Mapping) else getattr(value, name, default)
def valhalla_type(maneuver_type: str, modifier: str) -> int | None:
if maneuver_type == 'merge':
return None
if maneuver_type in VALHALLA_SIDED_TYPES:
straight, right, left = VALHALLA_SIDED_TYPES[maneuver_type]
return right if 'right' in modifier else left if 'left' in modifier else straight
return VALHALLA_TURN_TYPES.get(modifier, 8)
def navigation_input(maneuvers, traveled: float = 0.) -> np.ndarray:
"""The next non-depart/arrive Valhalla command plus distance in100m units."""
result = np.zeros(NAVIGATION_SIZE, dtype=np.float32)
if not math.isfinite(traveled) or traveled < 0:
return result
for maneuver in maneuvers:
maneuver_type, modifier = _field(maneuver, 'type', ''), _field(maneuver, 'modifier', '')
distance = _field(maneuver, 'distance')
if (not isinstance(maneuver_type, str) or not isinstance(modifier, str) or
isinstance(distance, bool) or not isinstance(distance, (int, float, np.floating)) or not math.isfinite(distance)):
continue
command, remaining = valhalla_type(maneuver_type, modifier), distance - traveled
if command is not None and command > 6 and remaining >= 0:
result[command], result[-1] = 1, remaining / 100
break
return result
def supports_navigation(numpy_inputs) -> bool:
"""Fail closed on CTV3/small fallback or an incompatible nav tensor contract."""
expected = {'nav_map': (1, 1, NAV_MAP_SIZE, NAV_MAP_SIZE), 'navigation': (1, NAVIGATION_SIZE)}
return all(name in numpy_inputs and numpy_inputs[name].shape == shape and numpy_inputs[name].dtype == np.float32
for name, shape in expected.items())
def navigation_capability(numpy_inputs, bundle, *, native: bool):
if not supports_navigation(numpy_inputs):
return False, 'model_unsupported'
if bundle is None:
return False, 'model_bundle_required'
if not native or bundle.generation != 12 or not bundle.is20hz:
return False, 'model_bundle_incompatible'
return True, 'supported'
def destination_route_context(destination, context, now):
"""An explicit destination must own the exact map generation, including at boot."""
missing = {'ready': False, 'reason': 'destination_route_unavailable'}
if not isinstance(destination, dict) or not isinstance(context, dict):
return missing
snapshot = context.get('destination')
updated, generation = context.get('updatedAt'), context.get('routeGeneration')
if (context.get('routeOwner') != 'destination' or context.get('status') != 'ready' or
not isinstance(snapshot, dict) or isinstance(updated, bool) or not isinstance(updated, (int, float)) or
not math.isfinite(updated) or not 0 <= now - updated <= .35 or
isinstance(generation, bool) or not isinstance(generation, int) or generation <= 0):
return missing
for name in ('latitude', 'longitude'):
value = destination.get(name)
if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value) or snapshot.get(name) != value:
return missing
return {'ready': True, 'reason': 'ready', 'routeGeneration': generation}
class NavigationInputBuffer:
"""Latest real map frame with bounded capture and instruction ages."""
def __init__(self):
self.map = None
self.map_received_at = 0.
self.map_captured_at = 0.
self.route_generation = None
self.reason = 'map_missing'
def observe_map(self, data, width: int, height: int, *, timestamp_ns: int, received_at: float,
route_generation: int | None = None) -> bool:
# A newly received invalid/black-with-zero-timestamp frame must invalidate
# the preceding map rather than make its old geometry appear fresh.
self.map = None
self.route_generation = None
self.reason = 'map_invalid'
if (width != NAV_MAP_SIZE or height != NAV_MAP_SIZE or timestamp_ns <= 0 or
not math.isfinite(received_at) or received_at < 0):
return False
captured_at = timestamp_ns / 1e9
if not 0 <= received_at - captured_at <= MAP_MAX_AGE:
return False
try:
pixels = np.frombuffer(data, dtype=np.uint8, count=NAV_MAP_SIZE * NAV_MAP_SIZE)
except (TypeError, ValueError):
return False
# The verified nav ONNX expects float32 raw raster values0..255, not a
# normalized camera image. Assignment in comma's runner performs this cast.
self.map = pixels.astype(np.float32).reshape(1, 1, NAV_MAP_SIZE, NAV_MAP_SIZE)
self.map_received_at, self.map_captured_at = received_at, captured_at
self.route_generation = route_generation
self.reason = 'ready'
return True
def prepare(self, *, now: float, enabled: bool, instruction, instruction_valid: bool,
instruction_received_at: float, speed: float, instruction_published_ns: int | None = None,
expected_route_generation: int | None = None):
zero = {'nav_map': np.zeros((1, 1, NAV_MAP_SIZE, NAV_MAP_SIZE), dtype=np.float32),
'navigation': np.zeros((1, NAVIGATION_SIZE), dtype=np.float32)}
if not enabled:
self.reason = 'disabled'
return zero
if self.map is None:
self.reason = 'map_missing'
return zero
if not all(math.isfinite(value) for value in (now, instruction_received_at, speed)):
self.reason = 'invalid_time_or_speed'
return zero
if not (0 <= now - self.map_received_at < MAP_MAX_AGE and 0 <= now - self.map_captured_at < MAP_MAX_AGE):
self.reason = 'map_stale'
return zero
if (not isinstance(self.route_generation, int) or self.route_generation <= 0 or
not isinstance(instruction_published_ns, int) or instruction_published_ns < self.route_generation or
(expected_route_generation is not None and self.route_generation != expected_route_generation)):
self.reason = 'route_generation_mismatch'
return zero
elapsed = now - instruction_received_at
if not instruction_valid or not 0 <= elapsed < INSTRUCTION_MAX_AGE:
self.reason = 'instruction_stale'
return zero
vector = navigation_input(_field(instruction, 'allManeuvers', []), max(speed, 0.) * elapsed)
self.reason = 'ready'
return {'nav_map': self.map, 'navigation': vector.reshape(1, NAVIGATION_SIZE)}
class SignalRouteGuard:
"""Bind automatic inputs to the current signal and exact rendered route.
The sole remembered identifier is a route this model actually consumed. It
permits the router's explicitly bounded outgoing continuation after normal
signal auto-cancel; a first-time retained route or another session is rejected.
No timing, steering, maneuver selection or route-completion policy lives here.
"""
def __init__(self):
self.last_consumed = None
def reset(self):
self.last_consumed = None
def consumed(self, session, route_generation):
self.last_consumed = (session, route_generation)
def check(self, *, now, enabled, lateral_active, left_blinker, right_blinker, intent, context):
def unavailable(reason, *, retire=False):
if retire:
self.reset()
return {'ready': False, 'reason': reason}
if not enabled or not lateral_active:
return unavailable('signal_navigation_disabled', retire=True)
if isinstance(intent, dict) and self.last_consumed is not None and intent.get('session') != self.last_consumed[0]:
self.reset()
if not isinstance(context, dict):
return unavailable('route_context_missing')
updated = context.get('updatedAt')
generation, session = context.get('routeGeneration'), context.get('session')
direction = context.get('direction')
if context.get('status') in ('passed', 'completed', 'arrived', 'cancelled', 'idle', 'disabled'):
return unavailable('route_context_unavailable', retire=True)
if self.last_consumed is not None and ((session is not None and session != self.last_consumed[0]) or
(generation is not None and generation != self.last_consumed[1])):
self.reset()
if (context.get('status') != 'ready' or context.get('routeOwner') != 'signal' or
context.get('locationValid') is not True or context.get('headingValid') is not True or
isinstance(updated, bool) or not isinstance(updated, (int, float)) or not math.isfinite(updated) or
not 0 <= now - updated <= .35 or isinstance(generation, bool) or not isinstance(generation, int) or generation <= 0 or
not isinstance(session, str) or not session or direction not in ('left', 'right')):
return unavailable('route_context_unavailable')
physical_direction = 'left' if left_blinker and not right_blinker else 'right' if right_blinker and not left_blinker else None
matches = (isinstance(intent, dict) and intent.get('session') == session and intent.get('direction') == direction and
intent.get('mode') == 'turn' and intent.get('modelSupportsNavigation') is True and intent.get('lateralActive') is True)
held_signal = matches and physical_direction == direction
retained = (not left_blinker and not right_blinker and intent is None and
context.get('retainedAfterSignalCancel') is True and self.last_consumed == (session, generation))
if not (held_signal or retained):
# The camera loop can see auto-cancel before navd's next10Hz decision.
# With neither indicator, hold only the identity (never the inputs) until
# the router explicitly retains or cancels that same owned route.
return unavailable('signal_session_mismatch', retire=bool(left_blinker or right_blinker))
return {'ready': True, 'reason': 'ready', 'session': session, 'routeGeneration': generation}
class NavigationStatePublisher:
"""Runtime capability and consumed-input status, without a control claim."""
def __init__(self, params):
self.params = params
self.last_published = float('-inf')
def publish(self, *, now: float, supported: bool, active: bool, reason: str, model_name: str = '', force: bool = False,
session=None, route_generation=None):
if not force and now - self.last_published < .1:
return False
value = {'updatedAt': now, 'supported': bool(supported), 'active': bool(supported and active), 'reason': reason}
if model_name:
value['modelName'] = model_name
if session is not None:
value.update(session=session, routeGeneration=route_generation)
self.params.put('NavModelState', value)
self.last_published = now
return True
@@ -0,0 +1,307 @@
"""Navigation conditioning contracts without camera/GPU/process dependencies."""
import ast
from collections.abc import Callable
import importlib.util
import math
from pathlib import Path
from types import SimpleNamespace
import unittest
import numpy as np
SOURCE = Path(__file__).resolve().parents[1]
spec = importlib.util.spec_from_file_location('nav_inputs_under_test', SOURCE / 'navigation_input.py')
nav = importlib.util.module_from_spec(spec)
spec.loader.exec_module(nav)
def maneuver(kind='turn', modifier='right', distance=50.):
return SimpleNamespace(type=kind, modifier=modifier, distance=distance)
class NavigationEncodingTests(unittest.TestCase):
def test_right_left_and_distance_use_training_encoding(self):
for modifier, index in (('right', 10), ('left', 15)):
result = nav.navigation_input([maneuver(modifier=modifier, distance=50.)], traveled=8.)
self.assertEqual(result.dtype, np.float32)
self.assertEqual(result.shape, (47,))
self.assertEqual(result[index], 1.)
self.assertAlmostEqual(float(result[-1]), .42)
self.assertEqual(np.count_nonzero(result), 2)
def test_depart_arrive_merge_and_passed_steps_do_not_mask_next_turn(self):
steps = [maneuver('depart'), maneuver('merge'), maneuver(distance=2.), maneuver(modifier='left', distance=90.), maneuver('arrive')]
result = nav.navigation_input(steps, traveled=5.)
self.assertEqual(result[15], 1.)
self.assertAlmostEqual(float(result[-1]), .85)
def test_first_eligible_step_preserves_order_and_roundabout_type(self):
result = nav.navigation_input([maneuver('roundabout', '', 100.), maneuver(modifier='left', distance=120.)])
self.assertEqual(result[26], 1.)
self.assertEqual(result[15], 0.)
def test_nonfinite_distance_cannot_enter_model(self):
result = nav.navigation_input([maneuver(distance=float('inf')), maneuver(distance=float('nan'))])
self.assertTrue(np.array_equal(result, np.zeros(47, np.float32)))
self.assertFalse(nav.navigation_input([maneuver()], traveled=-1).any())
def test_small_and_ctv3_are_not_navigation_models(self):
good = {'nav_map': np.zeros((1, 1, 256, 256), np.float32), 'navigation': np.zeros((1, 47), np.float32)}
self.assertTrue(nav.supports_navigation(good))
self.assertFalse(nav.supports_navigation({}))
self.assertFalse(nav.supports_navigation({'navigation': good['navigation']}))
self.assertFalse(nav.supports_navigation({**good, 'nav_map': good['nav_map'].astype(np.uint8)}))
self.assertFalse(nav.supports_navigation({**good, 'navigation': np.zeros((1, 46), np.float32)}))
def test_navigation_needs_a_compatible_real_bundle_not_an_override(self):
inputs = {'nav_map': np.zeros((1, 1, 256, 256), np.float32), 'navigation': np.zeros((1, 47), np.float32)}
self.assertEqual(nav.navigation_capability(inputs, None, native=True), (False, 'model_bundle_required'))
bundle = SimpleNamespace(generation=12, is20hz=True)
self.assertEqual(nav.navigation_capability(inputs, bundle, native=True), (True, 'supported'))
self.assertFalse(nav.navigation_capability(inputs, SimpleNamespace(generation=9, is20hz=True), native=True)[0])
self.assertFalse(nav.navigation_capability(inputs, bundle, native=False)[0])
class NavigationFreshnessTests(unittest.TestCase):
def setUp(self):
self.buffer = nav.NavigationInputBuffer()
self.pixels = np.zeros((256, 256), np.uint8)
self.pixels[5, :3] = [0, 128, 255]
self.instruction = SimpleNamespace(allManeuvers=[maneuver(distance=50.)])
def observe(self, *, t=10., timestamp_ns=10_000_000_000):
return self.buffer.observe_map(self.pixels, 256, 256, timestamp_ns=timestamp_ns, received_at=t, route_generation=9_000_000_000)
def prepare(self, **changes):
arguments = {'now': 10.1, 'enabled': True, 'instruction': self.instruction, 'instruction_valid': True,
'instruction_received_at': 10., 'speed': 10., 'instruction_published_ns': 10_000_000_000,
'expected_route_generation': 9_000_000_000}
return self.buffer.prepare(**(arguments | changes))
def assert_empty(self, result):
self.assertFalse(result['nav_map'].any())
self.assertFalse(result['navigation'].any())
def test_real_raster_scale_and_elapsed_distance_are_preserved(self):
self.assertTrue(self.observe())
result = self.prepare()
self.assertEqual(result['nav_map'].dtype, np.float32)
np.testing.assert_array_equal(result['nav_map'][0, 0, 5, :3], [0., 128., 255.])
self.assertAlmostEqual(float(result['navigation'][0, -1]), .49)
self.assertEqual(self.buffer.reason, 'ready')
def test_missing_stale_disabled_or_invalid_context_clears_both_inputs(self):
self.assert_empty(self.prepare())
self.observe()
for changes in ({'enabled': False}, {'now': 10.51}, {'instruction_valid': False},
{'instruction_received_at': 8.}, {'instruction_received_at': 10.2}, {'speed': float('nan')}):
self.assert_empty(self.prepare(**changes))
def test_new_invalid_map_cannot_refresh_or_keep_previous_geometry(self):
self.observe()
self.assertTrue(self.prepare()['nav_map'].any())
self.assertFalse(self.observe(timestamp_ns=0))
self.assert_empty(self.prepare())
def test_old_capture_is_rejected_even_if_just_received(self):
self.assertFalse(self.observe(t=11., timestamp_ns=10_000_000_000))
self.assert_empty(self.prepare(now=11.))
self.assertFalse(self.observe(t=10., timestamp_ns=11_000_000_000))
def test_wrong_sized_or_truncated_raster_is_rejected(self):
self.assertFalse(self.buffer.observe_map(b'', 256, 256, timestamp_ns=10_000_000_000, received_at=10.))
self.assertFalse(self.buffer.observe_map(self.pixels, 128, 256, timestamp_ns=10_000_000_000, received_at=10.))
def test_old_route_map_or_pre_route_instruction_cannot_condition_model(self):
self.observe()
self.assert_empty(self.prepare(expected_route_generation=9_500_000_000))
self.assertEqual(self.buffer.reason, 'route_generation_mismatch')
self.assert_empty(self.prepare(instruction_published_ns=8_999_999_999))
self.assertTrue(self.prepare()['nav_map'].any())
class SignalRouteGuardTests(unittest.TestCase):
def setUp(self):
self.guard = nav.SignalRouteGuard()
self.intent = {'session': 'signal1', 'direction': 'right', 'mode': 'turn', 'modelSupportsNavigation': True, 'lateralActive': True}
self.context = {'status': 'ready', 'session': 'signal1', 'direction': 'right', 'routeGeneration': 9_000_000_000,
'updatedAt': 10., 'routeOwner': 'signal', 'locationValid': True, 'headingValid': True}
def check(self, **changes):
arguments = {'now': 10.1, 'enabled': True, 'lateral_active': True, 'left_blinker': False, 'right_blinker': True,
'intent': self.intent, 'context': self.context}
return self.guard.check(**(arguments | changes))
def test_current_signal_session_and_generation_are_returned(self):
result = self.check()
self.assertTrue(result['ready'])
self.assertEqual((result['session'], result['routeGeneration']), ('signal1', 9_000_000_000))
def test_cancel_opposite_hazards_inactive_and_unsupported_are_blocked(self):
for changes in ({'right_blinker': False}, {'left_blinker': True, 'right_blinker': False}, {'left_blinker': True},
{'lateral_active': False}, {'intent': self.intent | {'modelSupportsNavigation': False}},
{'intent': self.intent | {'mode': 'lane_change'}}, {'intent': self.intent | {'mode': 'pending'}},
{'intent': self.intent | {'session': 'new-session'}}):
self.assertFalse(self.check(**changes)['ready'])
def test_missing_stale_and_unowned_contexts_fail_closed(self):
for context in (None, self.context | {'updatedAt': 9.}, self.context | {'routeOwner': 'destination'},
self.context | {'status': 'passed'}, self.context | {'routeGeneration': None},
self.context | {'headingValid': False}):
self.assertFalse(self.check(context=context)['ready'])
def test_retention_needs_previously_consumed_same_route_and_explicit_router_flag(self):
retained = self.context | {'retainedAfterSignalCancel': True}
self.assertFalse(self.check(right_blinker=False, intent=None, context=retained)['ready'])
self.guard.consumed('signal1', 9_000_000_000)
self.assertTrue(self.check(right_blinker=False, intent=None, context=retained)['ready'])
self.assertFalse(self.check(right_blinker=False, intent=None)['ready'])
def test_lateral_loss_or_new_session_revokes_retention(self):
retained = self.context | {'retainedAfterSignalCancel': True}
self.guard.consumed('signal1', 9_000_000_000)
self.check(lateral_active=False)
self.assertFalse(self.check(right_blinker=False, intent=None, context=retained)['ready'])
self.guard.consumed('signal1', 9_000_000_000)
self.check(intent=self.intent | {'session': 'new-session'})
self.assertFalse(self.check(right_blinker=False, intent=None, context=retained)['ready'])
def test_temporary_stale_context_zeros_inputs_but_can_resume_same_consumed_exit(self):
retained = self.context | {'retainedAfterSignalCancel': True}
self.guard.consumed('signal1', 9_000_000_000)
self.assertFalse(self.check(now=11., right_blinker=False, intent=None, context=retained)['ready'])
self.assertEqual(self.guard.last_consumed, ('signal1', 9_000_000_000))
self.assertTrue(self.check(now=11.1, right_blinker=False, intent=None, context=retained | {'updatedAt': 11.})['ready'])
self.assertFalse(self.check(context=self.context | {'status': 'completed'})['ready'])
self.assertIsNone(self.guard.last_consumed)
def test_auto_cancel_waits_zeroed_for_router_confirmation_then_resumes(self):
self.guard.consumed('signal1', 9_000_000_000)
waiting = self.check(right_blinker=False, intent=None)
self.assertFalse(waiting['ready'])
self.assertEqual(self.guard.last_consumed, ('signal1', 9_000_000_000))
retained = self.context | {'retainedAfterSignalCancel': True, 'updatedAt': 10.2}
self.assertTrue(self.check(now=10.21, right_blinker=False, intent=None, context=retained)['ready'])
self.check(right_blinker=False, intent=None, context=self.context | {'status': 'idle'})
self.assertIsNone(self.guard.last_consumed)
class DestinationOwnershipTests(unittest.TestCase):
def setUp(self):
self.destination = {'latitude': 40., 'longitude': -80., 'place_name': 'example'}
self.context = {'routeOwner': 'destination', 'status': 'ready', 'routeGeneration': 9_000_000_000,
'updatedAt': 10., 'destination': {'latitude': 40., 'longitude': -80.}}
def test_matching_existing_destination_is_usable_at_boot(self):
result = nav.destination_route_context(self.destination, self.context, 10.1)
self.assertTrue(result['ready'])
self.assertEqual(result['routeGeneration'], 9_000_000_000)
def test_signal_to_manual_or_changed_destination_never_reuses_old_route(self):
self.assertFalse(nav.destination_route_context(self.destination, self.context | {'routeOwner': 'signal'}, 10.1)['ready'])
changed = self.destination | {'longitude': -79.}
self.assertFalse(nav.destination_route_context(changed, self.context, 10.1)['ready'])
self.assertFalse(nav.destination_route_context(self.destination, self.context | {'destination': None}, 10.1)['ready'])
self.assertFalse(nav.destination_route_context(self.destination, self.context, 11.)['ready'])
class SourceModelInputTests(unittest.TestCase):
@staticmethod
def source_run():
tree = ast.parse((SOURCE / 'modeld.py').read_text())
model_class = next(node for node in tree.body if isinstance(node, ast.ClassDef) and node.name == 'ModelState')
run = next(node for node in model_class.body if isinstance(node, ast.FunctionDef) and node.name == 'run')
namespace = {'np': np, 'Callable': Callable, 'VisionBuf': object}
exec(compile(ast.fix_missing_locations(ast.Module(body=[run], type_ignores=[])), 'actual_modeld_run', 'exec'), namespace)
return namespace['run']
@staticmethod
def model(*, navigation=True):
arrays = {'desire': np.zeros(8, np.float32), 'traffic_convention': np.zeros(2, np.float32),
'action_t': np.zeros(2, np.float32)}
if navigation:
arrays.update(nav_map=np.full((1, 1, 256, 256), 255., np.float32), navigation=np.ones((1, 47), np.float32))
class Output:
def numpy(self):
return np.zeros(1, np.float32)
model = SimpleNamespace(numpy_inputs=arrays, prev_desire=np.zeros(8, np.float32), desire_key='desire', chestnut=False,
_vision_input_names=[], _combined_model_type='supercombo', vision_output_slices={'plan': slice(0, 1)},
parser=SimpleNamespace(parse_outputs=lambda _raw: {}))
model.adapter = SimpleNamespace(copy_frames=lambda _frames: None, is_native=True, run=lambda: Output())
return model
def test_actual_run_copies_nav_inputs_and_preserves_native_desire_edges(self):
run, model = self.source_run(), self.model()
request = np.zeros(8, np.float32)
request[2] = 1
inputs = {'desire': request, 'traffic_convention': np.array([1., 0.]), 'action_t': np.array([.295, .525]),
'nav_map': np.full((1, 1, 256, 256), 128., np.float32),
'navigation': nav.navigation_input([maneuver()]).reshape(1, 47)}
run(model, {}, {}, inputs)
np.testing.assert_array_equal(model.numpy_inputs['nav_map'], inputs['nav_map'])
np.testing.assert_array_equal(model.numpy_inputs['navigation'], inputs['navigation'])
self.assertEqual(model.numpy_inputs['desire'][2], 1.)
run(model, {}, {}, inputs)
self.assertEqual(model.numpy_inputs['desire'][2], 0.)
np.testing.assert_array_equal(model.numpy_inputs['action_t'], np.array([.295, .525], np.float32))
def test_actual_run_zeros_absent_nav_and_accepts_nonnav_fallback(self):
run = self.source_run()
inputs = {'desire': np.zeros(8, np.float32)}
model = self.model()
run(model, {}, {}, inputs)
self.assertFalse(model.numpy_inputs['nav_map'].any())
self.assertFalse(model.numpy_inputs['navigation'].any())
fallback = self.model(navigation=False)
run(fallback, {}, {}, inputs)
self.assertNotIn('nav_map', fallback.numpy_inputs)
def test_existing_float_host_layout_also_fits_verified_nav_graph(self):
tree = ast.parse((SOURCE / 'frame_fusion.py').read_text())
layout = next(node for node in tree.body if isinstance(node, ast.FunctionDef) and node.name == 'pack_layout')
namespace = {'math': math}
exec(compile(ast.fix_missing_locations(ast.Module(body=[layout], type_ignores=[])), 'source_layout', 'exec'), namespace)
old = {'desire': (8,), 'traffic_convention': (1, 2), 'action_t': (1, 2)}
before, old_size = namespace['pack_layout'](old, 4096)
expanded, new_size = namespace['pack_layout'](old | {'nav_map': (1, 1, 256, 256), 'navigation': (1, 47)}, 4096)
for name in ('tfm', *old):
self.assertEqual(before[name], expanded[name])
self.assertEqual(expanded['nav_map'], before['frames'])
self.assertGreaterEqual(expanded['navigation'] - expanded['nav_map'], 256 * 256 * 4)
self.assertGreater(new_size, old_size)
class NavigationStatusTests(unittest.TestCase):
def test_status_is_throttled_but_load_and_fallback_can_publish_immediately(self):
writes = []
publisher = nav.NavigationStatePublisher(SimpleNamespace(put=lambda key, value: writes.append((key, value))))
self.assertTrue(publisher.publish(now=10., supported=True, active=True, reason='ready', model_name='Nav N16'))
self.assertFalse(publisher.publish(now=10.05, supported=True, active=True, reason='ready'))
self.assertTrue(publisher.publish(now=10.11, supported=True, active=False, reason='map_stale'))
self.assertTrue(publisher.publish(now=10.12, supported=False, active=False, reason='model_unsupported', force=True))
self.assertEqual(len(writes), 3)
self.assertEqual(writes[0][0], 'NavModelState')
self.assertEqual(writes[0][1]['modelName'], 'Nav N16')
self.assertFalse(writes[-1][1]['supported'])
self.assertFalse(writes[-1][1]['active'])
def test_status_identifiers_belong_to_the_consumed_automatic_route(self):
writes = []
publisher = nav.NavigationStatePublisher(SimpleNamespace(put=lambda key, value: writes.append(value)))
publisher.publish(now=10., supported=True, active=True, reason='ready', session='signal1', route_generation=9_000_000_000)
self.assertEqual(writes[-1]['session'], 'signal1')
self.assertEqual(writes[-1]['routeGeneration'], 9_000_000_000)
publisher.publish(now=10.2, supported=False, active=False, reason='model_unsupported')
self.assertNotIn('session', writes[-1])
def test_unsupported_model_can_never_report_active_nav(self):
writes = []
publisher = nav.NavigationStatePublisher(SimpleNamespace(put=lambda key, value: writes.append(value)))
publisher.publish(now=10., supported=False, active=True, reason='model_unsupported')
self.assertFalse(writes[-1]['active'])
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,36 @@
"""
Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
import pytest
from openpilot.sunnypilot.modeld_v2.turn_plan_floor import TURN_PLAN_FLOOR, apply_turn_plan_floor
RIGHT, LEFT, NONE = 1., -1., 0.
def test_no_turn_leaves_action_alone():
assert apply_turn_plan_floor(0.01, 0.2, NONE) == 0.01
@pytest.mark.parametrize("sign", [RIGHT, LEFT])
def test_weak_action_is_raised_to_scaled_plan(sign):
# curvature is + to the right, so a left turn is negative
assert apply_turn_plan_floor(sign * 0.03, sign * 0.10, sign) == pytest.approx(sign * TURN_PLAN_FLOOR * 0.10)
@pytest.mark.parametrize("sign", [RIGHT, LEFT])
def test_strong_action_is_kept(sign):
assert apply_turn_plan_floor(sign * 0.12, sign * 0.10, sign) == sign * 0.12
@pytest.mark.parametrize("sign", [RIGHT, LEFT])
def test_plan_against_the_turn_never_weakens_or_reverses(sign):
assert apply_turn_plan_floor(sign * 0.05, -sign * 0.10, sign) == sign * 0.05
def test_floor_only_pushes_toward_the_requested_side():
# a right-turn request must never add left curvature, even if action and plan both point left
assert apply_turn_plan_floor(-0.02, -0.10, RIGHT) == -0.02
@@ -0,0 +1,23 @@
"""
Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
# Share of the plan-derived curvature the command may not fall below during a requested turn. The plan-derived
# curvature reads ~15-20% tighter than turns actually driven, so it is only trusted as a scaled-down floor.
TURN_PLAN_FLOOR = 0.75
def apply_turn_plan_floor(action_curvature: float, plan_curvature: float, turn_sign: float,
scale: float = TURN_PLAN_FLOOR) -> float:
"""Keep the steering command from falling below a scaled share of the model's own plan during a requested turn.
Curvature is + to the right. turn_sign is +1 for a right turn, -1 for a left turn and 0 when no turn is requested.
The floor only acts in the turn direction: it can strengthen a turn the action head under-commits to, but never
weakens or reverses the action.
"""
if turn_sign == 0.:
return action_curvature
return turn_sign * max(turn_sign * action_curvature, scale * turn_sign * plan_curvature)
@@ -0,0 +1,69 @@
# Experimental learned navigation model
Install this branch, park the device with the supported Chestnut GPU connected,
then open **Settings → Models → Prepare Navigation Model**. Keep the device
parked and powered throughout preparation. This downloads approximately 815 MB,
checks its official size and SHA-256, and builds the device artifact locally.
Download progress and the build stage appear in the Models screen. Compilation
has no measured time estimate yet. **Cancel** stops the preparation job and its
compiler; entering onroad also stops it. Failure leaves an explanation in Models
and does not activate or register an incomplete model.
After the screen says the model is ready, explicitly choose **RL Navigation
80eedcdd (experimental)** in the existing Chestnut model selector. Preparation
never changes the current model. Selecting this model is a whole-drive choice;
signal-created navigation routes do not swap models on each turn. The ordinary
small model remains the fallback and does not support navigation conditioning.
The build checks for the reviewed compiler, libusb and LLVM 18 or newer with the
AMDGPU backend before downloading. It requires at least 4 GiB of free model
storage. Missing device dependencies produce an error; this feature does not
install system libraries. The compiler log is
`Paths.model_root()/rl_nav_80eedcdd_build.log`. An interrupted verified download
can be reused on the next attempt; incomplete downloads are never accepted.
If a completed compilation fails protocol validation, its artifact, descriptor
and `validation_failure.json` are retained in the single
`Paths.model_root()/.rl_nav_failed_build` directory for diagnosis. A later rejected
build replaces that directory. It is outside normal model registration and is
never automatically selected or reused as a validated model.
## Exact checkpoint and build
The source is the official ONNX SHA-256
`1d18c42ea4cd95c8bffab4635844b78b8e2d4e25b9de44a9c5ac2e0961371a2e`,
814,591,106 bytes, referenced by comma commit
`afbf2e547e9319e2eb9f515e1e4330af63705cec`. This is the earlier learned navigation
checkpoint **80eedcdd/12864**, not the separate N16 compiled PKL updated in that
commit. The download uses the public `happymaj11r/openpilot-models` release as an
alternate transport because the official LFS object is unavailable. Only bytes
matching the authoritative upstream pointer are accepted.
The existing reviewed `compile_modeld_v2.py` is invoked with
`DEV=USB+AMD:LLVM`, `FRAME_DEV=CPU`, `FLOAT16=1`, `GMMU=0`, `TC_OPT=2`,
`TC_MIN_GLOBALS=32`, all inputs on-device and one benchmark run. The generated
plain artifact must match its hash, AMD device specification, all five recurrent
state pairs and the navigation/action output protocol before the descriptor is
published. No artifact means no selectable local model. Existing image warp
artifacts are reused; no new warp build is required.
The bundle uses generation 12 as the native action protocol, not a training
checkpoint number. Normal remote catalogs remain unchanged. Removing or changing
the local artifact makes its entry unavailable. Preparing and selecting it use
the existing model manager request, status and cancel mechanism; no new Params
keys are required.
## Validation limits
The exact ONNX has run on Mac CPU with real recorded camera frames and real
mini-route/map inputs. A same-model Nav versus NoNav test changed the predicted
turn response. That is a fixed-view neural response test, not proof of physical
turn completion, receiving lane choice or curb clearance. The full model has not
been compiled or run on the user's device here; device compilation time, memory
requirements beyond the preliminary storage check and source/device numerical
parity remain unmeasured. A successful preparation is an artifact/protocol check,
not a road-test readiness or safety verdict.
Existing action conversion, the 1.6 m/s curvature hold, steering and cooperative
steering are preserved. Navigation conditioning requires matching route ownership
and fresh map/instruction inputs. `NavModelState.active` reports consumed inputs,
not successful autonomous maneuver completion.
@@ -0,0 +1,211 @@
"""Parked preparation of the exact RL Navigation model; never selects it."""
import argparse
import ctypes
import hashlib
import json
import os
from pathlib import Path
import shutil
import subprocess
import sys
import tempfile
import time
import requests
from openpilot.sunnypilot.models.navigation_model import (MANIFEST_NAME, SOURCE_COMMIT, SOURCE_ONNX_BYTES, SOURCE_ONNX_SHA256,
SOURCE_ONNX_URL, file_sha256, installed_bundle)
COMPILE_ENV = {'DEBUG': '1', 'DEV': 'USB+AMD:LLVM', 'FRAME_DEV': 'CPU', 'FLOAT16': '1',
'GMMU': '0', 'TC_OPT': '2', 'TC_MIN_GLOBALS': '32'}
COMPILER_SHA256 = '0ff7415619aa267914f858c24b3f3c32583f75aceebd89ec4f56e70e9daf32ac'
FAILED_BUILD_DIRECTORY = '.rl_nav_failed_build'
class PreparationCancelled(RuntimeError):
pass
def preserve_failed_build(temporary, root, reason):
"""Keep one completed rejected build for inspection, outside the selector root."""
diagnostic = {'reason': reason, 'sourceOnnxSha256': SOURCE_ONNX_SHA256, 'compilerSha256': COMPILER_SHA256,
'registered': False, 'automaticReuseAllowed': False}
try:
descriptor = json.loads((temporary / MANIFEST_NAME).read_text())
artifact = temporary / descriptor['artifact']['fileName']
diagnostic['artifact'] = descriptor['artifact'] | {'bytes': artifact.stat().st_size}
from openpilot.sunnypilot.modeld_v2.helpers import read_parts
_parts, extras = read_parts(str(artifact))
metadata = {key: extras.get(key) for key in ('input_specs', 'output_specs', 'metadata')}
encoded = json.dumps(metadata, default=repr)
diagnostic['compiledMetadata'] = metadata if len(encoded) <= 65536 else {'truncated': True, 'head': encoded[:65536]}
except Exception as error:
diagnostic['diagnosticReadError'] = str(error)[:2048]
(temporary / 'validation_failure.json').write_text(json.dumps(diagnostic, indent=2, default=repr) + '\n')
failed = root / FAILED_BUILD_DIRECTORY
if failed.is_symlink():
failed.unlink()
elif failed.exists():
shutil.rmtree(failed)
temporary.replace(failed)
return failed
def check_allowed(params, queued):
if not params.get_bool('IsOffroad'):
raise PreparationCancelled('Stopped because the device is not confirmed offroad. Park before preparing again.')
if queued and params.get('ModelManager_DownloadRef') is None:
raise PreparationCancelled('Navigation model preparation canceled.')
def preflight(basedir, hardware):
if not hardware:
raise RuntimeError('Preparation requires a comma device with a supported Chestnut GPU.')
compiler = Path(basedir) / 'openpilot/sunnypilot/modeld_v2/compile_modeld_v2.py'
if not compiler.is_file() or file_sha256(compiler) != COMPILER_SHA256:
raise RuntimeError('Target compiler is missing or differs from the reviewed source. Update this branch first.')
# Use the same search order and optional paths as the target runtime.
from tinygrad.runtime.support.c import DLL
usb = DLL.findlib('libusb', ['usb-1.0'])
if not usb:
raise RuntimeError('Required libusb is missing. No model was built; update the device system software.')
ctypes.CDLL(usb)
library = DLL.findlib('llvm', [f'LLVM-{version}' for version in range(21, 13, -1)] + ['LLVM'])
if library:
llvm = ctypes.CDLL(library)
if hasattr(llvm, 'LLVMGetVersion') and hasattr(llvm, 'LLVMInitializeAMDGPUTarget'):
major, minor, patch = ctypes.c_uint(), ctypes.c_uint(), ctypes.c_uint()
llvm.LLVMGetVersion(ctypes.byref(major), ctypes.byref(minor), ctypes.byref(patch))
if major.value >= 18:
return compiler
raise RuntimeError('LLVM 18 or newer with AMDGPU support is missing. No model was built; update the device system software.')
def download_source(path, check, report):
"""Bounded stream, exact length and authoritative hash; partial bytes never become a source."""
if path.is_file() and path.stat().st_size == SOURCE_ONNX_BYTES:
report('verifying', 'Verifying cached navigation model', 0)
if file_sha256(path) == SOURCE_ONNX_SHA256:
check()
return
partial = path.with_suffix('.partial')
try:
check()
report('downloading', 'Downloading navigation model', 0)
digest, total, started = hashlib.sha256(), 0, time.monotonic()
with requests.get(SOURCE_ONNX_URL, stream=True, timeout=(15, 5), headers={'Accept-Encoding': 'identity'}) as response:
response.raise_for_status()
length = response.headers.get('Content-Length')
if length is not None and int(length) != SOURCE_ONNX_BYTES:
raise RuntimeError('Navigation model download has an unexpected length.')
with partial.open('wb') as stream:
for block in response.iter_content(chunk_size=256 * 1024):
check()
if not block:
continue
total += len(block)
if total > SOURCE_ONNX_BYTES:
raise RuntimeError('Navigation model download exceeded its pinned size.')
stream.write(block)
digest.update(block)
elapsed = max(.001, time.monotonic() - started)
report('downloading', 'Downloading navigation model', min(99, total / SOURCE_ONNX_BYTES * 100), total / elapsed)
if total != SOURCE_ONNX_BYTES or digest.hexdigest() != SOURCE_ONNX_SHA256:
raise RuntimeError('Navigation model download failed its official size/SHA-256 verification.')
check()
partial.replace(path)
finally:
partial.unlink(missing_ok=True)
def prepare(root, basedir, params, hardware, *, queued=False, onnx=None, report=lambda *_args: None):
def check():
check_allowed(params, queued)
check()
report('checking', 'Checking target build requirements', 0)
compiler = preflight(basedir, hardware)
root.mkdir(parents=True, exist_ok=True)
# Source + compiled output + compiler scratch; fail before the large transfer.
if shutil.disk_usage(root).free < 4 * 1024 ** 3:
raise RuntimeError('At least 4 GiB of free model storage is required for preparation.')
if installed_bundle(root) is not None:
report('ready', 'Navigation model ready — select it in the Chestnut model selector', 100)
return
source = Path(onnx) if onnx else root / 'rl_nav_80eedcdd_source.onnx'
if onnx:
if source.stat().st_size != SOURCE_ONNX_BYTES or file_sha256(source) != SOURCE_ONNX_SHA256:
raise RuntimeError('The ONNX bytes do not match the official RL Navigation artifact.')
else:
download_source(source, check, report)
check()
with tempfile.TemporaryDirectory(prefix='.rl-nav-build-', dir=root) as directory:
temporary = Path(directory)
output = temporary / 'navigation.pkl'
report('building', 'Building navigation model — keep the device parked', 0)
command = [sys.executable, str(compiler), str(source.resolve()), str(output), '--device-input', '*', '--benchmark-runs', '1']
log_path = root / 'rl_nav_80eedcdd_build.log'
with log_path.open('w') as log, subprocess.Popen(command, cwd=basedir, env=os.environ | COMPILE_ENV,
stdout=log, stderr=subprocess.STDOUT) as process:
try:
while process.poll() is None:
check()
time.sleep(.2)
if process.returncode:
raise RuntimeError(f'Target compilation failed (exit {process.returncode}); see {log_path.name}. No model was registered.')
finally:
if process.poll() is None:
process.terminate()
try:
process.wait(timeout=2)
except subprocess.TimeoutExpired:
process.kill()
process.wait()
check()
report('verifying', 'Verifying compiled navigation model', 99)
digest = file_sha256(output)
name = 'rl_nav_80eedcdd_' + digest[:16] + '_tinygrad.pkl'
descriptor = {'version': 1, 'target': 'chestnut', 'sourceOnnxSha256': SOURCE_ONNX_SHA256,
'sourceRef': 'commaai/openpilot:' + SOURCE_COMMIT, 'sourceCheckpoint': '80eedcdd/12864 (not the separate N16 PKL)',
'artifact': {'fileName': name, 'sha256': digest}, 'compilerEnvironment': COMPILE_ENV,
'compilerSha256': file_sha256(compiler)}
output.rename(temporary / name)
(temporary / MANIFEST_NAME).write_text(json.dumps(descriptor, indent=2) + '\n')
if installed_bundle(temporary) is None:
reason = 'Compiled model failed its plain-format/navigation protocol checks.'
failed = preserve_failed_build(temporary, root, reason)
raise RuntimeError(f'{reason} No model was registered. Completed build and validation details kept in {failed.name}; see the build log.')
check()
(temporary / name).replace(root / name)
check()
(temporary / MANIFEST_NAME).replace(root / MANIFEST_NAME)
report('ready', 'Navigation model ready — select it in the Chestnut model selector', 100)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--onnx', type=Path, help='Optional already downloaded official-SHA ONNX')
parser.add_argument('--queued', action='store_true', help='Honor the existing ModelManager cancel request')
parser.add_argument('--status-file', type=Path)
args = parser.parse_args()
from openpilot.common.basedir import BASEDIR
from openpilot.common.hardware import COMMA_HARDWARE
from openpilot.common.hardware.hw import Paths
from openpilot.common.params import Params
def report(stage, message, progress, speed=0):
payload = {'stage': stage, 'message': message, 'progress': progress, 'speed': speed}
if args.status_file:
temporary = args.status_file.with_suffix('.tmp')
temporary.write_text(json.dumps(payload))
temporary.replace(args.status_file)
else:
print(message, flush=True)
try:
prepare(Path(Paths.model_root()), BASEDIR, Params(), COMMA_HARDWARE, queued=args.queued, onnx=args.onnx, report=report)
except Exception as error:
report('failed', str(error), 0)
raise SystemExit(1) from error
if __name__ == '__main__':
main()
+17 -3
View File
@@ -14,6 +14,17 @@ from openpilot.sunnypilot.models.helpers import is_bundle_version_compatible
from openpilot.cereal import custom
def _with_local_navigation(bundles, source):
if source == 'chestnut':
from openpilot.common.hardware.hw import Paths
from openpilot.sunnypilot.models.navigation_model import installed_bundle
local = installed_bundle(Paths.model_root())
if local is not None:
parsed = ModelParser._parse_bundle(local)
return [bundle for bundle in bundles if bundle.ref != parsed.ref] + [parsed]
return bundles
class ModelParser:
"""Handles parsing of model data into cereal objects"""
@@ -179,6 +190,9 @@ class ModelFetcher:
return not any(bundle.get("is_big") is True for bundle in bundles)
def get_bundles_for_source(self, source: str) -> list[custom.ModelManagerSP.ModelBundle]:
return _with_local_navigation(self._get_remote_bundles_for_source(source), source)
def _get_remote_bundles_for_source(self, source: str) -> list[custom.ModelManagerSP.ModelBundle]:
if source not in self.MODEL_SOURCES:
cloudlog.warning(f"Unknown model source: {source}")
return []
@@ -226,12 +240,12 @@ def get_cached_bundles(params: Params, source: str) -> list[custom.ModelManagerS
_, suffix = ModelFetcher.MODEL_SOURCES[source]
cached_data = params.get(f"ModelManager_ModelsCache{suffix}")
if not cached_data:
return []
return _with_local_navigation([], source)
try:
return ModelParser.parse_models(cached_data)
return _with_local_navigation(ModelParser.parse_models(cached_data), source)
except Exception as e:
cloudlog.warning(f"Failed to parse cached models for source {source}: {e}")
return []
return _with_local_navigation([], source)
if __name__ == "__main__":
+107
View File
@@ -6,8 +6,14 @@ See the LICENSE.md file in the root directory for more details.
"""
import asyncio
import json
import os
from pathlib import Path
import re
import signal
import subprocess
import sys
import tempfile
import threading
import time
from concurrent.futures import ThreadPoolExecutor
@@ -20,6 +26,7 @@ from openpilot.common.hardware.hw import Paths
from openpilot.cereal import messaging, custom
from openpilot.sunnypilot.models.fetcher import ModelFetcher
from openpilot.sunnypilot.models.navigation_model import BUILD_REF, MANIFEST_NAME, installed_bundle
from openpilot.sunnypilot.models.helpers import (ACTIVE_BUNDLE_KEYS, get_active_bundle, get_selected_bundle,
resolve_bundle_by_ref, validate_active_bundles, verify_file)
@@ -32,6 +39,7 @@ PIECE_ATTEMPTS = 3
PIECE_BACKOFF = 1.0 # seconds, growing linearly: waits 1 s then 2 s between a piece's three attempts
TRANSIENT_HTTP_STATUS = frozenset({408, 429, 500, 502, 503, 504})
REPORT_INTERVAL = 0.5 # seconds between progress publications
NAV_PREPARE_STARTUP_TIMEOUT = 30.0 # First worker status, not a compilation time limit.
# byte ranges count encoded bytes but iter_content yields decoded ones, so ask for an unencoded body
NO_COMPRESSION = {"Accept-Encoding": "identity"}
SPEED_SMOOTHING = 0.7 # weight of the previous speed sample in the published speed
@@ -42,6 +50,23 @@ class DownloadCancelled(Exception):
pass
def _stop_job(process) -> None:
"""The detached worker and its compiler share one process group."""
try:
os.killpg(process.pid, signal.SIGTERM)
except ProcessLookupError:
pass
try:
process.wait(timeout=2)
except subprocess.TimeoutExpired:
pass
try:
os.killpg(process.pid, signal.SIGKILL)
except ProcessLookupError:
pass
process.wait()
def _is_transient(e: BaseException) -> bool:
if isinstance(e, requests.HTTPError):
return e.response is not None and e.response.status_code in TRANSIENT_HTTP_STATUS
@@ -345,6 +370,75 @@ class ModelManagerSP:
"""Main entry point for downloading a model bundle"""
asyncio.run(self._download_bundle(model_bundle, destination_path, source))
def _prepare_navigation(self) -> None:
# This is a preparation job, never an activation or a normal download bundle.
bundle = custom.ModelManagerSP.ModelBundle()
bundle.ref = BUILD_REF
bundle.models = [{'type': 'chunked', 'artifact': {'fileName': 'rl_nav_80eedcdd_source.onnx'}}]
self.selected_bundle = bundle
artifact = bundle.models[0].artifact
status = custom.ModelManagerSP.DownloadStatus
def update(payload):
stage = payload.get('stage', 'checking')
bundle.internalName = stage
bundle.displayName = payload.get('message', 'Preparing navigation model')[:2048]
bundle.status = status.failed if stage == 'failed' else status.downloaded if stage == 'ready' else status.downloading
artifact.downloadProgress.status = (status.downloading if stage == 'downloading' else status.verifying
if stage in ('checking', 'building', 'verifying') else bundle.status)
artifact.downloadProgress.progress = payload.get('progress', 0)
artifact.downloadProgress.speed = payload.get('speed', 0)
artifact.downloadProgress.eta = 0 # Compilation duration has not been measured on this target.
self._report_status()
process = None
try:
update({'stage': 'checking', 'message': 'Checking navigation model preparation requirements'})
if not self.params.get_bool('IsOffroad') or not self.chestnut_present:
update({'stage': 'failed', 'message': 'Park the device with a connected supported Chestnut GPU before preparing the navigation model.'})
return
from openpilot.common.basedir import BASEDIR
root = Path(Paths.model_root())
root.mkdir(parents=True, exist_ok=True)
with tempfile.TemporaryDirectory(prefix='.nav-prepare-', dir=root) as temporary, (root / 'rl_nav_80eedcdd_prepare.log').open('w') as log:
status_path = Path(temporary) / 'status.json'
command = [sys.executable, '-u', '-m', 'openpilot.sunnypilot.models.build_navigation_model', '--queued', '--status-file', str(status_path)]
process = subprocess.Popen(command, cwd=BASEDIR, start_new_session=True, stdout=log, stderr=subprocess.STDOUT)
previous = None
started = time.monotonic()
while process.poll() is None:
if not self.params.get_bool('IsOffroad') or self._download_interrupted():
artifact.downloadProgress.canceling = True
message = 'Preparation stopped: device is not offroad.' if not self.params.get_bool('IsOffroad') else 'Navigation model preparation canceled.'
raise DownloadCancelled(message)
try:
payload = status_path.read_text()
if payload != previous:
update(json.loads(payload))
previous = payload
except (OSError, ValueError):
pass
if previous is None and time.monotonic() - started >= NAV_PREPARE_STARTUP_TIMEOUT:
raise RuntimeError('Preparation worker did not report startup within 30 seconds. See rl_nav_80eedcdd_prepare.log.')
time.sleep(.2)
if status_path.exists():
update(json.loads(status_path.read_text()))
if not self.params.get_bool('IsOffroad') or self._download_interrupted():
raise DownloadCancelled('Preparation stopped before completion; park and prepare again.')
if process.returncode or installed_bundle(root) is None:
if bundle.status != status.failed:
update({'stage': 'failed', 'message': f'Navigation model preparation failed (exit {process.returncode}). ' +
'See rl_nav_80eedcdd_prepare.log and rl_nav_80eedcdd_build.log.'})
else:
self.source_models['chestnut'] = self.model_fetcher.get_bundles_for_source('chestnut')
self.available_models = self.source_models[ModelFetcher.active_source(self.chestnut_present)]
update({'stage': 'ready', 'message': 'Navigation model ready — select it in the Chestnut model selector', 'progress': 100})
except Exception as error:
update({'stage': 'failed', 'message': str(error)})
finally:
if process is not None:
_stop_job(process)
def _process_download_requests(self) -> None:
# loops so a ref queued during a download starts in the same tick, without
# the bar dropping to idle for a tick between the two transfers
@@ -353,6 +447,15 @@ class ModelManagerSP:
if ref_to_download == last_ref: # a repeating ref falls back to the next tick instead of spinning
return
last_ref = ref_to_download
if ref_to_download in (BUILD_REF, BUILD_REF.encode()):
self._download_ref = ref_to_download
try:
self._prepare_navigation()
except Exception as e:
cloudlog.exception(e)
finally:
self._release_download_ref()
continue
resolved = resolve_bundle_by_ref(ref_to_download, self.source_models)
if not resolved:
return
@@ -407,6 +510,10 @@ class ModelManagerSP:
active_files.append(model.artifact.fileName)
model_dir = Paths.model_root()
local_navigation = installed_bundle(model_dir)
if local_navigation and local_navigation['models'][0]['artifact']['file_name'] in active_files:
# Its descriptor is part of the installed local bundle, not a remote catalog cache.
active_files.append(MANIFEST_NAME)
try:
for filename in os.listdir(model_dir):
if filename not in active_files:
@@ -0,0 +1,122 @@
"""Explicitly installed RL Navigation model, appended to the normal catalog.
No artifact is bundled or selected automatically. The installer verifies the
official ONNX bytes before target compilation; this reader rechecks the resulting
plain artifact hash, device and native protocol before exposing a selectable row.
"""
import base64
import builtins
import hashlib
import io
import json
from pathlib import Path
import pickle
import struct
import numpy as np
SOURCE_COMMIT = 'afbf2e547e9319e2eb9f515e1e4330af63705cec'
SOURCE_ONNX_SHA256 = '1d18c42ea4cd95c8bffab4635844b78b8e2d4e25b9de44a9c5ac2e0961371a2e'
SOURCE_ONNX_BYTES = 814591106
# Alternate transport; accepted only when the bytes match the official upstream LFS pointer above.
SOURCE_ONNX_URL = ('https://github.com/happymaj11r/openpilot-models/releases/download/' +
'big-nav-model-integration-1d18c42e/big_driving_supercombo.onnx')
BUILD_REF = 'local-prepare-rl-navigation-80eedcdd'
MODEL_NAME = 'RL Navigation 80eedcdd (experimental)'
MANIFEST_NAME = 'rl_nav_80eedcdd_bundle.json'
SELECTOR_VERSION = 21
_validated = {}
class _SliceMetadata(pickle.Unpickler):
def find_class(self, module, name):
if module == 'builtins' and name == 'slice':
return builtins.slice
raise pickle.UnpicklingError('Unsupported output-slice metadata class')
def file_sha256(path):
with Path(path).open('rb') as stream:
return hashlib.file_digest(stream, 'sha256').hexdigest()
def compatible_metadata(extras):
try:
specs, outputs = extras['input_specs'], extras['metadata']['output_shapes']
expected = {
'new_img': ((2, 6, 128, 256), np.uint8), 'desire': ((8,), np.float32),
'traffic_convention': ((1, 2), np.float32), 'action_t': ((1, 2), np.float32),
'nav_map': ((1, 1, 256, 256), np.float32), 'navigation': ((1, 47), np.float32),
'state_img_q': ((2, 5, 6, 128, 256), np.uint8), 'state_desire_q': ((132, 1, 8), np.float32),
'state_feat_q': ((128, 1, 16384), np.float32), 'state_map_features_q': ((128, 1, 64), np.float32),
'state_navigation_features_q': ((128, 1, 64), np.float32),
}
if set(specs) != set(expected):
return False
for name, (shape, dtype) in expected.items():
if tuple(specs[name][0]) != shape or np.dtype(specs[name][1]) != dtype or 'AMD' not in specs[name][2]:
return False
if name.startswith('state_') and tuple(outputs.get('next_' + name, ())) != shape:
return False
expected_outputs = {'outputs': ((1, 18580), np.float32)} | {
'next_' + name: value for name, value in expected.items() if name.startswith('state_')}
if set(outputs) != set(expected_outputs) or set(extras['output_specs']) != set(expected_outputs):
return False
for name, (shape, dtype) in expected_outputs.items():
actual = extras['output_specs'][name]
if tuple(outputs[name]) != shape or tuple(actual[0]) != shape or np.dtype(actual[1]) != dtype or 'AMD' not in actual[2]:
return False
encoded = extras['metadata']['metadata']['output_slices']
# Upstream ONNX metadata uses wrapped Base64. Permit only ASCII whitespace
# around its encoded bytes; keep strict decoding and restricted unpickling.
encoded = encoded.translate(str.maketrans('', '', ' \t\r\n\v\f'))
slices = _SliceMetadata(io.BytesIO(base64.b64decode(encoded, validate=True))).load()
required = {'plan': slice(917, 1907), 'action': slice(2062, 2066),
'desire_state': slice(2054, 2062), 'desire_pred': slice(855, 887)}
if any(slices.get(name) != value for name, value in required.items()):
return False
except (ValueError, KeyError, TypeError, IndexError, AttributeError, EOFError, pickle.UnpicklingError):
return False
return True
def installed_bundle(model_root):
"""Return a normal selector bundle only for a present verified target build."""
root = Path(model_root)
try:
manifest_path = root / MANIFEST_NAME
if manifest_path.stat().st_size > 16384:
return None
manifest = json.loads(manifest_path.read_text())
artifact = manifest['artifact']
name, expected = artifact['fileName'], artifact['sha256']
if (manifest.get('version') != 1 or manifest.get('target') != 'chestnut' or
manifest.get('sourceOnnxSha256') != SOURCE_ONNX_SHA256 or
not isinstance(name, str) or Path(name).name != name or not name.endswith('.pkl') or
not isinstance(expected, str) or len(expected) != 64 or any(c not in '0123456789abcdef' for c in expected)):
return None
path = root / name
if path.resolve().parent != root.resolve():
return None
stat = path.stat()
key = (str(path.resolve()), stat.st_size, stat.st_mtime_ns, expected)
if key not in _validated:
if file_sha256(path) != expected:
return None
from openpilot.sunnypilot.modeld_v2.helpers import read_parts
parts, extras = read_parts(str(path))
if 'run' not in parts or not compatible_metadata(extras):
return None
_validated.clear()
_validated[key] = True
return {
'index': 2147483000, 'short_name': 'RLNAV80', 'display_name': MODEL_NAME,
'ref': 'local-rl-nav-80eedcdd-' + expected[:16], 'is_big': True,
'environment': 'development', 'runner': 'tinygrad', 'generation': 12,
'is_20hz': True, 'minimum_selector_version': SELECTOR_VERSION,
'overrides': {'folder': 'Local experimental models', 'lat': '.0', 'long': '.3'},
'models': [{'type': 'chunked', 'artifact': {'file_name': name,
'download_uri': {'url': path.resolve().as_uri(), 'sha256': expected}}}],
}
except (OSError, ValueError, KeyError, TypeError, IndexError, AttributeError, EOFError, pickle.UnpicklingError, struct.error):
return None
@@ -0,0 +1,264 @@
{
"provenance": {
"sourceRepository": "commaai/openpilot",
"sourceRef": "afbf2e547e9319e2eb9f515e1e4330af63705cec",
"sourceModelPath": "openpilot/selfdrive/modeld/models/big_driving_supercombo.onnx",
"officialOnnxSha256": "1d18c42ea4cd95c8bffab4635844b78b8e2d4e25b9de44a9c5ac2e0961371a2e",
"officialOnnxBytes": 814591106,
"sourceCheckpoint": "cf8f6120-49ee-48f8-b2d7-d663a4708271/56320/80eedcdd-f97b-46d0-9b94-780685292acf/12864",
"extractedIoMetadataSha256": "10b55d548b2f662f214c5e7c52c5e7864b18675b295ac51064661dd207e93e39",
"scope": "Graph declarations and literal metadata only; no model weights or executable pickle."
},
"input_specs": {
"new_img": [
[
2,
6,
128,
256
],
"uint8",
"AMD"
],
"desire": [
[
8
],
"float32",
"AMD"
],
"traffic_convention": [
[
1,
2
],
"float32",
"AMD"
],
"action_t": [
[
1,
2
],
"float32",
"AMD"
],
"nav_map": [
[
1,
1,
256,
256
],
"float32",
"AMD"
],
"navigation": [
[
1,
47
],
"float32",
"AMD"
],
"state_img_q": [
[
2,
5,
6,
128,
256
],
"uint8",
"AMD"
],
"state_desire_q": [
[
132,
1,
8
],
"float32",
"AMD"
],
"state_feat_q": [
[
128,
1,
16384
],
"float32",
"AMD"
],
"state_map_features_q": [
[
128,
1,
64
],
"float32",
"AMD"
],
"state_navigation_features_q": [
[
128,
1,
64
],
"float32",
"AMD"
]
},
"output_specs": {
"outputs": [
[
1,
18580
],
"float32",
"AMD"
],
"next_state_img_q": [
[
2,
5,
6,
128,
256
],
"uint8",
"AMD"
],
"next_state_desire_q": [
[
132,
1,
8
],
"float32",
"AMD"
],
"next_state_feat_q": [
[
128,
1,
16384
],
"float32",
"AMD"
],
"next_state_map_features_q": [
[
128,
1,
64
],
"float32",
"AMD"
],
"next_state_navigation_features_q": [
[
128,
1,
64
],
"float32",
"AMD"
]
},
"metadata": {
"metadata": {
"output_slices": "gASV0gEAAAAAAAB9lCiMCmxhbmVfbGluZXOUjAhidWlsdGluc5SMBXNsaWNllJOUSwBNEAJOh5RS\nlIwPbGFuZV9saW5lc19wcm9ilGgETRACTRgCToeUUpSMCnJvYWRfZWRnZXOUaARNGAJNIANOh5RS\nlIwEbWV0YZRoBE0gA01XA06HlFKUjAtkZXNpcmVfcHJlZJRoBE1XA013A06HlFKUjARwb3NllGgE\nTXcDTYMDToeUUpSMFndpZGVfZnJvbV9kZXZpY2VfZXVsZXKUaARNgwNNiQNOh5RSlIwOcm9hZF90\ncmFuc2Zvcm2UaARNiQNNlQNOh5RSlIwEcGxhbpRoBE2VA01zB06HlFKUjARsZWFklGgETXMHTQMI\nToeUUpSMCWxlYWRfcHJvYpRoBE0DCE0GCE6HlFKUjAxkZXNpcmVfc3RhdGWUaARNBghNDghOh5RS\nlIwGYWN0aW9ulGgETQ4ITRIIToeUUpSMDGhpZGRlbl9zdGF0ZZRoBE0SCE0SSE6HlFKUjAxtYXBf\nZmVhdHVyZXOUaARNEkhNUkhOh5RSlIwTbmF2aWdhdGlvbl9mZWF0dXJlc5RoBE1SSE2SSE6HlFKU\njANwYWSUaARNkkhNlEhOh5RSlHUu\n",
"model_checkpoint": "cf8f6120-49ee-48f8-b2d7-d663a4708271/56320/80eedcdd-f97b-46d0-9b94-780685292acf/12864"
},
"input_shapes": {
"new_img": [
2,
6,
128,
256
],
"desire": [
8
],
"traffic_convention": [
1,
2
],
"action_t": [
1,
2
],
"nav_map": [
1,
1,
256,
256
],
"navigation": [
1,
47
],
"state_img_q": [
2,
5,
6,
128,
256
],
"state_desire_q": [
132,
1,
8
],
"state_feat_q": [
128,
1,
16384
],
"state_map_features_q": [
128,
1,
64
],
"state_navigation_features_q": [
128,
1,
64
]
},
"output_shapes": {
"outputs": [
1,
18580
],
"next_state_img_q": [
2,
5,
6,
128,
256
],
"next_state_desire_q": [
132,
1,
8
],
"next_state_feat_q": [
128,
1,
16384
],
"next_state_map_features_q": [
128,
1,
64
],
"next_state_navigation_features_q": [
128,
1,
64
]
}
}
}
@@ -0,0 +1,267 @@
"""Local selector entries require an installed verified target artifact."""
import ast
import base64
import builtins
import hashlib
import importlib.util
import json
import io
import mmap
from pathlib import Path
import pickle
import sys
import struct
import subprocess
from tempfile import TemporaryDirectory
from types import SimpleNamespace
import unittest
from unittest.mock import patch
SOURCE = Path(__file__).resolve().parents[1]
spec = importlib.util.spec_from_file_location('local_nav_model_under_test', SOURCE / 'navigation_model.py')
nav = importlib.util.module_from_spec(spec)
spec.loader.exec_module(nav)
def metadata(*, device='USB+AMD:LLVM'):
inputs = {'new_img': ((2, 6, 128, 256), '|u1', device),
'desire': ((8,), '<f4', device), 'traffic_convention': ((1, 2), '<f4', device), 'action_t': ((1, 2), '<f4', device),
'nav_map': ((1, 1, 256, 256), '<f4', device), 'navigation': ((1, 47), '<f4', device),
'state_img_q': ((2, 5, 6, 128, 256), '|u1', device), 'state_desire_q': ((132, 1, 8), '<f4', device),
'state_feat_q': ((128, 1, 16384), '<f4', device),
'state_map_features_q': ((128, 1, 64), '<f4', device),
'state_navigation_features_q': ((128, 1, 64), '<f4', device)}
slices = {'plan': slice(917, 1907), 'action': slice(2062, 2066),
'desire_state': slice(2054, 2062), 'desire_pred': slice(855, 887)}
outputs = {'outputs': ((1, 18580), '<f4', device)} | {'next_' + name: value for name, value in inputs.items() if name.startswith('state_')}
return {'input_specs': inputs, 'output_specs': outputs,
'metadata': {'metadata': {'output_slices': base64.b64encode(pickle.dumps(slices)).decode()},
'output_shapes': {name: value[0] for name, value in outputs.items()}}}
class LocalNavigationModelTests(unittest.TestCase):
def setUp(self):
self.temporary = TemporaryDirectory()
self.root = Path(self.temporary.name)
self.artifact = self.root / 'local_nav_test.pkl'
self.artifact.write_bytes(b'test-plain-artifact-no-kernel-execution')
self.descriptor = {'version': 1, 'target': 'chestnut', 'sourceOnnxSha256': nav.SOURCE_ONNX_SHA256,
'artifact': {'fileName': self.artifact.name, 'sha256': hashlib.sha256(self.artifact.read_bytes()).hexdigest()}}
self.extras = metadata()
nav._validated.clear()
self.reader = SimpleNamespace(read_parts=lambda _path: ({'run': ({}, b'')}, self.extras))
self.patch = patch.dict(sys.modules, {'openpilot.sunnypilot.modeld_v2.helpers': self.reader})
self.patch.start()
def tearDown(self):
self.patch.stop()
self.temporary.cleanup()
def write(self, changes=None):
(self.root / nav.MANIFEST_NAME).write_text(json.dumps(self.descriptor | (changes or {})))
def test_no_installed_manifest_means_no_selectable_entry(self):
self.assertIsNone(nav.installed_bundle(self.root))
self.write()
self.artifact.unlink()
self.assertIsNone(nav.installed_bundle(self.root))
def test_verified_native_target_is_normal_explicit_bundle_not_n16_claim(self):
self.write()
bundle = nav.installed_bundle(self.root)
self.assertEqual(bundle['generation'], 12)
self.assertTrue(bundle['is_20hz'])
self.assertEqual(bundle['minimum_selector_version'], 21)
self.assertEqual(bundle['runner'], 'tinygrad')
self.assertIn('80eedcdd', bundle['display_name'])
self.assertNotIn('N16', bundle['display_name'])
self.assertEqual(bundle['overrides']['lat'], '.0')
self.assertEqual(bundle['models'][0]['artifact']['download_uri']['sha256'], self.descriptor['artifact']['sha256'])
self.assertNotIn('selected', bundle)
def test_model_hash_source_or_path_tamper_never_registers(self):
for change in ({'sourceOnnxSha256': '0' * 64}, {'target': 'metal'},
{'artifact': {'fileName': '../escape.pkl', 'sha256': 'a' * 64}},
{'artifact': {'fileName': self.artifact.name, 'sha256': 'a' * 64}}):
self.write(change)
self.assertIsNone(nav.installed_bundle(self.root))
def test_mac_cpu_small_or_missing_nav_metadata_never_registers(self):
self.write()
for extras in (metadata(device='METAL'), metadata(device='CPU'), {'input_specs': {}, 'metadata': {}},
metadata() | {'metadata': {'output_shapes': {'outputs': (1, 2576)}}}):
self.extras = extras
nav._validated.clear()
self.assertIsNone(nav.installed_bundle(self.root))
def test_changed_installed_bytes_are_reverified(self):
self.write()
self.assertIsNotNone(nav.installed_bundle(self.root))
self.artifact.write_bytes(b'changed artifact bytes')
self.assertIsNone(nav.installed_bundle(self.root))
def test_incompatible_action_layout_cannot_claim_generation12(self):
self.write()
slices = {'plan': slice(917, 1907), 'action': slice(2062, 2070),
'desire_state': slice(2054, 2062), 'desire_pred': slice(855, 887)}
self.extras['metadata']['metadata']['output_slices'] = base64.b64encode(pickle.dumps(slices)).decode()
self.assertIsNone(nav.installed_bundle(self.root))
def test_every_state_pair_needs_matching_output_shape_and_dtype(self):
for name in ('state_img_q', 'state_desire_q', 'state_feat_q', 'state_map_features_q', 'state_navigation_features_q'):
for change in ('missing', 'shape', 'dtype'):
extras = metadata()
output = 'next_' + name
if change == 'missing':
extras['metadata']['output_shapes'].pop(output)
elif change == 'shape':
extras['metadata']['output_shapes'][output] = (1,)
else:
shape, _, device = extras['output_specs'][output]
extras['output_specs'][output] = (shape, '<f8', device)
self.assertFalse(nav.compatible_metadata(extras), (name, change))
def test_host_input_and_malformed_metadata_contracts_fail_closed(self):
for name in ('desire', 'traffic_convention', 'action_t'):
extras = metadata()
extras['input_specs'][name] = ((99,), '<f4', 'USB+AMD:LLVM')
self.assertFalse(nav.compatible_metadata(extras))
for extras in (None, [], {}, {'input_specs': []}, metadata() | {'input_specs': {'new_img': ()}}):
self.assertFalse(nav.compatible_metadata(extras))
def test_invalid_plain_metadata_fails_closed_without_code_execution(self):
self.write()
def invalid(_path):
raise pickle.UnpicklingError('Rejected arbitrary metadata class')
self.reader.read_parts = invalid
self.assertIsNone(nav.installed_bundle(self.root))
def real_navigation_metadata():
fixture = json.loads((SOURCE / 'tests/fixtures/rl_nav_80eedcdd_metadata.json').read_text())
for name, value in fixture['input_specs'].items():
fixture['input_specs'][name] = (tuple(value[0]), nav.np.dtype(value[1]).str, value[2])
for name, value in fixture['output_specs'].items():
fixture['output_specs'][name] = (tuple(value[0]), nav.np.dtype(value[1]).name, value[2])
for kind in ('input_shapes', 'output_shapes'):
fixture['metadata'][kind] = {name: tuple(shape) for name, shape in fixture['metadata'][kind].items()}
return fixture
def plain_io_functions():
"""Execute the exact plain writer/reader definitions, with no GPU imports."""
path = SOURCE.parents[0] / 'modeld_v2/helpers.py'
text = path.read_text() if path.exists() else subprocess.check_output(
['git', '-C', str(SOURCE.parents[2]), 'show', 'HEAD:openpilot/sunnypilot/modeld_v2/helpers.py'], text=True)
names = {'Blob', '_MetadataUnpickler', '_relocate', 'write_bundle', 'read_parts'}
constants = {'MAGIC', 'ALIGNMENT', 'METADATA_TYPES'}
selected = []
for node in ast.parse(text).body:
if isinstance(node, (ast.FunctionDef, ast.ClassDef)) and node.name in names:
selected.append(node)
elif isinstance(node, ast.Assign) and any(isinstance(t, ast.Name) and t.id in constants for t in node.targets):
selected.append(node)
namespace = {'hashlib': hashlib, 'builtins': builtins, 'pickle': pickle, 'struct': struct, 'json': json,
'io': io, 'mmap': mmap}
exec(compile(ast.Module(body=selected, type_ignores=[]), str(path), 'exec'), namespace)
return namespace['write_bundle'], namespace['read_parts']
class RealNavigationMetadataTests(unittest.TestCase):
def test_exact_official_wrapped_metadata_survives_actual_plain_roundtrip(self):
fixture = real_navigation_metadata()
self.assertEqual(fixture['provenance']['officialOnnxSha256'], nav.SOURCE_ONNX_SHA256)
self.assertEqual(fixture['provenance']['officialOnnxBytes'], nav.SOURCE_ONNX_BYTES)
self.assertEqual(fixture['metadata']['metadata']['output_slices'].count('\n'), 9)
write_bundle, read_parts = plain_io_functions()
with TemporaryDirectory() as directory:
path = Path(directory) / 'real_metadata_plain_fixture.pkl'
# An empty, non-executable run part is sufficient to test the actual format.
write_bundle(path, {'run': ({'calls': [], 'buffers': []}, b'')}, fixture)
parts, decoded = read_parts(path)
self.assertIn('run', parts)
self.assertEqual(decoded, fixture)
self.assertTrue(nav.compatible_metadata(decoded))
(Path(directory) / nav.MANIFEST_NAME).write_text(json.dumps({'version': 1, 'target': 'chestnut',
'sourceOnnxSha256': nav.SOURCE_ONNX_SHA256,
'artifact': {'fileName': path.name, 'sha256': nav.file_sha256(path)}}))
with patch.dict(sys.modules, {'openpilot.sunnypilot.modeld_v2.helpers': SimpleNamespace(read_parts=read_parts)}):
self.assertIsNotNone(nav.installed_bundle(directory))
def test_ascii_wrapping_is_allowed_but_corrupt_base64_is_rejected(self):
fixture = real_navigation_metadata()
original = fixture['metadata']['metadata']['output_slices']
compact = ''.join(original.split())
fixture['metadata']['metadata']['output_slices'] = ' \t\r\n'.join(compact[i:i+16] for i in range(0, len(compact), 16))
self.assertTrue(nav.compatible_metadata(fixture))
for encoded in (compact[:8] + '!' + compact[8:], compact[:-1], compact[:8] + '\u00a0' + compact[8:]):
with self.subTest(encoded=encoded[:20]):
fixture['metadata']['metadata']['output_slices'] = encoded
self.assertFalse(nav.compatible_metadata(fixture))
def test_wrapping_does_not_permit_executable_or_wrong_slice_metadata(self):
class ExecutableMetadata:
def __reduce__(self):
return builtins.eval, ('1 + 1',)
fixture = real_navigation_metadata()
payload = base64.encodebytes(pickle.dumps(ExecutableMetadata())).decode('ascii')
fixture['metadata']['metadata']['output_slices'] = payload
with patch.object(builtins, 'eval', side_effect=AssertionError('Metadata must never execute')) as execute:
self.assertFalse(nav.compatible_metadata(fixture))
execute.assert_not_called()
original = real_navigation_metadata()['metadata']['metadata']['output_slices']
slices = nav._SliceMetadata(io.BytesIO(base64.b64decode(original))).load()
slices['action'] = slice(2062, 2070)
fixture['metadata']['metadata']['output_slices'] = base64.encodebytes(pickle.dumps(slices)).decode('ascii')
self.assertFalse(nav.compatible_metadata(fixture))
fixture['metadata']['metadata']['output_slices'] = base64.encodebytes(pickle.dumps([])).decode('ascii')
self.assertFalse(nav.compatible_metadata(fixture))
class CatalogHookTests(unittest.TestCase):
def test_cached_offline_model_is_selectable_without_changing_remote_cache(self):
tree = ast.parse((SOURCE / 'fetcher.py').read_text())
methods = [node for node in tree.body if isinstance(node, ast.FunctionDef) and node.name in ('_with_local_navigation', 'get_cached_bundles')]
local, remote = SimpleNamespace(ref='local-verified'), SimpleNamespace(ref='remote')
payload = {'bundles': [{'ref': 'remote'}]}
namespace = {'Params': object, 'custom': SimpleNamespace(ModelManagerSP=SimpleNamespace(ModelBundle=object)),
'ModelFetcher': SimpleNamespace(MODEL_SOURCES={'chestnut': ('url', '_Chestnut'), 'qcom': ('url', '')}),
'ModelParser': SimpleNamespace(parse_models=lambda value: [remote], _parse_bundle=lambda value: local),
'cloudlog': SimpleNamespace(warning=lambda *_args: None)}
exec(compile(ast.fix_missing_locations(ast.Module(body=methods, type_ignores=[])), 'actual_catalog_hook', 'exec'), namespace)
modules = {'openpilot.common.hardware.hw': SimpleNamespace(Paths=SimpleNamespace(model_root=lambda: '/test/models')),
'openpilot.sunnypilot.models.navigation_model': SimpleNamespace(installed_bundle=lambda path: {'ref': local.ref})}
with patch.dict(sys.modules, modules):
cached = namespace['get_cached_bundles'](SimpleNamespace(get=lambda _key: payload), 'chestnut')
self.assertEqual(cached, [remote, local])
self.assertEqual(payload, {'bundles': [{'ref': 'remote'}]})
offline = namespace['get_cached_bundles'](SimpleNamespace(get=lambda _key: None), 'chestnut')
self.assertEqual(offline, [local])
self.assertEqual(namespace['_with_local_navigation']([remote], 'qcom'), [remote])
class OffroadBuilderTests(unittest.TestCase):
def test_cli_failure_is_nonzero_and_never_registers(self):
# Detailed stream/build/cancel behavior is covered in test_navigation_preparation.
with TemporaryDirectory() as directory:
root = Path(directory)
modules = {
'openpilot.sunnypilot.models.navigation_model': nav,
'openpilot.common.basedir': SimpleNamespace(BASEDIR=str(SOURCE.parents[2])),
'openpilot.common.hardware': SimpleNamespace(COMMA_HARDWARE=False),
'openpilot.common.hardware.hw': SimpleNamespace(Paths=SimpleNamespace(model_root=lambda: str(root))),
'openpilot.common.params': SimpleNamespace(Params=lambda: SimpleNamespace(get_bool=lambda _key: False)),
}
with patch.dict(sys.modules, modules), patch.object(sys, 'argv', ['build_navigation_model']):
builder_spec = importlib.util.spec_from_file_location('nav_builder_under_test', SOURCE / 'build_navigation_model.py')
builder = importlib.util.module_from_spec(builder_spec)
builder_spec.loader.exec_module(builder)
with patch.object(builder.subprocess, 'Popen', side_effect=AssertionError('Compiler must not start')):
with self.assertRaises(SystemExit) as error:
builder.main()
self.assertEqual(error.exception.code, 1)
self.assertFalse((root / nav.MANIFEST_NAME).exists())
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,417 @@
"""Preparation lifecycle tests; no GPU kernels or downloaded model code execute."""
import ast
import asyncio
import hashlib
import importlib.util
import json
import os
from pathlib import Path
import re
import signal
import subprocess
import sys
import tempfile
import time
from types import SimpleNamespace as NS
import unittest
from unittest.mock import Mock, patch
import requests
SOURCE = Path(__file__).resolve().parents[1]
KEY_SOURCE = SOURCE.parents[1] / 'common/params_keys.h'
REGISTERED_KEYS = set(re.findall(r'\{"([A-Za-z0-9_]+)"', KEY_SOURCE.read_text()))
def load(name, path):
spec = importlib.util.spec_from_file_location(name, path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
nav = load('preparation_nav', SOURCE / 'navigation_model.py')
with patch.dict(sys.modules, {'openpilot.sunnypilot.models.navigation_model': nav}):
builder = load('preparation_builder', SOURCE / 'build_navigation_model.py')
class Params:
def __init__(self):
self.onroad, self.ref = False, nav.BUILD_REF.encode()
def get_bool(self, key):
if key not in REGISTERED_KEYS:
raise KeyError(f'UnknownKeyName: {key}')
if key != 'IsOffroad':
raise AssertionError(f'Unexpected preparation key: {key}')
return not self.onroad
def get(self, _key):
return self.ref
class Response:
def __init__(self, chunks, length=None):
self.chunks = chunks
self.headers = {} if length is None else {'Content-Length': str(length)}
def __enter__(self):
return self
def __exit__(self, *_args):
pass
def raise_for_status(self):
pass
def iter_content(self, **_kwargs):
yield from self.chunks
class SourceDownloadTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.path = Path(self.temp.name) / 'source.onnx'
self.body = b'official-source-stand-in'
self.report = Mock()
self.enterContext(patch.object(builder, 'SOURCE_ONNX_BYTES', len(self.body)))
self.enterContext(patch.object(builder, 'SOURCE_ONNX_SHA256', hashlib.sha256(self.body).hexdigest()))
def fetch(self, response, check=lambda: None):
with patch.object(builder.requests, 'get', return_value=response) as get:
builder.download_source(self.path, check, self.report)
return get
def test_exact_stream_and_cached_retry_skip_network(self):
self.fetch(Response([self.body[:5], self.body[5:]], len(self.body)))
self.assertEqual(self.path.read_bytes(), self.body)
self.assertFalse(self.path.with_suffix('.partial').exists())
with patch.object(builder.requests, 'get', side_effect=AssertionError('cached source must skip network')):
builder.download_source(self.path, lambda: None, self.report)
def test_wrong_size_short_oversized_and_corrupt_fail_closed(self):
for response in (Response([self.body], 1), Response([self.body[:-1]]), Response([self.body + b'x']), Response([b'x' * len(self.body)])):
with self.subTest(response=response), self.assertRaises(RuntimeError):
self.fetch(response)
self.assertFalse(self.path.exists())
self.assertFalse(self.path.with_suffix('.partial').exists())
def test_cancellation_during_stream_removes_partial(self):
checks = iter([False, False, True])
def check():
if next(checks, True):
raise builder.PreparationCancelled('canceled')
with self.assertRaises(builder.PreparationCancelled):
self.fetch(Response([self.body[:5], self.body[5:]]), check)
self.assertFalse(self.path.exists())
self.assertFalse(self.path.with_suffix('.partial').exists())
def test_queued_cancel_and_onroad_both_abort(self):
params = Params()
builder.check_allowed(params, True)
params.ref = None
with self.assertRaises(builder.PreparationCancelled):
builder.check_allowed(params, True)
def test_offroad_guard_uses_actual_registered_key_and_missing_is_fail_closed(self):
self.assertIn('IsOffroad', REGISTERED_KEYS)
self.assertNotIn('IsOnroad', REGISTERED_KEYS)
params = Params()
builder.check_allowed(params, True)
with self.assertRaises(KeyError):
params.get_bool('IsOnroad')
with self.assertRaises(builder.PreparationCancelled):
builder.check_allowed(NS(get_bool=lambda key: False, get=lambda key: nav.BUILD_REF), True)
params.ref, params.onroad = b'queued-other-model', True
with self.assertRaises(builder.PreparationCancelled):
builder.check_allowed(params, True)
class BuildLifecycleTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.params = Params()
self.report = Mock()
self.compiler = self.root / 'compiler.py'
self.compiler.write_text('reviewed source')
self.enterContext(patch.object(builder, 'preflight', return_value=self.compiler))
self.enterContext(patch.object(builder.shutil, 'disk_usage', return_value=NS(free=8 * 1024 ** 3)))
self.enterContext(patch.object(builder, 'download_source'))
def process(self, command, **_kwargs):
Path(command[3]).write_bytes(b'target-plain-model')
returncode = getattr(self, 'returncode', 0)
process = Mock(returncode=returncode)
process.poll.return_value = returncode
process.__enter__ = Mock(return_value=process)
process.__exit__ = Mock(return_value=False)
return process
def run_build(self, validator=None):
validate = validator or (lambda path: {'valid': True} if (Path(path) / nav.MANIFEST_NAME).exists() else None)
with patch.object(builder, 'installed_bundle', side_effect=validate), patch.object(builder.subprocess, 'Popen', side_effect=self.process):
builder.prepare(self.root, self.root, self.params, True, queued=True, report=self.report)
def test_success_registers_validated_artifact_without_selecting_model(self):
self.run_build()
descriptor = json.loads((self.root / nav.MANIFEST_NAME).read_text())
artifact = self.root / descriptor['artifact']['fileName']
self.assertEqual(nav.file_sha256(artifact), descriptor['artifact']['sha256'])
self.assertEqual(descriptor['sourceOnnxSha256'], nav.SOURCE_ONNX_SHA256)
self.assertEqual(self.params.ref, nav.BUILD_REF.encode())
self.assertEqual(self.report.call_args.args[0], 'ready')
def test_compile_failure_or_bad_protocol_never_registers(self):
self.returncode = 1
with self.assertRaises(RuntimeError):
self.run_build()
self.assertFalse((self.root / nav.MANIFEST_NAME).exists())
self.returncode = 0
with self.assertRaises(RuntimeError):
self.run_build(lambda _path: None)
self.assertFalse((self.root / nav.MANIFEST_NAME).exists())
def test_rejected_completed_build_is_preserved_once_without_normal_registration(self):
failed = self.root / builder.FAILED_BUILD_DIRECTORY
for attempt in range(2):
if attempt:
(failed / 'old-build-marker').write_text('must be replaced')
with self.assertRaisesRegex(RuntimeError, 'Completed build and validation details kept'):
self.run_build(lambda _path: None)
descriptor = json.loads((failed / nav.MANIFEST_NAME).read_text())
artifact = failed / descriptor['artifact']['fileName']
self.assertEqual(artifact.read_bytes(), b'target-plain-model')
self.assertEqual(nav.file_sha256(artifact), descriptor['artifact']['sha256'])
diagnostic = json.loads((failed / 'validation_failure.json').read_text())
self.assertFalse(diagnostic['registered'])
self.assertFalse(diagnostic['automaticReuseAllowed'])
self.assertEqual(diagnostic['artifact']['sha256'], descriptor['artifact']['sha256'])
self.assertFalse((failed / 'old-build-marker').exists())
self.assertFalse((self.root / nav.MANIFEST_NAME).exists())
self.assertIsNone(nav.installed_bundle(self.root))
self.assertEqual([path.name for path in self.root.glob('.rl_nav_failed*')], [builder.FAILED_BUILD_DIRECTORY])
def test_onroad_before_publication_never_registers(self):
def process(*args, **kwargs):
result = self.process(*args, **kwargs)
self.params.onroad = True
return result
with patch.object(builder, 'installed_bundle', return_value=None), patch.object(builder.subprocess, 'Popen', side_effect=process):
with self.assertRaises(builder.PreparationCancelled):
builder.prepare(self.root, self.root, self.params, True, queued=True, report=self.report)
self.assertFalse((self.root / nav.MANIFEST_NAME).exists())
def test_low_storage_does_not_download_or_compile(self):
with patch.object(builder.shutil, 'disk_usage', return_value=NS(free=1)), patch.object(builder.subprocess, 'Popen') as process:
with self.assertRaisesRegex(RuntimeError, '4 GiB'):
builder.prepare(self.root, self.root, self.params, True)
builder.download_source.assert_not_called()
process.assert_not_called()
class PrerequisiteTests(unittest.TestCase):
def test_hardware_compiler_usb_and_llvm_fail_before_download(self):
with tempfile.TemporaryDirectory() as directory:
basedir = Path(directory)
compiler = basedir / 'openpilot/sunnypilot/modeld_v2/compile_modeld_v2.py'
compiler.parent.mkdir(parents=True)
compiler.write_text('reviewed compiler stand-in')
with self.assertRaisesRegex(RuntimeError, 'comma device'):
builder.preflight(basedir, False)
with self.assertRaisesRegex(RuntimeError, 'reviewed source'):
builder.preflight(basedir, True)
dll = NS(findlib=Mock(return_value=None))
with patch.object(builder, 'file_sha256', return_value=builder.COMPILER_SHA256), \
patch.dict(sys.modules, {'tinygrad.runtime.support.c': NS(DLL=dll)}):
with self.assertRaisesRegex(RuntimeError, 'libusb'):
builder.preflight(basedir, True)
dll.findlib.side_effect = ['usb.so', None]
with patch.object(builder.ctypes, 'CDLL'), self.assertRaisesRegex(RuntimeError, 'LLVM 18'):
builder.preflight(basedir, True)
def test_runtime_loader_and_actual_llvm_version_are_used(self):
with tempfile.TemporaryDirectory() as directory:
basedir = Path(directory)
compiler = basedir / 'openpilot/sunnypilot/modeld_v2/compile_modeld_v2.py'
compiler.parent.mkdir(parents=True)
compiler.touch()
dll = NS(findlib=Mock(side_effect=['runtime-usb-path', 'runtime-unversioned-llvm-path']))
def version(major, _minor, _patch):
major._obj.value = 18
llvm = NS(LLVMGetVersion=version, LLVMInitializeAMDGPUTarget=lambda: None)
with patch.object(builder, 'file_sha256', return_value=builder.COMPILER_SHA256), \
patch.dict(sys.modules, {'tinygrad.runtime.support.c': NS(DLL=dll)}), \
patch.object(builder.ctypes, 'CDLL', side_effect=[NS(), llvm]):
self.assertEqual(builder.preflight(basedir, True), compiler)
self.assertEqual(dll.findlib.call_args_list[0].args, ('libusb', ['usb-1.0']))
class Progress:
def __init__(self):
self.status, self.progress, self.speed, self.eta, self.canceling = 0, 0, 0, 0, False
class Bundle:
@property
def models(self):
return self._models
@models.setter
def models(self, values):
self._models = [NS(artifact=NS(fileName=value['artifact']['fileName'], downloadProgress=Progress())) for value in values]
def manager_namespace():
# Actual manager methods, with platform transport imports stubbed for Mac.
tree = ast.parse((SOURCE / 'manager.py').read_text())
tree.body = [node for node in tree.body if not isinstance(node, (ast.Import, ast.ImportFrom)) and not isinstance(node, ast.If)]
status = NS(downloading=1, downloaded=2, cached=3, failed=4, verifying=5)
namespace = {'__name__': 'manager_under_test', 'asyncio': asyncio, 'json': json, 'os': os, 'Path': Path,
're': __import__('re'), 'signal': signal, 'subprocess': subprocess, 'sys': sys, 'tempfile': tempfile,
'time': time, 'threading': __import__('threading'), 'requests': requests,
'custom': NS(ModelManagerSP=NS(ModelBundle=Bundle, DownloadStatus=status)),
'BUILD_REF': nav.BUILD_REF, 'MANIFEST_NAME': nav.MANIFEST_NAME, 'installed_bundle': Mock(return_value={'valid': True}),
'Paths': NS(model_root=lambda: ''), 'cloudlog': Mock()}
namespace['ModelFetcher'] = NS(active_source=lambda _present: 'chestnut')
exec(compile(tree, str(SOURCE / 'manager.py'), 'exec'), namespace)
return namespace
class ManagerPreparationTests(unittest.TestCase):
def setUp(self):
self.ns = manager_namespace()
self.manager = self.ns['ModelManagerSP'].__new__(self.ns['ModelManagerSP'])
self.manager.params = Params()
self.manager.chestnut_present = True
self.manager.active_bundle = 'unchanged-active-model'
self.manager.source_models = {}
self.manager.model_fetcher = NS(get_bundles_for_source=Mock(return_value=['new-selectable-local-row']))
self.manager._report_status = Mock()
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.ns['Paths'].model_root = lambda: self.temp.name
self.enterContext(patch.dict(sys.modules, {'openpilot.common.basedir': NS(BASEDIR=self.temp.name)}))
def test_missing_hardware_or_onroad_is_visible_failure_without_worker(self):
for chestnut, onroad in ((False, False), (True, True)):
self.manager.chestnut_present, self.manager.params.onroad = chestnut, onroad
with patch.object(subprocess, 'Popen') as process:
self.manager._prepare_navigation()
process.assert_not_called()
self.assertEqual(self.manager.selected_bundle.status, 4)
self.assertIn('Park', self.manager.selected_bundle.displayName)
def test_startup_exceptions_are_visible_failed_status_through_real_dispatch(self):
for failure in ('model-directory', 'status-directory', 'worker-exec'):
with self.subTest(failure=failure):
store = {'ModelManager_DownloadRef': nav.BUILD_REF.encode()}
params = self.manager.params = Mock()
params.get.side_effect = store.get
params.get_bool.side_effect = Params().get_bool
params.remove.side_effect = lambda key, current=store: current.pop(key, None)
self.manager._download_ref = None
error = OSError(f'fixture {failure} startup failure')
target, method = ((Path, 'mkdir') if failure == 'model-directory' else
(tempfile, 'TemporaryDirectory') if failure == 'status-directory' else (subprocess, 'Popen'))
with patch.object(target, method, side_effect=error):
self.manager._process_download_requests()
self.assertEqual(self.manager.active_bundle, 'unchanged-active-model')
self.assertIsNone(store.get('ModelManager_DownloadRef'))
self.assertEqual(self.manager.selected_bundle.status, 4, 'Startup failure must not leave initial checking forever')
self.assertIn('startup failure', self.manager.selected_bundle.displayName)
def test_success_remains_explicit_selection_and_stops_entire_job_group(self):
process = Mock(pid=99999999, returncode=0)
process.poll.return_value = 0
with patch.object(subprocess, 'Popen', return_value=process) as start, patch.object(os, 'killpg') as kill:
self.manager._prepare_navigation()
self.assertTrue(start.call_args.kwargs['start_new_session'])
self.assertEqual(self.manager.active_bundle, 'unchanged-active-model')
self.assertEqual(self.manager.selected_bundle.status, 2)
self.assertEqual(self.manager.available_models, ['new-selectable-local-row'])
self.assertIn('select', self.manager.selected_bundle.displayName)
kill.assert_any_call(process.pid, signal.SIGTERM)
kill.assert_any_call(process.pid, signal.SIGKILL)
def test_canceled_job_is_failed_and_kills_group(self):
process = Mock(pid=99999999, returncode=0)
process.poll.return_value = None
self.manager.params.ref = None
with patch.object(subprocess, 'Popen', return_value=process), patch.object(os, 'killpg') as kill:
self.manager._prepare_navigation()
self.assertEqual(self.manager.selected_bundle.status, 4)
self.assertIn('canceled', self.manager.selected_bundle.displayName)
self.assertTrue(kill.called)
def test_special_request_dispatches_without_normal_download_or_auto_activation(self):
self.manager.params = Mock()
self.manager.params.get.side_effect = [nav.BUILD_REF.encode(), nav.BUILD_REF.encode(), None]
self.manager._prepare_navigation = Mock()
self.manager.download = Mock()
self.manager._download_ref = None
self.manager._process_download_requests()
self.manager._prepare_navigation.assert_called_once()
self.manager.download.assert_not_called()
self.manager.params.put.assert_not_called()
class NativeSelectorTests(unittest.TestCase):
def test_real_parser_and_verified_local_file_activation_use_no_network(self):
try:
if schema := os.environ.get('NAVIGATION_TEST_SCHEMA'):
import capnp
custom = capnp.load(schema, imports=[os.environ['NAVIGATION_TEST_INCLUDE']])
else:
from openpilot.cereal import custom
except (ImportError, OSError) as error:
self.skipTest(f'Native cereal unavailable: {error}')
fixture = load('navigation_metadata_fixture', SOURCE / 'tests/test_navigation_model.py')
tree = ast.parse((SOURCE / 'fetcher.py').read_text())
parser_namespace = {'custom': custom, 'is_bundle_version_compatible': lambda _value: True}
parser = next(node for node in tree.body if isinstance(node, ast.ClassDef) and node.name == 'ModelParser')
exec(compile(ast.Module(body=[parser], type_ignores=[]), 'actual_model_parser', 'exec'), parser_namespace)
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
artifact = root / 'test_navigation.pkl'
artifact.write_bytes(b'plain-target-stand-in-no-code-execution')
digest = nav.file_sha256(artifact)
(root / nav.MANIFEST_NAME).write_text(json.dumps({'version': 1, 'target': 'chestnut',
'sourceOnnxSha256': nav.SOURCE_ONNX_SHA256, 'artifact': {'fileName': artifact.name, 'sha256': digest}}))
nav._validated.clear()
with patch.dict(sys.modules, {'openpilot.sunnypilot.modeld_v2.helpers': NS(read_parts=lambda _path: ({'run': ({}, b'')}, fixture.metadata()))}):
bundle = parser_namespace['ModelParser']._parse_bundle(nav.installed_bundle(root))
self.assertEqual(bundle.index, 2147483000)
self.assertTrue(bundle.ref.startswith('local-rl-nav-80eedcdd-'))
self.assertEqual(bundle.models[0].artifact.downloadUri.uri, artifact.resolve().as_uri())
namespace = manager_namespace()
helper_path = SOURCE / 'helpers.py'
helper_source = (helper_path.read_text() if helper_path.exists() else subprocess.check_output(
['git', '-C', str(SOURCE.parents[2]), 'show', 'HEAD:openpilot/sunnypilot/models/helpers.py'], text=True))
helper_tree = ast.parse(helper_source)
functions = [node for node in helper_tree.body if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
and node.name in ('verify_file', '_compute_hash')]
helper_namespace = {'hashlib': hashlib}
exec(compile(ast.Module(body=functions, type_ignores=[]), 'actual_file_verifier', 'exec'), helper_namespace)
namespace.update(custom=custom, verify_file=helper_namespace['verify_file'], ACTIVE_BUNDLE_KEYS={'chestnut': 'ModelManager_ActiveBundleChestnut'},
get_active_bundle=Mock(return_value=bundle))
manager = namespace['ModelManagerSP'].__new__(namespace['ModelManagerSP'])
manager.params, manager.chestnut_present = Mock(), True
manager._report_status = Mock()
manager._download_file = Mock(side_effect=AssertionError('Verified local artifact must never attempt file:// HTTP'))
asyncio.run(manager._download_bundle(bundle, directory, 'chestnut'))
manager._download_file.assert_not_called()
manager.params.put.assert_called_once()
self.assertEqual(manager.params.put.call_args.args[0], 'ModelManager_ActiveBundleChestnut')
# Clear cache must retain both parts of the active local installation.
(root / 'unused-model.pkl').write_bytes(b'unused')
namespace.update(get_selected_bundle=Mock(return_value=bundle), installed_bundle=nav.installed_bundle,
Paths=NS(model_root=lambda: directory))
manager.clear_model_cache()
self.assertFalse((root / 'unused-model.pkl').exists())
self.assertTrue(artifact.exists())
self.assertIsNotNone(nav.installed_bundle(root))
namespace['get_selected_bundle'].return_value = None
manager.clear_model_cache()
self.assertFalse(artifact.exists())
self.assertFalse((root / nav.MANIFEST_NAME).exists())
self.assertIsNone(nav.installed_bundle(root))
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,128 @@
"""Actual worker subprocess checks with the device's checkout/module layout.
Only native platform interfaces and unavailable GPU libraries are fixtures. The
worker, navigation descriptor and BASEDIR sources execute unchanged. No network,
model kernels, GPU build or Popen mock is used.
"""
import importlib.util
import os
from pathlib import Path
import runpy
import subprocess
import sys
import tempfile
import time
from types import SimpleNamespace as NS
import unittest
from unittest.mock import Mock, patch
SOURCE = Path(__file__).resolve().parents[1]
spec = importlib.util.spec_from_file_location('preparation_test_support', SOURCE / 'tests/test_navigation_preparation.py')
support = importlib.util.module_from_spec(spec)
spec.loader.exec_module(support)
def tracked_source(relative):
path = SOURCE.parents[2] / relative
return path.read_text() if path.exists() else subprocess.check_output(
['git', '-C', str(SOURCE.parents[2]), 'show', 'HEAD:' + relative], text=True)
class ActualWorkerStartupTests(unittest.TestCase):
def setUp(self):
self.temporary = tempfile.TemporaryDirectory()
self.addCleanup(self.temporary.cleanup)
self.root = Path(self.temporary.name).resolve() / 'checkout'
self.root.mkdir()
self.models = Path(self.temporary.name) / 'models'
self.models.mkdir()
for relative in ('openpilot/sunnypilot/models/build_navigation_model.py',
'openpilot/sunnypilot/models/navigation_model.py',
'openpilot/common/basedir.py',
'openpilot/sunnypilot/modeld_v2/compile_modeld_v2.py'):
self.write(relative, tracked_source(relative))
# Regular packages must win over any editable-install finder in the host
# interpreter. Namespace-only fixtures can accidentally import the workspace.
for package in ('openpilot', 'openpilot/common', 'openpilot/sunnypilot', 'openpilot/sunnypilot/models',
'openpilot/sunnypilot/modeld_v2', 'tinygrad', 'tinygrad/runtime', 'tinygrad/runtime/support'):
self.write(package + '/__init__.py', '')
self.write('openpilot/common/params.py', '''import os
class Params:
def get_bool(self, key):
if key != 'IsOffroad': raise KeyError('UnknownKeyName: ' + key)
return True
def get(self, key):
if key != 'ModelManager_DownloadRef': raise KeyError('UnknownKeyName: ' + key)
return 'local-prepare-rl-navigation-80eedcdd'
''')
self.write('openpilot/common/hardware/__init__.py', '''import os
if os.environ.get('NAV_STARTUP_FIXTURE') == 'import-error':
raise ImportError('fixture-worker-platform-import-error')
COMMA_HARDWARE = True
''')
self.write('openpilot/common/hardware/hw.py', '''import os
class Paths:
@staticmethod
def model_root(): return os.environ['NAV_TEST_MODEL_ROOT']
''')
self.write('tinygrad/runtime/support/c.py', '''class DLL:
@staticmethod
def findlib(_name, _paths): return None
''')
self.write('sitecustomize.py', '''import os, time
if os.environ.get('NAV_STARTUP_FIXTURE') == 'stall-before-worker':
time.sleep(30)
''')
# The production launcher exports checkout-root PYTHONPATH before starting manager.
self.environment = {'PYTHONPATH': str(self.root), 'NAV_TEST_MODEL_ROOT': str(self.models),
'NAV_STARTUP_FIXTURE': 'normal'}
self.basedir = runpy.run_path(str(self.root / 'openpilot/common/basedir.py'))['BASEDIR']
self.assertEqual(Path(self.basedir), self.root)
self.ns = support.manager_namespace()
self.manager = self.ns['ModelManagerSP'].__new__(self.ns['ModelManagerSP'])
self.manager.params = support.Params()
self.manager.chestnut_present = True
self.manager.active_bundle = 'unchanged-active-model'
self.manager.source_models = {}
self.manager.model_fetcher = NS(get_bundles_for_source=Mock(return_value=[]))
self.manager._report_status = Mock()
self.ns['Paths'].model_root = lambda: str(self.models)
self.ns['installed_bundle'].return_value = None
self.enterContext(patch.dict(sys.modules, {'openpilot.common.basedir': NS(BASEDIR=self.basedir)}))
def write(self, relative, text):
path = self.root / relative
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(text)
def run_actual_worker(self, mode='normal'):
self.environment['NAV_STARTUP_FIXTURE'] = mode
with patch.dict(os.environ, self.environment):
self.manager._prepare_navigation()
self.assertEqual(self.manager.active_bundle, 'unchanged-active-model')
self.assertEqual(self.manager.selected_bundle.status, 4)
def test_actual_layout_imports_and_first_worker_status_reach_prerequisite_check(self):
self.run_actual_worker()
logs = '\n'.join(path.read_text() for path in self.models.glob('*.log'))
self.assertIn('libusb', self.manager.selected_bundle.displayName, logs[-4096:])
self.assertNotIn('compiler is missing', self.manager.selected_bundle.displayName)
self.assertFalse((self.models / support.nav.MANIFEST_NAME).exists())
def test_before_status_import_error_is_failed_and_persisted_in_worker_log(self):
self.run_actual_worker('import-error')
logs = '\n'.join(path.read_text() for path in self.models.glob('*.log'))
self.assertIn('fixture-worker-platform-import-error', logs)
self.assertNotIn('Checking navigation model preparation requirements', self.manager.selected_bundle.displayName)
def test_no_first_status_has_bounded_timeout_and_kills_worker(self):
self.ns['NAV_PREPARE_STARTUP_TIMEOUT'] = .3
started = time.monotonic()
self.run_actual_worker('stall-before-worker')
self.assertLess(time.monotonic() - started, 3)
self.assertIn('start', self.manager.selected_bundle.displayName.lower())
self.assertFalse((self.models / support.nav.MANIFEST_NAME).exists())
if __name__ == '__main__':
unittest.main()
@@ -1,67 +0,0 @@
"""
Copyright (c) 2021-, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
import numpy as np
from openpilot.cereal import log
from openpilot.common.params import Params
from openpilot.common.realtime import DT_MDL
from openpilot.sunnypilot import PARAMS_UPDATE_PERIOD
ACCEL_BOOST_MIN_SPEED = 1.0
ACCEL_BOOST_MAX = (0.4, 0.25, 0.1) # aggressive, standard, relaxed, m/s^2
ACCEL_BOOST_BUILD, ACCEL_BOOST_RELEASE = 0.2, 1.0 # m/s^3
BRAKE_PROB = (0.1, 0.3) # the model's brake press in 4 s: it wants to slow
GAS_PROB = (0.6, 0.8) # predicts a human gas press at this hypothesis at about 23%
CURVE_LAT_ACCEL = (1.5, 2.5)
def get_boost_max(personality=log.LongitudinalPersonality.standard):
if personality == log.LongitudinalPersonality.relaxed:
return ACCEL_BOOST_MAX[2]
elif personality == log.LongitudinalPersonality.standard:
return ACCEL_BOOST_MAX[1]
elif personality == log.LongitudinalPersonality.aggressive:
return ACCEL_BOOST_MAX[0]
else:
raise NotImplementedError("Longitudinal personality not supported")
class AccelBoostSP:
def __init__(self):
self.params = Params()
self.frame = -1
self.enabled = False
self.boost = 0.0
self.total_boost = 0.0
def update_params(self):
self.frame += 1
if self.frame % int(PARAMS_UPDATE_PERIOD / DT_MDL) != 0:
return False
enabled = self.params.get_bool("AccelBoostSP")
if enabled == self.enabled:
return False
self.enabled = enabled
self.boost = self.total_boost = 0.0
return True
def update(self, sm, output_a_target_e2e, output_a_target_mpc, a_cruise):
model = sm['modelV2']
brake_probs = model.meta.disengagePredictions.brakePressProbs
gas_probs = model.meta.disengagePredictions.gasPressProbs
if not sm['selfdriveState'].enabled or len(brake_probs) < 2 or len(gas_probs) < 2:
self.boost = self.total_boost = 0.0
return output_a_target_e2e
want = max(float(np.interp(brake_probs[1], BRAKE_PROB, [1.0, 0.0])), float(np.interp(gas_probs[1], GAS_PROB, [0.0, 1.0])))
want = 0.0 if model.action.shouldStop else want
want *= float(np.interp(abs(model.action.desiredCurvature) * sm['carState'].vEgo ** 2, CURVE_LAT_ACCEL, [1.0, 0.0]))
target = want * get_boost_max(sm['selfdriveState'].personality)
self.boost = float(np.clip(target, self.boost - ACCEL_BOOST_RELEASE * DT_MDL, self.boost + ACCEL_BOOST_BUILD * DT_MDL))
speed_scale = np.interp(sm['carState'].vEgo, [ACCEL_BOOST_MIN_SPEED, 2 * ACCEL_BOOST_MIN_SPEED], [0.0, 1.0])
self.total_boost = self.boost * speed_scale * np.interp(output_a_target_e2e, [-0.7, -0.2, 5.0], [0.0, 1.0, 1.0], right=0.0)
return output_a_target_e2e + self.total_boost
@@ -8,38 +8,122 @@ from openpilot.cereal import custom
from openpilot.common.constants import CV
from openpilot.common.params import Params
from openpilot.sunnypilot.selfdrive.controls.lib.nav_turn_intent import NavTurnIntent, turn_route_ready
import time
TurnDirection = custom.ModelDataV2SP.TurnDirection
LANE_CHANGE_SPEED_MIN = 20 * CV.MPH_TO_MS
# "Wait for the turn": the blinker only arms the turn, and the one-shot turn desire is sent when one of these says
# the turn is here. The model treats the desire as "turn now", so sending it early makes it take the first opening.
MODEL_TURN_PROB = 0.35 # model's own prediction that a turn starts within the next horizons
MODEL_TURN_HORIZONS = 2 # desirePrediction horizons checked, soonest first
DESIRE_PRED_WIDTH = 8
WAIT_FOR_TURN = True # always on in this build: prebuilt branches can't add a param for it
def model_turn_probs(desire_prediction) -> tuple[float, float]:
"""Highest predicted (turnLeft, turnRight) probability over the soonest horizons of modelV2.meta.desirePrediction."""
if desire_prediction is None or len(desire_prediction) < DESIRE_PRED_WIDTH * MODEL_TURN_HORIZONS:
return 0., 0.
desire_prediction = list(desire_prediction) # a capnp list can't be sliced
rows = [desire_prediction[h * DESIRE_PRED_WIDTH:(h + 1) * DESIRE_PRED_WIDTH] for h in range(MODEL_TURN_HORIZONS)]
return max(r[1] for r in rows), max(r[2] for r in rows)
class LaneTurnController:
def __init__(self, desire_helper):
self.DH = desire_helper
self.turn_direction = TurnDirection.none
self.armed_direction = TurnDirection.none
self.cue_sent = False
self.params = Params()
self.nav_intent = NavTurnIntent(self.params)
self.navigation_enabled = False
self.navigation_owns_signal = False
self.lane_turn_value = float(self.params.get("LaneTurnValue", return_default=True)) * CV.MPH_TO_MS
self.param_read_counter = 0
self.enabled = self.params.get_bool("LaneTurnDesire")
self.wait_for_turn = WAIT_FOR_TURN
def read_params(self):
self.enabled = self.params.get_bool("LaneTurnDesire")
value = float(self.params.get("LaneTurnValue", return_default=True)) * CV.MPH_TO_MS
self.lane_turn_value = min(float(LANE_CHANGE_SPEED_MIN), value)
self.navigation_enabled = self.params.get_bool("MapboxNextTurnEnabled")
def update_params(self) -> None:
if self.param_read_counter % 50 == 0:
self.read_params()
self.param_read_counter += 1
def update_lane_turn(self, blindspot_left: bool, blindspot_right: bool, left_blinker: bool, right_blinker: bool, v_ego: float) -> None:
if left_blinker and not right_blinker and v_ego < self.lane_turn_value and not blindspot_left:
self.turn_direction = TurnDirection.turnLeft
elif right_blinker and not left_blinker and v_ego < self.lane_turn_value and not blindspot_right:
self.turn_direction = TurnDirection.turnRight
else:
def update_lane_turn(self, blindspot_left: bool, blindspot_right: bool, left_blinker: bool, right_blinker: bool, v_ego: float,
steering_pressed: bool = False, steering_torque: float = 0., turn_probs: tuple[float, float] = (0., 0.),
lateral_active: bool = True, model_position=None, map_turn_supported: bool = False) -> None:
signal = 'left' if left_blinker and not right_blinker else 'right' if right_blinker and not left_blinker else None
self.nav_intent.update(signal, self.navigation_enabled, map_turn_supported, lateral_active)
self.navigation_owns_signal = (self.navigation_enabled and map_turn_supported and
self.nav_intent.mode == 'turn' and signal is not None)
self.turn_blindspot = blindspot_left if signal == 'left' else blindspot_right
# Navigation-conditioned models consume the short route continuously. Sending
# the legacy turn-now desire as well would introduce a second timing source.
if self.navigation_enabled and map_turn_supported:
self.turn_direction = TurnDirection.none
self.armed_direction = TurnDirection.none
self.cue_sent = False
return
if not left_blinker and not right_blinker:
self.armed_direction = TurnDirection.none
self.cue_sent = False
elif left_blinker != right_blinker:
direction = TurnDirection.turnLeft if left_blinker else TurnDirection.turnRight
if direction != self.armed_direction:
self.armed_direction = direction
self.turn_direction = TurnDirection.none
self.cue_sent = False
if left_blinker and not right_blinker and v_ego < self.lane_turn_value and not blindspot_left:
requested = TurnDirection.turnLeft
elif right_blinker and not left_blinker and v_ego < self.lane_turn_value and not blindspot_right:
requested = TurnDirection.turnRight
else:
requested = TurnDirection.none
if not self.wait_for_turn or requested == TurnDirection.none:
self.turn_direction = requested
return
if not self.enabled and self.cue_sent:
self.turn_direction = TurnDirection.none
# Eligibility can clear the request, but only a canceled or switched signal permits another cue.
if self.turn_direction == TurnDirection.none and not self.cue_sent:
ready = self.turn_is_here(requested, v_ego, steering_pressed, steering_torque, turn_probs)
if ready:
self.turn_direction = requested
if self.enabled and self.turn_direction != TurnDirection.none:
self.cue_sent = True
@staticmethod
def turn_is_here(direction, v_ego: float, steering_pressed: bool, steering_torque: float, turn_probs: tuple[float, float]) -> bool:
left = direction == TurnDirection.turnLeft
nudged = steering_pressed and (steering_torque > 0 if left else steering_torque < 0)
model_sees_turn = (turn_probs[0] if left else turn_probs[1]) >= MODEL_TURN_PROB
return nudged or model_sees_turn
def finalize_navigation_intent(self, lane_change_requested, speed, lateral_active, model_supported, lane_change_completed=False):
if not self.navigation_enabled or not model_supported or not lateral_active:
return
if lane_change_requested:
self.nav_intent.commit('lane_change')
else:
if lane_change_completed:
self.nav_intent.resume_after_lane_change()
if (self.nav_intent.mode == 'pending' and 0. <= speed < self.lane_turn_value and not self.turn_blindspot and
turn_route_ready(self.params.get('MapboxTurnContext'), self.nav_intent.session, self.nav_intent.direction, time.monotonic())):
self.nav_intent.commit('turn')
self.navigation_owns_signal = self.nav_intent.mode == 'turn'
def get_turn_direction(self):
if not self.enabled:
@@ -9,8 +9,6 @@ from openpilot.cereal import messaging, custom
from opendbc.car import structs
from openpilot.common.constants import CV
from openpilot.selfdrive.car.cruise import V_CRUISE_MAX
from openpilot.selfdrive.controls.lib.accel_boost import AccelBoost
from openpilot.sunnypilot.selfdrive.controls.lib.accel_boost import AccelBoostSP
from openpilot.sunnypilot.selfdrive.controls.lib.e2e_alerts_helper import E2EAlertsHelper
from openpilot.sunnypilot.selfdrive.controls.lib.smart_cruise_control.smart_cruise_control import SmartCruiseControl
from openpilot.sunnypilot.selfdrive.controls.lib.speed_limit.speed_limit_assist import SpeedLimitAssist
@@ -31,8 +29,6 @@ class LongitudinalPlannerSP:
self.generation = int(model_bundle.generation) if (model_bundle := get_active_bundle()) else None
self.source = LongitudinalPlanSource.cruise
self.e2e_alerts_helper = E2EAlertsHelper()
self.accel_boost_sp = AccelBoostSP()
self.openpilot_longitudinal = CP.openpilotLongitudinalControl
self.output_v_target = 0.
self.output_a_target = 0.
@@ -68,9 +64,6 @@ class LongitudinalPlannerSP:
return self.output_v_target, self.output_a_target
def update(self, sm: messaging.SubMaster) -> None:
if self.accel_boost_sp.update_params():
self.accel_boost = self.accel_boost_sp if self.accel_boost_sp.enabled and self.openpilot_longitudinal else AccelBoost()
self.events_sp.clear()
self.e2e_alerts_helper.update(sm, self.events_sp)
@@ -0,0 +1,118 @@
"""Legacy cue-experiment helpers; live signal navigation uses nav_turn_intent."""
from dataclasses import dataclass
import math
import time
from openpilot.sunnypilot.selfdrive.controls.lib.turn_motion import TurnMotionReadiness, predicted_travel
@dataclass(frozen=True)
class MapTurnDecision:
allowed: bool
reason: str
distance_m: float | None = None
def evaluate_map_turn(context, session, direction, now, speed, lead_m, allow_stationary=False):
"""Check a fresh, session-matched approach. Distance is not curb clearance."""
def reject(reason):
return MapTurnDecision(False, reason)
if not isinstance(context, dict) or not session or direction not in ('left', 'right'):
return reject('no_target')
if context.get('session') != session or context.get('direction') != direction:
return reject('different_signal')
if context.get('status') != 'ready' or context.get('locationValid') is not True or context.get('headingValid') is not True:
return reject('target_unavailable')
names = ('updatedAt', 'gpsAgeSeconds', 'gpsHorizontalAccuracyM', 'crossTrackM', 'distanceM')
values = [context.get(name) for name in names]
if any(isinstance(v, bool) or not isinstance(v, (int, float)) or not math.isfinite(v) for v in (*values, now, speed, lead_m)):
return reject('invalid_measurement')
updated, gps_age, accuracy, cross_track, distance = values
age = now - updated
if not -.05 <= age <= .35 or not 0. <= gps_age <= .5 or max(age, 0.) + gps_age > .5:
return reject('stale_location')
if not 0. < accuracy <= 3. or not 0. <= cross_track <= 6.:
return reject('uncertain_location')
if not 4. <= lead_m <= 30.:
return reject('invalid_lead')
if allow_stationary is not True and speed <= .3:
return reject('not_moving')
# The publisher projects the last GPS fix, not a predicted position. Limit
# extrapolation to the short freshness interval; never extend past junction.
distance -= speed * (max(age, 0.) + gps_age)
if distance <= 0.:
return MapTurnDecision(False, 'junction_passed', distance)
if distance > lead_m:
return MapTurnDecision(False, 'approaching', distance)
return MapTurnDecision(True, 'cue', distance)
class MapTurnGate:
"""Own one intent identifier per physical signal; navd owns map requests."""
def __init__(self, params, clock=time.monotonic, session_clock=time.monotonic_ns, use_motion=True, motion_threshold=1.5):
self.params, self.clock, self.session_clock = params, clock, session_clock
self.enabled = False
self.session = None
self.direction = None
self.lead_m = 20.
self.use_motion = bool(use_motion)
self.motion = TurnMotionReadiness(threshold=motion_threshold)
self.last_mode = None
self.last_decision = MapTurnDecision(False, 'disabled')
# A model-process restart retires the previous process's physical intent.
self.params.remove('MapboxTurnIntent')
def configure(self, enabled, lead_m):
self.enabled = bool(enabled)
self.lead_m = lead_m
def update_signal(self, direction, assist_enabled):
direction = direction if self.enabled and assist_enabled else None
if direction == self.direction:
return
self.direction = direction
self.motion.reset()
self.session = str(self.session_clock()) if direction else None
if direction:
self.params.put('MapboxTurnIntent', {'session': self.session, 'direction': direction})
else:
self.params.remove('MapboxTurnIntent')
self.last_decision = MapTurnDecision(False, 'searching' if direction else 'disabled')
def reset_motion(self):
self.motion.reset()
def ready(self, speed, model_position=None, lateral_active=True):
now = self.clock()
self.last_mode = None
if lateral_active is not True:
self.motion.reset()
self.last_decision = MapTurnDecision(False, 'lateral_inactive')
return False
spatial = evaluate_map_turn(self.params.get('MapboxTurnContext'), self.session,
self.direction, now, speed, self.lead_m, allow_stationary=self.use_motion)
self.last_decision = spatial
if not spatial.allowed:
self.motion.reset()
return False
movement = predicted_travel(model_position, start=2., end=3.) if self.use_motion else None
if self.use_motion and movement is None:
self.motion.reset()
self.last_decision = MapTurnDecision(False, 'model_movement_unavailable', spatial.distance_m)
return False
if not self.use_motion or speed > 1.6:
self.motion.reset()
self.last_mode = 'moving'
return True
if not self.motion.update(self.session, now, True, movement):
self.last_decision = MapTurnDecision(False, 'model_movement_pending', spatial.distance_m)
return False
self.last_mode = 'forecast_movement'
return True
def record_cue(self):
self.params.put('MapboxTurnCue', {'session': self.session, 'direction': self.direction,
'cueAt': self.clock(), 'distanceM': self.last_decision.distance_m, 'leadM': self.lead_m,
'cueMode': self.last_mode, 'forecastTravelM': self.motion.last_motion_m,
'forecastTravelThresholdM': self.motion.threshold, 'forecastIntervalSeconds': [2., 3.]})
@@ -0,0 +1,70 @@
"""Keep a signal's route prepared through completed lane changes and a turn."""
import math
import time
def turn_route_ready(context, session, direction, now):
if not isinstance(context, dict):
return False
updated, generation, distance = context.get('updatedAt'), context.get('routeGeneration'), context.get('distanceM')
return (context.get('session') == session and context.get('direction') == direction and
context.get('status') == 'ready' and context.get('routeOwner') == 'signal' and
context.get('locationValid') is True and context.get('headingValid') is True and
isinstance(generation, int) and not isinstance(generation, bool) and generation > 0 and
isinstance(updated, (int, float)) and not isinstance(updated, bool) and math.isfinite(updated) and
math.isfinite(now) and 0. <= now-updated <= .35 and
isinstance(distance, (int, float)) and not isinstance(distance, bool) and math.isfinite(distance) and distance > 0.)
class NavTurnIntent:
def __init__(self, params, session_clock=time.monotonic_ns):
self.params = params
self.session_clock = session_clock
self.direction = None
self.session = None
self.physical_direction = None
self.mode = None
self._supported, self._active = False, False
self._published = None
self.params.remove('MapboxTurnIntent')
def update(self, direction, enabled, model_supported, lateral_active):
physical = direction if enabled and direction in ('left', 'right') else None
if physical != self.physical_direction:
self.physical_direction = physical
self.mode = 'pending' if physical else None
self._supported, self._active = bool(model_supported), bool(lateral_active)
self._publish()
def commit(self, mode):
if self.mode != 'pending' or self.physical_direction is None or mode not in ('turn', 'lane_change'):
return False
self.mode = mode
self._publish()
return True
def resume_after_lane_change(self):
"""Called only for the helper's normal completion, never for a speed drop."""
if self.mode != 'lane_change' or self.physical_direction is None or not self._active or not self._supported:
return False
self.mode = 'pending'
self._publish()
return True
def _publish(self):
# LC keeps the same session and route warm; its mode never authorizes map
# inputs. Normal completion can then use the route without another lookup.
direction = self.physical_direction if self._active and self.mode in ('pending', 'lane_change', 'turn') else None
if direction != self.direction:
self.direction = direction
self.session = str(self.session_clock()) if direction else None
value = {'session': self.session, 'direction': direction, 'mode': self.mode,
'modelSupportsNavigation': self._supported, 'lateralActive': True} if direction else None
if value != self._published:
if value is None:
self.params.remove('MapboxTurnIntent')
else:
# This event is rare. Persist it before a synchronous cancel/remove can
# race a queued write and resurrect a retired signal session.
self.params.put('MapboxTurnIntent', value, block=True)
self._published = value
@@ -1,81 +0,0 @@
"""
Copyright (c) 2021-, sunnypilot, and a number of other contributors.
This file is part of sunnypilot and is licensed under the MIT License.
See the LICENSE.md file in the root directory for more details.
"""
from types import SimpleNamespace
from openpilot.common.parameterized import parameterized
from openpilot.common.params import Params
from openpilot.common.realtime import DT_MDL
from openpilot.common.test import OpenpilotTestCase
from openpilot.sunnypilot import PARAMS_UPDATE_PERIOD
from openpilot.sunnypilot.selfdrive.controls.lib.accel_boost import AccelBoostSP, ACCEL_BOOST_MAX
def make_sm(brake_prob=0.05, gas_prob=0.3, lat_accel=0.0, v_ego=30.0, personality=0, enabled=True, should_stop=False):
meta = SimpleNamespace(disengagePredictions=SimpleNamespace(brakePressProbs=[0.0, brake_prob, 0.0, 0.0, 0.0, 0.0],
gasPressProbs=[0.0, gas_prob, 0.0, 0.0, 0.0, 0.0]))
action = SimpleNamespace(shouldStop=should_stop, desiredCurvature=lat_accel / v_ego ** 2)
return {'modelV2': SimpleNamespace(meta=meta, action=action), 'carState': SimpleNamespace(vEgo=v_ego),
'selfdriveState': SimpleNamespace(enabled=enabled, personality=personality)}
def settle(sm, frames=100):
boost = AccelBoostSP()
for _ in range(frames):
boost.update(sm, 0.0, 9.0, 9.0)
return boost.total_boost
class TestAccelBoostSP(OpenpilotTestCase):
@parameterized.expand([(0,), (1,), (2,)])
def test_personality_max(self, personality):
assert abs(settle(make_sm(personality=personality)) - ACCEL_BOOST_MAX[personality]) < 1e-6
def test_brake_gate_closes(self):
assert settle(make_sm(brake_prob=0.5)) == 0.0
def test_gas_reopens(self):
assert abs(settle(make_sm(brake_prob=0.5, gas_prob=0.9)) - ACCEL_BOOST_MAX[0]) < 1e-6
def test_should_stop(self):
assert settle(make_sm(gas_prob=0.9, should_stop=True)) == 0.0
@parameterized.expand([(1.5, 1.0), (2.0, 0.5), (2.5, 0.0), (-2.0, 0.5)])
def test_curve_fade(self, lat_accel, scale):
assert abs(settle(make_sm(lat_accel=lat_accel)) - scale * ACCEL_BOOST_MAX[0]) < 1e-6
def test_low_speed(self):
assert settle(make_sm(v_ego=0.5)) == 0.0
def test_disabled_resets(self):
boost = AccelBoostSP()
for _ in range(100):
boost.update(make_sm(), 0.0, 9.0, 9.0)
assert boost.update(make_sm(enabled=False), 0.0, 9.0, 9.0) == 0.0
assert boost.boost == 0.0 and boost.total_boost == 0.0
def test_build_and_release_rates(self):
boost = AccelBoostSP()
boost.update(make_sm(), 0.0, 9.0, 9.0)
assert abs(boost.total_boost - 0.01) < 1e-6
for _ in range(100):
boost.update(make_sm(), 0.0, 9.0, 9.0)
boost.update(make_sm(brake_prob=0.5), 0.0, 9.0, 9.0)
assert abs(boost.total_boost - (ACCEL_BOOST_MAX[0] - 0.05)) < 1e-6
def test_param_switch(self):
params = Params()
boost = AccelBoostSP()
params.put_bool("AccelBoostSP", True, block=True)
assert boost.update_params() and boost.enabled
for _ in range(100):
boost.update(make_sm(), 0.0, 9.0, 9.0)
params.put_bool("AccelBoostSP", False, block=True)
for _ in range(int(PARAMS_UPDATE_PERIOD / DT_MDL) - 1):
assert not boost.update_params()
assert boost.update_params() and not boost.enabled
assert boost.boost == 0.0 and boost.total_boost == 0.0
@@ -4,8 +4,10 @@ from openpilot.common.parameterized import parameterized
from openpilot.common.test import OpenpilotTestCase
from openpilot.selfdrive.controls.lib.desire_helper import DesireHelper, LaneChangeState, LaneChangeDirection
from openpilot.sunnypilot.selfdrive.controls.lib.lane_turn_desire import LaneTurnController, LANE_CHANGE_SPEED_MIN
from openpilot.sunnypilot.selfdrive.controls.lib.lane_turn_desire import LaneTurnController, LANE_CHANGE_SPEED_MIN, MODEL_TURN_PROB, \
DESIRE_PRED_WIDTH, model_turn_probs
from openpilot.sunnypilot.selfdrive.controls.lib.auto_lane_change import AutoLaneChangeMode
from openpilot.sunnypilot.selfdrive.controls.lib import lane_turn_desire
TurnDirection = custom.ModelDataV2SP.TurnDirection
@@ -25,6 +27,7 @@ class TestLaneTurnDesire(OpenpilotTestCase):
dh = DesireHelper()
controller = LaneTurnController(dh)
controller.enabled = True
controller.wait_for_turn = False
controller.lane_turn_value = LANE_CHANGE_SPEED_MIN
controller.turn_direction = TurnDirection.none
controller.update_lane_turn(blindspot_left, blindspot_right, left_blinker, right_blinker, v_ego)
@@ -34,6 +37,7 @@ class TestLaneTurnDesire(OpenpilotTestCase):
dh = DesireHelper()
controller = LaneTurnController(dh)
controller.enabled = False
controller.wait_for_turn = False
controller.lane_turn_value = LANE_CHANGE_SPEED_MIN
controller.turn_direction = TurnDirection.none
controller.update_lane_turn(False, False, True, False, 7)
@@ -43,6 +47,7 @@ class TestLaneTurnDesire(OpenpilotTestCase):
dh = DesireHelper()
controller = LaneTurnController(dh)
controller.enabled = True
controller.wait_for_turn = False
controller.lane_turn_value = LANE_CHANGE_SPEED_MIN
controller.turn_direction = TurnDirection.none
# left turn desire
@@ -64,6 +69,7 @@ class TestLaneTurnDesire(OpenpilotTestCase):
dh = DesireHelper()
controller = LaneTurnController(dh)
controller.enabled = True
controller.wait_for_turn = False
controller.lane_turn_value = LANE_CHANGE_SPEED_MIN
controller.turn_direction = TurnDirection.none
controller.update_lane_turn(False, True, True, False, v_ego)
@@ -84,12 +90,46 @@ class DummyCarState:
def set_lane_turn_params():
lane_turn_desire.WAIT_FOR_TURN = False # these cases cover the immediate turn desire
params = Params()
params.put("LaneTurnDesire", True)
params.put("LaneTurnValue", 20.0)
class TestDesireHelperIntegration(OpenpilotTestCase):
@parameterized.expand([(True,), (False,)], names=["left"])
def test_turn_cue_is_not_repeated_after_lateral_reengagement(self, left, set_lane_turn_params):
dh = DesireHelper()
dh.lane_turn_controller.wait_for_turn = True
carstate = DummyCarState(vEgo=6, leftBlinker=left, rightBlinker=not left)
direction = log.Desire.turnLeft if left else log.Desire.turnRight
prediction = [0.] * (DESIRE_PRED_WIDTH * 4)
prediction[1 if left else 2] = MODEL_TURN_PROB
requests = [log.Desire.none]
def update(active):
dh.update(carstate, active, 1., desire_prediction=prediction)
requests.append(dh.desire)
update(True)
carstate.vEgo = LANE_CHANGE_SPEED_MIN + .1
update(False)
carstate.vEgo = 6
carstate.steeringPressed = True
carstate.steeringTorque = 2.08 if left else -2.08
update(False)
carstate.steeringPressed = False
update(True)
assert requests == [log.Desire.none, direction, log.Desire.none, log.Desire.none, log.Desire.none]
carstate.leftBlinker = carstate.rightBlinker = False
update(True)
carstate.leftBlinker, carstate.rightBlinker = left, not left
update(True)
pulses = sum(current == direction and previous != direction for previous, current in zip(requests, requests[1:], strict=False))
assert pulses == 2
assert requests[-1] == direction
@parameterized.expand([
# Lane turn desire overrides lane change desire
(DummyCarState(vEgo=5, leftBlinker=True, rightBlinker=False, leftBlindspot=False, rightBlindspot=False), True, 1.0,
@@ -166,3 +206,200 @@ class TestDesireHelperIntegration(OpenpilotTestCase):
dh.update(carstate, True, 1.0, left_edge_detected=False, right_edge_detected=True)
assert dh.lane_change_state == LaneChangeState.laneChangeStarting
assert dh.desire == log.Desire.laneChangeLeft
def make_wait_controller():
controller = LaneTurnController(DesireHelper())
controller.enabled = True
controller.wait_for_turn = True
controller.lane_turn_value = LANE_CHANGE_SPEED_MIN
return controller
class TestWaitForTurn(OpenpilotTestCase):
def test_blinker_only_arms(self):
controller = make_wait_controller()
for _ in range(20):
controller.update_lane_turn(False, False, True, False, 6)
assert controller.get_turn_direction() == TurnDirection.none
@parameterized.expand([
("nudge", {"steering_pressed": True, "steering_torque": 1.5}),
("model", {"turn_probs": (MODEL_TURN_PROB, 0.)}),
])
def test_trigger_fires_and_latches(self, _, trigger):
controller = make_wait_controller()
controller.update_lane_turn(False, False, True, False, 6)
assert controller.get_turn_direction() == TurnDirection.none
args = {"v_ego": 6} | trigger
controller.update_lane_turn(False, False, True, False, args.pop('v_ego'), **args)
assert controller.get_turn_direction() == TurnDirection.turnLeft
# stays on after the trigger goes away, so the model sees a single pulse
controller.update_lane_turn(False, False, True, False, 6)
assert controller.get_turn_direction() == TurnDirection.turnLeft
@parameterized.expand([(-0.05,), (0.,), (0.2,), (0.49,)])
def test_stopped_speed_alone_does_not_trigger(self, v_ego):
for left_blinker in (True, False):
controller = make_wait_controller()
for _ in range(20):
controller.update_lane_turn(False, False, left_blinker, not left_blinker, v_ego)
assert controller.get_turn_direction() == TurnDirection.none
@parameterized.expand([
("nudge_wrong_way", {"steering_pressed": True, "steering_torque": 1.5}),
("model_other_side", {"turn_probs": (0.9, 0.)}),
("model_low", {"turn_probs": (0., MODEL_TURN_PROB - 0.01)}),
])
def test_wrong_side_triggers_ignored(self, _, trigger):
controller = make_wait_controller()
for _ in range(5):
controller.update_lane_turn(False, False, False, True, 6, **trigger)
assert controller.get_turn_direction() == TurnDirection.none
def test_blinker_off_rearms(self):
controller = make_wait_controller()
controller.update_lane_turn(False, False, True, False, 0., turn_probs=(MODEL_TURN_PROB, 0.))
assert controller.get_turn_direction() == TurnDirection.turnLeft
controller.update_lane_turn(False, False, False, False, 6)
controller.update_lane_turn(False, False, True, False, 6)
assert controller.get_turn_direction() == TurnDirection.none
@parameterized.expand([(True, True), (True, False), (False, True), (False, False)])
def test_speed_crossing_does_not_send_another_cue(self, left, model_trigger):
controller = make_wait_controller()
direction = TurnDirection.turnLeft if left else TurnDirection.turnRight
probs = (MODEL_TURN_PROB, 0.) if left else (0., MODEL_TURN_PROB)
controller.update_lane_turn(False, False, left, not left, 6, turn_probs=probs)
assert controller.get_turn_direction() == direction
retry = {"turn_probs": probs} if model_trigger else {"steering_pressed": True, "steering_torque": 2.08 if left else -2.08}
for speed in (LANE_CHANGE_SPEED_MIN + .1, LANE_CHANGE_SPEED_MIN, 6, 6):
controller.update_lane_turn(False, False, left, not left, speed, **retry)
assert controller.get_turn_direction() == TurnDirection.none
@parameterized.expand([(True, True), (True, False), (False, True), (False, False)])
def test_blindspot_interruption_does_not_send_another_cue(self, left, model_trigger):
controller = make_wait_controller()
direction = TurnDirection.turnLeft if left else TurnDirection.turnRight
probs = (MODEL_TURN_PROB, 0.) if left else (0., MODEL_TURN_PROB)
controller.update_lane_turn(False, False, left, not left, 6, turn_probs=probs)
assert controller.get_turn_direction() == direction
controller.update_lane_turn(left, not left, left, not left, 6, turn_probs=probs)
assert controller.get_turn_direction() == TurnDirection.none
retry = {"turn_probs": probs} if model_trigger else {"steering_pressed": True, "steering_torque": 2.08 if left else -2.08}
controller.update_lane_turn(False, False, left, not left, 6, **retry)
assert controller.get_turn_direction() == TurnDirection.none
@parameterized.expand([(True, "speed"), (False, "speed"), (True, "blindspot"), (False, "blindspot")])
def test_blocking_before_first_cue_does_not_consume_signal(self, left, block):
controller = make_wait_controller()
probs = (MODEL_TURN_PROB, 0.) if left else (0., MODEL_TURN_PROB)
controller.update_lane_turn(block == "blindspot" and left, block == "blindspot" and not left,
left, not left, LANE_CHANGE_SPEED_MIN if block == "speed" else 6, turn_probs=probs)
assert controller.get_turn_direction() == TurnDirection.none
controller.update_lane_turn(False, False, left, not left, 6)
assert controller.get_turn_direction() == TurnDirection.none
controller.update_lane_turn(False, False, left, not left, 6, turn_probs=probs)
assert controller.get_turn_direction() == (TurnDirection.turnLeft if left else TurnDirection.turnRight)
@parameterized.expand([(True,), (False,)])
def test_canceling_consumed_signal_allows_next_turn(self, left):
controller = make_wait_controller()
direction = TurnDirection.turnLeft if left else TurnDirection.turnRight
probs = (MODEL_TURN_PROB, 0.) if left else (0., MODEL_TURN_PROB)
controller.update_lane_turn(False, False, left, not left, 6, turn_probs=probs)
assert controller.get_turn_direction() == direction
controller.update_lane_turn(False, False, left, not left, LANE_CHANGE_SPEED_MIN)
controller.update_lane_turn(False, False, False, False, 6)
controller.update_lane_turn(False, False, left, not left, 6)
assert controller.get_turn_direction() == TurnDirection.none
controller.update_lane_turn(False, False, left, not left, 6, turn_probs=probs)
assert controller.get_turn_direction() == direction
@parameterized.expand([(True,), (False,)])
def test_switching_signal_side_allows_next_turn(self, left):
controller = make_wait_controller()
probs = (MODEL_TURN_PROB, 0.) if left else (0., MODEL_TURN_PROB)
controller.update_lane_turn(False, False, left, not left, 6, turn_probs=probs)
assert controller.get_turn_direction() == (TurnDirection.turnLeft if left else TurnDirection.turnRight)
controller.update_lane_turn(False, False, not left, left, 6)
assert controller.get_turn_direction() == TurnDirection.none
controller.update_lane_turn(False, False, not left, left, 6, turn_probs=probs[::-1])
assert controller.get_turn_direction() == (TurnDirection.turnRight if left else TurnDirection.turnLeft)
@parameterized.expand([(True,), (False,)])
def test_hazards_do_not_rearm_same_signal(self, left):
controller = make_wait_controller()
probs = (MODEL_TURN_PROB, 0.) if left else (0., MODEL_TURN_PROB)
controller.update_lane_turn(False, False, left, not left, 6, turn_probs=probs)
assert controller.get_turn_direction() == (TurnDirection.turnLeft if left else TurnDirection.turnRight)
controller.update_lane_turn(False, False, True, True, 6, turn_probs=probs)
assert controller.get_turn_direction() == TurnDirection.none
controller.update_lane_turn(False, False, left, not left, 6, turn_probs=probs)
assert controller.get_turn_direction() == TurnDirection.none
def test_disabled_feature_does_not_consume_unemitted_cue(self):
controller = make_wait_controller()
controller.enabled = False
controller.update_lane_turn(False, False, True, False, 6, turn_probs=(MODEL_TURN_PROB, 0.))
assert controller.get_turn_direction() == TurnDirection.none
controller.update_lane_turn(False, False, True, False, LANE_CHANGE_SPEED_MIN)
controller.enabled = True
controller.update_lane_turn(False, False, True, False, 6, turn_probs=(MODEL_TURN_PROB, 0.))
assert controller.get_turn_direction() == TurnDirection.turnLeft
def test_enabling_latched_turn_remembers_emitted_cue(self):
controller = make_wait_controller()
controller.enabled = False
controller.update_lane_turn(False, False, True, False, 6, turn_probs=(MODEL_TURN_PROB, 0.))
controller.enabled = True
controller.update_lane_turn(False, False, True, False, 6)
assert controller.get_turn_direction() == TurnDirection.turnLeft
controller.update_lane_turn(False, False, True, False, LANE_CHANGE_SPEED_MIN)
controller.update_lane_turn(False, False, True, False, 6, turn_probs=(MODEL_TURN_PROB, 0.))
assert controller.get_turn_direction() == TurnDirection.none
def test_feature_toggle_does_not_repeat_emitted_cue(self):
controller = make_wait_controller()
controller.update_lane_turn(False, False, True, False, 6, turn_probs=(MODEL_TURN_PROB, 0.))
assert controller.get_turn_direction() == TurnDirection.turnLeft
controller.enabled = False
controller.update_lane_turn(False, False, True, False, 6)
assert controller.get_turn_direction() == TurnDirection.none
controller.enabled = True
controller.update_lane_turn(False, False, True, False, 6, turn_probs=(MODEL_TURN_PROB, 0.))
assert controller.get_turn_direction() == TurnDirection.none
def test_model_turn_probs(self):
pred = [0.] * (DESIRE_PRED_WIDTH * 4)
pred[DESIRE_PRED_WIDTH + 1] = 0.6 # turnLeft, second horizon
pred[2] = 0.2 # turnRight, first horizon
pred[DESIRE_PRED_WIDTH * 3 + 2] = 0.9 # turnRight, beyond the horizons checked
assert model_turn_probs(pred) == (0.6, 0.2)
assert model_turn_probs(None) == (0., 0.)
def test_model_turn_probs_capnp(self):
# modeld passes modelV2.meta.desirePrediction, a capnp list, which can't be sliced
msg = log.ModelDataV2.new_message()
pred = [0.] * (DESIRE_PRED_WIDTH * 4)
pred[1] = 0.5
msg.meta.desirePrediction = pred
assert model_turn_probs(msg.meta.desirePrediction) == (0.5, 0.)
class TestWaitForTurnDesireHelper(OpenpilotTestCase):
def test_model_prediction_releases_turn(self):
params = Params()
params.put("LaneTurnDesire", True)
lane_turn_desire.WAIT_FOR_TURN = True
params.put("LaneTurnValue", 20.0)
dh = DesireHelper()
carstate = DummyCarState(vEgo=5, leftBlinker=True)
pred = [0.] * (DESIRE_PRED_WIDTH * 4)
for _ in range(10):
dh.update(carstate, True, 0., desire_prediction=pred)
assert dh.desire == log.Desire.none
pred[log.Desire.turnLeft] = 0.9
dh.update(carstate, True, 0., desire_prediction=pred)
assert dh.desire == log.Desire.turnLeft
@@ -0,0 +1,92 @@
"""Map cue/session behavior; no vehicle control or network dependency."""
import importlib.util
from pathlib import Path
import sys
import unittest
ROOT = Path(__file__).resolve().parents[1]
MOTION_SPEC = importlib.util.spec_from_file_location('openpilot.sunnypilot.selfdrive.controls.lib.turn_motion', ROOT/'turn_motion.py')
MOTION = importlib.util.module_from_spec(MOTION_SPEC)
sys.modules[MOTION_SPEC.name] = MOTION
MOTION_SPEC.loader.exec_module(MOTION)
SPEC = importlib.util.spec_from_file_location('map_turn_desire_test_subject', ROOT/'map_turn_desire.py')
MAP = importlib.util.module_from_spec(SPEC)
sys.modules[SPEC.name] = MAP
SPEC.loader.exec_module(MAP)
class FakeParams:
def __init__(self):
self.values = {'LaneTurnDesire': True, 'LaneTurnValue': 19., 'MapboxNextTurnEnabled': False,
'MapboxTurnLeadDistance': 8.}
def get(self, name, return_default=False):
return self.values.get(name)
def get_bool(self, name):
return bool(self.values.get(name))
def put(self, name, value):
self.values[name] = value
def remove(self, name):
self.values.pop(name, None)
def context(session='1', direction='right', **changes):
return {'session': session, 'direction': direction, 'status': 'ready',
'updatedAt': 10., 'gpsAgeSeconds': .1, 'gpsHorizontalAccuracyM': 1.5,
'crossTrackM': 2., 'distanceM': 7., 'locationValid': True, 'headingValid': True, **changes}
class TestMapTurnGate(unittest.TestCase):
def evaluate(self, packet=None, **changes):
args = {'session': '1', 'direction': 'right', 'now': 10.1, 'speed': 3., 'lead_m': 8., **changes}
return MAP.evaluate_map_turn(context() if packet is None else packet, **args)
def test_near_target_and_short_pose_extrapolation(self):
decision = self.evaluate()
self.assertTrue(decision.allowed)
self.assertAlmostEqual(decision.distance_m, 6.4)
self.assertFalse(self.evaluate(context(distanceM=20.)).allowed)
self.assertFalse(self.evaluate(context(distanceM=.5)).allowed)
def test_unavailable_and_wrong_session_fail_closed(self):
for change in ({'status': 'unavailable'}, {'locationValid': False}, {'headingValid': False},
{'direction': 'left'}, {'session': 'old'}):
with self.subTest(change=change):
self.assertFalse(self.evaluate(context(**change)).allowed)
def test_stale_or_uncertain_or_invalid_location_fails_closed(self):
for change in ({'updatedAt': 9.7}, {'updatedAt': 11.}, {'gpsAgeSeconds': .51}, {'gpsAgeSeconds': -.1},
{'gpsHorizontalAccuracyM': 3.1}, {'gpsHorizontalAccuracyM': 0.}, {'crossTrackM': 6.1},
{'distanceM': float('nan')}, {'distanceM': float('inf')}, {'distanceM': True},
{'updatedAt': 9.8, 'gpsAgeSeconds': .3}):
with self.subTest(change=change):
self.assertFalse(self.evaluate(context(**change)).allowed)
def test_standstill_does_not_create_a_cue(self):
self.assertFalse(self.evaluate(speed=0.).allowed)
self.assertFalse(self.evaluate(speed=.3).allowed)
self.assertTrue(self.evaluate(speed=.301).allowed)
def test_cancel_and_opposite_signal_invalidate_old_targets(self):
params = FakeParams()
sessions = iter([1, 2, 3])
gate = MAP.MapTurnGate(params, clock=lambda: 10.1, session_clock=lambda: next(sessions), use_motion=False)
gate.configure(True, 8.)
gate.update_signal('right', True)
params.put('MapboxTurnContext', context())
self.assertTrue(gate.ready(3.))
gate.update_signal(None, True)
self.assertNotIn('MapboxTurnIntent', params.values)
self.assertFalse(gate.ready(3.))
gate.update_signal('right', True)
self.assertEqual(gate.session, '2')
self.assertFalse(gate.ready(3.), 'Same-direction re-signal must reject the previous target')
gate.update_signal('left', True)
self.assertEqual(gate.session, '3')
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,408 @@
"""Exercise source maneuver commitment without native Params or vehicle IO."""
import ast
from enum import IntEnum
import importlib.util
from pathlib import Path
from types import SimpleNamespace
import unittest
SOURCE = Path(__file__).resolve().parents[1]
spec = importlib.util.spec_from_file_location('nav_turn_intent', SOURCE/'nav_turn_intent.py')
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
NOW = 100.
class Params:
def __init__(self):
self.values = {}
self.writes = 0
def put(self, key, value, **kwargs):
self.values[key] = value
self.writes += 1
def remove(self, key):
self.values.pop(key, None)
def get(self, key, **kwargs):
return self.values.get(key, 19. if key == 'LaneTurnValue' else None)
def get_bool(self, key):
return self.values.get(key) is True
class Direction(IntEnum):
none, turnLeft, turnRight = 0, 1, 2
class ChangeState(IntEnum):
off, preLaneChange, laneChangeStarting = 0, 1, 2
class ChangeDirection(IntEnum):
none, left, right = 0, 1, 2
DESIRE = SimpleNamespace(none=0, turnLeft=1, turnRight=2, laneChangeLeft=3, laneChangeRight=4)
def source_namespace(path, namespace):
tree = ast.parse(path.read_text())
tree.body = [node for node in tree.body if not isinstance(node, (ast.Import, ast.ImportFrom))]
exec(compile(tree, str(path), 'exec'), namespace)
return namespace
def controller(dh=None):
namespace = {'custom': SimpleNamespace(ModelDataV2SP=SimpleNamespace(TurnDirection=Direction)),
'CV': SimpleNamespace(MPH_TO_MS=.44704), 'Params': Params, 'NavTurnIntent': module.NavTurnIntent,
'turn_route_ready': module.turn_route_ready, 'time': SimpleNamespace(monotonic=lambda: NOW)}
obj = source_namespace(SOURCE/'lane_turn_desire.py', namespace)['LaneTurnController'](dh)
obj.navigation_enabled, obj.enabled = True, True
return obj
def ready_context(obj, **overrides):
value = {'session': obj.nav_intent.session, 'direction': obj.nav_intent.direction, 'status': 'ready',
'routeOwner': 'signal', 'locationValid': True, 'headingValid': True, 'updatedAt': NOW,
'routeGeneration': 123, 'distanceM': 25.}
value.update(overrides)
return value
class ALC:
"""Only external ALC eligibility is doubled; the actual DesireHelper FSM executes."""
def __init__(self, _dh):
self.lane_change_set_timer = 1
self.auto_lane_change_allowed = True
self.lane_change_blocked = False
def update_params(self):
pass
def update_lane_change(self, blindspot, edge, brake, torque):
self.lane_change_blocked = blindspot or edge
def update_state(self):
pass
def helper(enabled=True, supported=True):
path = SOURCE.parents[3]/'selfdrive/controls/lib/desire_helper.py'
namespace = {'log': SimpleNamespace(LaneChangeState=ChangeState, LaneChangeDirection=ChangeDirection, Desire=DESIRE),
'custom': SimpleNamespace(ModelDataV2SP=SimpleNamespace(TurnDirection=Direction)),
'CV': SimpleNamespace(MPH_TO_MS=.44704), 'DT_MDL': .05, 'AutoLaneChangeController': ALC,
'AutoLaneChangeMode': SimpleNamespace(OFF=0), 'LaneTurnController': controller,
'model_turn_probs': lambda _prediction: (0., 0.)}
dh = source_namespace(path, namespace)['DesireHelper']()
dh.lane_turn_controller.params.values.update(MapboxNextTurnEnabled=enabled, LaneTurnDesire=True, LaneTurnValue=19.)
car = SimpleNamespace(vEgo=12., leftBlinker=False, rightBlinker=True, leftBlindspot=False,
rightBlindspot=False, steeringPressed=False, steeringTorque=0., brakePressed=False)
def step(speed=None, active=True, lane_change_prob=.5, **kwargs):
if speed is not None:
car.vEgo = speed
dh.update(car, active, lane_change_prob, map_turn_supported=supported, **kwargs)
return dh.desire
return dh, car, step
class NavigationSignalTests(unittest.TestCase):
def test_delayed_parameter_writes_cannot_resurrect_canceled_signal(self):
class DeferredParams(Params):
def __init__(self):
super().__init__()
self.pending = []
def put(self, key, value, block=False):
if block:
super().put(key, value)
else:
self.pending.append((key, value))
params = DeferredParams()
intent = module.NavTurnIntent(params)
intent.update('right', True, True, True)
intent.commit('turn')
intent.update(None, True, True, True)
for key, value in params.pending:
params.values[key] = value
self.assertNotIn('MapboxTurnIntent', params.values)
def test_held_signal_does_not_repeat_and_new_signal_gets_new_session(self):
params, counter = Params(), iter(range(100))
intent = module.NavTurnIntent(params, session_clock=lambda: next(counter))
for _ in range(100):
intent.update('right', True, True, True)
self.assertEqual(params.writes, 1)
old = intent.session
intent.update(None, True, True, True)
self.assertNotIn('MapboxTurnIntent', params.values)
intent.update('right', True, True, True)
self.assertNotEqual(intent.session, old)
def test_disable_disengage_or_hazards_retires_published_intent(self):
for args in [('right', False, True, True), ('right', True, True, False), (None, True, True, True)]:
intent = module.NavTurnIntent(Params())
intent.update('right', True, True, True)
intent.update(*args)
self.assertNotIn('MapboxTurnIntent', intent.params.values)
def test_fallback_updates_capability_without_rearming_signal(self):
intent = module.NavTurnIntent(Params())
intent.update('left', True, True, True)
session = intent.session
intent.update('left', True, False, True)
self.assertEqual(intent.session, session)
self.assertIs(intent.params.values['MapboxTurnIntent']['modelSupportsNavigation'], False)
def test_pending_requires_explicit_commit_and_keeps_session(self):
intent = module.NavTurnIntent(Params())
intent.update('right', True, True, True)
session = intent.session
self.assertEqual(intent.mode, 'pending')
self.assertTrue(intent.commit('turn'))
self.assertEqual(intent.session, session)
self.assertEqual(intent.params.values['MapboxTurnIntent']['mode'], 'turn')
self.assertFalse(intent.commit('lane_change'))
def test_lane_change_commit_keeps_prefetch_and_cannot_flip_without_completion(self):
intent = module.NavTurnIntent(Params())
intent.update('right', True, True, True)
self.assertTrue(intent.commit('lane_change'))
self.assertEqual(intent.params.values.get('MapboxTurnIntent', {}).get('mode'), 'lane_change')
intent.update('right', True, True, True)
self.assertFalse(intent.commit('turn'))
intent.update(None, True, True, True)
intent.update('right', True, True, True)
self.assertEqual(intent.mode, 'pending')
def test_direct_opposite_signal_starts_new_pending_session(self):
intent = module.NavTurnIntent(Params())
intent.update('left', True, True, True)
intent.commit('turn')
old = intent.session
intent.update('right', True, True, True)
self.assertEqual(intent.mode, 'pending')
self.assertNotEqual(intent.session, old)
def test_real_navigation_never_also_sends_a_turn_now_desire(self):
obj = controller()
for speed in (0., 2., 7., 12.):
obj.update_lane_turn(False, False, False, True, speed, steering_pressed=True,
steering_torque=-5., turn_probs=(0., 1.), map_turn_supported=True)
self.assertEqual(obj.get_turn_direction(), Direction.none)
self.assertEqual(obj.params.values['MapboxTurnIntent']['mode'], 'pending')
def test_navigation_does_not_depend_on_legacy_turn_desire_toggle(self):
obj = controller()
obj.enabled = False
obj.update_lane_turn(False, False, True, False, 2., map_turn_supported=True)
obj.params.values['MapboxTurnContext'] = ready_context(obj)
obj.finalize_navigation_intent(False, 2., True, True)
self.assertEqual(obj.nav_intent.mode, 'turn')
self.assertEqual(obj.get_turn_direction(), Direction.none)
def test_nonnav_model_keeps_legacy_behavior_and_reports_unsupported(self):
obj = controller()
obj.update_lane_turn(False, False, True, False, 2., turn_probs=(1., 0.), map_turn_supported=False)
self.assertEqual(obj.get_turn_direction(), Direction.turnLeft)
self.assertIs(obj.params.values['MapboxTurnIntent']['modelSupportsNavigation'], False)
obj.finalize_navigation_intent(False, 2., True, False)
self.assertEqual(obj.nav_intent.mode, 'pending')
def test_ready_context_rejects_stale_wrong_session_passed_or_invalid(self):
obj = controller()
obj.update_lane_turn(False, False, False, True, 2., map_turn_supported=True)
for overrides in ({'updatedAt': NOW-.351}, {'updatedAt': NOW+.01}, {'session': 'wrong'}, {'direction': 'left'},
{'distanceM': 0.}, {'distanceM': -1.}, {'distanceM': float('nan')}, {'status': 'passed'},
{'routeOwner': 'destination'}, {'headingValid': False}, {'locationValid': False}, {'routeGeneration': True}):
with self.subTest(overrides=overrides):
obj.params.values['MapboxTurnContext'] = ready_context(obj, **overrides)
obj.finalize_navigation_intent(False, 2., True, True)
self.assertEqual(obj.nav_intent.mode, 'pending')
def test_speed_bsm_and_lateral_permission_block_turn_commit(self):
for speed, bsm, active in ((9., False, True), (2., True, True), (2., False, False)):
obj = controller()
obj.update_lane_turn(False, bsm, False, True, speed, map_turn_supported=True)
obj.params.values['MapboxTurnContext'] = ready_context(obj)
obj.finalize_navigation_intent(False, speed, active, True)
self.assertEqual(obj.nav_intent.mode, 'pending')
def test_early_fast_signal_can_slow_to_turn_before_lc_request(self):
dh, _car, step = helper()
self.assertEqual(step(), DESIRE.none)
obj = dh.lane_turn_controller
session = obj.nav_intent.session
self.assertEqual(dh.lane_change_state, ChangeState.preLaneChange)
self.assertEqual(obj.nav_intent.mode, 'pending')
obj.params.values['MapboxTurnContext'] = ready_context(obj)
self.assertEqual(step(2.), DESIRE.none)
self.assertEqual(obj.nav_intent.mode, 'turn')
self.assertEqual(obj.nav_intent.session, session)
self.assertTrue(obj.navigation_owns_signal)
self.assertEqual(step(12.), DESIRE.none)
self.assertEqual(dh.lane_change_state, ChangeState.off)
def test_actual_lane_change_first_remains_lane_change_after_slowing(self):
dh, _car, step = helper()
step()
obj = dh.lane_turn_controller
obj.params.values['MapboxTurnContext'] = ready_context(obj)
self.assertEqual(step(), DESIRE.laneChangeRight)
self.assertEqual(obj.nav_intent.mode, 'lane_change')
self.assertEqual(obj.params.values.get('MapboxTurnIntent', {}).get('mode'), 'lane_change')
step(2.)
self.assertEqual(obj.nav_intent.mode, 'lane_change')
self.assertFalse(obj.navigation_owns_signal)
def test_normal_lane_change_completion_allows_mapped_turn_with_same_signal_and_route(self):
for left in (False, True):
with self.subTest(left=left):
dh, car, step = helper()
car.leftBlinker, car.rightBlinker = left, not left
step()
obj = dh.lane_turn_controller
session = obj.nav_intent.session
context = ready_context(obj)
obj.params.values['MapboxTurnContext'] = context
self.assertEqual(step(), DESIRE.laneChangeLeft if left else DESIRE.laneChangeRight)
self.assertEqual(obj.nav_intent.mode, 'lane_change')
# Low speed alone cannot terminate an already Starting maneuver.
self.assertEqual(step(2.), DESIRE.laneChangeLeft if left else DESIRE.laneChangeRight)
self.assertFalse(obj.navigation_owns_signal)
dh.lane_change_timer = .5
self.assertEqual(step(lane_change_prob=0.), DESIRE.none)
self.assertEqual(dh.lane_change_state, ChangeState.preLaneChange)
self.assertEqual(obj.nav_intent.mode, 'turn')
self.assertEqual(obj.nav_intent.session, session)
self.assertIs(obj.params.values['MapboxTurnContext'], context)
self.assertTrue(obj.navigation_owns_signal)
def test_active_lane_change_keeps_prefetch_intent_without_turn_authority(self):
dh, _car, step = helper()
step()
obj = dh.lane_turn_controller
session = obj.nav_intent.session
step()
self.assertIn('MapboxTurnIntent', obj.params.values)
self.assertEqual(obj.params.values.get('MapboxTurnIntent', {}).get('mode'), 'lane_change')
self.assertEqual(obj.nav_intent.session, session)
self.assertFalse(obj.navigation_owns_signal)
def test_non_normal_resets_do_not_release_lane_change_lock(self):
for reason in ('timeout', 'disengage', 'alc_disabled', 'low_speed', 'too_early'):
with self.subTest(reason=reason):
dh, _car, step = helper()
step()
obj = dh.lane_turn_controller
obj.params.values['MapboxTurnContext'] = ready_context(obj)
step()
dh.lane_change_timer = 10.01 if reason == 'timeout' else 0. if reason == 'too_early' else .5
if reason == 'alc_disabled':
dh.alc.lane_change_set_timer = 0
step(2., active=reason != 'disengage', lane_change_prob=.5 if reason == 'low_speed' else 0.)
self.assertEqual(obj.nav_intent.mode, 'lane_change')
self.assertFalse(obj.navigation_owns_signal)
if reason == 'disengage':
step(2.)
self.assertEqual(obj.nav_intent.mode, 'lane_change')
def test_completion_after_signal_cancel_or_switch_cannot_use_old_route(self):
for opposite in (False, True):
with self.subTest(opposite=opposite):
dh, car, step = helper()
step()
obj = dh.lane_turn_controller
old_session = obj.nav_intent.session
obj.params.values['MapboxTurnContext'] = ready_context(obj)
step()
dh.lane_change_timer = .5
car.rightBlinker, car.leftBlinker = False, opposite
step(2., lane_change_prob=0.)
self.assertNotEqual(obj.nav_intent.mode, 'turn')
self.assertNotEqual(obj.nav_intent.session, old_session)
self.assertFalse(obj.navigation_owns_signal)
def test_completed_lc_waits_pending_and_new_lc_remains_authoritative(self):
dh, _car, step = helper()
step()
step()
obj = dh.lane_turn_controller
dh.lane_change_timer = .5
step(12., lane_change_prob=0.)
self.assertEqual(obj.nav_intent.mode, 'pending')
self.assertEqual(dh.lane_change_state, ChangeState.preLaneChange)
# Still above turn speed: an actual new LC can start and must win.
step()
self.assertEqual(obj.nav_intent.mode, 'lane_change')
obj.params.values['MapboxTurnContext'] = ready_context(obj)
step(2.)
self.assertEqual(dh.lane_change_state, ChangeState.laneChangeStarting)
self.assertEqual(obj.nav_intent.mode, 'lane_change')
self.assertFalse(obj.navigation_owns_signal)
def test_pre_lane_change_blocked_by_bsm_does_not_commit(self):
dh, car, step = helper()
step()
car.rightBlindspot = True
self.assertEqual(step(), DESIRE.none)
self.assertEqual(dh.lane_turn_controller.nav_intent.mode, 'pending')
car.rightBlindspot = False
dh.lane_turn_controller.params.values['MapboxTurnContext'] = ready_context(dh.lane_turn_controller)
step(2.)
self.assertEqual(dh.lane_turn_controller.nav_intent.mode, 'turn')
def test_waiting_alc_timer_or_detected_edge_does_not_commit_lane_change(self):
for edge in (False, True):
dh, _car, step = helper()
step()
dh.alc.auto_lane_change_allowed = edge
self.assertEqual(step(right_edge_detected=edge), DESIRE.none)
self.assertEqual(dh.lane_turn_controller.nav_intent.mode, 'pending')
def test_left_and_right_turn_commit_requires_matching_route(self):
for left in (False, True):
dh, car, step = helper()
car.leftBlinker, car.rightBlinker = left, not left
step(2.)
obj = dh.lane_turn_controller
obj.params.values['MapboxTurnContext'] = ready_context(obj)
self.assertEqual(step(), DESIRE.none)
self.assertEqual(obj.nav_intent.mode, 'turn')
self.assertEqual(obj.nav_intent.direction, 'left' if left else 'right')
def test_lane_change_request_wins_same_frame_ready_turn_tie(self):
obj = controller()
obj.update_lane_turn(False, False, False, True, 2., map_turn_supported=True)
obj.params.values['MapboxTurnContext'] = ready_context(obj)
obj.finalize_navigation_intent(True, 2., True, True)
self.assertEqual(obj.nav_intent.mode, 'lane_change')
self.assertEqual(obj.params.values.get('MapboxTurnIntent', {}).get('mode'), 'lane_change')
def test_feature_enabled_mid_existing_lane_change_preserves_lane_change(self):
dh, _car, step = helper(enabled=False)
step()
self.assertEqual(step(), DESIRE.laneChangeRight)
obj = dh.lane_turn_controller
obj.navigation_enabled = True
self.assertEqual(step(), DESIRE.laneChangeRight)
self.assertEqual(obj.nav_intent.mode, 'lane_change')
def test_feature_off_and_unsupported_preserve_same_lane_change_sequence(self):
traces = []
for enabled, supported in ((False, True), (True, False)):
dh, car, step = helper(enabled, supported)
trace = []
for speed, signal in ((12., True), (12., True), (2., True), (2., False), (12., True), (12., True)):
car.rightBlinker = signal
trace.append((step(speed), dh.lane_change_state, dh.lane_change_direction))
traces.append(trace)
self.assertEqual(*traces)
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,33 @@
import importlib.util
from pathlib import Path
from types import SimpleNamespace
import unittest
SPEC = importlib.util.spec_from_file_location('turn_motion_test_subject', Path(__file__).resolve().parents[1]/'turn_motion.py')
MODULE = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(MODULE)
class TestTurnMotion(unittest.TestCase):
def test_nonuniform_horizons_interpolate_boundary_points(self):
position = SimpleNamespace(t=[0., .5, 1.5, 3.], x=[0., .5, 1.5, 3.], y=[0., 0., 0., 0.])
self.assertAlmostEqual(MODULE.predicted_travel(position), 1.)
def test_turning_sideways_does_not_look_stationary(self):
position = SimpleNamespace(t=[0., 1., 1.5, 2., 3.], x=[0., 1., 1.5, 1.5, 1.5], y=[0., 0., .5, 1.5, 2.])
self.assertAlmostEqual(MODULE.predicted_travel(position), .5**.5+1.)
def test_stationary_trajectory_has_zero_travel(self):
position = SimpleNamespace(t=[0., 1., 2., 3.], x=[1., 1., 1., 1.], y=[2., 2., 2., 2.])
self.assertEqual(MODULE.predicted_travel(position), 0.)
def test_invalid_or_incomplete_forecast_is_unavailable(self):
for position in (None, SimpleNamespace(t=[0., 1.], x=[0., 1.], y=[0., 0.]),
SimpleNamespace(t=[0., 1., 1., 3.], x=[0.]*4, y=[0.]*4),
SimpleNamespace(t=[0., 1., 2., 3.], x=[0., 0., float('nan'), 1.], y=[0.]*4)):
with self.subTest(position=position):
self.assertIsNone(MODULE.predicted_travel(position))
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,94 @@
"""Temporal forecast permission only; no model inference or vehicle controls."""
import importlib.util
from pathlib import Path
import unittest
PATH = Path(__file__).resolve().parents[1]/'turn_motion.py'
spec = importlib.util.spec_from_file_location('turn_motion_readiness_under_test', PATH)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
TurnMotionReadiness = module.TurnMotionReadiness
class TestTurnMotionReadiness(unittest.TestCase):
def test_requires_elapsed_persistence_with_contiguous_model_frames(self):
readiness = TurnMotionReadiness()
for t in (100., 100.05, 100.1, 100.15):
self.assertFalse(readiness.update('one', t, True, 1.5))
self.assertTrue(readiness.update('one', 100.2, True, 1.5))
self.assertEqual(readiness.since, 100.)
self.assertEqual(readiness.last_motion_m, 1.5)
self.assertTrue(readiness.update('one', 100.25, True, 2.))
def test_single_spike_and_threshold_dropout_cannot_prepare_later(self):
readiness = TurnMotionReadiness()
self.assertFalse(readiness.update('one', 123.45, True, 1.8))
self.assertFalse(readiness.update('one', 123.5, True, 1.49))
self.assertIsNone(readiness.since)
for t in (134.146, 134.196, 134.246, 134.296):
self.assertFalse(readiness.update('one', t, True, 1.8))
self.assertTrue(readiness.update('one', 134.346, True, 1.8))
def test_session_change_and_explicit_reset_retire_prior_history(self):
readiness = TurnMotionReadiness()
for t in (0., .05, .1, .15, .2):
readiness.update('one', t, True, 2.)
self.assertFalse(readiness.update('two', .25, True, 2.))
self.assertEqual(readiness.since, .25)
readiness.reset()
self.assertIsNone(readiness.since)
self.assertIsNone(readiness.last_motion_m)
self.assertIsNone(readiness.session)
self.assertFalse(readiness.update('two', .45, True, 2.))
def test_invalid_map_or_missing_forecast_requires_a_new_full_window(self):
invalids = ((False, 2.), (None, 2.), ('true', 2.), (1, 2.), (True, None),
(True, float('nan')), (True, float('inf')), (True, -.01), (True, True))
for permission, motion in invalids:
with self.subTest(permission=permission, motion=motion):
readiness = TurnMotionReadiness()
for t in (0., .05, .1, .15):
readiness.update('one', t, True, 2.)
self.assertFalse(readiness.update('one', .2, permission, motion))
self.assertIsNone(readiness.since)
self.assertIsNone(readiness.last_motion_m)
self.assertFalse(readiness.update('one', .25, True, 2.))
self.assertEqual(readiness.since, .25)
def test_backward_duplicate_and_missing_native_update_restart_window(self):
for next_time in (.1, .2, .276):
with self.subTest(next_time=next_time):
readiness = TurnMotionReadiness()
for t in (0., .05, .1, .15, .2):
readiness.update('one', t, True, 2.)
self.assertFalse(readiness.update('one', next_time, True, 2.))
self.assertEqual(readiness.since, next_time)
self.assertFalse(readiness.update('one', next_time+.05, True, 2.))
def test_invalid_clock_or_session_cannot_keep_previous_readiness(self):
for session, timestamp in (('one', True), ('one', float('nan')), ('one', float('inf')),
('one', None), ('one', '1.0'), ('one', 10**400), (None, .2), ('', .2), (True, .2)):
with self.subTest(session=session, timestamp=timestamp):
readiness = TurnMotionReadiness()
readiness.update('one', 0., True, 2.)
self.assertFalse(readiness.update(session, timestamp, True, 2.))
self.assertIsNone(readiness.since)
def test_gap_boundary_and_nonuniform_valid_updates_use_time_not_frame_count(self):
readiness = TurnMotionReadiness()
for t in (0., .075, .15):
self.assertFalse(readiness.update('one', t, True, 2.))
self.assertFalse(readiness.update('one', .199, True, 2.))
self.assertTrue(readiness.update('one', .2, True, 2.))
self.assertFalse(readiness.update('one', .276, True, 2.))
def test_config_validation_and_explicit_zero_duration(self):
for kwargs in ({'threshold': 0}, {'threshold': True}, {'threshold': float('nan')},
{'persistence': -1.}, {'persistence': True}, {'persistence': float('inf')}):
with self.subTest(kwargs=kwargs), self.assertRaises(ValueError):
TurnMotionReadiness(**kwargs)
self.assertTrue(TurnMotionReadiness(persistence=0.).update('one', 0., True, 1.5))
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,82 @@
"""Read predicted travel from an existing model trajectory, without altering it."""
from bisect import bisect_right
import math
from numbers import Real
def predicted_travel(position, start=1., end=2.):
"""XY arc length over model time [start,end], or None for invalid input.
Travel follows the planned curve rather than only forward x, so an ordinary
corner does not falsely look stationary. This is a motion forecast, not a
traffic permission or a curb-clearance estimate.
"""
try:
times, xs, ys = list(position.t), list(position.x), list(position.y)
if not 2 <= len(times) == len(xs) == len(ys) <= 256:
return None
if not all(math.isfinite(v) for values in (times, xs, ys) for v in values):
return None
if not times[0] <= start < end <= times[-1] or any(b <= a for a, b in zip(times[:-1], times[1:], strict=True)):
return None
def point(t):
index = min(bisect_right(times, t)-1, len(times)-2)
ratio = (t-times[index])/(times[index+1]-times[index])
return (xs[index]+ratio*(xs[index+1]-xs[index]), ys[index]+ratio*(ys[index+1]-ys[index]))
samples = [point(start), *((x, y) for t, x, y in zip(times, xs, ys, strict=True) if start < t < end), point(end)]
return sum(math.hypot(b[0]-a[0], b[1]-a[1]) for a, b in zip(samples[:-1], samples[1:], strict=True))
except (AttributeError, TypeError, ValueError, IndexError, OverflowError):
return None
class TurnMotionReadiness:
"""Require a continuous forecast predicate within one valid map session.
Pass the current native model frame's timestamp and motion forecast. The
timestamp checks detect missing/out-of-order updates, not geometry reused by
a caller with an invented newer timestamp. This class grants no actuator or
traffic permission and does not inspect current vehicle speed.
"""
MAX_NATIVE_GAP = .075
TIME_TOLERANCE = 1e-9
def __init__(self, threshold=1.5, persistence=.2):
threshold, persistence = self._scalar(threshold), self._scalar(persistence)
if threshold is None or threshold <= 0. or persistence is None or persistence < 0.:
raise ValueError('Motion threshold must be positive and persistence nonnegative; both must be finite numbers')
self.threshold, self.persistence = threshold, persistence
self.reset()
@staticmethod
def _scalar(value):
if isinstance(value, bool) or not isinstance(value, Real):
return None
try:
return float(value) if math.isfinite(value) else None
except (TypeError, ValueError, OverflowError):
return None
def reset(self):
self.session = None
self.last_time = None
self.last_motion_m = None
self.since = None
def update(self, session, now, spatial_allowed, motion_m):
now, motion = self._scalar(now), self._scalar(motion_m)
if not isinstance(session, str) or not session or now is None or spatial_allowed is not True or motion is None or motion < 0.:
self.reset()
return False
if session != self.session:
self.reset()
if self.last_time is not None and (now <= self.last_time or now-self.last_time > self.MAX_NATIVE_GAP+self.TIME_TOLERANCE):
self.since = None
self.session, self.last_time, self.last_motion_m = session, now, motion
if motion < self.threshold:
self.since = None
return False
if self.since is None:
self.since = now
return now-self.since >= self.persistence-self.TIME_TOLERANCE
Binary file not shown.
File diff suppressed because it is too large Load Diff
@@ -10,29 +10,29 @@ void live_update_32(double *in_x, double *in_P, double *in_z, double *in_R, doub
void live_update_13(double *in_x, double *in_P, double *in_z, double *in_R, double *in_ea);
void live_update_14(double *in_x, double *in_P, double *in_z, double *in_R, double *in_ea);
void live_update_33(double *in_x, double *in_P, double *in_z, double *in_R, double *in_ea);
void live_H(double *in_vec, double *out_5667407676628393486);
void live_err_fun(double *nom_x, double *delta_x, double *out_8668485348903095543);
void live_inv_err_fun(double *nom_x, double *true_x, double *out_5202107570088414963);
void live_H_mod_fun(double *state, double *out_6130977875155903016);
void live_f_fun(double *state, double dt, double *out_2092095269018721650);
void live_F_fun(double *state, double dt, double *out_6131485590610354164);
void live_h_4(double *state, double *unused, double *out_9169265721936850834);
void live_H_4(double *state, double *unused, double *out_8583382569498834716);
void live_h_9(double *state, double *unused, double *out_1190771968094934266);
void live_H_9(double *state, double *unused, double *out_1296163634234387246);
void live_h_10(double *state, double *unused, double *out_518897998535379324);
void live_H_10(double *state, double *unused, double *out_3441208344363766057);
void live_h_12(double *state, double *unused, double *out_5953485100376480286);
void live_H_12(double *state, double *unused, double *out_3563926161466872921);
void live_h_35(double *state, double *unused, double *out_4415523909377844667);
void live_H_35(double *state, double *unused, double *out_5216720512126227340);
void live_h_32(double *state, double *unused, double *out_1202921583661580157);
void live_H_32(double *state, double *unused, double *out_584583860081385218);
void live_h_13(double *state, double *unused, double *out_5949023650374730472);
void live_H_13(double *state, double *unused, double *out_8461368207358977669);
void live_h_14(double *state, double *unused, double *out_1190771968094934266);
void live_H_14(double *state, double *unused, double *out_1296163634234387246);
void live_h_33(double *state, double *unused, double *out_4507703079326906254);
void live_H_33(double *state, double *unused, double *out_2066163507487369736);
void live_H(double *in_vec, double *out_2772122295686760629);
void live_err_fun(double *nom_x, double *delta_x, double *out_6423955222691101398);
void live_inv_err_fun(double *nom_x, double *true_x, double *out_719794691562319521);
void live_H_mod_fun(double *state, double *out_126498273849915804);
void live_f_fun(double *state, double dt, double *out_309423444720901128);
void live_F_fun(double *state, double dt, double *out_1032010961160371620);
void live_h_4(double *state, double *unused, double *out_390901647692918293);
void live_H_4(double *state, double *unused, double *out_5503350273553476542);
void live_h_9(double *state, double *unused, double *out_7608991673321319985);
void live_H_9(double *state, double *unused, double *out_5262160626923885897);
void live_h_10(double *state, double *unused, double *out_1303176786126841938);
void live_H_10(double *state, double *unused, double *out_163299894457592615);
void live_h_12(double *state, double *unused, double *out_9079169607037095226);
void live_H_12(double *state, double *unused, double *out_483893865521514747);
void live_h_35(double *state, double *unused, double *out_2930488141765413477);
void live_H_35(double *state, double *unused, double *out_2261669166803498962);
void live_h_32(double *state, double *unused, double *out_2212806671509093227);
void live_H_32(double *state, double *unused, double *out_2079335608929607926);
void live_h_13(double *state, double *unused, double *out_1599053777318742698);
void live_H_13(double *state, double *unused, double *out_4211853044970865788);
void live_h_14(double *state, double *unused, double *out_7608991673321319985);
void live_H_14(double *state, double *unused, double *out_5262160626923885897);
void live_h_33(double *state, double *unused, double *out_4867598165434144742);
void live_H_33(double *state, double *unused, double *out_5412226171442356566);
void live_predict(double *in_x, double *in_P, double *in_Q, double dt);
}
@@ -121,7 +121,7 @@ def ws_recv(ws: WebSocket, end_event: threading.Event) -> None:
if opcode == ABNF.OPCODE_TEXT:
data = data.decode("utf-8")
recv_queue.put_nowait(data)
cloudlog.debug(f"sunnylinkd.ws_recv.recv {data}")
cloudlog.debug("sunnylinkd.ws_recv.received")
elif opcode in (ABNF.OPCODE_PING, ABNF.OPCODE_PONG):
cloudlog.debug("sunnylinkd.ws_recv.pong")
last_ping = int(time.monotonic() * 1e9)
@@ -620,19 +620,6 @@
}
]
},
{
"key": "AccelBoostSP",
"widget": "toggle",
"title": "Acceleration Boost",
"description": "Adds a small push on top of sunnypilot's own acceleration, stronger in Aggressive and gentler in Relaxed. Eases off in curves and when sunnypilot slows or stops. Never goes past your set speed, and lead-car following still comes first.",
"enablement": [
{
"type": "capability",
"field": "has_longitudinal_control",
"equals": true
}
]
},
{
"key": "IntelligentCruiseButtonManagement",
"widget": "toggle",
@@ -30,12 +30,6 @@ sections:
label: Relaxed
enablement:
- $ref: '#/macros/longitudinal'
- key: AccelBoostSP
widget: toggle
title: Acceleration Boost
description: "Adds a small push on top of sunnypilot's own acceleration, stronger in Aggressive and gentler in Relaxed. Eases off in curves and when sunnypilot slows or stops. Never goes past your set speed, and lead-car following still comes first."
enablement:
- $ref: '#/macros/longitudinal'
- key: IntelligentCruiseButtonManagement
widget: toggle
title: Intelligent Cruise Button Management (ICBM) (Alpha)
+31 -1
View File
@@ -224,7 +224,7 @@ def jsonrpc_handler(end_event: threading.Event, localProxyHandler = None) -> Non
data = recv_queue.get(timeout=1)
msg = loads(data)
if is_call(msg):
cloudlog.event("athena.jsonrpc_handler.call_method", data=data)
cloudlog.event("athena.jsonrpc_handler.call_method", method=msg.get("method"))
send_queue_push(handle(msg, dispatcher), SEND_PRIORITY_HIGH)
elif is_response(msg):
log_recv_queue.put_nowait(data)
@@ -707,6 +707,36 @@ def cancelUpload(upload_id: str | list[str]) -> dict[str, int | str]:
cancelled_uploads.update(cancelled_ids)
return {"success": 1}
MAPBOX_TOKEN_PATH = "/data/navigation/mapbox_token"
@dispatcher.add_method
def setMapboxToken(token: str) -> dict[str, int | str]:
if not isinstance(token, str) or len(token) > 4096 or re.fullmatch(r"pk\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+", token) is None:
return {"success": 0, "error": "Invalid public Mapbox token"}
temporary = None
try:
directory = os.path.dirname(MAPBOX_TOKEN_PATH)
os.makedirs(directory, mode=0o700, exist_ok=True)
os.chmod(directory, 0o700)
fd, temporary = tempfile.mkstemp(prefix=".mapbox_token.", dir=directory)
with os.fdopen(fd, "w") as stream:
os.fchmod(stream.fileno(), 0o600)
stream.write(token)
stream.flush()
os.fsync(stream.fileno())
os.replace(temporary, MAPBOX_TOKEN_PATH)
temporary = None
return {"success": 1}
except OSError:
return {"success": 0, "error": "Unable to save Mapbox token"}
finally:
if temporary is not None:
with suppress(OSError):
os.unlink(temporary)
@dispatcher.add_method
def setNavDestination(latitude: float, longitude: float) -> dict[str, int | str]:
if not (-90 <= latitude <= 90 and -180 <= longitude <= 180):
@@ -0,0 +1,120 @@
"""Exercise the credential RPC without loading device hardware dependencies."""
import ast
from contextlib import suppress
from functools import partial
import importlib.util
import json
import os
from pathlib import Path
import queue
import re
import stat
import tempfile
import threading
import time
from types import SimpleNamespace
import unittest
from unittest.mock import Mock, patch
ATHENA = Path(__file__).resolve().parents[1]
ROOT = ATHENA.parents[2]
TOKEN = 'pk.test_payload.test_signature'
def function_from_source(path, name, namespace):
function = next(node for node in ast.parse(path.read_text()).body if isinstance(node, ast.FunctionDef) and node.name == name)
function.decorator_list = []
module = ast.Module(body=[*ast.parse('from __future__ import annotations').body, function], type_ignores=[])
exec(compile(module, str(path), 'exec'), namespace)
return namespace[name]
class TestMapboxTokenRpc(unittest.TestCase):
def setUp(self):
self.directory = tempfile.TemporaryDirectory()
self.addCleanup(self.directory.cleanup)
self.target = Path(self.directory.name) / 'navigation' / 'mapbox_token'
self.namespace = {'os': os, 'tempfile': tempfile, 're': re, 'suppress': suppress, 'MAPBOX_TOKEN_PATH': str(self.target)}
self.setter = function_from_source(ATHENA / 'athenad.py', 'setMapboxToken', self.namespace)
spec = importlib.util.spec_from_file_location('mapbox_rpc_under_test', ATHENA / 'rpc.py')
self.rpc = importlib.util.module_from_spec(spec)
spec.loader.exec_module(self.rpc)
def test_invalid_probe_never_creates_files(self):
for token in ('', 'sk.payload.signature', 'pk.onlyonepart', 'pk.a.b c', 'pk.a.b.c', 'pk.' + 'a' * 4096 + '.b', None, 1):
with self.subTest(token_type=type(token).__name__):
self.assertEqual(self.setter(token), {'success': 0, 'error': 'Invalid public Mapbox token'})
self.assertFalse(self.target.parent.exists())
def test_private_atomic_install_and_replacement(self):
self.assertEqual(self.setter(TOKEN), {'success': 1})
self.assertEqual(self.target.read_text(), TOKEN)
self.assertEqual(stat.S_IMODE(self.target.stat().st_mode), 0o600)
self.assertEqual(stat.S_IMODE(self.target.parent.stat().st_mode), 0o700)
self.assertEqual(self.setter('pk.new_payload.new_signature'), {'success': 1})
self.assertEqual(self.target.read_text(), 'pk.new_payload.new_signature')
self.assertEqual(list(self.target.parent.iterdir()), [self.target])
def test_failed_replace_preserves_previous_token_and_sanitizes_rpc_error(self):
self.setter(TOKEN)
with patch.object(os, 'replace', side_effect=OSError(TOKEN)):
response = self.rpc.handle(
{'jsonrpc': '2.0', 'id': 1, 'method': 'setMapboxToken', 'params': {'token': 'pk.new_payload.new_signature'}}, {'setMapboxToken': self.setter}
)
self.assertEqual(json.loads(response)['result'], {'success': 0, 'error': 'Unable to save Mapbox token'})
self.assertNotIn(TOKEN, response)
self.assertEqual(self.target.read_text(), TOKEN)
self.assertEqual(list(self.target.parent.iterdir()), [self.target])
def test_athena_logs_method_without_credentials(self):
end = threading.Event()
raw = json.dumps({'jsonrpc': '2.0', 'id': 1, 'method': 'setMapboxToken', 'params': {'token': TOKEN}})
incoming = Mock()
incoming.get.side_effect = lambda **kwargs: (end.set(), raw)[1]
logger = Mock()
namespace = {
'threading': threading,
'partial': partial,
'startLocalProxy': Mock(),
'dispatcher': {'setMapboxToken': self.setter},
'recv_queue': incoming,
'loads': self.rpc.loads,
'is_call': self.rpc.is_call,
'is_response': self.rpc.is_response,
'cloudlog': logger,
'send_queue_push': Mock(),
'handle': self.rpc.handle,
'SEND_PRIORITY_HIGH': 0,
'queue': queue,
'json': json,
}
function_from_source(ATHENA / 'athenad.py', 'jsonrpc_handler', namespace)(end)
logger.event.assert_called_once_with('athena.jsonrpc_handler.call_method', method='setMapboxToken')
self.assertNotIn(TOKEN, repr(logger.mock_calls))
self.assertNotIn(TOKEN, namespace['send_queue_push'].call_args.args[0])
def test_sunnylink_receives_credentials_without_logging_them(self):
end = threading.Event()
raw = json.dumps({'method': 'setMapboxToken', 'params': {'token': TOKEN}})
ws = Mock()
ws.recv_data.side_effect = lambda **kwargs: (end.set(), (1, raw.encode()))[1]
logger = Mock()
incoming = queue.Queue()
namespace = {
'time': time,
'threading': threading,
'ABNF': SimpleNamespace(OPCODE_TEXT=1, OPCODE_BINARY=2, OPCODE_PING=9, OPCODE_PONG=10),
'recv_queue': incoming,
'cloudlog': logger,
}
path = ROOT / 'openpilot/sunnypilot/sunnylink/athena/sunnylinkd.py'
function_from_source(path, 'ws_recv', namespace)(ws, end)
self.assertEqual(incoming.get_nowait(), raw)
logger.debug.assert_called_once_with('sunnylinkd.ws_recv.received')
self.assertNotIn(TOKEN, repr(logger.mock_calls))
if __name__ == '__main__':
unittest.main()
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+7
View File
@@ -0,0 +1,7 @@
FROM debian:bookworm-slim@sha256:99c0b25c46d0214d5a2091092e7908907dcaecd6a962cde0ac700acee222b6f3
RUN apt-get update && apt-get install -y --no-install-recommends \
g++=4:12.2.0-3 g++-12=12.2.0-14+deb12u1 gcc-12=12.2.0-14+deb12u1 \
libstdc++-12-dev=12.2.0-14+deb12u1 libc6-dev=2.36-9+deb12u14 \
binutils=2.40-2 python3 ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /build
+34
View File
@@ -0,0 +1,34 @@
This prebuilt release requires rebuilding `openpilot/common/libparams_c.so` when
the compiled Params registry changes. Editing `params_keys.h` alone does not
register a key with the shipped Python API. The Python wrapper still validates
every key through the native registry.
From the checkout root, with Python 3.11+ and a Docker ARM64 Linux builder:
```sh
python3 openpilot/tools/params/rebuild.py --work-dir /tmp/sunnypilot-params-arm64
```
The script copies only the required source into that temporary directory,
verifies the three ARM64 dependency wheels against this checkout's `uv.lock`,
and builds with the pinned Debian image/compiler. The compile and test
containers have networking disabled and no repository mount. The output is
`/tmp/sunnypilot-params-arm64/out/libparams_c.so`; installation is deliberately
separate. `provenance.json`, `out/build.json` and `out/verification.json` record
the inputs, command, binary hash and checks. Existing wheel downloads are
verified and reused. An unavailable pinned package is a build error.
The checked-in `build_provenance.json` describes the shipped feature build.
The previous build preserved all 272 original key names, types and raw defaults
and added five Mapbox keys. The navigation build adds only `NavModelState`;
all 277 previous registry entries were compared against the real ARM64 library
and remain identical. The status JSON clears on manager startup and going offroad. The C API exports and dynamic
library dependencies match the base release, with no increased symbol-version
requirements. The `params_keys_by_flag` correction returns distinct buffers
owned by the handle instead of aliasing one temporary string.
Verification executes all 13 original Params test method bodies, the three
buffer regressions, typed new-key round trips, asynchronous JSON writes, and
both lifecycle clears against the real ARM64 Linux library. Only logging and
the test-directory fixture are substituted. These are Linux container tests;
the library has not been executed on the vehicle device.
@@ -0,0 +1,246 @@
{
"schemaVersion": 1,
"baseCommit": "3b1816ae6c1521b91f98907e2034f7ef1bda8c8b",
"sourceTreeIncludesFeatureChanges": true,
"builder": {
"architecture": "native aarch64 Linux",
"baseImage": "debian:bookworm-slim@sha256:99c0b25c46d0214d5a2091092e7908907dcaecd6a962cde0ac700acee222b6f3",
"testedImageId": "sha256:34f11b746cf5e80960a40db5f5d27ad6d50455f8db1790d14db7db5adf562d4e",
"compiler": "g++ (Debian 12.2.0-14+deb12u1) 12.2.0",
"compileAndTestNetwork": "none",
"repositoryMountedInContainer": false
},
"sources": {
"openpilot/common/params.h": "3c7b87d050c79275a401597dc7d15ae9fa57c2316c816ff5472cbdd84588a6e6",
"openpilot/common/params.cc": "0074bdf766e9bf410f21820b8f730ed9403117c54725b6193176f28a14233737",
"openpilot/common/params_c.cc": "7c3c86301ab3d65aa0eecc3f7b61c3baf4730baf72a9166cbb4de4237419a48d",
"openpilot/common/params_keys.h": "738e4fa928acbd2980a9aac590320f1b8bed4fe1a98ed06db45b890114f37fdc",
"openpilot/common/queue.h": "f7c75c847ae63dfd6e864406e722248540a79908270e85ca2674b43407408fcf",
"openpilot/common/timing.h": "66a580e240fa4fe4af03bc3073ddd0f4db5d1027720e8adfa92ba448f7c8bc71",
"openpilot/common/util.cc": "4479ecf72465e8f453d8af78447f7715f02d9397c58a49048f2bbc87a96d6b8a",
"openpilot/common/util.h": "2a366c25a26a8c11fa0a1c746a00c02cb28546fa027786ee0f2a8a27a39a8896",
"openpilot/common/swaglog.cc": "9c2f88a2f1c3c4253b73defb264cc367a13ade23e02928e1d469b5c5833df176",
"openpilot/common/swaglog.h": "bcc65e78e40b45366f393fea51286b83ec75792e38c44914f89e3912f5a6841b",
"openpilot/common/hardware/hw.h": "defd8c4ae343f3f89573ae3628c3ba0618783fc5f2be128cea51fb0caa550ef5",
"openpilot/common/hardware/base.h": "b7e7fda7bf7deb437421ac590fa67a26b3c52411d085aad3e0caa9f86e86853e",
"openpilot/common/hardware/comma/hardware.h": "10c14aa9e6556e2a5148c3315e2e0e8ffc11feb97f6f33eb21fbefeb6e6eac7f",
"openpilot/common/hardware/pc/hardware.h": "4389449b911037873559375c4ca1faf06ea6909e885db7a597bdeb4c064fcab5",
"openpilot/common/version.h": "7de18e09df11f7f5e55f9d182b869d79ee07c91f3fc5a66d71eac645aa21102d",
"openpilot/sunnypilot/common/version.h": "2be5a00ae12dc260e4b5ab3cbd4c45631a34c3ce9bfc89b39e345b5b9ce037bf",
"openpilot/cereal/gen/cpp/log.capnp.h": "5b9738bee43b2787823ab5dd2d33bcfed201a794068e3003d0b3df09a51f1712",
"openpilot/cereal/gen/cpp/car.capnp.h": "6a7c186812386e191cba79fd88f8a77dd9bd75189c474cfae99344def613585f",
"openpilot/cereal/gen/cpp/custom.capnp.h": "4197fad27d5cd4ccd9dc7085a0e798e4cf4918e7171d1c9aa7111f85345f6532",
"openpilot/cereal/gen/cpp/deprecated.capnp.h": "4158c7af29c10cb7e480f14625f661639fa797efe26e2fab336450e3e85c5fc1",
"openpilot/common/params.py": "c7333c07046b075d99ff343392f25982691b66d57eab037eee55581c8b5ae8c4",
"openpilot/common/tests/test_params.py": "565db228b44eba46898c3887fb2864204ed57c7a1abe5352043b3cc7543efda0",
"openpilot/common/tests/test_params_flag_buffers.py": "ee878cdebf25a9e99d55b3443a0022e007a8b68dd0c95c9cf9d1e2117dcef90d"
},
"uvLockDependencies": [
{
"name": "comma-deps-capnproto",
"version": "1.0.1.post103",
"url": "https://files.pythonhosted.org/packages/c4/53/38b0e046aab3f0d371d0fa6eb3d4eb3cf787e7d270f429a485e53080416c/comma_deps_capnproto-1.0.1.post103-py3-none-manylinux_2_28_aarch64.whl",
"sha256": "4a115a789af5835dde75371abdd0c1766df40764eaccfd912de4212522935d08"
},
{
"name": "comma-deps-json11",
"version": "20170411.0.post103",
"url": "https://files.pythonhosted.org/packages/6c/be/51c3dd8f80ee0a968a90b4737e0362e7b621366c24c9c55613386ba0aa14/comma_deps_json11-20170411.0.post103-py3-none-manylinux_2_28_aarch64.whl",
"sha256": "0f7764609db411d98c540d985c289a2c37f282fbf2a75e2503017f1243546653"
},
{
"name": "comma-deps-zeromq",
"version": "4.3.5.post103",
"url": "https://files.pythonhosted.org/packages/0e/72/522bdc7b93d9c84b2600e9ed84588c94acf698a8aa3186e304c6d54b4371/comma_deps_zeromq-4.3.5.post103-py3-none-manylinux_2_28_aarch64.whl",
"sha256": "d25d7b336707f39a6c92f0b01b40ca761d067564b39c6c56a6a7f4cca7fd512d"
}
],
"rebuildTools": {
"Dockerfile": "c60c96043ea7b6d8ad94ab4e9181033606ae35e4fb43e034ab38b7505d867cff",
"README.md": "65b95ef339e8a0183c7030cf759d042c128c0ba846080fecfb5c19740db1cda7",
"verify.py": "b068578579c1694c7b1594deca244917894ba43d69a80651058280c9a60a1b9f",
"rebuild.py": "62f25ff2e6bd7daa9a635d6a6f49b7fdf03533370f2f75296d77003bed01d765"
},
"build": {
"command": [
"g++",
"-std=c++17",
"-D__COMMA_HARDWARE__",
"-mcpu=cortex-a57",
"-O2",
"-g",
"-fPIC",
"-shared",
"-pthread",
"-I/build/openpilot",
"-I/build",
"-I/build/openpilot/cereal/gen/cpp",
"-I/build/deps/comma_deps_capnproto-1.0.1.post103.data/purelib/capnproto/install/include",
"-I/build/deps/comma_deps_json11-20170411.0.post103.data/purelib/json11/install/include",
"-I/build/deps/comma_deps_zeromq-4.3.5.post103.data/purelib/zeromq/install/include",
"/build/openpilot/common/params_c.cc",
"/build/openpilot/common/params.cc",
"/build/openpilot/common/util.cc",
"/build/openpilot/common/swaglog.cc",
"/build/deps/comma_deps_json11-20170411.0.post103.data/purelib/json11/install/lib/libjson11.a",
"/build/deps/comma_deps_zeromq-4.3.5.post103.data/purelib/zeromq/install/lib/libzmq.a",
"/build/deps/comma_deps_capnproto-1.0.1.post103.data/purelib/capnproto/install/lib/libcapnp.a",
"/build/deps/comma_deps_capnproto-1.0.1.post103.data/purelib/capnproto/install/lib/libkj.a",
"-ldl",
"-lrt",
"-Wl,-z,defs",
"-o",
"/build/out/libparams_c.so"
],
"compiler": "g++ (Debian 12.2.0-14+deb12u1) 12.2.0",
"machine": "aarch64",
"sha256": "4391c55759edb28c41987914e5b02c7fa3df7a4c499a884d67be1f1b16cac511",
"bytes": 4132776
},
"verification": {
"originalTestMethods": [
"test_get_bool",
"test_params_all_keys",
"test_params_default_value",
"test_params_get_block",
"test_params_get_cleared_manager_start",
"test_params_get_type",
"test_params_non_ascii",
"test_params_put_and_get",
"test_params_two_things",
"test_params_unknown_key_fails",
"test_put_bool_non_blocking_with_get_block",
"test_put_non_blocking_with_get_block",
"test_remove_not_there"
],
"originalTestCount": 13,
"flagBufferTestCount": 3,
"newKeysTypedRoundTrip": true,
"lifecycleClears": true,
"asynchronousJson": true,
"maxSymbolVersions": {
"GLIBC": "2.34",
"GLIBCXX": "3.4.30",
"CXXABI": "1.3.13"
},
"needed": [
"libstdc++.so.6",
"libgcc_s.so.1",
"libc.so.6"
],
"cApi": [
"params_check_key",
"params_clear_all",
"params_create",
"params_destroy",
"params_get",
"params_get_bool",
"params_get_default",
"params_get_key_type",
"params_get_path",
"params_key_at",
"params_keys_by_flag",
"params_keys_size",
"params_last_error",
"params_put",
"params_put_bool",
"params_remove"
],
"hardwareExecuted": false
},
"abiComparison": {
"baseline": {
"sha256": "4ae3277dccb111ae65db482f3b04e85735eb215158d41160742aadf2231605b0",
"needed": [
"libstdc++.so.6",
"libgcc_s.so.1",
"libc.so.6"
],
"maxVersions": {
"GLIBC": "2.38",
"GLIBCXX": "3.4.30",
"CXXABI": "1.3.13"
},
"cApi": [
"params_check_key",
"params_clear_all",
"params_create",
"params_destroy",
"params_get",
"params_get_bool",
"params_get_default",
"params_get_key_type",
"params_get_path",
"params_key_at",
"params_keys_by_flag",
"params_keys_size",
"params_last_error",
"params_put",
"params_put_bool",
"params_remove"
]
},
"rebuilt": {
"sha256": "4391c55759edb28c41987914e5b02c7fa3df7a4c499a884d67be1f1b16cac511",
"needed": [
"libstdc++.so.6",
"libgcc_s.so.1",
"libc.so.6"
],
"maxVersions": {
"GLIBC": "2.34",
"GLIBCXX": "3.4.30",
"CXXABI": "1.3.13"
},
"cApi": [
"params_check_key",
"params_clear_all",
"params_create",
"params_destroy",
"params_get",
"params_get_bool",
"params_get_default",
"params_get_key_type",
"params_get_path",
"params_key_at",
"params_keys_by_flag",
"params_keys_size",
"params_last_error",
"params_put",
"params_put_bool",
"params_remove"
]
},
"sameCApi": true,
"noHigherSymbolVersionRequirement": true,
"hardwareExecuted": false
},
"limits": [
"Executed under ARM64 Linux containers, not on the vehicle device.",
"Original Params method bodies use isolated test-directory scaffolding and a logging stub.",
"No model, steering, Tesla, cooperative steering, CAN or safety binaries were rebuilt."
],
"previousRegistryCompatibility": {
"originalKeyCount": 272,
"rebuiltKeyCount": 277,
"exactAddedKeys": [
"MapboxNextTurnEnabled",
"MapboxTurnContext",
"MapboxTurnCue",
"MapboxTurnIntent",
"MapboxTurnLeadDistance"
],
"allOriginalNamesTypesDefaultsUnchanged": true,
"executionArchitecture": "nativeaarch64Linux"
},
"registryCompatibility": {
"originalKeyCount": 277,
"rebuiltKeyCount": 278,
"exactAddedKeys": [
"NavModelState"
],
"allOriginalNamesTypesDefaultsUnchanged": true,
"executionArchitecture": "nativeaarch64Linux",
"baselineCommit": "3b1816ae6c1521b91f98907e2034f7ef1bda8c8b"
}
}
+115
View File
@@ -0,0 +1,115 @@
#!/usr/bin/env python3
"""Rebuild only the release's ARM64 Linux Params C API, in an isolated container."""
import argparse
import hashlib
import json
from pathlib import Path
import shutil
import subprocess
import tomllib
import urllib.request
import zipfile
SOURCE_FILES = [
'openpilot/common/params.h', 'openpilot/common/params.cc', 'openpilot/common/params_c.cc',
'openpilot/common/params_keys.h', 'openpilot/common/queue.h', 'openpilot/common/timing.h',
'openpilot/common/util.cc', 'openpilot/common/util.h', 'openpilot/common/swaglog.cc', 'openpilot/common/swaglog.h',
'openpilot/common/hardware/hw.h', 'openpilot/common/hardware/base.h',
'openpilot/common/hardware/comma/hardware.h', 'openpilot/common/hardware/pc/hardware.h',
'openpilot/common/version.h', 'openpilot/sunnypilot/common/version.h',
'openpilot/cereal/gen/cpp/log.capnp.h', 'openpilot/cereal/gen/cpp/car.capnp.h',
'openpilot/cereal/gen/cpp/custom.capnp.h', 'openpilot/cereal/gen/cpp/deprecated.capnp.h',
'openpilot/common/params.py',
]
PACKAGES = {'comma-deps-capnproto', 'comma-deps-json11', 'comma-deps-zeromq'}
IMAGE = 'sunnypilot-params-arm64:bookworm-gcc12'
def digest(path):
return hashlib.sha256(path.read_bytes()).hexdigest()
def tracked_bytes(repo, name):
path = repo / name
return path.read_bytes() if path.exists() else subprocess.check_output(['git', 'show', f'HEAD:{name}'], cwd=repo)
def prepare(repo, staging):
hashes = {}
for name in SOURCE_FILES + ['openpilot/common/tests/test_params.py', 'openpilot/common/tests/test_params_flag_buffers.py']:
path = staging / name
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(tracked_bytes(repo, name))
hashes[name] = digest(path)
dependencies = []
lock = tomllib.loads(tracked_bytes(repo, 'uv.lock').decode())
for package in lock['package']:
if package['name'] not in PACKAGES:
continue
wheel = next(item for item in package['wheels'] if 'aarch64' in item['url'])
path = staging / 'wheels' / wheel['url'].split('/')[-1]
path.parent.mkdir(exist_ok=True)
if not path.exists():
with urllib.request.urlopen(wheel['url'], timeout=120) as response:
path.write_bytes(response.read())
if digest(path) != wheel['hash'].removeprefix('sha256:'):
raise ValueError(f"Dependency hash mismatch: {package['name']}")
with zipfile.ZipFile(path) as archive:
archive.extractall(staging / 'deps')
dependencies.append({'name': package['name'], 'version': package['version'], 'url': wheel['url'], 'sha256': digest(path)})
for name in ('rebuild.py', 'verify.py'):
shutil.copyfile(Path(__file__).with_name(name), staging / name)
provenance = {'sourceCommit': subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=repo, text=True).strip(),
'files': hashes, 'dependencies': dependencies}
(staging / 'provenance.json').write_text(json.dumps(provenance, indent=2) + '\n')
def compile_library():
root = Path('/build')
deps = root / 'deps'
capnp = next(deps.glob('comma_deps_capnproto-*.data/purelib/capnproto/install'))
json11 = next(deps.glob('comma_deps_json11-*.data/purelib/json11/install'))
zmq = next(deps.glob('comma_deps_zeromq-*.data/purelib/zeromq/install'))
(root / 'out').mkdir(exist_ok=True)
command = ['g++', '-std=c++17', '-D__COMMA_HARDWARE__', '-mcpu=cortex-a57', '-O2', '-g', '-fPIC', '-shared', '-pthread',
'-I' + str(root / 'openpilot'), '-I' + str(root), '-I' + str(root / 'openpilot/cereal/gen/cpp'),
'-I' + str(capnp / 'include'), '-I' + str(json11 / 'include'), '-I' + str(zmq / 'include')]
command += [str(root / 'openpilot/common' / name) for name in ('params_c.cc', 'params.cc', 'util.cc', 'swaglog.cc')]
command += [str(json11 / 'lib/libjson11.a'), str(zmq / 'lib/libzmq.a'), str(capnp / 'lib/libcapnp.a'), str(capnp / 'lib/libkj.a'),
'-ldl', '-lrt', '-Wl,-z,defs', '-o', str(root / 'out/libparams_c.so')]
subprocess.run(command, check=True)
result = {'command': command, 'compiler': subprocess.check_output(['g++', '--version'], text=True).splitlines()[0],
'machine': subprocess.check_output(['uname', '-m'], text=True).strip(),
'sha256': digest(root / 'out/libparams_c.so'), 'bytes': (root / 'out/libparams_c.so').stat().st_size}
(root / 'out/build.json').write_text(json.dumps(result, indent=2) + '\n')
print(json.dumps(result))
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--work-dir', type=Path)
parser.add_argument('--inside', action='store_true', help=argparse.SUPPRESS)
parser.add_argument('--skip-image-build', action='store_true', help='Use an already built, audited builder image')
parser.add_argument('--image', default=IMAGE)
args = parser.parse_args()
if args.inside:
compile_library()
return
if args.work_dir is None:
parser.error('--work-dir is required (outside the checkout)')
repo = Path(__file__).resolve().parents[3]
staging = args.work_dir.resolve()
if staging == repo or repo in staging.parents:
parser.error('--work-dir must be outside the checkout')
staging.mkdir(parents=True, exist_ok=True)
prepare(repo, staging)
if not args.skip_image_build:
subprocess.run(['docker', 'build', '--platform', 'linux/arm64', '-t', args.image, str(Path(__file__).parent)], check=True)
for command in (['python3', '/build/rebuild.py', '--inside'], ['python3', '/build/verify.py']):
subprocess.run(['docker', 'run', '--rm', '--network', 'none', '--cap-drop', 'ALL', '--platform', 'linux/arm64',
'-v', f'{staging}:/build', '-w', '/build', args.image, *command], check=True)
print(f"Verified library: {staging / 'out/libparams_c.so'} (not installed automatically)")
if __name__ == '__main__':
main()
+104
View File
@@ -0,0 +1,104 @@
"""Run the actual Params C/Python API in the isolated ARM64 Linux builder.
Only the project's cloud logger and test-root fixture are replaced. The Params
implementation and original test method bodies are executed unchanged.
"""
import ast
import datetime
import gc
import importlib.util
import json
import os
from pathlib import Path
import re
import shutil
import subprocess
import sys
import tempfile
import threading
import time
import types
import unittest
import uuid
def run():
root = Path('/build')
shutil.copyfile(root / 'openpilot/common/params.py', root / 'out/params.py')
logger = types.ModuleType('openpilot.common.swaglog')
logger.cloudlog = types.SimpleNamespace(warning=lambda *args, **kwargs: None)
sys.modules[logger.__name__] = logger
spec = importlib.util.spec_from_file_location('isolated_native_params', root / 'out/params.py')
native = importlib.util.module_from_spec(spec)
spec.loader.exec_module(native)
namespace = {'__name__': 'original_params_tests', 'datetime': datetime, 'os': os, 'threading': threading,
'time': time, 'uuid': uuid, 'OpenpilotTestCase': unittest.TestCase, 'Params': native.Params,
'ParamKeyFlag': native.ParamKeyFlag, 'UnknownKeyName': native.UnknownKeyName}
tree = ast.parse((root / 'openpilot/common/tests/test_params.py').read_text())
tree.body = [node for node in tree.body if not isinstance(node, (ast.Import, ast.ImportFrom))]
exec(compile(tree, 'original_test_params.py', 'exec'), namespace)
cls = namespace['TestParams']
original_tests = []
for name in sorted(name for name in vars(cls) if name.startswith('test_')):
with tempfile.TemporaryDirectory() as directory:
os.environ['PARAMS_ROOT'], os.environ['OPENPILOT_PREFIX'] = directory, 'native-test'
case = cls(name)
case.setup_method()
getattr(case, name)()
case.params._finalizer()
original_tests.append(name)
gc.collect()
tree = ast.parse((root / 'openpilot/common/tests/test_params_flag_buffers.py').read_text())
tree.body = [node for node in tree.body if not (isinstance(node, ast.ImportFrom) and node.module == 'openpilot.common')]
namespace = {'__name__': 'flag_buffer_tests', 'native': native}
exec(compile(tree, 'test_params_flag_buffers.py', 'exec'), namespace)
result = unittest.TextTestRunner(verbosity=2).run(unittest.defaultTestLoader.loadTestsFromTestCase(namespace['TestParamsFlagBuffers']))
if not result.wasSuccessful():
raise RuntimeError('Native flag-buffer tests failed')
events = ['MapboxTurnIntent', 'MapboxTurnContext', 'MapboxTurnCue']
with tempfile.TemporaryDirectory() as directory:
params = native.Params(directory)
expected = {'MapboxNextTurnEnabled': ('BOOL', False), 'MapboxTurnLeadDistance': ('FLOAT', 20.0),
**dict.fromkeys(events + ['NavModelState'], ('JSON', None))}
assert {key: (params.get_type(key).name, params.get_default_value(key)) for key in expected} == expected
params.put_bool('MapboxNextTurnEnabled', True, block=True)
params.put('MapboxTurnLeadDistance', 20., block=True)
for flag in (native.ParamKeyFlag.CLEAR_ON_MANAGER_START, native.ParamKeyFlag.CLEAR_ON_ONROAD_TRANSITION):
for key in events:
params.put(key, {'session': 'native-test'}, block=True)
assert params.get(key) == {'session': 'native-test'}
params.clear_all(flag)
assert all(params.get(key) is None for key in events)
assert params.get_bool('MapboxNextTurnEnabled') is True
assert params.get('MapboxTurnLeadDistance') == 20.
for flag in (native.ParamKeyFlag.CLEAR_ON_MANAGER_START, native.ParamKeyFlag.CLEAR_ON_OFFROAD_TRANSITION):
params.put('NavModelState', {'supported': True, 'active': False}, block=True)
assert params.get('NavModelState') == {'supported': True, 'active': False}
params.clear_all(flag)
assert params.get('NavModelState') is None
params.put(events[0], {'session': 'async-test'}, block=False)
deadline = time.monotonic() + 2.
while params.get(events[0]) != {'session': 'async-test'} and time.monotonic() < deadline:
time.sleep(.01)
assert params.get(events[0]) == {'session': 'async-test'}
params._finalizer()
library = root / 'out/libparams_c.so'
symbols = subprocess.check_output(['readelf', '--dyn-syms', '--wide', str(library)], text=True)
strings = subprocess.check_output(['strings', str(library)], text=True)
versions = {tag: max(set(re.findall(tag + r'_([0-9.]+)', strings)), key=lambda value: tuple(map(int, value.split('.'))))
for tag in ('GLIBC', 'GLIBCXX', 'CXXABI')}
ceilings = {'GLIBC': '2.38', 'GLIBCXX': '3.4.30', 'CXXABI': '1.3.13'}
assert all(tuple(map(int, versions[tag].split('.'))) <= tuple(map(int, value.split('.'))) for tag, value in ceilings.items())
dynamic = subprocess.check_output(['readelf', '-d', str(library)], text=True)
report = {'originalTestMethods': original_tests, 'originalTestCount': len(original_tests),
'flagBufferTestCount': result.testsRun, 'newKeysTypedRoundTrip': True,
'lifecycleClears': True, 'asynchronousJson': True, 'maxSymbolVersions': versions,
'needed': re.findall(r'\(NEEDED\).*\[([^]]+)\]', dynamic),
'cApi': sorted(set(re.findall(r'\b(params_[a-z_]+)\s*$', symbols, re.M))),
'hardwareExecuted': False}
(root / 'out/verification.json').write_text(json.dumps(report, indent=2) + '\n')
print(json.dumps(report))
if __name__ == '__main__':
run()