models: pin tg to @02e6981 (#2071)

* models: download whole-file artifacts with parallel byte ranges

HuggingFace's Xet CAS throttles each connection to roughly 1-2 MB/s but does
not limit connection count, so the single-stream _download_file left the link
idle most of the time. Replace it with a downloader that probes the size with
a one-byte Range request, creates the file at its final size and streams each
8 MB piece straight to its offset from one of 12 worker threads. The file is
never copied or assembled and only ever occupies its own size on disk. A
server that answers the probe with 200 gets one plain stream instead.
Every request sends Accept-Encoding: identity, because byte ranges count
encoded bytes while requests hands back decoded ones, so a compressed
response would land at the wrong offsets and fail as a short read.

Workers only write bytes; a reporter coroutine on the event loop publishes
progress every 0.5 s (instead of every 128 KB), re-raises the first worker
failure and polls the cancel ref. Transient errors (connection, timeout,
truncated body, HTTP 408/429/500/502/503/504) give a piece up to three
attempts; any other HTTP error fails the download at once. The whole-file
path now also publishes "verifying" while hashing before and after the
transfer, and the download is hashed against the manifest exactly as before.

A cancel has to wait for in-flight pieces to unwind (a blocked socket read
cannot be interrupted), so the row shows "canceling..." through a new
DownloadProgress.canceling flag instead of a frozen bar, and the mici cancel
button hides once the ref is gone, as it already does on the big UI.

DownloadProgress gains speed, canceling and resuming, appended to the capnp so
older readers are unaffected. All three are declared here so the wire format
lands in one place; speed and resuming are filled in by the follow-up speed
and resume changes.

Builds on #2050, which drops chunking and publishes models as whole files;
this makes that whole-file download parallel.

Measured on a comma 3X on WiFi: ~2 MB/s single stream, 17-24 MB/s steady and
30-60 MB/s peak with 12 range connections, the count picked from an on-device
sweep. Disk is not the bottleneck (eMMC writes at 56-67 MB/s with fdatasync).

Tests drive the real downloader against a local threaded range server: piece
tiling, actual parallelism (a stalled piece does not block the others), the
no-range fallback, per-piece retry, 4xx, cancel via ModelManager_DownloadRef
and mid-piece, verify/cached/downloaded state, a probe failure moving
past verifying, and hash mismatch cleanup.

Also drops two chunk-era leftovers in the tests: a stale "no chunks" note on
manifest_bundle and a one-item URL loop in the live manifest check.

Assisted-by: Claude Code (Claude Opus 4.8)
Assisted-by: Claude Code (Claude Opus 5.5)

* remove unneeded comments

* models: trim the manager tests to the regressions they catch

Each remaining test is there because a deliberate break in manager.py,
helpers.py or fetcher.py fails it. Tests that only re-checked what another
already catches are merged into a happy-path download or a table, and
checks of constants rather than behaviour are dropped. The fake server
now starts per test, keeps its state in one dataclass, and can cut a
response short.

Against 66 deliberate breaks, every one the old tests caught still fails,
and a 404 being retried, a short read and its retry, and the display
flags choosing the big model without a GPU are now caught too.

Assisted-by: Claude Code (Claude Opus 5.5)

* Update test_manager_download.py

* Update manager.py

---------

Co-authored-by: James Vecellio-Grant <159560811+Discountchubbs@users.noreply.github.com>

openpilot/cereal/custom.capnp

@@ -138,6 +138,9 @@ struct ModelManagerSP @0xaedffd8f31e7b55d {
     status @0 :DownloadStatus;
     progress @1 :Float32;
     eta @2 :UInt32;
+    speed @3 :Float32;
+    canceling @4 :Bool;
+    resuming @5 :Bool;
   }
 
   struct Chunk {
@@ -465,11 +468,6 @@ struct ModelDataV2SP @0xa1680744031fdb2d {
   laneTurnDirection @0 :TurnDirection;
   leftLaneChangeEdgeBlock @1 :Bool;
   rightLaneChangeEdgeBlock @2 :Bool;
-  lanefulCurvature @3 :Float32;
-  lanefulTrust @4 :Float32;
-  lanefulLaneWidth @5 :Float32;
-  lanefulLeftProb @6 :Float32;
-  lanefulRightProb @7 :Float32;
 
   enum TurnDirection {
     none @0;

openpilot/common/params_keys.h

@@ -256,7 +256,6 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
     {"LagdValueCache", {PERSISTENT, FLOAT, "0.2"}},
     {"LaneTurnDesire", {PERSISTENT | BACKUP, BOOL, "0"}},
     {"LaneTurnValue", {PERSISTENT | BACKUP, FLOAT, "19.0"}},
-    {"Laneful", {PERSISTENT | BACKUP, BOOL, "0"}},
     {"PlanplusControl", {PERSISTENT | BACKUP, FLOAT, "1.0"}},
 
     // mapd

openpilot/selfdrive/ui/mici/onroad/augmented_road_view.py

@@ -206,8 +206,6 @@ class AugmentedRoadView(CameraView):
   def _handle_mouse_release(self, mouse_pos: MousePos):
     # Don't trigger click callback if bookmark was triggered
     if not self._bookmark_icon.interacting():
-      if self._model_renderer.laneful.handle_tap(mouse_pos):
-        return
       super()._handle_mouse_release(mouse_pos)
 
   def _render(self, _):

openpilot/selfdrive/ui/mici/onroad/model_renderer.py

@@ -141,7 +141,6 @@ class ModelRenderer(Widget, ModelRendererSP):
     if ui_state.status != UIStatus.DISENGAGED:
       self._draw_lane_lines()
       self._draw_path()
-      self.laneful.draw(self._rect)
 
     if render_lead_indicator:
       self._update_leads(sm)
@@ -220,7 +219,6 @@ class ModelRenderer(Widget, ModelRendererSP):
     # Update road edges using raw points
     for road_edge in self._road_edges:
       road_edge.projected_points = self._map_line_to_polygon(road_edge.raw_points, line_width_factor, 0.0, max_idx)
-    self.laneful.update(self, max_idx)
 
     # Update path using raw points
     if lead and lead.present:

openpilot/selfdrive/ui/sunnypilot/layouts/settings/models.py

@@ -235,6 +235,8 @@ class ModelsLayout(Widget):
     progress = sum(p.progress for p in progresses) / len(progresses)
     ds = custom.ModelManagerSP.DownloadStatus
 
+    if any(p.canceling for p in progresses):
+      return {"name": name, "downloading": True, "progress": progress, "status_text": tr("canceling...")}
     if ds.failed in statuses:
       # close.png is authored black and a tint cannot lift it, hence close2
       return {"name": name, "status_text": tr("download failed"), "text_color": rl.RED, "icon": "icons/close2.png"}

openpilot/selfdrive/ui/sunnypilot/mici/layouts/laneful.py

@@ -1,21 +0,0 @@
-"""
-Copyright (c) 2021-, sunnypilot, and a number of other contributors.
-
-This file is part of sunnypilot and is licensed under the MIT License.
-See the LICENSE.md file in the root directory for more details.
-"""
-from openpilot.selfdrive.ui.mici.widgets.button import BigParamControl
-from openpilot.system.ui.widgets.scroller import NavScroller
-
-
-class LanefulLayoutMici(NavScroller):
-  def __init__(self):
-    super().__init__()
-    self._toggle = BigParamControl("laneful", "Laneful",
-                                   description="Keeps the car centered in its lane.\n" +
-                                               "Leans on the lane lines when they are clear, drives laneless when they are not.")
-    self._scroller.add_widget(self._toggle)
-
-  def show_event(self):
-    super().show_event()
-    self._toggle.refresh()

openpilot/selfdrive/ui/sunnypilot/mici/layouts/models.py

@@ -251,11 +251,12 @@ class ModelsLayoutMici(NavScroller):
       self.current_model_info.info_text.set_text(tr("download failed"))
 
     elif manager.selectedBundle and manager.selectedBundle.status == custom.ModelManagerSP.DownloadStatus.downloading:
-      self.cancel_download_btn.set_visible(True)
+      self.cancel_download_btn.set_visible(ui_state.params.get("ModelManager_DownloadRef") is not None)
       device.set_override_interactive_timeout(5)
       progress = 0.0
       count = 0
       verifying = False
+      canceling = False
       for model in manager.selectedBundle.models:
         count += 1
         p = model.artifact.downloadProgress
@@ -263,11 +264,13 @@ class ModelsLayoutMici(NavScroller):
                         custom.ModelManagerSP.DownloadStatus.verifying):
           progress += p.progress
           verifying = verifying or p.status == custom.ModelManagerSP.DownloadStatus.verifying
+          canceling = canceling or p.canceling
         elif p.status in (custom.ModelManagerSP.DownloadStatus.downloaded,
                           custom.ModelManagerSP.DownloadStatus.cached):
           progress += 100.0
 
-      self.current_model_info.current_model_header.set_text(tr("verifying") if verifying else tr("downloading"))
+      header = tr("canceling...") if canceling else tr("verifying") if verifying else tr("downloading")
+      self.current_model_info.current_model_header.set_text(header)
       self.cancel_download_btn.set_text(tr("cancel verification") if verifying else tr("cancel download"))
       self.current_model_info.current_model_header._shimmer = True
       name_text = manager.selectedBundle.internalName.lower()

openpilot/selfdrive/ui/sunnypilot/mici/layouts/settings.py

@@ -11,7 +11,6 @@ from openpilot.selfdrive.ui.mici.widgets.button import BigCircleButton
 from openpilot.selfdrive.ui.mici.widgets.dialog import BigConfirmationDialog, BigDialog
 from openpilot.selfdrive.ui.sunnypilot.mici.layouts.sunnylink import SunnylinkLayoutMici
 from openpilot.selfdrive.ui.sunnypilot.mici.layouts.models import ModelsLayoutMici
-from openpilot.selfdrive.ui.sunnypilot.mici.layouts.laneful import LanefulLayoutMici
 from openpilot.selfdrive.ui.ui_state import ui_state
 from openpilot.system.ui.lib.application import gui_app, FontWeight
 from openpilot.system.ui.lib.multilang import tr
@@ -51,10 +50,6 @@ class SettingsLayoutSP(OP.SettingsLayout):
     models_btn = SettingsBigButton(tr("models"), "", gui_app.texture("../../sunnypilot/selfdrive/assets/offroad/icon_models.png", ICON_SIZE, ICON_SIZE))
     models_btn.set_click_callback(lambda: gui_app.push_widget(models_panel))
 
-    laneful_panel = LanefulLayoutMici()
-    laneful_btn = SettingsBigButton(tr("laneful"), "", gui_app.texture("icons_mici/settings/device/lkas.png", ICON_SIZE, ICON_SIZE))
-    laneful_btn.set_click_callback(lambda: gui_app.push_widget(laneful_panel))
-
     # onroad: enable button sits at the front (left of toggles)
     self._enable_offroad_btn_onroad = BigCircleButton(self.icon_offroad_enable, red=True)
     self._enable_offroad_btn_onroad.set_click_callback(lambda: self._handle_always_offroad(True))
@@ -72,8 +67,7 @@ class SettingsLayoutSP(OP.SettingsLayout):
     items = self._scroller._items.copy()
 
     items.insert(1, models_btn)
-    items.insert(2, laneful_btn)
-    items.insert(6, sunnylink_btn)
+    items.insert(5, sunnylink_btn)
 
     # front slots (only one ever visible at a time): exit-always-offroad, then enable-onroad
     items.insert(0, self._enable_offroad_btn_onroad)

openpilot/selfdrive/ui/sunnypilot/mici/onroad/laneful.py

@@ -1,120 +0,0 @@
-"""
-Copyright (c) 2021-, sunnypilot, and a number of other contributors.
-
-This file is part of sunnypilot and is licensed under the MIT License.
-See the LICENSE.md file in the root directory for more details.
-
-Laneful visuals: faint cyan lane-center target, thin white line down the middle of the model path,
-and a top-center readout of the pull, from the curvature, trust, line probs and lane width modeld publishes in modelDataV2SP.
-"""
-import numpy as np
-import pyray as rl
-from openpilot.common.filter_simple import FirstOrderFilter
-from openpilot.common.realtime import DT_MDL
-from openpilot.selfdrive.ui.ui_state import ui_state
-from openpilot.sunnypilot.modeld_v2.laneful import LOOKAHEAD_TIME, MIN_LOOKAHEAD, MAX_ERROR
-from openpilot.system.ui.lib.application import gui_app, FontWeight
-from openpilot.system.ui.lib.shader_polygon import draw_polygon, Gradient
-from openpilot.system.ui.lib.text_measure import measure_text_cached
-
-CENTER_COLOR = (0, 200, 255)
-PATH_COLOR = (255, 255, 255)
-
-
-
-class LanefulRenderer:
-  def __init__(self):
-    self._center_points = np.empty((0, 2), dtype=np.float32)
-    self._path_points = np.empty((0, 2), dtype=np.float32)
-    self._font = gui_app.font(FontWeight.BOLD)
-    # smoothed copies for the readout so the numbers don't flicker
-    self._pull_display = FirstOrderFilter(0.0, 0.5, DT_MDL)
-    self._offset_display = FirstOrderFilter(0.0, 0.5, DT_MDL)
-    self._target_frac = 0.0
-    self._alpha = FirstOrderFilter(0.0, 0.3, DT_MDL)
-    self._readout_rect: rl.Rectangle | None = None
-    self._settings_page = None
-
-  def update(self, renderer, max_idx: int) -> None:
-    """Called from the model renderer's update with its lane lines and path already in raw points."""
-    laneful = ui_state.sm['modelDataV2SP']
-    laneful_curvature = laneful.lanefulCurvature
-    self._alpha.update(float(laneful.lanefulTrust > 0.0))
-
-    left, right = renderer._lane_lines[1].raw_points, renderer._lane_lines[2].raw_points
-    stds = list(ui_state.sm['modelV2'].laneLineStds)
-    if self._alpha.x < 0.01 or len(left) == 0 or len(right) != len(left) or len(stds) < 3:
-      self._center_points = np.empty((0, 2), dtype=np.float32)
-      self._path_points = np.empty((0, 2), dtype=np.float32)
-      return
-
-    lookahead = max(ui_state.sm['carState'].vEgo * LOOKAHEAD_TIME, MIN_LOOKAHEAD)
-    l_var = stds[1] ** 2 / max(laneful.lanefulLeftProb, 1e-4)
-    r_var = stds[2] ** 2 / max(laneful.lanefulRightProb, 1e-4)
-    half_width = laneful.lanefulLaneWidth / 2
-    center_y = (r_var * (left[:, 1] + half_width) + l_var * (right[:, 1] - half_width)) / (l_var + r_var)
-
-    # target and middle of the model path share the path's points, so both end at the same distance
-    path = renderer._path.raw_points
-    center = path.copy()
-    center[:, 1] = np.interp(path[:, 0], left[:, 0], center_y)
-    end_idx = renderer._get_path_length_idx(path[:, 0], left[max_idx, 0])
-    self._center_points = renderer._map_line_to_polygon(center, 0.15, renderer._path_offset_z, end_idx)
-    self._path_points = renderer._map_line_to_polygon(path, 0.15, renderer._path_offset_z, end_idx)
-    self._pull_display.update(laneful_curvature * lookahead ** 2 / 2)  # how far laneful moves the car at the lookahead
-    if laneful.lanefulTrust > 0.0:
-      self._target_frac = self._offset_display.update(float(np.clip(np.interp(lookahead, path[:, 0], center[:, 1] - path[:, 1]) / MAX_ERROR, -1.0, 1.0)))
-
-  def _color(self, rgb, alpha: float) -> rl.Color:
-    return rl.Color(*rgb, int(alpha * self._alpha.x))
-
-  def _fade(self, rgb, alpha: float) -> Gradient:
-    return Gradient(start=(0.0, 1.0), end=(0.0, 0.0), colors=[self._color(rgb, alpha), self._color(rgb, alpha), self._color(rgb, 0)], stops=[0.0, 0.6, 1.0])
-
-  def draw(self, rect: rl.Rectangle) -> None:
-    if self._path_points.size:
-      draw_polygon(rect, self._center_points, gradient=self._fade(CENTER_COLOR, 255))
-      draw_polygon(rect, self._path_points, gradient=self._fade(PATH_COLOR, 255))
-    self._readout_rect = None
-    if self._path_points.size:
-      self._draw_readout(rect)
-
-  def handle_tap(self, pos) -> bool:
-    """Tapping the readout opens the laneful settings page. Returns True if the tap was consumed."""
-    r = self._readout_rect
-    if r is None or not (r.x - 20 <= pos.x <= r.x + r.width + 20 and r.y - 20 <= pos.y <= r.y + r.height + 20):
-      return False
-    if self._settings_page is None:
-      from openpilot.selfdrive.ui.sunnypilot.mici.layouts.laneful import LanefulLayoutMici
-      self._settings_page = LanefulLayoutMici()
-    gui_app.push_widget(self._settings_page)
-    return True
-
-  def _draw_readout(self, rect: rl.Rectangle) -> None:
-    """Top-center pill: pull toward lane center in cm at the lookahead, arrow = side,
-    bar grows from center toward the lane-center target, full at laneful's 1 m cap."""
-    pull = self._pull_display.x
-    w, h = 190, 76
-    x, y = rect.x + (rect.width - w) / 2, rect.y + 10
-    cx, cy = x + w / 2, y + 30
-    self._readout_rect = rl.Rectangle(x, y, w, h)
-    rl.draw_rectangle_rounded(self._readout_rect, 0.5, 12, self._color((0, 0, 0), 170))
-
-    text = f"{abs(pull) * 100:.0f}"
-    text_w = measure_text_cached(self._font, text, 44, 0).x
-    rl.draw_text_ex(self._font, text, rl.Vector2(cx - text_w / 2, cy - 24), 44, 0, self._color((255, 255, 255), 255))
-
-    active = self._color(CENTER_COLOR, np.clip(abs(pull) / 0.1, 0.0, 1.0) * 255)
-    idle = self._color((255, 255, 255), 40)
-    lx, rx = x + 18, x + w - 18
-    rl.draw_triangle(rl.Vector2(lx, cy), rl.Vector2(lx + 16, cy + 12), rl.Vector2(lx + 16, cy - 12), active if pull < 0 else idle)
-    rl.draw_triangle(rl.Vector2(rx - 16, cy - 12), rl.Vector2(rx - 16, cy + 12), rl.Vector2(rx, cy), active if pull > 0 else idle)
-
-    track = rl.Rectangle(x + 24, y + h - 16, w - 48, 6)
-    rl.draw_rectangle_rounded(track, 1.0, 6, self._color((255, 255, 255), 40))
-    rl.draw_rectangle(int(cx - 1), int(track.y - 2), 2, int(track.height + 4), self._color((255, 255, 255), 120))
-    fill = track.width / 2 * abs(self._target_frac)
-    if fill > 1:
-      fill_x = cx if self._target_frac > 0 else cx - fill
-      rl.draw_rectangle_rounded(rl.Rectangle(fill_x, track.y, fill, track.height), 1.0, 6, self._color(CENTER_COLOR, 255))
-

openpilot/selfdrive/ui/sunnypilot/mici/onroad/model_renderer.py

@@ -6,7 +6,6 @@ See the LICENSE.md file in the root directory for more details.
 """
 import pyray as rl
 from openpilot.selfdrive.ui.ui_state import UIStatus
-from openpilot.selfdrive.ui.sunnypilot.mici.onroad.laneful import LanefulRenderer
 from openpilot.selfdrive.ui.sunnypilot.onroad.rainbow_path import RainbowPath
 
 LANE_LINE_COLORS_SP = {
@@ -18,4 +17,3 @@ LANE_LINE_COLORS_SP = {
 class ModelRendererSP:
   def __init__(self):
     self.rainbow_path = RainbowPath()
-    self.laneful = LanefulRenderer()

openpilot/selfdrive/ui/sunnypilot/ui_state.py

@@ -36,7 +36,7 @@ class UIStateSP:
     self.is_sp_release: bool = self.params.get_bool("IsReleaseSpBranch")
     self.sm_services_ext = [
       "modelManagerSP", "selfdriveStateSP", "longitudinalPlanSP", "backupManagerSP",
-      "lateralTorqueParameters", "carStateSP", "liveMapDataSP", "carParamsSP", "lateralDelay", "modelDataV2SP"
+      "lateralTorqueParameters", "carStateSP", "liveMapDataSP", "carParamsSP", "lateralDelay"
     ]
 
     self.sunnylink_state = SunnylinkState()

openpilot/sunnypilot/modeld_v2/laneful.py

@@ -1,121 +0,0 @@
-"""
-Copyright (c) 2021-, Haibin Wen, sunnypilot, and a number of other contributors.
-
-This file is part of sunnypilot and is licensed under the MIT License.
-See the LICENSE.md file in the root directory for more details.
-"""
-import numpy as np
-
-from openpilot.common.filter_simple import FirstOrderFilter
-from openpilot.common.realtime import DT_MDL
-from openpilot.sunnypilot.modeld_v2.constants import ModelConstants
-
-LANE_WIDTH_DEFAULT = 3.6  # m (reseeded to model's lane width upon initialization)
-LANE_WIDTH_CLIP = (2.8, 4.5)  # m
-LOOKAHEAD_TIME = 1.5  # s
-MIN_LOOKAHEAD = 10.0  # m
-MIN_SPEED = 5.0  # m/s
-LINE_STD = (0.2, 0.6)
-POINT_STD = 0.1
-MAX_ERROR = 1.0
-YIELD = (0.2, 0.45)
-X_IDXS = np.array(ModelConstants.X_IDXS)
-
-
-class Laneful: # TODO-SP: Rename this to blendful or something similar because it is not pure laneful anymore.
-  def __init__(self):
-    self.lane_width = FirstOrderFilter(LANE_WIDTH_DEFAULT, 10.0, DT_MDL, initialized=False)
-    self.line_gone = FirstOrderFilter(np.zeros(2), 2.0, DT_MDL)
-    self.line_probs_slow = FirstOrderFilter(np.zeros(2), 3.0, DT_MDL)
-    self.line_probs = FirstOrderFilter(np.zeros(2), 0.5, DT_MDL)
-    self.usual_offset = FirstOrderFilter(0.0, 10.0, DT_MDL, initialized=False)
-    self.going_around = False
-    self.trust = 0.0
-    self.curvature = 0.0
-    self.l_prob = 0.0
-    self.r_prob = 0.0
-
-  def update(self, model_output: dict[str, np.ndarray], v_ego: float, active: bool, model_curvature: float) -> float:
-    self.l_prob = self.r_prob = 0.0
-    fading = not active or v_ego < MIN_SPEED
-    if fading:
-      self.trust = 0.0
-      self.usual_offset.initialized = False
-      self.lane_width.initialized = False
-      self.going_around = False
-    target = 0.0 if fading else self._target(model_output, v_ego, model_curvature)
-    releasing = not fading and (abs(target) < abs(self.curvature) or target * self.curvature < 0)
-    step = (2.0 if releasing else 0.5) / max(v_ego, 1.0) ** 2 * DT_MDL  # jerk over 1 second
-    self.curvature = float(np.clip(target, self.curvature - step, self.curvature + step))
-    return self.curvature
-
-  def _target(self, model_output: dict[str, np.ndarray], v_ego: float, model_curvature: float) -> float:
-    if v_ego < MIN_SPEED or 'lane_lines' not in model_output:
-      return 0.0
-
-    lookahead = max(v_ego * LOOKAHEAD_TIME, MIN_LOOKAHEAD)
-    lines = model_output['lane_lines'][0, :, :, 0]
-    stds = model_output['lane_lines_stds'][0, :, :, 0]
-
-    probs = self.line_probs.update(model_output['lane_lines_prob'][0, 1::2][1:3])
-    gone = self.line_gone.update((model_output['lane_lines_prob'][0, 1::2][1:3] < 0.3).astype(float))
-    dropping = bool(np.any(probs < 0.6 * self.line_probs_slow.update(model_output['lane_lines_prob'][0, 1::2][1:3])))
-    l_prob = float(probs[0] * np.interp(np.interp(0.5 * lookahead, X_IDXS, stds[1]), LINE_STD, [1.0, 0.0]))
-    r_prob = float(probs[1] * np.interp(np.interp(0.5 * lookahead, X_IDXS, stds[2]), LINE_STD, [1.0, 0.0]))
-    # one line steadily on its own still centers at half say conf. a line that is disappearing gives control back to laneless
-    if gone[1] > 0.8 and l_prob > 0.5:
-      lane_trust = l_prob / 2
-    elif gone[0] > 0.8 and r_prob > 0.5:
-      lane_trust = r_prob / 2
-    elif dropping:
-      lane_trust = min(l_prob, r_prob)
-    else:
-      lane_trust = l_prob + r_prob - l_prob * r_prob
-    self.l_prob, self.r_prob = l_prob, r_prob
-    if lane_trust <= 0.0:
-      self.trust = 0.0
-      return 0.0
-
-    win = (X_IDXS >= 0.5 * lookahead) & (X_IDXS <= 1.5 * lookahead)
-    x = X_IDXS[win]
-    half_width = self.lane_width.x / 2
-    if l_prob * r_prob > 0.5:
-      width = np.interp(lookahead, X_IDXS, lines[2]) - np.interp(lookahead, X_IDXS, lines[1])
-      self.lane_width.update(float(np.clip(width, *LANE_WIDTH_CLIP)))
-      half_width = np.clip(lines[2, win] - lines[1, win], *LANE_WIDTH_CLIP) / 2
-    l_var = stds[1, win] ** 2 / max(l_prob, 1e-4)
-    r_var = stds[2, win] ** 2 / max(r_prob, 1e-4)
-    center = (r_var * (lines[1, win] + half_width) + l_var * (lines[2, win] - half_width)) / (l_var + r_var)
-    weight = 1.0 / (l_var * r_var / (l_var + r_var) + POINT_STD ** 2)
-
-    # Fit curvature to place the plant in center: e ~ dk x^2 / 2
-    plan_x = np.interp(x, model_output['plan'][0, :, 0], model_output['plan'][0, :, 1])
-    plan_share = np.interp(abs(model_curvature) * lookahead ** 2 / 2, [0.5, 2.0], [0.3, 0.5]) # blending up to 50% plan with action during sharp curvature
-    error = np.clip(center - ((1 - plan_share) * model_curvature * x ** 2 / 2 + plan_share * plan_x), -1.0, 1.0)
-    dk = 2.0 * np.sum(weight * error * x ** 2) / np.sum(weight * x ** 4)
-
-    # A model planning to end up further from its usual spot in the lane than the plant is now is likely going around something
-    hw_now = np.clip(lines[2, 0] - lines[1, 0], *LANE_WIDTH_CLIP) / 2 if l_prob * r_prob > 0.5 else self.lane_width.x / 2
-    center_now = (l_prob * (lines[1, 0] + hw_now) + r_prob * (lines[2, 0] - hw_now)) / (l_prob + r_prob)
-    plan_y = np.interp(lookahead, model_output['plan'][0, :, 0], model_output['plan'][0, :, 1])
-    offset = plan_y - np.interp(lookahead, x, center)
-    usual = self.usual_offset.x if self.usual_offset.initialized else offset
-    swerve = abs(offset - usual) - 0.25 * abs(model_curvature) * lookahead ** 2 / 2
-    away = swerve - abs(-center_now - usual)
-    lane_curvature = 2.0 * np.polyfit(x, center, 2)[0]
-    inside_curve = abs(lane_curvature) * lookahead ** 2 / 2 > 0.1 and (offset - usual) * lane_curvature > 0
-    if inside_curve:
-      self.going_around = False
-    elif away > YIELD[0]:
-      self.going_around = True
-    elif swerve < YIELD[0] / 2:
-      self.going_around = False
-    self.usual_offset.update_alpha(20.0 if self.going_around else 10.0)
-    self.usual_offset.update(offset)
-    if inside_curve:
-      lane_trust *= np.interp(abs(offset) - abs(center_now) - 0.25 * abs(model_curvature) * lookahead ** 2 / 2, (0.5, 1.0), [1.0, 0.0])
-    else:
-      lane_trust *= np.interp(swerve, (YIELD[0] / 2, YIELD[0]), [1.0, 0.0]) if self.going_around else np.interp(away, YIELD, [1.0, 0.0])
-
-    self.trust = lane_trust if lane_trust < self.trust else self.trust + (lane_trust - self.trust) * DT_MDL / (1.5 + DT_MDL)
-    return float(self.trust * dk)

openpilot/sunnypilot/modeld_v2/modeld.py

@@ -7,7 +7,6 @@ See the LICENSE.md file in the root directory for more details.
 """
 
 from collections.abc import Callable
-from functools import partial
 import gc
 import os
 os.environ['GMMU'] = '0'
@@ -40,7 +39,6 @@ from openpilot.selfdrive.modeld.modeld import ChestnutGpuState
 
 from openpilot.sunnypilot.modeld_v2.fill_model_msg import fill_model_msg, fill_pose_msg, PublishState, get_curvature_from_output
 from openpilot.sunnypilot.modeld_v2.parse_model_outputs import Parser
-from openpilot.sunnypilot.modeld_v2.laneful import Laneful
 from openpilot.sunnypilot.modeld_v2.constants import ModelConstants, Plan
 from openpilot.sunnypilot.modeld_v2.meta_helper import load_meta_constants
 from openpilot.sunnypilot.modeld_v2.camera_offset_helper import CameraOffsetHelper
@@ -230,8 +228,7 @@ class ModelState(ModelStateBase):
     return outputs
 
   def get_action_from_model(self, model_output: dict[str, np.ndarray], prev_action: log.ModelDataV2.Action,
-                            lat_action_t: float, long_action_t: float, v_ego: float,
-                            laneful: Callable[[float], float] | None = None) -> log.ModelDataV2.Action:
+                            lat_action_t: float, long_action_t: float, v_ego: float) -> log.ModelDataV2.Action:
     if 'action' not in model_output:
       plan = model_output['plan'][0]
       desired_accel = get_accel_from_plan(plan[:, Plan.VELOCITY][:, 0], plan[:, Plan.ACCELERATION][:, 0], self.constants.T_IDXS,
@@ -243,8 +240,6 @@ class ModelState(ModelStateBase):
     else:
       desired_accel = model_output['action'][0, 1]
       desired_curvature = model_output['action'][0, 0] / (max(1.0, v_ego))**2
-    if laneful is not None:
-      desired_curvature += laneful(desired_curvature)
 
     stop = v_ego < 0.3 and desired_accel < 0.1
     desired_accel = smooth_value(desired_accel, prev_action.desiredAcceleration, self.LONG_SMOOTH_SECONDS)
@@ -371,8 +366,6 @@ def main(demo=False):
   DH = DesireHelper()
   meta_constants = load_meta_constants()
   RELC = RoadEdgeLaneChangeController()
-  laneful = Laneful()
-  laneful_enabled = False
 
   while True:
     # Keep receiving frames until we are at least 1 frame ahead of previous extra frame
@@ -416,7 +409,6 @@ def main(demo=False):
       model.lat_delay = get_lat_delay(params, sm["lateralDelay"].lateralDelay)
       model.PLANPLUS_CONTROL = params.get("PlanplusControl", return_default=True)
       camera_offset_helper.set_offset(params.get("CameraOffset", return_default=True))
-      laneful_enabled = params.get_bool("Laneful")
     lat_delay = model.lat_delay + model.LAT_SMOOTH_SECONDS
     if sm.updated["extrinsicsCalibration"] and sm.seen['narrowRoadCameraState'] and sm.seen['deviceState']:
       device_from_calib_euler = np.array(sm["extrinsicsCalibration"].rpyCalib, dtype=np.float32)
@@ -490,9 +482,7 @@ def main(demo=False):
 
       if not sm['carControl'].latActive:
         prev_action.desiredCurvature = sm['controlsState'].curvature
-      laneful_active = laneful_enabled and desire == log.Desire.none and DH.lane_change_state == log.LaneChangeState.off
-      action = model.get_action_from_model(model_output, prev_action, lat_action_t, long_action_t, v_ego,
-                                           partial(laneful.update, model_output, v_ego, laneful_active))
+      action = model.get_action_from_model(model_output, prev_action, lat_action_t, long_action_t, v_ego)
       prev_action = action
       fill_model_msg(drivingdata_send, modelv2_send, model_output, action,
                      publish_state, meta_main.frame_id, meta_extra.frame_id, frame_id,
@@ -509,11 +499,6 @@ def main(demo=False):
       modelv2_send.modelV2.meta.laneChangeDirection = DH.lane_change_direction
       mdv2sp_send.valid = modelv2_send.valid
       mdv2sp_send.modelDataV2SP.laneTurnDirection = DH.lane_turn_direction
-      mdv2sp_send.modelDataV2SP.lanefulCurvature = laneful.curvature
-      mdv2sp_send.modelDataV2SP.lanefulTrust = float(laneful.trust)
-      mdv2sp_send.modelDataV2SP.lanefulLaneWidth = float(laneful.lane_width.x)
-      mdv2sp_send.modelDataV2SP.lanefulLeftProb = laneful.l_prob
-      mdv2sp_send.modelDataV2SP.lanefulRightProb = laneful.r_prob
       drivingdata_send.drivingModelData.meta.laneChangeState = DH.lane_change_state
       drivingdata_send.drivingModelData.meta.laneChangeDirection = DH.lane_change_direction
 

openpilot/sunnypilot/models/manager.py

@@ -7,7 +7,10 @@ See the LICENSE.md file in the root directory for more details.
 
 import asyncio
 import os
+import re
+import threading
 import time
+from concurrent.futures import ThreadPoolExecutor
 
 import requests
 from openpilot.common.params import Params
@@ -22,12 +25,99 @@ from openpilot.sunnypilot.models.helpers import (ACTIVE_BUNDLE_KEYS, get_active_
 
 # (connect, read) seconds. read is per-request inactivity, not a total cap
 DOWNLOAD_TIMEOUT = (30, 30)
+# HuggingFace throttles each connection to ~1-2 MB/s but not the connection count; 12 saturates the device link
+MAX_CONCURRENT_PIECES = 12
+PIECE_SIZE = 8 * 1024 * 1024
+PIECE_ATTEMPTS = 3
+PIECE_BACKOFF = 1.0  # seconds, growing linearly: waits 1 s then 2 s between a piece's three attempts
+TRANSIENT_HTTP_STATUS = frozenset({408, 429, 500, 502, 503, 504})
+REPORT_INTERVAL = 0.5  # seconds between progress publications
+# byte ranges count encoded bytes but iter_content yields decoded ones, so ask for an unencoded body
+NO_COMPRESSION = {"Accept-Encoding": "identity"}
 
 
 class DownloadCancelled(Exception):
   pass
 
 
+def _is_transient(e: BaseException) -> bool:
+  if isinstance(e, requests.HTTPError):
+    return e.response is not None and e.response.status_code in TRANSIENT_HTTP_STATUS
+  return isinstance(e, (requests.ConnectionError, requests.Timeout, requests.exceptions.ChunkedEncodingError))
+
+
+class _Progress:
+  def __init__(self):
+    self._bytes = 0
+    self._lock = threading.Lock()
+
+  def add_bytes(self, n: int) -> None:
+    with self._lock:
+      self._bytes += n
+
+  def snapshot(self) -> int:
+    with self._lock:
+      return self._bytes
+
+
+def _piece_ranges(total: int, piece_size: int) -> list[tuple[int, int]]:
+  return [(start, min(start + piece_size, total)) for start in range(0, total, piece_size)]
+
+
+def _prepare_target(path: str, total: int) -> None:
+  with open(path, "wb") as f:
+    f.truncate(total)
+
+
+def _content_range_total(response: requests.Response) -> int | None:
+  if response.status_code != 206:
+    return None
+  match = re.fullmatch(r"bytes \d+-\d+/(\d+)", response.headers.get("Content-Range", ""))
+  return int(match.group(1)) if match else None
+
+
+def _probe_size(url: str) -> tuple[int, bool]:
+  with requests.get(url, headers={**NO_COMPRESSION, "Range": "bytes=0-0"}, stream=True, timeout=DOWNLOAD_TIMEOUT) as response:
+    response.raise_for_status()
+    total = _content_range_total(response)
+    if total is not None:
+      return total, True
+    return int(response.headers.get("Content-Length", 0)), False
+
+
+def _fetch_piece(url: str, path: str, index: int, start: int, end: int | None, progress: _Progress,
+                 cancel: threading.Event, block_size: int) -> None:
+  headers = {**NO_COMPRESSION, "Range": f"bytes={start}-{end - 1}"} if end is not None else NO_COMPRESSION
+  label = f"{os.path.basename(path)} piece {index}"
+  for attempt in range(PIECE_ATTEMPTS):
+    written = 0
+    try:
+      with requests.get(url, headers=headers, stream=True, timeout=DOWNLOAD_TIMEOUT) as response:
+        response.raise_for_status()
+        if end is not None and response.status_code != 206:
+          raise ValueError(f"server ignored byte range for {label}")
+        with open(path, "r+b") as f:
+          f.seek(start)
+          for block in response.iter_content(chunk_size=block_size):
+            if cancel.is_set():
+              raise DownloadCancelled("Download cancelled")
+            f.write(block)
+            written += len(block)
+            progress.add_bytes(len(block))
+          if end is None:
+            f.truncate()  # unknown length: the body defines the file size
+          if end is not None and written != end - start:
+            raise requests.exceptions.ConnectionError(f"short read: {written} of {end - start} bytes")
+      return
+    except requests.RequestException as e:
+      progress.add_bytes(-written)  # the piece restarts from scratch
+      if not _is_transient(e) or attempt == PIECE_ATTEMPTS - 1:
+        raise
+      cloudlog.warning(f"retrying {label} after {type(e).__name__}: {e}")
+      if cancel.wait(PIECE_BACKOFF * (1 + attempt)):
+        raise DownloadCancelled("Download cancelled") from None
+
+
 class ModelManagerSP:
   """Manages model downloads and status reporting"""
 
@@ -41,7 +131,7 @@ class ModelManagerSP:
     self.source_models: dict[str, list[custom.ModelManagerSP.ModelBundle]] = {}
     self.selected_bundle: custom.ModelManagerSP.ModelBundle = None
     self.active_bundle: custom.ModelManagerSP.ModelBundle = get_active_bundle(self.params, chestnut=self.chestnut_present)
-    self._chunk_size = 128 * 1000  # 128 KB chunks
+    self._block_size = 128 * 1000  # 128 KB network read blocks
     self._download_start_times: dict[str, float] = {}  # Track start time per model
     self._download_ref: bytes | str | None = None
 
@@ -56,7 +146,6 @@ class ModelManagerSP:
     self._download_ref = None
 
   def _sync_artifact_progress(self, source_artifact) -> None:
-    """Mirror download progress to all artifacts sharing the same filename in the selected bundle."""
     if not self.selected_bundle:
       return
     for model in self.selected_bundle.models:
@@ -67,7 +156,6 @@ class ModelManagerSP:
         artifact.downloadProgress.eta = source_artifact.downloadProgress.eta
 
   def _calculate_eta(self, filename: str, progress: float) -> int:
-    """Calculate ETA based on elapsed time and current progress"""
     if filename not in self._download_start_times or progress <= 0:
       return 60  # Default ETA for new downloads
 
@@ -75,39 +163,58 @@ class ModelManagerSP:
     if elapsed_time <= 0:
       return 60
 
-    # If we're at X% after Y seconds, we can estimate total time as (Y / X) * 100
     total_estimated_time = (elapsed_time / progress) * 100
     eta = total_estimated_time - elapsed_time
-
     return max(1, int(eta))  # Return at least 1 second if download is ongoing
 
-  async def _download_file(self, url: str, path: str, model) -> None:
-    """Downloads a file with progress tracking"""
-    self._download_start_times[model.fileName] = time.monotonic()
+  def _set_progress(self, artifact, status, progress: float, eta: int = 0) -> None:
+    artifact.downloadProgress.status = status
+    artifact.downloadProgress.progress = progress
+    artifact.downloadProgress.eta = eta
+    self._sync_artifact_progress(artifact)
+    self._report_status()
 
-    with requests.get(url, stream=True, timeout=DOWNLOAD_TIMEOUT) as response:  # noqa: ASYNC210
-      response.raise_for_status()
-      total_size = int(response.headers.get("content-length", 0))
-      bytes_downloaded = 0
+  def _publish_progress(self, artifact, done_bytes: int, total: int) -> None:
+    # held at 99 until the file passes its hash check
+    progress = min(99.0, done_bytes / total * 100) if total > 0 else 0.0
+    eta = self._calculate_eta(artifact.fileName, progress)
+    self._set_progress(artifact, custom.ModelManagerSP.DownloadStatus.downloading, progress, eta)
 
-      with open(path, 'wb') as f:  # noqa: ASYNC230
-        for chunk in response.iter_content(chunk_size=self._chunk_size):  # type: bytes
-          f.write(chunk)
-          bytes_downloaded += len(chunk)
+  async def _report_until_done(self, tasks: list[asyncio.Future], artifact, progress: _Progress, total: int) -> None:
+    pending = set(tasks)
+    while pending:
+      done, pending = await asyncio.wait(pending, timeout=REPORT_INTERVAL)
+      for task in done:
+        task.result()  # re-raises a worker failure
+      if self._download_interrupted():
+        raise DownloadCancelled("Download cancelled")
+      self._publish_progress(artifact, progress.snapshot(), total)
 
-          if self._download_interrupted():
-            raise DownloadCancelled("Download cancelled")
+  async def _download_file(self, url: str, path: str, artifact) -> None:
+    self._download_start_times[artifact.fileName] = time.monotonic()
+    loop = asyncio.get_running_loop()
 
-          if total_size > 0:
-            progress = (bytes_downloaded / total_size) * 100
-            model.downloadProgress.status = custom.ModelManagerSP.DownloadStatus.downloading
-            model.downloadProgress.progress = progress
-            model.downloadProgress.eta = self._calculate_eta(model.fileName, progress)
-            self._sync_artifact_progress(model)
-            self._report_status()
+    total, ranged = await loop.run_in_executor(None, _probe_size, url)
+    pieces: list[tuple[int, int | None]] = list(_piece_ranges(total, PIECE_SIZE)) if ranged else [(0, None)]
+    await loop.run_in_executor(None, _prepare_target, path, total)
+    progress = _Progress()
+    cancel = threading.Event()
 
-    # Clean up start time after download completes
-    del self._download_start_times[model.fileName]
+    with ThreadPoolExecutor(max_workers=MAX_CONCURRENT_PIECES) as pool:
+      tasks = [loop.run_in_executor(pool, _fetch_piece, url, path, i, start, end, progress, cancel, self._block_size)
+               for i, (start, end) in enumerate(pieces)]
+      try:
+        await self._report_until_done(tasks, artifact, progress, total)
+      except BaseException as e:
+        cancel.set()
+        if isinstance(e, DownloadCancelled):
+          artifact.downloadProgress.canceling = True
+          self._set_progress(artifact, custom.ModelManagerSP.DownloadStatus.downloading, artifact.downloadProgress.progress)
+        for task in tasks:
+          task.cancel()  # drops queued pieces; running ones see the event at their next block
+        await asyncio.gather(*tasks, return_exceptions=True)
+        raise
+    del self._download_start_times[artifact.fileName]
 
   async def _process_artifact(self, artifact, destination_path: str) -> None:
     if not artifact.downloadUri.uri:
@@ -119,33 +226,30 @@ class ModelManagerSP:
     expected_hash = artifact.downloadUri.sha256
     filename = artifact.fileName
     full_path = os.path.join(destination_path, filename)
+    status = custom.ModelManagerSP.DownloadStatus
+    artifact.downloadProgress.canceling = False
 
     try:
+      self._set_progress(artifact, status.verifying, 0)
       if await verify_file(full_path, expected_hash):
-        artifact.downloadProgress.status = custom.ModelManagerSP.DownloadStatus.cached
-        artifact.downloadProgress.progress = 100
-        artifact.downloadProgress.eta = 0
-        self._sync_artifact_progress(artifact)
-        self._report_status()
+        self._set_progress(artifact, status.cached, 100)
         return
 
+      self._set_progress(artifact, status.downloading, 0)
       await self._download_file(url, full_path, artifact)
+      self._set_progress(artifact, status.verifying, 99)
       if not await verify_file(full_path, expected_hash):
         raise ValueError(f"Hash validation failed for {filename}")
 
-      artifact.downloadProgress.status = custom.ModelManagerSP.DownloadStatus.downloaded
-      artifact.downloadProgress.progress = 100
-      artifact.downloadProgress.eta = 0
-      self._sync_artifact_progress(artifact)
-      self._report_status()
+      self._set_progress(artifact, status.downloaded, 100)
 
     except DownloadCancelled:
       self._download_start_times.pop(artifact.fileName, None)
-      artifact.downloadProgress.status = custom.ModelManagerSP.DownloadStatus.failed
+      artifact.downloadProgress.status = status.failed
       artifact.downloadProgress.eta = 0
       self._sync_artifact_progress(artifact)
       if self.selected_bundle:
-        self.selected_bundle.status = custom.ModelManagerSP.DownloadStatus.failed
+        self.selected_bundle.status = status.failed
       self._report_status()
       raise
 
@@ -153,11 +257,11 @@ class ModelManagerSP:
       cloudlog.error(f"Error downloading {filename}: {str(e)}")
       if os.path.isfile(full_path):  # noqa: ASYNC240
         os.remove(full_path)
-      artifact.downloadProgress.status = custom.ModelManagerSP.DownloadStatus.failed
+      artifact.downloadProgress.status = status.failed
       artifact.downloadProgress.eta = 0
       self._sync_artifact_progress(artifact)
       if self.selected_bundle:
-        self.selected_bundle.status = custom.ModelManagerSP.DownloadStatus.failed
+        self.selected_bundle.status = status.failed
       self._report_status()
       self._download_start_times.pop(artifact.fileName, None)
       raise
@@ -272,12 +376,7 @@ class ModelManagerSP:
         rk.keep_time()
 
   def clear_model_cache(self) -> None:
-    """
-    Clears the model cache directory of all files except those in the active model bundle.
-    """
-
-    # Get list of files used by both slots' selected bundles (either may become
-    # the truly active bundle depending on hardware availability)
+    """ Clears the model cache directory of all files except those in the active model bundle."""
     active_files = []
     for source in ACTIVE_BUNDLE_KEYS:
       if selected_bundle := get_selected_bundle(self.params, source):
@@ -285,7 +384,6 @@ class ModelManagerSP:
           if model.artifact.fileName:
             active_files.append(model.artifact.fileName)
 
-    # Remove all files except active ones
     model_dir = Paths.model_root()
     try:
       for filename in os.listdir(model_dir):

openpilot/system/ui/sunnypilot/widgets/download_status.py

@@ -57,7 +57,7 @@ class DownloadStatusAction(ItemAction):
     self.icon: str | None = None
     self.icon_color: rl.Color | None = None
     self._font = gui_app.font(FontWeight.NORMAL)
-    # raw progress arrives in steps, one per 128KB chunk the manager publishes
+    # raw progress arrives in steps, one per progress report from the manager
     self._progress = FirstOrderFilter(0.0, 0.5, 1 / gui_app.target_fps)
     # integrated per frame; (t * speed) % span jumps whenever the fill width changes
     self._sweep = 0.0