Compare commits

..

5 Commits

Author SHA1 Message Date
Amy Jeanes 9489b2f481 models: retry a failed transfer from its resume state, keep the failure on the row (#2014)
* models: retry a failed transfer from its resume state, keep the failure on the row

A whole-file transfer whose pieces exhaust their retries (dropped link, DNS
blip, overloaded server) now gets up to four attempts, with a cancellable
2/4/8 s backoff between them. Every retry resumes from the sidecar, so nothing
already on disk is fetched again; speed is published as 0 while stalled.
Permanent HTTP errors raise at once, as before.

A failed download used to vanish from the Models row on the next manager tick
because the selection was cleared in `finally`. It now stays on the row, with
the failed status, until the next request; a finished or cancelled download
still clears it. The mici panel's error state names the failed model instead
of animating dots that never resolve.

With failures now sticky, a user cancel no longer marks the artifact or bundle
failed: it would otherwise flash "download failed" while the request handler
clears the row.

Tests: a transient failure resumes from the recorded pieces on retry and
fetches nothing twice, a persistent 503 gives up after every attempt, a 404
is never retried, a cancel during the backoff ends it at once, a cancel
leaves the bundle not failed, and the row state after a failed, finished and
cancelled request.

Assisted-by: Claude Code (Claude Opus 4.8)
Assisted-by: Claude Code (Claude Opus 5.5)

* remove unneeded comments

Assisted-by: Claude Code (Claude Opus 5.5)
2026-11-18 10:06:45 -08:00
Amy Jeanes c270c7835e models: resume an interrupted whole-file download where it stopped (#2013) 2026-11-17 09:08:22 -08:00
James Vecellio-Grant c6e564cdcd modeld_v2: native QCOM reads frames in place (#2100) 2026-11-14 17:44:33 -08:00
Jason Wen b55c4be825 bump opendbc and update CARS.md 2026-11-13 12:53:13 -05:00
Jason Wen dbba01d933 ci: add --local to upload_ci_routes (#2101)
* upload_ci_routes: add --local option

* upload_ci_routes: read --local via Route
2026-11-13 12:52:54 -05:00
15 changed files with 390 additions and 74 deletions
+1 -1
View File
@@ -42,7 +42,7 @@ jobs:
WARP_PKL="openpilot/sunnypilot/modeld_v2/models/driving_warp_${res}_tinygrad.pkl"
echo "Compiling $WARP_PKL on QCOM"
taskset -c 7 env DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1 python3 openpilot/sunnypilot/modeld_v2/compile_warp.py \
--frame ${FRAME_PARAM} --warp-to ${MODEL_SIZE} --output "${WARP_PKL}"
--frame ${FRAME_PARAM} --warp-to ${MODEL_SIZE} --split-frames --output "${WARP_PKL}"
BIG_WARP_PKL="openpilot/sunnypilot/modeld_v2/models/big_driving_warp_${res}_tinygrad.pkl"
echo "Compiling $BIG_WARP_PKL on AMD"
+3 -2
View File
@@ -4,7 +4,7 @@
A supported vehicle is one that just works when you install a comma device. All supported cars provide a better experience than any stock system. Supported vehicles reference the US market unless otherwise specified.
# 356 Supported Cars
# 357 Supported Cars
|Make|Model|Supported Package|ACC|No ACC accel below|No ALC below|Steering Torque|Resume from stop|<a href="##"><img width=2000></a>Hardware Needed<br>&nbsp;|Video|Setup Video|
|---|---|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
@@ -153,8 +153,9 @@ A supported vehicle is one that just works when you install a comma device. All
|Hyundai|Santa Fe Plug-in Hybrid 2022-23|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai L connector<br>- 1 OBD-C cable (2 ft)<br>- 1 comma four<br>- 1 comma power v3<br>- 1 harness box<br>- 1 mount<br><a href="https://comma.ai/shop/comma-3x?harness=Hyundai Santa Fe Plug-in Hybrid 2022-23">Buy Here</a></sub></details>|||
|Hyundai|Sonata 2018-19|Smart Cruise Control (SCC)|Stock|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai E connector<br>- 1 OBD-C cable (2 ft)<br>- 1 comma four<br>- 1 comma power v3<br>- 1 harness box<br>- 1 mount<br><a href="https://comma.ai/shop/comma-3x?harness=Hyundai Sonata 2018-19">Buy Here</a></sub></details>|||
|Hyundai|Sonata 2020-23|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai A connector<br>- 1 OBD-C cable (2 ft)<br>- 1 comma four<br>- 1 comma power v3<br>- 1 harness box<br>- 1 mount<br><a href="https://comma.ai/shop/comma-3x?harness=Hyundai Sonata 2020-23">Buy Here</a></sub></details>|<a href="https://www.youtube.com/watch?v=ix63r9kE3Fw" target="_blank"><img height="18px" src="assets/icon-youtube.svg" /></a>||
|Hyundai|Sonata (without HDA II) 2024-26|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai A connector<br>- 1 OBD-C cable (2 ft)<br>- 1 comma four<br>- 1 comma power v3<br>- 1 harness box<br>- 1 mount<br><a href="https://comma.ai/shop/comma-3x?harness=Hyundai Sonata (without HDA II) 2024-26">Buy Here</a></sub></details>|||
|Hyundai|Sonata Hybrid 2020-23|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai A connector<br>- 1 OBD-C cable (2 ft)<br>- 1 comma four<br>- 1 comma power v3<br>- 1 harness box<br>- 1 mount<br><a href="https://comma.ai/shop/comma-3x?harness=Hyundai Sonata Hybrid 2020-23">Buy Here</a></sub></details>|||
|Hyundai|Sonata Hybrid (without HDA II) 2024-26|Smart Cruise Control (SCC)|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai A connector<br>- 1 OBD-C cable (2 ft)<br>- 1 comma four<br>- 1 comma power v3<br>- 1 harness box<br>- 1 mount<br><a href="https://comma.ai/shop/comma-3x?harness=Hyundai Sonata Hybrid (without HDA II) 2024-26">Buy Here</a></sub></details>|||
|Hyundai|Sonata Hybrid (without HDA II) 2024-26|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai A connector<br>- 1 OBD-C cable (2 ft)<br>- 1 comma four<br>- 1 comma power v3<br>- 1 harness box<br>- 1 mount<br><a href="https://comma.ai/shop/comma-3x?harness=Hyundai Sonata Hybrid (without HDA II) 2024-26">Buy Here</a></sub></details>|||
|Hyundai|Staria 2023|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai K connector<br>- 1 OBD-C cable (2 ft)<br>- 1 comma four<br>- 1 comma power v3<br>- 1 harness box<br>- 1 mount<br><a href="https://comma.ai/shop/comma-3x?harness=Hyundai Staria 2023">Buy Here</a></sub></details>|||
|Hyundai|Tucson 2021|Smart Cruise Control (SCC)|openpilot available[<sup>1</sup>](#footnotes)|19 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai L connector<br>- 1 OBD-C cable (2 ft)<br>- 1 comma four<br>- 1 comma power v3<br>- 1 harness box<br>- 1 mount<br><a href="https://comma.ai/shop/comma-3x?harness=Hyundai Tucson 2021">Buy Here</a></sub></details>|||
|Hyundai|Tucson 2022|Smart Cruise Control (SCC)|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai N connector<br>- 1 OBD-C cable (2 ft)<br>- 1 comma four<br>- 1 comma power v3<br>- 1 harness box<br>- 1 mount<br><a href="https://comma.ai/shop/comma-3x?harness=Hyundai Tucson 2022">Buy Here</a></sub></details>|||
@@ -249,7 +249,11 @@ class ModelsLayout(Widget):
return {"name": name, "downloading": True, "progress": progress, "status_text": tr("verifying")}
if ds.downloading in statuses:
speed = max((p.speed for p in progresses), default=0.0)
status_text = f"{speed / 1024 ** 2:.1f} MB/s" if speed > 0 else "--.- MB/s"
speed_text = f"{speed / 1024 ** 2:.1f} MB/s" if speed > 0 else "--.- MB/s"
if any(p.resuming for p in progresses):
status_text = tr("resuming...") + f" {speed_text}"
else:
status_text = speed_text
return {"name": name, "downloading": True, "progress": progress, "status_text": status_text}
if statuses <= {ds.downloaded, ds.cached}:
return {"name": name, "text_color": ON_COLOR, "icon": "icons/checkmark.png"}
@@ -267,8 +267,8 @@ class ModelsLayoutMici(NavScroller):
self.current_model_info.info_text.set_text(info_text)
if manager.selectedBundle and manager.selectedBundle.status == custom.ModelManagerSP.DownloadStatus.failed:
self.current_model_info.info_header.set_text(tr("error") + self._download_progress)
self.current_model_info.info_text.set_text(tr("download failed"))
self.current_model_info.info_header.set_text(tr("error"))
self.current_model_info.info_text.set_text(f"{manager.selectedBundle.internalName.lower()} | {tr('download failed')}")
elif manager.selectedBundle and manager.selectedBundle.status == custom.ModelManagerSP.DownloadStatus.downloading:
self.cancel_download_btn.set_visible(ui_state.params.get("ModelManager_DownloadRef") is not None)
@@ -277,6 +277,7 @@ class ModelsLayoutMici(NavScroller):
count = 0
verifying = False
canceling = False
resuming = False
speed = 0.0
for model in manager.selectedBundle.models:
count += 1
@@ -286,12 +287,13 @@ class ModelsLayoutMici(NavScroller):
progress += p.progress
verifying = verifying or p.status == custom.ModelManagerSP.DownloadStatus.verifying
canceling = canceling or p.canceling
resuming = resuming or p.resuming
speed = max(speed, p.speed) # only one file downloads at a time, so the non-zero speed is the live one
elif p.status in (custom.ModelManagerSP.DownloadStatus.downloaded,
custom.ModelManagerSP.DownloadStatus.cached):
progress += 100.0
header = tr("canceling...") if canceling else tr("verifying") if verifying else tr("downloading")
header = tr("canceling...") if canceling else tr("verifying") if verifying else tr("resuming...") if resuming else tr("downloading")
self.current_model_info.current_model_header.set_text(header)
self.cancel_download_btn.set_text(tr("cancel verification") if verifying else tr("cancel download"))
self.current_model_info.current_model_header._shimmer = True
+12 -4
View File
@@ -17,16 +17,25 @@ def main():
parser.add_argument("--frame", required=True, help="width,height,stride,y_height,uv_height,buffer_size")
parser.add_argument("--warp-to", required=True, help="model input WxH")
parser.add_argument("--output", required=True)
parser.add_argument("--split-frames", action="store_true", help="one input per camera (frame, big_frame), bound to the camera buffers by pointer")
args = parser.parse_args()
from examples.openpilot.compile_warp import make_frame_prepare, parse_frame, parse_size
from tinygrad import Device, Tensor, TinyJit
frame = parse_frame(args.frame)
warp = make_frame_prepare(frame, *parse_size(args.warp_to))
run = TinyJit(lambda input_frame, M_inv: Tensor.stack(*(warp(input_frame[i], M_inv[i]) for i in range(2))), prune=True)
rng = np.random.default_rng(42)
inputs = {"input_frame": Tensor(rng.integers(0, 256, (2, frame.size), dtype=np.uint8), device=Device.DEFAULT).realize(),
"M_inv": Tensor((rng.standard_normal((2, 3, 3)) * 8).astype(np.float32), device=Device.DEFAULT).realize()}
frames = rng.integers(0, 256, (2, frame.size), dtype=np.uint8)
m_inv = Tensor((rng.standard_normal((2, 3, 3)) * 8).astype(np.float32), device=Device.DEFAULT).realize()
if args.split_frames:
run = TinyJit(lambda frame, big_frame, M_inv: Tensor.stack(warp(frame, M_inv[0]), warp(big_frame, M_inv[1])), prune=True)
inputs = {"frame": Tensor(frames[0], device=Device.DEFAULT).realize(), "big_frame": Tensor(frames[1], device=Device.DEFAULT).realize(), "M_inv": m_inv}
specs = {"frame": ((frame.size,), "|u1", Device.DEFAULT), "big_frame": ((frame.size,), "|u1", Device.DEFAULT),
"M_inv": ((2, 3, 3), "<f4", Device.DEFAULT)}
else:
run = TinyJit(lambda input_frame, M_inv: Tensor.stack(*(warp(input_frame[i], M_inv[i]) for i in range(2))), prune=True)
inputs = {"input_frame": Tensor(frames, device=Device.DEFAULT).realize(), "M_inv": m_inv}
specs = {"input_frame": ((2, frame.size), "|u1", Device.DEFAULT), "M_inv": ((2, 3, 3), "<f4", Device.DEFAULT)}
with record_kernel_linear() as recorded:
run(**inputs)
run(**inputs)
@@ -34,7 +43,6 @@ def main():
assert len(recorded) == 1, f"expected one captured kernel list, got {len(recorded)}"
part = serialize_jit(run, recorded[0], check)
PlainJit(*part)
specs = {"input_frame": ((2, frame.size), "|u1", Device.DEFAULT), "M_inv": ((2, 3, 3), "<f4", Device.DEFAULT)}
write_bundle(args.output, {"run": part}, {"input_specs": specs, "metadata": {}})
print(f"Saved warp to {args.output} ({len(part[0]['calls'])} kernels)")
@@ -12,6 +12,8 @@ from functools import partial
from openpilot.common.basedir import BASEDIR
from openpilot.sunnypilot.modeld_v2.helpers import _dtype, read_parts
SPLIT_FRAMES = ("big_frame", "frame")
def pack_layout(shapes: dict[str, tuple], frame_size: int) -> tuple[dict[str, int], int]:
offsets, offset = {}, 0
@@ -39,29 +41,30 @@ def frame_part(warp_path: str, parts: dict, extras: dict) -> dict:
(model, model_blob), (warp, warp_blob) = parts["run"], read_parts(warp_path)[0]["run"]
states = sorted(name for name in extras["input_specs"] if f"next_{name}" in extras["metadata"]["output_shapes"])
host = {name: shape for name, (shape, _, _) in extras["input_specs"].items() if name not in states and name != "new_img"}
frame_size = next(s["numel"] for s in warp["inputs"] if s["name"] == "input_frame") // 2
offsets, packed = pack_layout(host, frame_size)
frame_specs = {s["name"]: s for s in warp["inputs"] if s["name"] in SPLIT_FRAMES}
offsets, packed = pack_layout(host, 0 if frame_specs else next(s["numel"] for s in warp["inputs"] if s["name"] == "input_frame") // 2)
shift, upload, u8 = len(model_blob), len(model["buffers"]) + len(warp["buffers"]), repr(_dtype("dtypes.uint8"))
def view(name: str, nbytes: int, dtype: str) -> dict:
return {"buffer": upload, "offset": offsets[name], "nbytes": nbytes, "dtype": dtype}
outputs = sorted(s["slot"] for s in model["inputs"] if s["name"] is None)
names = sorted(["host", *states])
names = sorted(["host", *states, *frame_specs])
slots = {s["slot"]: names.index(s["name"]) for s in model["inputs"] if s["name"] in states} | \
{slot: len(names) + i for i, slot in enumerate(outputs)}
returned = warp["returns"]["tensors"][0]
model_inputs = {s["slot"]: {"input": slots[s["slot"]]} if s["slot"] in slots else
{k: v for k, v in returned.items() if k != "shape"} | {"buffer": returned["buffer"] + len(model["buffers"])} if s["name"] == "new_img" else
view(s["name"], s["numel"] * _dtype(s["dtype"]).itemsize, s["dtype"]) for s in model["inputs"]}
warp_inputs = {s["slot"]: view("frames" if s["name"] == "input_frame" else "tfm", s["numel"] * _dtype(s["dtype"]).itemsize, s["dtype"])
for s in warp["inputs"]}
warp_inputs = {s["slot"]: {"input": names.index(s["name"])} if s["name"] in frame_specs else
view("frames" if s["name"] == "input_frame" else "tfm", s["numel"] * _dtype(s["dtype"]).itemsize, s["dtype"]) for s in warp["inputs"]}
def remap(calls: list, inputs: dict, buffers: int, lib: int) -> list:
return [call | ({"lib": [call["lib"][0] + lib, call["lib"][1] + lib]} if "lib" in call else {}) |
{"args": [dict(inputs[a["input"]]) if "input" in a else a | {"buffer": a["buffer"] + buffers} for a in call["args"]]} for call in calls]
header = {"input_names": names, "returns": {"kind": "NoneType", "tensors": []}, "check": None,
"inputs": [{"slot": names.index("host"), "name": "host", "dtype": u8, "numel": packed, "device": "NPY"}] +
[s | {"slot": slots[s["slot"]]} for s in model["inputs"] if s["slot"] in slots],
[s | {"slot": slots[s["slot"]]} for s in model["inputs"] if s["slot"] in slots] +
[s | {"slot": names.index(name)} for name, s in frame_specs.items()],
"buffers": model["buffers"] + [b | ({"init": [b["init"][0] + shift, b["init"][1] + shift]} if "init" in b else {}) for b in warp["buffers"]] +
[{"nbytes": packed, "device": model["inputs"][0]["device"], "dtype": u8}],
"calls": [{"kind": "copy", "args": [{"buffer": upload, "offset": 0, "nbytes": packed, "dtype": u8}, {"input": names.index("host")}]}] +
+18 -1
View File
@@ -270,15 +270,32 @@ def _program_uop(call: dict, lib: bytes):
UOp(Ops.SOURCE, arg=""), UOp(Ops.BINARY, arg=lib)), arg=info)
def _input_tensors(kwargs: dict) -> list: # in _prepare_jit_inputs' order
from tinygrad import Tensor
tensors = [t for _, t in sorted(kwargs.items()) if t.__class__ is Tensor]
for value in kwargs.values():
tensors += [t for t in (value.values() if isinstance(value, dict) else value if isinstance(value, (tuple, list)) else ())
if t.__class__ is Tensor and not any(t is y for y in tensors)]
return tensors
class _Jit:
def __init__(self, captured, input_names: list, buffers: list):
self.captured, self.input_names, self.buffers = captured, input_names, buffers
def __call__(self, *args, **kwargs):
import tinygrad.engine.jit as jit
from tinygrad.uop.ops import Ops
# same names, no variables: the binding is the tensors' own uops, without _prepare_jit_inputs' graph rewrites
if not args and self.__dict__.get("_names") == sorted(kwargs):
tensors = _input_tensors(kwargs)
if all(t.uop.is_realized for t in tensors):
return self.captured([t.uop.src[0] if t.uop.op is Ops.UNSHARD else t.uop for t in tensors], {})
input_uops, var_vals, names, _ = jit._prepare_jit_inputs(args, kwargs)
if names != self.input_names:
raise ValueError(f"expected inputs {self.input_names}, got {names}")
if not args and not var_vals:
self._names = sorted(kwargs)
return self.captured(input_uops, var_vals)
@@ -311,7 +328,7 @@ class PlainJit(_Jit):
calls.append(_program_uop(call, bytes(blob[slice(*call["lib"])])).call(*arguments))
else:
calls.append(arguments[0].store_call(arguments[1]))
# stand-ins for JIT inputs: copy staging to GPU is decided from the input buffers, the real ones bind per call
# allocate gpu buffers
stand_ins = {slot: UOp.from_buffer(Buffer(_device(spec["device"], device_map), spec["numel"] * _dtype(spec["dtype"]).itemsize).ensure_allocated(),
_dtype(spec["dtype"])) for slot, spec in inputs.items()}
input_uops = tuple(stand_ins.get(slot, next(iter(stand_ins.values()), None)) for slot in range(max(stand_ins, default=-1) + 1))
@@ -11,7 +11,7 @@ import pickle
import numpy as np
from openpilot.common.basedir import BASEDIR
from openpilot.sunnypilot.modeld_v2.frame_fusion import pack_layout
from openpilot.sunnypilot.modeld_v2.frame_fusion import pack_layout, SPLIT_FRAMES
from openpilot.sunnypilot.modeld_v2.helpers import load_model
from openpilot.sunnypilot.modeld_v2.compile_modeld import (POLICY_INPUTS, derive_frame_skip,
make_split_input_queues, make_supercombo_input_queues)
@@ -169,16 +169,18 @@ class NativeTinygradAdapter(BaseModelAdapter):
self.warp_frame_size = self.frame_copy_size
self._init_common()
self.run_model = self.jits['run']
self.split = [name for name in SPLIT_FRAMES if name in self.run_model.input_names]
self.frame_inputs: dict[str, Tensor] = {}
self.input_queues = {name: Tensor(np.zeros(shape, dtype=dtype), device=self.model_device).realize()
for name, (shape, dtype) in self.input_shapes.items() if name in self.state_pairs}
shapes = {name: shape for name, (shape, _) in self.input_shapes.items() if name not in self.state_pairs and name != 'new_img'}
offsets, packed = pack_layout(shapes, self.warp_frame_size)
offsets, packed = pack_layout(shapes, 0 if self.split else self.warp_frame_size)
self.packed_input = np.zeros(packed, dtype=np.uint8)
self.host = Tensor(self.packed_input, device='NPY')
self.numpy_inputs = {name: np.ndarray(shape, dtype=np.float32, buffer=self.packed_input, offset=offsets[name])
for name, shape in ({'tfm': (2, 3, 3)} | shapes).items()}
self.frames = self.packed_input[offsets['frames']:].reshape(2, self.warp_frame_size)
self.frames = self.packed_input[offsets['frames']:].reshape(2, -1)
self.outputs = {name: Tensor(np.zeros(shape, dtype=dtype), device=device).realize() for name, (shape, dtype, device) in self.jits['output_specs'].items()}
for name, next_name in self.state_pairs.items():
@@ -187,17 +189,29 @@ class NativeTinygradAdapter(BaseModelAdapter):
def copy_frames(self, bufs):
for i, key in enumerate(self._vision_input_names):
if key in bufs:
if key not in bufs:
continue
if not self.split:
data = bufs[key].data if hasattr(bufs[key], 'data') else bufs[key]
np.copyto(self.frames[i], np.frombuffer(data, dtype=np.uint8, count=self.warp_frame_size))
elif (name := ('frame', 'big_frame')[i]) in self.split:
if isinstance(bufs[key], np.ndarray): # cached memory: the GPU would read stale lines in place
self.frame_inputs[name] = Tensor(np.ascontiguousarray(bufs[key][:self.warp_frame_size]), device=self.model_device).realize()
continue
ptr = np.frombuffer(bufs[key].data, dtype=np.uint8).ctypes.data # visionipc buffers are coherent
if (key, ptr) not in self._blob_cache:
self._blob_cache[(key, ptr)] = Tensor.from_blob(ptr, (self.warp_frame_size,), dtype='uint8', device=self.model_device)
self.frame_inputs[name] = self._blob_cache[(key, ptr)]
def reset_warmup_buffers(self) -> None:
self.packed_input[:] = 0
self.frame_inputs.clear()
self._blob_cache.clear()
for key in self.state_pairs:
self.input_queues[key].assign(0).realize()
def run(self):
self.run_model(host=self.host, **self.input_queues, output_buffers=self.outputs)
self.run_model(host=self.host, **self.input_queues, **self.frame_inputs, output_buffers=self.outputs)
return self.outputs['outputs']
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4b22fd6bcb3a3e8a642e6866449dd9c30c12e694ff0741cad9e34e4b00d8c5fa
size 3698799
oid sha256:d776c72125156b5dd3eb59768409401e28c8a52a3a619cd3b9f02c10a156fda6
size 3699072
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9a17fd6acece4fab23784ef0f3f8bba28a049e9599e5e9e5ee7b36e4b1d24a6f
size 7924847
oid sha256:a3f13ed12e087817a86afd48b2496c66c4f61a1925065b676b2945e81fabaacd
size 7924864
@@ -144,6 +144,21 @@ class TestPlainArtifact(OpenpilotTestCase):
for name in ("outputs", "next_state_feat_q"):
assert np.array_equal(inputs["output_buffers"][name].numpy(), expected["output_buffers"][name].numpy()), name
split_path = os.path.join(self.tmp.name, "warp_split_tinygrad.pkl")
with mock.patch.object(sys, "argv", ["compile_warp", "--frame", FRAME, "--warp-to", f"{MODEL_W}x{MODEL_H}", "--output", split_path, "--split-frames"]):
compile_warp.main()
split = load_model(path, cache=False, extend=functools.partial(frame_part, split_path))
assert split["run"].input_names == ["big_frame", "frame", "host", "state_feat_q"]
offsets, packed = pack_layout(HOST, 0)
host = np.zeros(packed, np.uint8)
for name, shape in ({"tfm": (2, 3, 3)} | HOST).items():
np.ndarray(shape, np.float32, host, offsets[name])[...] = tfm.numpy() if name == "tfm" else inputs[name].numpy()
out = {"outputs": Tensor.zeros(OUT).contiguous().realize(), "next_state_feat_q": Tensor.zeros(STATE).contiguous().realize()}
split["run"](host=Tensor(host, device="NPY"), frame=Tensor(frames.numpy()[0]).realize(), big_frame=Tensor(frames.numpy()[1]).realize(),
state_feat_q=expected["state_feat_q"], output_buffers=out)
for name in ("outputs", "next_state_feat_q"):
assert np.array_equal(out[name].numpy(), expected["output_buffers"][name].numpy()), name
def test_tinygrad_api_the_loader_reads(self):
import tinygrad.engine.jit as jit
from tinygrad.device import Buffer, TinyELF
+120 -29
View File
@@ -6,6 +6,7 @@ See the LICENSE.md file in the root directory for more details.
"""
import asyncio
import json
import os
import re
import threading
@@ -28,8 +29,11 @@ DOWNLOAD_TIMEOUT = (30, 30)
# HuggingFace throttles each connection to ~1-2 MB/s but not the connection count; 12 saturates the device link
MAX_CONCURRENT_PIECES = 12
PIECE_SIZE = 8 * 1024 * 1024
PIECE_ATTEMPTS = 3
PIECE_ATTEMPTS = 3 # per piece, before the transfer as a whole is retried
PIECE_BACKOFF = 1.0 # seconds, growing linearly: waits 1 s then 2 s between a piece's three attempts
DOWNLOAD_ATTEMPTS = 4 # whole-transfer attempts, each resuming from the sidecar
RETRY_BACKOFF = 2.0 # seconds, doubling: waits 2, 4 then 8 s between the four transfer attempts
RETRY_POLL = 0.25 # how often a backoff checks for a cancel
TRANSIENT_HTTP_STATUS = frozenset({408, 429, 500, 502, 503, 504})
REPORT_INTERVAL = 0.5 # seconds between progress publications
# byte ranges count encoded bytes but iter_content yields decoded ones, so ask for an unencoded body
@@ -49,26 +53,69 @@ def _is_transient(e: BaseException) -> bool:
class _Progress:
def __init__(self):
self._bytes = 0
def __init__(self, done: set[int], done_bytes: int):
self._bytes = done_bytes
self._done = set(done)
self._lock = threading.Lock()
def add_bytes(self, n: int) -> None:
with self._lock:
self._bytes += n
def snapshot(self) -> int:
def mark_done(self, index: int) -> None:
with self._lock:
return self._bytes
self._done.add(index)
def snapshot(self) -> tuple[int, set[int]]:
with self._lock:
return self._bytes, set(self._done)
def _piece_ranges(total: int, piece_size: int) -> list[tuple[int, int]]:
return [(start, min(start + piece_size, total)) for start in range(0, total, piece_size)]
def _prepare_target(path: str, total: int) -> None:
def _sidecar_path(path: str) -> str:
return f"{path}.download"
def _download_in_progress(path: str) -> bool:
return os.path.isfile(_sidecar_path(path))
def _save_resume_state(path: str, layout: dict, done: set[int]) -> None:
tmp = _sidecar_path(path) + ".tmp"
with open(tmp, "w") as f:
json.dump({**layout, "done": sorted(done)}, f)
f.flush()
os.fsync(f.fileno())
os.replace(tmp, _sidecar_path(path)) # a power loss leaves the old sidecar or the new one, never a torn file
dir_fd = os.open(os.path.dirname(path) or ".", os.O_RDONLY)
try:
os.fsync(dir_fd)
finally:
os.close(dir_fd)
def _remove_download_state(path: str) -> None:
for stale in (_sidecar_path(path), _sidecar_path(path) + ".tmp"):
if os.path.isfile(stale):
os.remove(stale)
def _prepare_target(path: str, layout: dict) -> set[int]:
total = layout["total"]
try:
with open(_sidecar_path(path)) as f:
state = json.load(f)
if isinstance(state, dict) and all(state.get(k) == v for k, v in layout.items()) and os.path.getsize(path) == total:
return {i for i in state.get("done", []) if isinstance(i, int)}
except (OSError, ValueError, TypeError): # missing, unreadable or malformed sidecar: start over
pass
_save_resume_state(path, layout, set())
with open(path, "wb") as f:
f.truncate(total)
return set()
def _content_range_total(response: requests.Response) -> int | None:
@@ -110,6 +157,9 @@ def _fetch_piece(url: str, path: str, index: int, start: int, end: int | None, p
f.truncate() # unknown length: the body defines the file size
if end is not None and written != end - start:
raise requests.exceptions.ConnectionError(f"short read: {written} of {end - start} bytes")
f.flush()
os.fsync(f.fileno()) # before it is recorded, so a power loss never leaves the sidecar naming unsynced bytes
progress.mark_done(index)
return
except requests.RequestException as e:
progress.add_bytes(-written) # the piece restarts from scratch
@@ -120,6 +170,11 @@ def _fetch_piece(url: str, path: str, index: int, start: int, end: int | None, p
raise DownloadCancelled("Download cancelled") from None
def _remove_file(path: str) -> None:
if os.path.isfile(path):
os.remove(path)
class ModelManagerSP:
"""Manages model downloads and status reporting"""
@@ -186,11 +241,12 @@ class ModelManagerSP:
else:
eta = self._calculate_eta(artifact.fileName, progress)
self._set_progress(artifact, custom.ModelManagerSP.DownloadStatus.downloading, progress, eta, speed)
artifact.downloadProgress.resuming = False # shown only for the first report, when the bar jumps to the resumed point
async def _report_until_done(self, tasks: list[asyncio.Future], artifact, progress: _Progress, total: int,
started: tuple[int, float]) -> None:
async def _report_until_done(self, tasks: list[asyncio.Future], artifact, progress: _Progress, path: str, layout: dict,
started: tuple[int, set[int], float]) -> None:
pending = set(tasks)
last_bytes, last_time = started
last_bytes, saved, last_time = started
speed = 0.0
while pending:
done, pending = await asyncio.wait(pending, timeout=REPORT_INTERVAL)
@@ -199,14 +255,17 @@ class ModelManagerSP:
if self._download_interrupted():
raise DownloadCancelled("Download cancelled")
now = time.monotonic()
done_bytes = progress.snapshot()
done_bytes, done_pieces = progress.snapshot()
if done_pieces != saved:
_save_resume_state(path, layout, done_pieces)
saved = done_pieces
if (dt := now - last_time) > 0:
instant = max(0.0, (done_bytes - last_bytes) / dt) # bytes/s this tick; clamped since retried pieces can lower the byte count
speed = instant if speed == 0 else SPEED_SMOOTHING * speed + (1 - SPEED_SMOOTHING) * instant
if speed < SPEED_FLOOR: # a stall decays toward 0 without reaching it, and a near-0 speed overflows the eta
speed = 0.0
last_time, last_bytes = now, done_bytes
self._publish_progress(artifact, done_bytes, total, speed)
self._publish_progress(artifact, done_bytes, layout["total"], speed)
async def _download_file(self, url: str, path: str, artifact) -> None:
self._download_start_times[artifact.fileName] = time.monotonic()
@@ -214,16 +273,18 @@ class ModelManagerSP:
total, ranged = await loop.run_in_executor(None, _probe_size, url)
pieces: list[tuple[int, int | None]] = list(_piece_ranges(total, PIECE_SIZE)) if ranged else [(0, None)]
await loop.run_in_executor(None, _prepare_target, path, total)
progress = _Progress()
layout = {"total": total, "piece_size": PIECE_SIZE, "ranged": ranged, "sha256": artifact.downloadUri.sha256}
done = await loop.run_in_executor(None, _prepare_target, path, layout)
resumed_bytes = sum(end - start for i, (start, end) in enumerate(pieces) if i in done and end is not None)
progress = _Progress(done, resumed_bytes) # the bar and eta start from the resumed pieces
cancel = threading.Event()
started = progress.snapshot(), time.monotonic() # before any worker can land bytes
started = *progress.snapshot(), time.monotonic() # before any worker can land bytes
with ThreadPoolExecutor(max_workers=MAX_CONCURRENT_PIECES) as pool:
tasks = [loop.run_in_executor(pool, _fetch_piece, url, path, i, start, end, progress, cancel, self._block_size)
for i, (start, end) in enumerate(pieces)]
for i, (start, end) in enumerate(pieces) if i not in done]
try:
await self._report_until_done(tasks, artifact, progress, total, started)
await self._report_until_done(tasks, artifact, progress, path, layout, started)
except BaseException as e:
cancel.set()
if isinstance(e, DownloadCancelled):
@@ -232,9 +293,32 @@ class ModelManagerSP:
for task in tasks:
task.cancel() # drops queued pieces; running ones see the event at their next block
await asyncio.gather(*tasks, return_exceptions=True)
await loop.run_in_executor(None, pool.shutdown) # let running pieces stop so none lands after the save
_save_resume_state(path, layout, progress.snapshot()[1])
raise
_remove_download_state(path)
artifact.downloadProgress.resuming = False
del self._download_start_times[artifact.fileName]
async def _download_with_retries(self, url: str, path: str, artifact) -> None:
for attempt in range(DOWNLOAD_ATTEMPTS):
try:
await self._download_file(url, path, artifact)
return
except Exception as e:
if not _is_transient(e) or attempt == DOWNLOAD_ATTEMPTS - 1:
raise
delay = RETRY_BACKOFF * 2 ** attempt
cloudlog.warning(f"Retrying {artifact.fileName} in {delay:g}s after {type(e).__name__}: {e}")
progress = artifact.downloadProgress
self._set_progress(artifact, progress.status, progress.progress, progress.eta, 0.0) # stalled: no speed
deadline = time.monotonic() + delay
while (remaining := deadline - time.monotonic()) > 0:
if self._download_interrupted():
raise DownloadCancelled("Download cancelled") from e
await asyncio.sleep(min(RETRY_POLL, remaining))
async def _process_artifact(self, artifact, destination_path: str) -> None:
if not artifact.downloadUri.uri:
return None
@@ -246,37 +330,39 @@ class ModelManagerSP:
filename = artifact.fileName
full_path = os.path.join(destination_path, filename)
status = custom.ModelManagerSP.DownloadStatus
artifact.downloadProgress.resuming = False
artifact.downloadProgress.canceling = False
try:
self._set_progress(artifact, status.verifying, 0)
if await verify_file(full_path, expected_hash):
self._set_progress(artifact, status.cached, 100)
return
if _download_in_progress(full_path):
cloudlog.info(f"Resuming interrupted download of {filename}")
artifact.downloadProgress.resuming = True
else:
self._set_progress(artifact, status.verifying, 0)
if await verify_file(full_path, expected_hash):
self._set_progress(artifact, status.cached, 100)
return
self._set_progress(artifact, status.downloading, 0)
await self._download_file(url, full_path, artifact)
await self._download_with_retries(url, full_path, artifact)
self._set_progress(artifact, status.verifying, 99)
if not await verify_file(full_path, expected_hash):
_remove_file(full_path)
_remove_download_state(full_path)
raise ValueError(f"Hash validation failed for {filename}")
self._set_progress(artifact, status.downloaded, 100)
except DownloadCancelled:
except DownloadCancelled: # not a failure: the request handler clears the row
self._download_start_times.pop(artifact.fileName, None)
artifact.downloadProgress.status = status.failed
artifact.downloadProgress.eta = 0
artifact.downloadProgress.speed = 0
self._sync_artifact_progress(artifact)
if self.selected_bundle:
self.selected_bundle.status = status.failed
self._report_status()
raise
except Exception as e:
cloudlog.error(f"Error downloading {filename}: {str(e)}")
if os.path.isfile(full_path): # noqa: ASYNC240
os.remove(full_path)
artifact.downloadProgress.status = status.failed
artifact.downloadProgress.eta = 0
artifact.downloadProgress.speed = 0
@@ -333,6 +419,8 @@ class ModelManagerSP:
self.params.put(ACTIVE_BUNDLE_KEYS[source], model_bundle.to_dict(), block=True)
self.active_bundle = get_active_bundle(self.params, chestnut=self.chestnut_present)
except DownloadCancelled:
raise # not a failure
except Exception:
if self.selected_bundle is not None:
self.selected_bundle.status = custom.ModelManagerSP.DownloadStatus.failed
@@ -360,11 +448,14 @@ class ModelManagerSP:
self._download_ref = ref_to_download
try:
self.download(model_to_download, Paths.model_root(), source)
except DownloadCancelled:
self.selected_bundle = None
except Exception as e:
cloudlog.exception(e)
cloudlog.exception(e) # a failure stays on the row until the next request
else:
self.selected_bundle = None
finally:
self._release_download_ref()
self.selected_bundle = None
def main_thread(self) -> None:
"""Main thread for model management"""
@@ -11,11 +11,13 @@ import contextlib
import dataclasses
import hashlib
import http.server
import json
import math
import os
import random
import re
import requests
import stat
import tempfile
import threading
import time
@@ -55,6 +57,8 @@ class ServerState:
fail_ranges: dict[tuple[int, int], int] = dataclasses.field(default_factory=dict) # range -> HTTP status, every time
fail_times: dict[tuple[int, int], int] = dataclasses.field(default_factory=dict) # range -> how many more requests 503
short_once: set[tuple[int, int]] = dataclasses.field(default_factory=set) # ranges whose first answer is cut short
fail_after_others: bool = False # hold each failure until every other piece has been sent
sent: set[tuple[int, int]] = dataclasses.field(default_factory=set) # ranges sent in full
stall_ranges: set[tuple[int, int]] = dataclasses.field(default_factory=set)
stall_event: threading.Event | None = None
stall_released_by_event: bool | None = None
@@ -88,6 +92,10 @@ class DownloadHandler(http.server.BaseHTTPRequestHandler):
state.fail_times[rng] -= 1
status = 503
if status is not None:
others = {piece_range(i) for i in range(NUM_PIECES)} - {rng}
deadline = time.monotonic() + 5
while state.fail_after_others and not others <= state.sent and time.monotonic() < deadline:
time.sleep(0.01)
self.send_response(status)
self.end_headers()
return
@@ -115,6 +123,8 @@ class DownloadHandler(http.server.BaseHTTPRequestHandler):
self.wfile.write(payload[100:])
else:
self.wfile.write(payload)
if rng is not None:
state.sent.add(rng)
@contextlib.contextmanager
@@ -133,7 +143,10 @@ def threaded_server(handler):
class InlineExecutor(concurrent.futures.Executor):
def submit(self, fn, /, *args, **kwargs):
future: concurrent.futures.Future = concurrent.futures.Future()
future.set_result(fn(*args, **kwargs)) # the piece lands before the reporter starts
try:
future.set_result(fn(*args, **kwargs)) # the piece lands before the reporter starts
except BaseException as e:
future.set_exception(e)
return future
@@ -147,11 +160,12 @@ class ManagerDownloadTestBase(OpenpilotTestCase):
self.addCleanup(self._tmp.cleanup)
self.dest = self._tmp.name
for name, value in (('PIECE_SIZE', PIECE_SIZE), ('REPORT_INTERVAL', 0.05), ('PIECE_BACKOFF', 0.01), ('SPEED_FLOOR', 100)):
for name, value in (('PIECE_SIZE', PIECE_SIZE), ('REPORT_INTERVAL', 0.05), ('PIECE_BACKOFF', 0.01), ('SPEED_FLOOR', 100), ('RETRY_BACKOFF', 0.01)):
self.enterContext(mock.patch.object(manager_module, name, value))
self.reported: list[float] = []
self.reported_statuses: list[int] = []
self.reported_resuming: list[bool] = []
self.manager = ModelManagerSP.__new__(ModelManagerSP)
self.params = self.manager.params = mock.MagicMock()
@@ -172,6 +186,7 @@ class ManagerDownloadTestBase(OpenpilotTestCase):
self.reported.append(float(artifact.downloadProgress.progress))
status = artifact.downloadProgress.status
self.reported_statuses.append(getattr(status, 'raw', status)) # .raw: _DynamicEnum equals its int but does not hash like it
self.reported_resuming.append(bool(artifact.downloadProgress.resuming))
def make_artifact(self):
bundle = custom.ModelManagerSP.ModelBundle.new_message()
@@ -188,6 +203,24 @@ class ManagerDownloadTestBase(OpenpilotTestCase):
def path(self) -> str:
return os.path.join(self.dest, FILE_NAME)
@property
def sidecar(self) -> str:
return manager_module._sidecar_path(self.path)
def resume_state(self) -> dict:
with open(self.sidecar) as f:
return json.load(f)
def write_resume_state(self, done: list[int], total: int = len(WHOLE_BODY), file_size: int | None = None,
file_sha256: str = sha256(WHOLE_BODY)) -> None:
with open(self.path, 'wb') as f:
f.truncate(len(WHOLE_BODY) if file_size is None else file_size)
for i in done:
f.seek(i * PIECE_SIZE)
f.write(WHOLE_BODY[i * PIECE_SIZE:(i + 1) * PIECE_SIZE])
with open(self.sidecar, 'w') as f:
json.dump({"total": total, "piece_size": PIECE_SIZE, "ranged": True, "sha256": file_sha256, "done": done}, f)
def piece_requests(self) -> list[tuple[int, int]]:
return [r for r in self.server.request_ranges if r is not None and r != (0, 0)]
@@ -209,6 +242,17 @@ class ManagerDownloadTestBase(OpenpilotTestCase):
def download_bundle(self, source: str):
asyncio.run(self.manager._download_bundle(self._bundle, self.dest, source))
def request_download(self) -> dict:
self.make_artifact()
self._bundle.ref = "ref"
self._bundle.minimumSelectorVersion = helpers.REQUIRED_JSON_VERSION
self.manager.source_models = {"qcom": [self._bundle], "chestnut": []}
self.manager.params, store = self._make_params_with_store()
store["ModelManager_DownloadRef"] = "ref"
with mock.patch.object(manager_module.Paths, 'model_root', return_value=self.dest):
self.manager._process_download_requests()
return store
def read_path(self) -> bytes:
with open(self.path, 'rb') as f:
return f.read()
@@ -225,6 +269,7 @@ class ManagerDownloadTestBase(OpenpilotTestCase):
params.get.side_effect = get
params.put.side_effect = put
params.remove.side_effect = lambda key: put(key, None)
return params, store
@@ -238,6 +283,7 @@ class TestManagerDownload(ManagerDownloadTestBase):
assert max(self.reported) <= 99.0, "progress holds below 100 until the hash check"
assert artifact.fileName not in self.manager._download_start_times
assert artifact.downloadProgress.speed > 0, "a running download shows its speed"
assert not os.path.exists(self.sidecar), "the resume state goes once every piece has landed"
def test_speed_counts_bytes_landed_before_the_first_report(self):
self.enterContext(mock.patch.object(manager_module, 'ThreadPoolExecutor', lambda max_workers: InlineExecutor()))
@@ -296,9 +342,42 @@ class TestManagerDownload(ManagerDownloadTestBase):
def test_permanent_failure_is_not_retried(self):
self.server.fail_ranges = {piece_range(1): 404}
self.enterContext(mock.patch.object(manager_module, 'ThreadPoolExecutor', lambda max_workers: InlineExecutor()))
with self.assertRaises(requests.exceptions.HTTPError):
self.download_file()
self.process_artifact()
assert self.piece_requests().count(piece_range(1)) == 1
assert os.path.isfile(self.path)
assert self.resume_state()["done"] == [i for i in range(NUM_PIECES) if i != 1]
def test_failed_transfer_resumes_on_retry(self):
self.server.fail_times = {piece_range(1): manager_module.PIECE_ATTEMPTS} # the first transfer gives up on piece 1
self.server.fail_after_others = True # by then every other piece is saved
self.process_artifact()
assert self.read_path() == WHOLE_BODY
assert self.piece_requests().count(piece_range(1)) == manager_module.PIECE_ATTEMPTS + 1, "the retry fetched piece 1"
others = [i for i in range(NUM_PIECES) if i != 1]
assert all(self.piece_requests().count(piece_range(i)) == 1 for i in others), "the retry skipped every saved piece"
def test_lasting_transient_failure_gives_up(self):
self.server.fail_ranges = {piece_range(1): 503}
with self.assertRaises(requests.exceptions.HTTPError):
self.process_artifact()
assert self.piece_requests().count(piece_range(1)) == manager_module.PIECE_ATTEMPTS * manager_module.DOWNLOAD_ATTEMPTS
def test_cancel_ends_retry_backoff(self):
self.enterContext(mock.patch.object(manager_module, 'RETRY_BACKOFF', 30.0))
self.manager._download_ref = b"ref"
def fail_then_cancel(*args):
self.params.get.return_value = None # the user cancels while the manager waits to retry
raise requests.exceptions.ConnectionError("dropped")
transfer = self.enterContext(mock.patch.object(self.manager, '_download_file', side_effect=fail_then_cancel))
started = time.monotonic()
with self.assertRaises(DownloadCancelled):
self.process_artifact()
assert time.monotonic() - started < 5, "the wait must end at the cancel, not after 30 s"
assert transfer.call_count == 1, "no second transfer after a cancel"
assert self.artifact.downloadProgress.status != custom.ModelManagerSP.DownloadStatus.failed
def test_removing_download_ref_cancels(self):
stall_event = self.stall(3)
@@ -316,6 +395,51 @@ class TestManagerDownload(ManagerDownloadTestBase):
assert self.server.stall_released_by_event is True, "the cancel must have caught the piece mid-transfer"
end = piece_range(3)[1] + 1
assert self.read_path()[end - 100:end] == bytes(100), "the cancelled piece must stop before its last bytes land"
done = self.resume_state()["done"]
assert done and 3 not in done, "a cancel records the finished pieces, and only those, for resume"
def test_resume_skips_complete_pieces(self):
self.write_resume_state(done=[0, 4])
self.download_file()
assert piece_range(0) not in self.piece_requests() and piece_range(4) not in self.piece_requests()
assert self.read_path() == WHOLE_BODY
assert min(self.reported) >= 2 * PIECE_SIZE / len(WHOLE_BODY) * 100 - 1, "progress must not restart below the resumed share"
def test_unusable_resume_state_starts_over(self):
def malformed(content: bytes):
self.write_resume_state(done=[0])
with open(self.sidecar, 'wb') as f:
f.write(content)
layout = {"total": len(WHOLE_BODY), "piece_size": PIECE_SIZE, "ranged": True, "sha256": sha256(WHOLE_BODY)}
cases = {
"other size": lambda: self.write_resume_state(done=[0], total=len(WHOLE_BODY) + 1),
"other bytes": lambda: self.write_resume_state(done=[0], file_sha256="0" * 64),
"truncated file": lambda: self.write_resume_state(done=[0], file_size=PIECE_SIZE),
"not an object": lambda: malformed(b'[]'),
"cut-off json": lambda: malformed(b'{"total": '),
"done not a list": lambda: malformed(json.dumps({**layout, "done": 3}).encode()),
"done not ints": lambda: malformed(json.dumps({**layout, "done": ["0", None]}).encode()),
}
for name, write_state in cases.items():
with self.subTest(name):
self.server.request_ranges = []
write_state()
self.download_file()
assert sorted(self.piece_requests()) == [piece_range(i) for i in range(NUM_PIECES)], "every piece is fetched again"
assert self.read_path() == WHOLE_BODY
def test_pieces_and_resume_state_are_synced_to_disk(self):
real_fsync, synced = os.fsync, []
def record_sync(fd):
st = os.fstat(fd)
synced.append("dir" if stat.S_ISDIR(st.st_mode) else "model" if os.path.exists(self.path) and st.st_ino == os.stat(self.path).st_ino else "sidecar")
real_fsync(fd)
with mock.patch.object(manager_module.os, 'fsync', side_effect=record_sync):
self.download_file()
assert synced.count("model") == NUM_PIECES, f"expected one fsync per piece, got {synced.count('model')}"
assert "sidecar" in synced and "dir" in synced, "sidecar file and directory must be fsync'd"
def test_replaced_download_ref_queues_instead_of_canceling(self):
self.params.get.return_value = b"other-ref"
@@ -345,9 +469,26 @@ class TestProcessArtifact(ManagerDownloadTestBase):
self.server.corrupt = True
with self.assertRaises(ValueError):
self.process_artifact()
assert not os.path.isfile(self.path)
assert not os.path.isfile(self.path) and not os.path.exists(self.sidecar), "a corrupt file must not be resumed from"
assert self.artifact.downloadProgress.status == custom.ModelManagerSP.DownloadStatus.failed
def test_interrupted_download_resumes_without_reverifying(self):
self.write_resume_state(done=[0, 1, 2])
artifact = self.process_artifact()
ds = custom.ModelManagerSP.DownloadStatus
assert self.reported_statuses[0] == ds.downloading, "an in-progress download is not verified first"
assert piece_range(0) not in self.piece_requests()
assert artifact.downloadProgress.status == ds.downloaded
assert any(self.reported_resuming), "a resumed download is flagged resuming until its first progress report"
assert not artifact.downloadProgress.resuming
def test_resume_with_every_piece_done_clears_resuming(self):
self.write_resume_state(done=list(range(NUM_PIECES)))
artifact = self.process_artifact()
assert self.piece_requests() == []
assert artifact.downloadProgress.status == custom.ModelManagerSP.DownloadStatus.downloaded
assert not artifact.downloadProgress.resuming
def test_probe_failure_moves_past_verifying(self):
self.server.fail_ranges = {(0, 0): 404} # the size probe
with self.assertRaises(requests.exceptions.HTTPError):
@@ -374,6 +515,23 @@ class TestProcessArtifact(ManagerDownloadTestBase):
self.download_bundle("qcom")
assert ACTIVE_BUNDLE_KEYS["qcom"] not in store
assert os.path.isfile(self.path), "cancel must not delete a cached model"
assert self.manager.selected_bundle.status != custom.ModelManagerSP.DownloadStatus.failed, "a cancel is not a failure"
def test_failed_download_stays_on_the_row(self):
self.server.fail_ranges = {piece_range(1): 404}
store = self.request_download()
selected = self.manager.selected_bundle
assert selected is not None and selected.status == custom.ModelManagerSP.DownloadStatus.failed, "a failure stays on the row"
assert store["ModelManager_DownloadRef"] is None, "the failed request is released"
def test_finished_download_clears_the_row(self):
self.request_download()
assert self.manager.selected_bundle is None
def test_cancelled_download_clears_the_row(self):
self.pm.send.side_effect = lambda *args: self.manager.params.remove("ModelManager_DownloadRef") # cancel at the first report
self.request_download()
assert self.manager.selected_bundle is None
def test_download_ref_is_released_only_if_unchanged(self):
for current, removed in ((b"new-ref", False), (b"ref", True)):
+16 -13
View File
@@ -19,10 +19,9 @@ from openpilot.tools.lib.route import Route
REPO_ID = os.environ.get("SUNNYPILOT_CAR_SEGMENTS_REPO_ID", "sunnypilot/sunnypilotCarSegments")
def upload_route(route_name: str, dry_run: bool = False) -> None:
route = Route(route_name)
log_paths = route.log_paths()
valid_segments = [(i, url) for i, url in enumerate(log_paths) if url is not None]
def upload_route(route_name: str, dry_run: bool = False, local_dir: str | None = None) -> None:
log_paths = Route(route_name, data_dir=local_dir).log_paths()
valid_segments = [(i, path) for i, path in enumerate(log_paths) if path is not None]
print(f"Route: {route_name}")
print(f"Segments: {len(valid_segments)}/{len(log_paths)}")
@@ -34,14 +33,17 @@ def upload_route(route_name: str, dry_run: bool = False) -> None:
api = HfApi()
with tempfile.TemporaryDirectory() as tmpdir:
for seg_idx, url in tqdm(valid_segments, desc="Uploading"):
filename = url.split("?")[0].rsplit("/", 1)[-1]
local_path = os.path.join(tmpdir, f"{seg_idx}_{filename}")
resp = requests.get(url, stream=True)
resp.raise_for_status()
with open(local_path, "wb") as f:
for chunk in resp.iter_content(chunk_size=8192):
f.write(chunk)
for seg_idx, src in tqdm(valid_segments, desc="Uploading"):
filename = src.split("?")[0].rsplit("/", 1)[-1]
if local_dir is not None:
local_path = src
else:
local_path = os.path.join(tmpdir, f"{seg_idx}_{filename}")
resp = requests.get(src, stream=True)
resp.raise_for_status()
with open(local_path, "wb") as f:
for chunk in resp.iter_content(chunk_size=8192):
f.write(chunk)
repo_path = f"segments/{route_name.replace('|', '/')}/{seg_idx}/{filename}"
@@ -63,6 +65,7 @@ if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Upload route rlogs to sunnypilot HuggingFace dataset")
parser.add_argument("route", help="Route ID (e.g. 5beb9b58bd12b691/0000010a--a51155e496)")
parser.add_argument("--dry-run", action="store_true", help="Download and show sizes without uploading")
parser.add_argument("--local", metavar="DIR", help="Upload rlogs from DIR (<dongle id>|<log id>--<segment>/) instead of the comma API")
args = parser.parse_args()
upload_route(args.route, dry_run=args.dry_run)
upload_route(args.route, dry_run=args.dry_run, local_dir=args.local)