diff --git a/openpilot/starpilot/speed_limits/vision/assets/PROVENANCE.md b/openpilot/starpilot/speed_limits/vision/assets/PROVENANCE.md index 701b3ef62e..cd3a5c2b03 100644 --- a/openpilot/starpilot/speed_limits/vision/assets/PROVENANCE.md +++ b/openpilot/starpilot/speed_limits/vision/assets/PROVENANCE.md @@ -1,12 +1,12 @@ The joint traffic-sign detector contains the original StarPilot US speed graph plus a separate STOP head. The vision producer uses one joint detector forward. Original speed graph weights, -speed decoding, numeric classifier and resource scheduling are preserved. +speed decoding, numeric digit logits and resource scheduling are preserved. The classifier adds a learned rejection-only residual. | File | Bytes | SHA-256 | | --- | ---: | --- | -| `traffic_signs_us_detector.onnx` | 9721639 | `e94f0ef60ce6c631fb69e449b86d50f07518d0031d7d4ec84db912618697d3ed` | +| `traffic_signs_us_detector.onnx` | 9725014 | `ac735518a3d32108dee9a112ec2f740c2ed573daac22798525ecbbbbc0ed6606` | | `speed_limit_us_detector.onnx` | 9,605,948 | `82408b68c79c269296f0af942130c5383cace4ee06c78e2a4690e8488720116a` | -| `speed_limit_us_value_classifier.onnx` | 6,229,930 | `07c6696e530eb940d2757d5849b4bc0f1d785cda704e5296e18c0a94959f30a5` | +| `speed_limit_us_value_classifier.onnx` | 6,240,936 | `5d861067ab8f50986488f983cbc24f78527775d37af0b9740c5545d5f3173902` | The joint detector accepts the existing 256×256 RGB letterbox. `output0` remains the original `(1, 5, 1344)` speed output. `stop/output0` is `(1, 5, 4096)`, with four @@ -28,16 +28,16 @@ Training used the recovered YOLO26n speed checkpoint with all backbone/speed weights frozen. LISA, GLARE, COCO and manually reviewed comma training scenes supplied STOP/negative examples. Original recording splits were retained for LISA/GLARE; augmented and low-light views inherited their source -split. COCO splits were assigned by image identity. Real driving approaches used -for evaluation were excluded from training. Public datasets overlap the pretrained +split. COCO splits were assigned by image identity. Evaluation recordings were kept separate except for the development scenes +explicitly promoted into training below. Public datasets overlap the pretrained backbone's ancestry and do not establish whole-model unseen-scene generalization. The STOP branch was trained for 30 epochs, selected epoch 20, then fine-tuned with mobile Hardswish activations; selected fine-tuning epoch 10. -A final frozen-backbone hard-negative fine-tune selected epoch 4 of 12. Three +The initial frozen-backbone hard-negative fine-tune selected epoch 4 of 12. Three previous development false positives (Do Not Enter / road closure) were explicitly promoted into training, with 195 augmented views; they are no longer held-out -evidence. The final STOP checkpoint SHA-256 is +evidence. That STOP checkpoint SHA-256 is `4fa350a2e7d177eb9b3f48cbeafd1d406560f7a7c77482eb3f61e90716f6f73c`, using Ultralytics 8.4.32, PyTorch 2.15.0, opset 12. Original Ultralytics author/license metadata is retained in the detector. Dataset source/license receipts and original recordings are retained with the training @@ -78,3 +78,75 @@ another device timing pass. A declared-cost replay showed small latency changes can shift speed-sign sampling (73 to 69 correct reads in a 100-scene +5% cost scenario). Frozen speed outputs preserve per-frame accuracy but do not establish identical sampling, loaded-device performance or onroad stopping reliability. + + +## Night and advisory rejection follow-up + +The STOP head was fine-tuned with frozen backbone/speed weights and frozen batch +normalization statistics; selected epoch 6. Six reviewed night images and eight +reviewed construction-warning negatives were explicitly assigned to training. +Those recordings are development/training evidence, not independent test data. +All existing source splits were retained, and original recordings were unchanged. + +A 313-parameter RGB rejection branch was added to the same detector forward: +4x4 average pooling, a 3-to-8 channel 3x3 convolution, Hardswish, an 8-channel +3x3 depthwise convolution, Hardswish, and an 8-to-1 pointwise convolution with +Softplus. Its nonnegative output is subtracted from STOP logits before their +sigmoid. The fine-tuned STOP head and box coordinates stay frozen. The rejection +branch used 3,000 initial updates followed by 2,000 updates with positive-score +margin preservation; no detector or confirmation threshold was relaxed. +The original 402 speed nodes and 204 initializers remain byte-for-byte identical. + +In development replay at one source frame per second, five sampling phases and +400ms assumed completion latency, the previously missed dark approach confirmed +in all five phases. Existing approach confirmation was preserved. Two reviewed +warning-sign sequences produced no confirmed STOP observations; isolated raw +warning proposals can remain. The selected model retained all 108 reviewed strong +STOP detections in an additional backlog sample and improved a teacher-selected +difficult sample from 3 to 14 detections among 46 signs inside the existing crop. +These are selected development samples, not exhaustive road recall estimates. +The retained GLARE test split remained 36/46 IoU50 hits, one false detection, +and no detections on its 50 negative images. Public pretraining ancestry limits +whole-model generalization claims. + +The numeric classifier preserves all 147 original nodes preceding the final +softmax and all 80 original initializers from classifier +`07c6696e530eb940d2757d5849b4bc0f1d785cda704e5296e18c0a94959f30a5`. +A 2,204-parameter residual uses frozen classifier features and original logits: +a 256-to-8 pointwise convolution, Hardswish, concatenated original logits, a +linear output and Softplus. The residual, scaled by 0.9, is added only to the +reject logit. Original numeric logits and their ordering are unchanged for the +same crop; numeric probabilities can decrease. Full-frame proposal selection +and classifier workload are therefore evaluated separately. + +Classifier training retained the original numeric weights and used reviewed +advisory negatives plus an immutable semantic-label overlay. Yellow advisory +numerals previously labeled as regulatory speeds were corrected by visual sign +identity, while warm-white regulatory and school signs were retained. Original +images and train/validation assignments were not changed; validation examples +were not used for gradient updates. The selected rejection checkpoint is update +1,800. Additional validation annotation review was used for model selection. + +The 1,486-frame development replay retained all 395 previously correct reads +among 472 regulatory frames and the same three wrong-value reads. Advisory false +reads dropped from 137 to 43 among 167 advisory frames. No new numeric reads were +introduced in the 845 reviewed reject-frame sample. All 1,858 previously accepted +regulatory crops in the reviewed classifier validation set were retained. +Classifier forwards were 6,108 before and 6,148 after in the full-frame replay; +static accuracy preservation does not establish unchanged processing cadence. +Resource placement, processing-cost cooldown, source freshness, confirmation, +STOP expiry and braking policy remain unchanged. Closed-loop road behavior and +co-load performance are not established by offline replay. + + +The runtime retrieves the original classifier logits in the same classifier +forward. A learned advisory veto can stop further crop expansions only for a +non-regulatory-looking class-0 proposal whose original numeric probability is +at least 0.98 and new reject probability at least 0.90. Previous tentative reads +for that object are discarded. Duplicate class-0 proposals with IoU at least +0.75 are skipped within that frame; other objects, school proposals and the next +camera frame remain eligible. White regulatory crops keep their existing +expansion fallback. Five focused tests cover the real classifier ABI, same-forward logits, invalid/generic +rejections, duplicate-object scope and regulatory fallback. This reduces repeated +classifier work after a confident advisory rejection; it does not change +acceptance, confirmation, expiry or processing-cost cooldown thresholds. diff --git a/openpilot/starpilot/speed_limits/vision/assets/speed_limit_us_value_classifier.onnx b/openpilot/starpilot/speed_limits/vision/assets/speed_limit_us_value_classifier.onnx index c3b2eb51ed..3090c34d0c 100644 Binary files a/openpilot/starpilot/speed_limits/vision/assets/speed_limit_us_value_classifier.onnx and b/openpilot/starpilot/speed_limits/vision/assets/speed_limit_us_value_classifier.onnx differ diff --git a/openpilot/starpilot/speed_limits/vision/assets/traffic_signs_us_detector.onnx b/openpilot/starpilot/speed_limits/vision/assets/traffic_signs_us_detector.onnx index 131c536295..19d489d115 100644 Binary files a/openpilot/starpilot/speed_limits/vision/assets/traffic_signs_us_detector.onnx and b/openpilot/starpilot/speed_limits/vision/assets/traffic_signs_us_detector.onnx differ diff --git a/openpilot/starpilot/speed_limits/vision/model.py b/openpilot/starpilot/speed_limits/vision/model.py index 7d1a749e41..784385a39d 100644 --- a/openpilot/starpilot/speed_limits/vision/model.py +++ b/openpilot/starpilot/speed_limits/vision/model.py @@ -272,7 +272,7 @@ class VisionModelCore: ASSET_DIR = Path(__file__).resolve().parent / "assets" DETECTOR_SHA256 = "82408b68c79c269296f0af942130c5383cace4ee06c78e2a4690e8488720116a" - CLASSIFIER_SHA256 = "07c6696e530eb940d2757d5849b4bc0f1d785cda704e5296e18c0a94959f30a5" + CLASSIFIER_SHA256 = "5d861067ab8f50986488f983cbc24f78527775d37af0b9740c5545d5f3173902" DETECTOR_SIZE = 256 CLASSIFIER_SIZE = 128 @@ -283,12 +283,15 @@ class VisionModelCore: self.net = self._load_net("speed_limit_us_detector.onnx", self.DETECTOR_SHA256) self.classifier_net = self._load_net("speed_limit_us_value_classifier.onnx", self.CLASSIFIER_SHA256) self._validate_model_abi() + self.classifier_logits_layer = ("onnx_node!/model.10/linear/Gemm" + if self.classifier_net.getLayerId("onnx_node!advisory/AddReject") > 0 else None) self.detector_input_size = self.DETECTOR_SIZE self.classifier_input_size = self.CLASSIFIER_SIZE self.last_detector_forward_count = 0 self.last_detector_forward_duration_s = 0.0 self.last_classifier_forward_count = 0 self.last_classifier_forward_duration_s = 0.0 + self.last_classifier_advisory_rejected = False self.temporal_tracking_enabled = False self.latest_detector_proposal = None self.history: deque[HistoryEntry] = deque() @@ -304,6 +307,7 @@ class VisionModelCore: def reset(self) -> None: """Discard evidence from a previous camera, stream or sign-unit setting.""" self.history.clear() + self.last_classifier_advisory_rejected = False self.latest_detector_proposal = None self.published_speed_limit_mph = 0 self.published_confidence = 0.0 @@ -667,6 +671,7 @@ class VisionModelCore: def _classify_speed_limit_from_model(self, sign_crop): + self.last_classifier_advisory_rejected = False if self.classifier_net is None or sign_crop.size == 0: return None @@ -678,7 +683,17 @@ class VisionModelCore: self.classifier_net.setInput(blob) forward_started_at = time.monotonic() - scores = np.array(self.classifier_net.forward()).reshape(-1) + logits_layer = getattr(self, "classifier_logits_layer", None) + original_probabilities = None + if logits_layer is not None: + output, original_logits = self.classifier_net.forward(['output0', logits_layer]) + original_logits = np.asarray(original_logits).reshape(-1) + if original_logits.size != speed_class_count + 1 or not np.isfinite(original_logits).all(): + raise ValueError("Invalid original classifier logits") + original_probabilities = self._normalize_classifier_output(original_logits) + else: + output = self.classifier_net.forward() + scores = np.asarray(output).reshape(-1) self.last_classifier_forward_count += 1 self.last_classifier_forward_duration_s += time.monotonic() - forward_started_at has_reject_class = scores.size == speed_class_count + 1 @@ -691,6 +706,10 @@ class VisionModelCore: confidence = float(speed_probabilities[class_index]) speed_limit = US_CLASSIFIER_SPEED_VALUES[class_index] if has_reject_class and float(probabilities[speed_class_count]) >= max(confidence, US_CLASSIFIER_REJECT_MIN_CONFIDENCE): + self.last_classifier_advisory_rejected = bool( + original_probabilities is not None and float(np.max(original_probabilities[:speed_class_count])) >= .98 and + float(probabilities[speed_class_count]) >= .90 + ) return None minimum_confidence = ( EXTENDED_CLASSIFIER_MIN_CONFIDENCE if speed_limit in EXTENDED_CLASSIFIER_SPEED_VALUES else US_CLASSIFIER_MIN_CONFIDENCE @@ -705,10 +724,21 @@ class VisionModelCore: frame_height, frame_width = frame_bgr.shape[:2] best_detection = None best_score = 0.0 + rejected_advisory_boxes: list[tuple[int, int, int, int]] = [] for proposal_confidence, class_id, (x1, y1, x2, y2) in self._collect_detector_classifier_proposals(frame_bgr): if class_id == 1: continue + if class_id == 0: + duplicate_advisory = False + for rx1, ry1, rx2, ry2 in rejected_advisory_boxes: + overlap = max(0, min(x2, rx2) - max(x1, rx1)) * max(0, min(y2, ry2) - max(y1, ry1)) + union = (x2 - x1) * (y2 - y1) + (rx2 - rx1) * (ry2 - ry1) - overlap + if union > 0 and overlap >= .75 * union: + duplicate_advisory = True + break + if duplicate_advisory: + continue box_width = x2 - x1 box_height = y2 - y1 @@ -814,6 +844,12 @@ class VisionModelCore: is_regulatory = True model_read = self._classify_speed_limit_from_model(sign_crop) + if (class_id == 0 and not raw_is_regulatory and model_read is None and + getattr(self, "last_classifier_advisory_rejected", False)): + # A confident advisory veto applies to this object, not another crop of it. + speed_scores.clear() + rejected_advisory_boxes.append((x1, y1, x2, y2)) + break trusted_model_read = ( class_id == 0 and model_read is not None and diff --git a/openpilot/starpilot/speed_limits/vision/observation.py b/openpilot/starpilot/speed_limits/vision/observation.py index 664281f81b..059e1d8ea2 100644 --- a/openpilot/starpilot/speed_limits/vision/observation.py +++ b/openpilot/starpilot/speed_limits/vision/observation.py @@ -13,7 +13,7 @@ MAX_FRAME_AGE_NS = 250_000_000 # Bound sign-processing latency independently of raw camera admission. MAX_RESULT_AGE_NS = 2_000_000_000 MAX_PAIR_SKEW_NS = 5_000_000 -MODEL_ID = "e94f0ef60ce6c631fb69e449b86d50f07518d0031d7d4ec84db912618697d3ed:07c6696e530eb940d2757d5849b4bc0f1d785cda704e5296e18c0a94959f30a5" +MODEL_ID = "ac735518a3d32108dee9a112ec2f740c2ed573daac22798525ecbbbbc0ed6606:5d861067ab8f50986488f983cbc24f78527775d37af0b9740c5545d5f3173902" def clock_pair_ns() -> tuple[int, int] | None: diff --git a/openpilot/starpilot/speed_limits/vision/tests/test_advisory_rejection.py b/openpilot/starpilot/speed_limits/vision/tests/test_advisory_rejection.py new file mode 100644 index 0000000000..1e375a27f3 --- /dev/null +++ b/openpilot/starpilot/speed_limits/vision/tests/test_advisory_rejection.py @@ -0,0 +1,94 @@ +import unittest +from unittest.mock import Mock, patch + +import numpy as np + +from openpilot.starpilot.speed_limits.vision import model +from openpilot.starpilot.speed_limits.vision.model import VisionModelCore + + +class TestAdvisoryRejection(unittest.TestCase): + def test_pinned_classifier_returns_original_logits_in_one_real_forward(self): + if model.cv2 is None: + self.skipTest('OpenCV DNN is not available on this host') + core = VisionModelCore(is_metric=False) + self.assertIsNotNone(core.classifier_logits_layer) + core._classify_speed_limit_from_model(np.zeros((32, 32, 3), np.uint8)) + self.assertEqual(core.last_classifier_forward_count, 1) + self.assertIsInstance(core.last_classifier_advisory_rejected, bool) + + def classifier(self, *, original_digit=True): + core = VisionModelCore.__new__(VisionModelCore) + logits = np.full((1, 19), -12., np.float32) + logits[0, 8 if original_digit else 18] = 12. + probabilities = np.zeros((1, 19), np.float32) + probabilities[0, 8], probabilities[0, 18] = .01, .99 + core.classifier_net = Mock() + core.classifier_net.forward.return_value = (probabilities, logits) + core.classifier_logits_layer = 'original_logits' + core.classifier_input_size = 128 + core.last_classifier_forward_count = 0 + core.last_classifier_forward_duration_s = 0. + return core + + def test_advisory_rejection_uses_original_logits_in_the_same_forward(self): + core = self.classifier() + self.assertIsNone(core._classify_speed_limit_from_model(np.zeros((32, 32, 3), np.uint8))) + self.assertTrue(core.last_classifier_advisory_rejected) + core.classifier_net.forward.assert_called_once_with(['output0', 'original_logits']) + self.assertEqual(core.last_classifier_forward_count, 1) + self.assertIsNone(core._classify_speed_limit_from_model(np.empty((0, 0, 3), np.uint8))) + self.assertFalse(core.last_classifier_advisory_rejected) + + def test_generic_rejection_and_invalid_logits_cannot_be_an_advisory_veto(self): + core = self.classifier(original_digit=False) + crop = np.zeros((32, 32, 3), np.uint8) + self.assertIsNone(core._classify_speed_limit_from_model(crop)) + self.assertFalse(core.last_classifier_advisory_rejected) + probabilities, logits = core.classifier_net.forward.return_value + for invalid in (logits[:, :18], np.full((1, 19), np.nan, np.float32)): + core.classifier_net.forward.return_value = probabilities, invalid + with self.assertRaisesRegex(ValueError, 'Invalid original classifier logits'): + core._classify_speed_limit_from_model(crop) + self.assertFalse(core.last_classifier_advisory_rejected) + + def detector(self, proposals): + core = VisionModelCore.__new__(VisionModelCore) + core._collect_detector_classifier_proposals = Mock(return_value=proposals) + core._remember_detector_proposal = Mock() + return core + + def test_confident_nonregulatory_rejection_stops_only_that_proposal(self): + core = self.detector([(.99, 0, (240, 30, 280, 90)), (.98, 0, (241, 31, 281, 91)), + (.97, 0, (310, 30, 350, 90))]) + calls = 0 + + def classify(_crop): + nonlocal calls + calls += 1 + core.last_classifier_advisory_rejected = calls == 2 + return None if calls == 2 else (35 if calls == 1 else 55, .99) + + with patch.object(core, '_classify_speed_limit_from_model', side_effect=classify), \ + patch.object(core, '_is_regulatory_speed_sign', side_effect=lambda _crop: calls >= 2): + result = core._detect_sign_from_detector_classifier(np.zeros((200, 400, 3), np.uint8)) + assert result is not None + self.assertEqual(result.speed_limit_mph, 55) + self.assertEqual(calls, 2 + len(model.DETECTOR_CLASSIFIER_EXPANSIONS)) + + def test_white_regulatory_crop_keeps_its_expansion_fallback(self): + core = self.detector([(.99, 0, (240, 30, 280, 90))]) + calls = 0 + + def classify(_crop): + nonlocal calls + calls += 1 + core.last_classifier_advisory_rejected = calls == 1 + return None if calls == 1 else (55, .99) + + with patch.object(core, '_classify_speed_limit_from_model', side_effect=classify), \ + patch.object(core, '_is_regulatory_speed_sign', return_value=True): + result = core._detect_sign_from_detector_classifier(np.zeros((200, 400, 3), np.uint8)) + assert result is not None + self.assertEqual(result.speed_limit_mph, 55) + self.assertEqual(calls, len(model.DETECTOR_CLASSIFIER_EXPANSIONS)) diff --git a/openpilot/starpilot/speed_limits/vision/traffic_signs.py b/openpilot/starpilot/speed_limits/vision/traffic_signs.py index 5c941c99bd..48f42eb70b 100644 --- a/openpilot/starpilot/speed_limits/vision/traffic_signs.py +++ b/openpilot/starpilot/speed_limits/vision/traffic_signs.py @@ -14,7 +14,7 @@ class SignDetections: class TrafficSignModelCore(model.VisionModelCore): - DETECTOR_SHA256 = 'e94f0ef60ce6c631fb69e449b86d50f07518d0031d7d4ec84db912618697d3ed' + DETECTOR_SHA256 = 'ac735518a3d32108dee9a112ec2f740c2ed573daac22798525ecbbbbc0ed6606' MODEL_ID = DETECTOR_SHA256 + ':' + model.VisionModelCore.CLASSIFIER_SHA256 STOP_THRESHOLD = .6 diff --git a/prebuilt.json b/prebuilt.json index 115eab6205..ce6a1d3a85 100644 --- a/prebuilt.json +++ b/prebuilt.json @@ -1 +1 @@ -{"protected_inventory_sha256": "5870a8ec20d98140ff1f527f8f7a8d3714eef42321538d8c90fb5a04591b0c84", "schema_version": 1} +{"protected_inventory_sha256": "fcd05ae38a6a2a27e0d5e38347a880f1eb9cee25539646aede1c9cdeeabd1bdb", "schema_version": 1}