mirror of
https://github.com/firestar5683/StarPilot.git
synced 2026-11-08 05:23:41 +08:00
Improve night stop detection and reject advisory speed signs
This commit is contained in:
firestar5683
@@ -1,12 +1,12 @@
|
||||
The joint traffic-sign detector contains the original StarPilot US speed graph
|
||||
plus a separate STOP head. The vision producer uses one joint detector forward. Original speed graph weights,
|
||||
speed decoding, numeric classifier and resource scheduling are preserved.
|
||||
speed decoding, numeric digit logits and resource scheduling are preserved. The classifier adds a learned rejection-only residual.
|
||||
|
||||
| File | Bytes | SHA-256 |
|
||||
| --- | ---: | --- |
|
||||
| `traffic_signs_us_detector.onnx` | 9721639 | `e94f0ef60ce6c631fb69e449b86d50f07518d0031d7d4ec84db912618697d3ed` |
|
||||
| `traffic_signs_us_detector.onnx` | 9725014 | `ac735518a3d32108dee9a112ec2f740c2ed573daac22798525ecbbbbc0ed6606` |
|
||||
| `speed_limit_us_detector.onnx` | 9,605,948 | `82408b68c79c269296f0af942130c5383cace4ee06c78e2a4690e8488720116a` |
|
||||
| `speed_limit_us_value_classifier.onnx` | 6,229,930 | `07c6696e530eb940d2757d5849b4bc0f1d785cda704e5296e18c0a94959f30a5` |
|
||||
| `speed_limit_us_value_classifier.onnx` | 6,240,936 | `5d861067ab8f50986488f983cbc24f78527775d37af0b9740c5545d5f3173902` |
|
||||
|
||||
The joint detector accepts the existing 256×256 RGB letterbox. `output0` remains the
|
||||
original `(1, 5, 1344)` speed output. `stop/output0` is `(1, 5, 4096)`, with four
|
||||
@@ -28,16 +28,16 @@ Training used the recovered YOLO26n speed checkpoint
|
||||
with all backbone/speed weights frozen. LISA, GLARE, COCO and manually reviewed
|
||||
comma training scenes supplied STOP/negative examples. Original recording splits
|
||||
were retained for LISA/GLARE; augmented and low-light views inherited their source
|
||||
split. COCO splits were assigned by image identity. Real driving approaches used
|
||||
for evaluation were excluded from training. Public datasets overlap the pretrained
|
||||
split. COCO splits were assigned by image identity. Evaluation recordings were kept separate except for the development scenes
|
||||
explicitly promoted into training below. Public datasets overlap the pretrained
|
||||
backbone's ancestry and do not establish whole-model unseen-scene generalization.
|
||||
|
||||
The STOP branch was trained for 30 epochs, selected epoch 20, then fine-tuned
|
||||
with mobile Hardswish activations; selected fine-tuning epoch 10.
|
||||
A final frozen-backbone hard-negative fine-tune selected epoch 4 of 12. Three
|
||||
The initial frozen-backbone hard-negative fine-tune selected epoch 4 of 12. Three
|
||||
previous development false positives (Do Not Enter / road closure) were explicitly
|
||||
promoted into training, with 195 augmented views; they are no longer held-out
|
||||
evidence. The final STOP checkpoint SHA-256 is
|
||||
evidence. That STOP checkpoint SHA-256 is
|
||||
`4fa350a2e7d177eb9b3f48cbeafd1d406560f7a7c77482eb3f61e90716f6f73c`, using Ultralytics 8.4.32, PyTorch 2.15.0, opset 12.
|
||||
Original Ultralytics author/license metadata is retained in the detector. Dataset
|
||||
source/license receipts and original recordings are retained with the training
|
||||
@@ -78,3 +78,75 @@ another device timing pass. A declared-cost replay showed small latency changes
|
||||
can shift speed-sign sampling (73 to 69 correct reads in a 100-scene +5% cost
|
||||
scenario). Frozen speed outputs preserve per-frame accuracy but do not establish
|
||||
identical sampling, loaded-device performance or onroad stopping reliability.
|
||||
|
||||
|
||||
## Night and advisory rejection follow-up
|
||||
|
||||
The STOP head was fine-tuned with frozen backbone/speed weights and frozen batch
|
||||
normalization statistics; selected epoch 6. Six reviewed night images and eight
|
||||
reviewed construction-warning negatives were explicitly assigned to training.
|
||||
Those recordings are development/training evidence, not independent test data.
|
||||
All existing source splits were retained, and original recordings were unchanged.
|
||||
|
||||
A 313-parameter RGB rejection branch was added to the same detector forward:
|
||||
4x4 average pooling, a 3-to-8 channel 3x3 convolution, Hardswish, an 8-channel
|
||||
3x3 depthwise convolution, Hardswish, and an 8-to-1 pointwise convolution with
|
||||
Softplus. Its nonnegative output is subtracted from STOP logits before their
|
||||
sigmoid. The fine-tuned STOP head and box coordinates stay frozen. The rejection
|
||||
branch used 3,000 initial updates followed by 2,000 updates with positive-score
|
||||
margin preservation; no detector or confirmation threshold was relaxed.
|
||||
The original 402 speed nodes and 204 initializers remain byte-for-byte identical.
|
||||
|
||||
In development replay at one source frame per second, five sampling phases and
|
||||
400ms assumed completion latency, the previously missed dark approach confirmed
|
||||
in all five phases. Existing approach confirmation was preserved. Two reviewed
|
||||
warning-sign sequences produced no confirmed STOP observations; isolated raw
|
||||
warning proposals can remain. The selected model retained all 108 reviewed strong
|
||||
STOP detections in an additional backlog sample and improved a teacher-selected
|
||||
difficult sample from 3 to 14 detections among 46 signs inside the existing crop.
|
||||
These are selected development samples, not exhaustive road recall estimates.
|
||||
The retained GLARE test split remained 36/46 IoU50 hits, one false detection,
|
||||
and no detections on its 50 negative images. Public pretraining ancestry limits
|
||||
whole-model generalization claims.
|
||||
|
||||
The numeric classifier preserves all 147 original nodes preceding the final
|
||||
softmax and all 80 original initializers from classifier
|
||||
`07c6696e530eb940d2757d5849b4bc0f1d785cda704e5296e18c0a94959f30a5`.
|
||||
A 2,204-parameter residual uses frozen classifier features and original logits:
|
||||
a 256-to-8 pointwise convolution, Hardswish, concatenated original logits, a
|
||||
linear output and Softplus. The residual, scaled by 0.9, is added only to the
|
||||
reject logit. Original numeric logits and their ordering are unchanged for the
|
||||
same crop; numeric probabilities can decrease. Full-frame proposal selection
|
||||
and classifier workload are therefore evaluated separately.
|
||||
|
||||
Classifier training retained the original numeric weights and used reviewed
|
||||
advisory negatives plus an immutable semantic-label overlay. Yellow advisory
|
||||
numerals previously labeled as regulatory speeds were corrected by visual sign
|
||||
identity, while warm-white regulatory and school signs were retained. Original
|
||||
images and train/validation assignments were not changed; validation examples
|
||||
were not used for gradient updates. The selected rejection checkpoint is update
|
||||
1,800. Additional validation annotation review was used for model selection.
|
||||
|
||||
The 1,486-frame development replay retained all 395 previously correct reads
|
||||
among 472 regulatory frames and the same three wrong-value reads. Advisory false
|
||||
reads dropped from 137 to 43 among 167 advisory frames. No new numeric reads were
|
||||
introduced in the 845 reviewed reject-frame sample. All 1,858 previously accepted
|
||||
regulatory crops in the reviewed classifier validation set were retained.
|
||||
Classifier forwards were 6,108 before and 6,148 after in the full-frame replay;
|
||||
static accuracy preservation does not establish unchanged processing cadence.
|
||||
Resource placement, processing-cost cooldown, source freshness, confirmation,
|
||||
STOP expiry and braking policy remain unchanged. Closed-loop road behavior and
|
||||
co-load performance are not established by offline replay.
|
||||
|
||||
|
||||
The runtime retrieves the original classifier logits in the same classifier
|
||||
forward. A learned advisory veto can stop further crop expansions only for a
|
||||
non-regulatory-looking class-0 proposal whose original numeric probability is
|
||||
at least 0.98 and new reject probability at least 0.90. Previous tentative reads
|
||||
for that object are discarded. Duplicate class-0 proposals with IoU at least
|
||||
0.75 are skipped within that frame; other objects, school proposals and the next
|
||||
camera frame remain eligible. White regulatory crops keep their existing
|
||||
expansion fallback. Five focused tests cover the real classifier ABI, same-forward logits, invalid/generic
|
||||
rejections, duplicate-object scope and regulatory fallback. This reduces repeated
|
||||
classifier work after a confident advisory rejection; it does not change
|
||||
acceptance, confirmation, expiry or processing-cost cooldown thresholds.
|
||||
|
||||
Binary file not shown.
Binary file not shown.
@@ -272,7 +272,7 @@ class VisionModelCore:
|
||||
|
||||
ASSET_DIR = Path(__file__).resolve().parent / "assets"
|
||||
DETECTOR_SHA256 = "82408b68c79c269296f0af942130c5383cace4ee06c78e2a4690e8488720116a"
|
||||
CLASSIFIER_SHA256 = "07c6696e530eb940d2757d5849b4bc0f1d785cda704e5296e18c0a94959f30a5"
|
||||
CLASSIFIER_SHA256 = "5d861067ab8f50986488f983cbc24f78527775d37af0b9740c5545d5f3173902"
|
||||
DETECTOR_SIZE = 256
|
||||
CLASSIFIER_SIZE = 128
|
||||
|
||||
@@ -283,12 +283,15 @@ class VisionModelCore:
|
||||
self.net = self._load_net("speed_limit_us_detector.onnx", self.DETECTOR_SHA256)
|
||||
self.classifier_net = self._load_net("speed_limit_us_value_classifier.onnx", self.CLASSIFIER_SHA256)
|
||||
self._validate_model_abi()
|
||||
self.classifier_logits_layer = ("onnx_node!/model.10/linear/Gemm"
|
||||
if self.classifier_net.getLayerId("onnx_node!advisory/AddReject") > 0 else None)
|
||||
self.detector_input_size = self.DETECTOR_SIZE
|
||||
self.classifier_input_size = self.CLASSIFIER_SIZE
|
||||
self.last_detector_forward_count = 0
|
||||
self.last_detector_forward_duration_s = 0.0
|
||||
self.last_classifier_forward_count = 0
|
||||
self.last_classifier_forward_duration_s = 0.0
|
||||
self.last_classifier_advisory_rejected = False
|
||||
self.temporal_tracking_enabled = False
|
||||
self.latest_detector_proposal = None
|
||||
self.history: deque[HistoryEntry] = deque()
|
||||
@@ -304,6 +307,7 @@ class VisionModelCore:
|
||||
def reset(self) -> None:
|
||||
"""Discard evidence from a previous camera, stream or sign-unit setting."""
|
||||
self.history.clear()
|
||||
self.last_classifier_advisory_rejected = False
|
||||
self.latest_detector_proposal = None
|
||||
self.published_speed_limit_mph = 0
|
||||
self.published_confidence = 0.0
|
||||
@@ -667,6 +671,7 @@ class VisionModelCore:
|
||||
|
||||
|
||||
def _classify_speed_limit_from_model(self, sign_crop):
|
||||
self.last_classifier_advisory_rejected = False
|
||||
if self.classifier_net is None or sign_crop.size == 0:
|
||||
return None
|
||||
|
||||
@@ -678,7 +683,17 @@ class VisionModelCore:
|
||||
self.classifier_net.setInput(blob)
|
||||
|
||||
forward_started_at = time.monotonic()
|
||||
scores = np.array(self.classifier_net.forward()).reshape(-1)
|
||||
logits_layer = getattr(self, "classifier_logits_layer", None)
|
||||
original_probabilities = None
|
||||
if logits_layer is not None:
|
||||
output, original_logits = self.classifier_net.forward(['output0', logits_layer])
|
||||
original_logits = np.asarray(original_logits).reshape(-1)
|
||||
if original_logits.size != speed_class_count + 1 or not np.isfinite(original_logits).all():
|
||||
raise ValueError("Invalid original classifier logits")
|
||||
original_probabilities = self._normalize_classifier_output(original_logits)
|
||||
else:
|
||||
output = self.classifier_net.forward()
|
||||
scores = np.asarray(output).reshape(-1)
|
||||
self.last_classifier_forward_count += 1
|
||||
self.last_classifier_forward_duration_s += time.monotonic() - forward_started_at
|
||||
has_reject_class = scores.size == speed_class_count + 1
|
||||
@@ -691,6 +706,10 @@ class VisionModelCore:
|
||||
confidence = float(speed_probabilities[class_index])
|
||||
speed_limit = US_CLASSIFIER_SPEED_VALUES[class_index]
|
||||
if has_reject_class and float(probabilities[speed_class_count]) >= max(confidence, US_CLASSIFIER_REJECT_MIN_CONFIDENCE):
|
||||
self.last_classifier_advisory_rejected = bool(
|
||||
original_probabilities is not None and float(np.max(original_probabilities[:speed_class_count])) >= .98 and
|
||||
float(probabilities[speed_class_count]) >= .90
|
||||
)
|
||||
return None
|
||||
minimum_confidence = (
|
||||
EXTENDED_CLASSIFIER_MIN_CONFIDENCE if speed_limit in EXTENDED_CLASSIFIER_SPEED_VALUES else US_CLASSIFIER_MIN_CONFIDENCE
|
||||
@@ -705,10 +724,21 @@ class VisionModelCore:
|
||||
frame_height, frame_width = frame_bgr.shape[:2]
|
||||
best_detection = None
|
||||
best_score = 0.0
|
||||
rejected_advisory_boxes: list[tuple[int, int, int, int]] = []
|
||||
|
||||
for proposal_confidence, class_id, (x1, y1, x2, y2) in self._collect_detector_classifier_proposals(frame_bgr):
|
||||
if class_id == 1:
|
||||
continue
|
||||
if class_id == 0:
|
||||
duplicate_advisory = False
|
||||
for rx1, ry1, rx2, ry2 in rejected_advisory_boxes:
|
||||
overlap = max(0, min(x2, rx2) - max(x1, rx1)) * max(0, min(y2, ry2) - max(y1, ry1))
|
||||
union = (x2 - x1) * (y2 - y1) + (rx2 - rx1) * (ry2 - ry1) - overlap
|
||||
if union > 0 and overlap >= .75 * union:
|
||||
duplicate_advisory = True
|
||||
break
|
||||
if duplicate_advisory:
|
||||
continue
|
||||
|
||||
box_width = x2 - x1
|
||||
box_height = y2 - y1
|
||||
@@ -814,6 +844,12 @@ class VisionModelCore:
|
||||
is_regulatory = True
|
||||
|
||||
model_read = self._classify_speed_limit_from_model(sign_crop)
|
||||
if (class_id == 0 and not raw_is_regulatory and model_read is None and
|
||||
getattr(self, "last_classifier_advisory_rejected", False)):
|
||||
# A confident advisory veto applies to this object, not another crop of it.
|
||||
speed_scores.clear()
|
||||
rejected_advisory_boxes.append((x1, y1, x2, y2))
|
||||
break
|
||||
trusted_model_read = (
|
||||
class_id == 0 and
|
||||
model_read is not None and
|
||||
|
||||
@@ -13,7 +13,7 @@ MAX_FRAME_AGE_NS = 250_000_000
|
||||
# Bound sign-processing latency independently of raw camera admission.
|
||||
MAX_RESULT_AGE_NS = 2_000_000_000
|
||||
MAX_PAIR_SKEW_NS = 5_000_000
|
||||
MODEL_ID = "e94f0ef60ce6c631fb69e449b86d50f07518d0031d7d4ec84db912618697d3ed:07c6696e530eb940d2757d5849b4bc0f1d785cda704e5296e18c0a94959f30a5"
|
||||
MODEL_ID = "ac735518a3d32108dee9a112ec2f740c2ed573daac22798525ecbbbbc0ed6606:5d861067ab8f50986488f983cbc24f78527775d37af0b9740c5545d5f3173902"
|
||||
|
||||
|
||||
def clock_pair_ns() -> tuple[int, int] | None:
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
import unittest
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
import numpy as np
|
||||
|
||||
from openpilot.starpilot.speed_limits.vision import model
|
||||
from openpilot.starpilot.speed_limits.vision.model import VisionModelCore
|
||||
|
||||
|
||||
class TestAdvisoryRejection(unittest.TestCase):
|
||||
def test_pinned_classifier_returns_original_logits_in_one_real_forward(self):
|
||||
if model.cv2 is None:
|
||||
self.skipTest('OpenCV DNN is not available on this host')
|
||||
core = VisionModelCore(is_metric=False)
|
||||
self.assertIsNotNone(core.classifier_logits_layer)
|
||||
core._classify_speed_limit_from_model(np.zeros((32, 32, 3), np.uint8))
|
||||
self.assertEqual(core.last_classifier_forward_count, 1)
|
||||
self.assertIsInstance(core.last_classifier_advisory_rejected, bool)
|
||||
|
||||
def classifier(self, *, original_digit=True):
|
||||
core = VisionModelCore.__new__(VisionModelCore)
|
||||
logits = np.full((1, 19), -12., np.float32)
|
||||
logits[0, 8 if original_digit else 18] = 12.
|
||||
probabilities = np.zeros((1, 19), np.float32)
|
||||
probabilities[0, 8], probabilities[0, 18] = .01, .99
|
||||
core.classifier_net = Mock()
|
||||
core.classifier_net.forward.return_value = (probabilities, logits)
|
||||
core.classifier_logits_layer = 'original_logits'
|
||||
core.classifier_input_size = 128
|
||||
core.last_classifier_forward_count = 0
|
||||
core.last_classifier_forward_duration_s = 0.
|
||||
return core
|
||||
|
||||
def test_advisory_rejection_uses_original_logits_in_the_same_forward(self):
|
||||
core = self.classifier()
|
||||
self.assertIsNone(core._classify_speed_limit_from_model(np.zeros((32, 32, 3), np.uint8)))
|
||||
self.assertTrue(core.last_classifier_advisory_rejected)
|
||||
core.classifier_net.forward.assert_called_once_with(['output0', 'original_logits'])
|
||||
self.assertEqual(core.last_classifier_forward_count, 1)
|
||||
self.assertIsNone(core._classify_speed_limit_from_model(np.empty((0, 0, 3), np.uint8)))
|
||||
self.assertFalse(core.last_classifier_advisory_rejected)
|
||||
|
||||
def test_generic_rejection_and_invalid_logits_cannot_be_an_advisory_veto(self):
|
||||
core = self.classifier(original_digit=False)
|
||||
crop = np.zeros((32, 32, 3), np.uint8)
|
||||
self.assertIsNone(core._classify_speed_limit_from_model(crop))
|
||||
self.assertFalse(core.last_classifier_advisory_rejected)
|
||||
probabilities, logits = core.classifier_net.forward.return_value
|
||||
for invalid in (logits[:, :18], np.full((1, 19), np.nan, np.float32)):
|
||||
core.classifier_net.forward.return_value = probabilities, invalid
|
||||
with self.assertRaisesRegex(ValueError, 'Invalid original classifier logits'):
|
||||
core._classify_speed_limit_from_model(crop)
|
||||
self.assertFalse(core.last_classifier_advisory_rejected)
|
||||
|
||||
def detector(self, proposals):
|
||||
core = VisionModelCore.__new__(VisionModelCore)
|
||||
core._collect_detector_classifier_proposals = Mock(return_value=proposals)
|
||||
core._remember_detector_proposal = Mock()
|
||||
return core
|
||||
|
||||
def test_confident_nonregulatory_rejection_stops_only_that_proposal(self):
|
||||
core = self.detector([(.99, 0, (240, 30, 280, 90)), (.98, 0, (241, 31, 281, 91)),
|
||||
(.97, 0, (310, 30, 350, 90))])
|
||||
calls = 0
|
||||
|
||||
def classify(_crop):
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
core.last_classifier_advisory_rejected = calls == 2
|
||||
return None if calls == 2 else (35 if calls == 1 else 55, .99)
|
||||
|
||||
with patch.object(core, '_classify_speed_limit_from_model', side_effect=classify), \
|
||||
patch.object(core, '_is_regulatory_speed_sign', side_effect=lambda _crop: calls >= 2):
|
||||
result = core._detect_sign_from_detector_classifier(np.zeros((200, 400, 3), np.uint8))
|
||||
assert result is not None
|
||||
self.assertEqual(result.speed_limit_mph, 55)
|
||||
self.assertEqual(calls, 2 + len(model.DETECTOR_CLASSIFIER_EXPANSIONS))
|
||||
|
||||
def test_white_regulatory_crop_keeps_its_expansion_fallback(self):
|
||||
core = self.detector([(.99, 0, (240, 30, 280, 90))])
|
||||
calls = 0
|
||||
|
||||
def classify(_crop):
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
core.last_classifier_advisory_rejected = calls == 1
|
||||
return None if calls == 1 else (55, .99)
|
||||
|
||||
with patch.object(core, '_classify_speed_limit_from_model', side_effect=classify), \
|
||||
patch.object(core, '_is_regulatory_speed_sign', return_value=True):
|
||||
result = core._detect_sign_from_detector_classifier(np.zeros((200, 400, 3), np.uint8))
|
||||
assert result is not None
|
||||
self.assertEqual(result.speed_limit_mph, 55)
|
||||
self.assertEqual(calls, len(model.DETECTOR_CLASSIFIER_EXPANSIONS))
|
||||
@@ -14,7 +14,7 @@ class SignDetections:
|
||||
|
||||
|
||||
class TrafficSignModelCore(model.VisionModelCore):
|
||||
DETECTOR_SHA256 = 'e94f0ef60ce6c631fb69e449b86d50f07518d0031d7d4ec84db912618697d3ed'
|
||||
DETECTOR_SHA256 = 'ac735518a3d32108dee9a112ec2f740c2ed573daac22798525ecbbbbc0ed6606'
|
||||
MODEL_ID = DETECTOR_SHA256 + ':' + model.VisionModelCore.CLASSIFIER_SHA256
|
||||
STOP_THRESHOLD = .6
|
||||
|
||||
|
||||
+1
-1
@@ -1 +1 @@
|
||||
{"protected_inventory_sha256": "5870a8ec20d98140ff1f527f8f7a8d3714eef42321538d8c90fb5a04591b0c84", "schema_version": 1}
|
||||
{"protected_inventory_sha256": "fcd05ae38a6a2a27e0d5e38347a880f1eb9cee25539646aede1c9cdeeabd1bdb", "schema_version": 1}
|
||||
|
||||
Reference in New Issue
Block a user