StarPilot/docs/how-to/train-speed-limit-vision.md

Train Speed-Limit Vision

This flow is for replacing the current imported ayoubsa_best checkpoint with a U.S.-focused detector plus a separate posted-speed classifier.

The intended recipe is:

1. bootstrap from public traffic-sign data
2. fine-tune on comma-specific bookmarked drives
3. export ONNX models
4. copy them onto the device for runtime testing

Why Two Models

The detector and the value reader solve different problems.

• detector: find the sign and decide whether it is a regulatory speed-limit sign
• classifier: read the posted value from the cropped sign

This is a better fit for U.S. roads than a single detector with baked-in classes like Speed Limit 10, Speed Limit 20, Speed Limit 30, and so on.

Suggested Public Data

• LISA for U.S. roadside sign geometry and sign style
• ARTS for U.S. MUTCD-style sign annotations and values
• GLARE for U.S. glare/lighting failures

Use public data to get the model into the right regime, then fine-tune on comma bookmarks and replay clips.

Install Training Deps

From the repo root:

uv sync --extra speedvision

The runtime device does not need these packages. Only the training machine does.

Initialize a Workspace

.venv/bin/python scripts/speed_limit_vision/init_workspace.py

This creates .tmp/speed_limit_training with:

• detector image/label folders
• classifier crop folders
• review/bookmark manifests
• raw-source manifests
• export and run directories

To keep the raw datasets off the internal disk, point --workspace at the SSD-backed workspace, for example:

.venv/bin/python scripts/speed_limit_vision/init_workspace.py \
  --workspace /Volumes/T5/starpilot_speed_limit/workspace/speed_limit_training_clean

The workspace now also tracks:

• manifests/raw_sources.csv
• manifests/public_detector_samples.csv
• manifests/public_classifier_samples.csv

Those manifests are the provenance record for every imported public sample.

Import Public Datasets

ARTS challenging subset:

.venv/bin/python scripts/speed_limit_vision/import_arts_challenging.py \
  --workspace /Volumes/T5/starpilot_speed_limit/workspace/speed_limit_training_clean

This imports mapped speed-limit signs from the raw challenging-dev.tar.gz archive and appends detector/classifier provenance rows into the manifest CSVs.

GLARE and LISA should be downloaded into the SSD raw tree first:

• /Volumes/T5/starpilot_speed_limit/raw/glare_official
• /Volumes/T5/starpilot_speed_limit/raw/lisa_official

Then import them into the same workspace so the detector/classifier datasets stay source-traceable.

For GLARE, do not pull the whole Drive tree blindly. Use the filtered raw downloader so only the Images/ and optional Tracks/ files land on disk:

.venv/bin/python scripts/speed_limit_vision/download_glare_raw.py \
  --workspace /Volumes/T5/starpilot_speed_limit/workspace/speed_limit_training_clean \
  --output-root /Volumes/T5/starpilot_speed_limit/raw/glare_raw \
  --prefix Images/ \
  --resume

Then import the completed image tree:

.venv/bin/python scripts/speed_limit_vision/import_glare_images.py \
  --workspace /Volumes/T5/starpilot_speed_limit/workspace/speed_limit_training_clean \
  --overwrite

Import Bookmarked Debug Sessions

After a drive, copy or mount the debug session directory locally, then import it:

.venv/bin/python scripts/speed_limit_vision/import_debug_sessions.py

Or point at specific sessions:

.venv/bin/python scripts/speed_limit_vision/import_debug_sessions.py 20260330_220102 20260330_223355

This writes:

• review/bookmarks.csv
• snapshot images under review/images

That manifest is the shortlist for labeling.

If the route only lives on comma connect, fetch it directly into the same clip layout the bookmark tools expect:

.venv/bin/python scripts/speed_limit_vision/download_connect_routes.py \
  <dongle_id>/<route_log_id> \
  --streams fcamera,qlog

This reads the JWT from ~/.comma/auth.json, downloads the requested route files from comma connect, and updates:

• /Volumes/T5/starpilot_speed_limit/live_route_clips/bookmark_windows/data/media/0/realdata/... when the SSD is mounted, otherwise the same .tmp/... paths under the repo
• /Volumes/T5/starpilot_speed_limit/live_routes_meta/qlog_mtimes.txt when the SSD is mounted, otherwise .tmp/live_routes_meta/qlog_mtimes.txt
• /Volumes/T5/starpilot_speed_limit/live_routes_meta/files.txt when the SSD is mounted, otherwise .tmp/live_routes_meta/files.txt

For multiple routes, either pass several ids on the command line or use --routes-file.

Evaluate Real Sign Lead-Ins

Bookmark stills are often too late. To score what matters, replay the real fcamera.hevc footage from the 5 seconds before each bookmark:

.venv/bin/python scripts/speed_limit_vision/evaluate_bookmark_leadins.py \
  --json-out .tmp/live_route_clips/bookmark_windows_report.json

This produces a per-bookmark report of whether the current runtime saw anything in the usable sign approach window.

The evaluator now reads a local session-to-route map from session_route_map.json under the same live_routes_meta root. Keep that file local or on the SSD so personal route ids never need to live in the repo.

Import Missed Lead-Ins for Labeling

Turn those lead-in misses into review frames and contact sheets:

.venv/bin/python scripts/speed_limit_vision/import_bookmark_leadins.py \
  --mode misses

This writes:

• review/bookmark_leadins.csv
• sampled frames under review/leadins/frames
• contact sheets under review/leadins/contact_sheets

That review set is the right source for labeling missed 55 mph, night, and town-sequence failures.

The bookmark/lead-in importers also accept source metadata fields such as region, device, and driver. Use those when importing debug sessions from multiple users so the comma-specific fine-tune can be sliced by contributor or geography instead of becoming one opaque pool.

To shrink that review set to the most promising frames per missed sign window:

.venv/bin/python scripts/speed_limit_vision/rank_bookmark_leadin_frames.py

This writes review/bookmark_leadin_shortlist.csv with the top-ranked sampled frames per bookmark.

Build the Detector Dataset

Take the imported review images and move or copy the ones you want into:

• detector/images/train
• detector/images/val

Label them in YOLO detect format into:

• detector/labels/train
• detector/labels/val

Recommended classes:

• regulatory_speed_limit
• advisory_speed_limit
• school_zone_speed_limit

The dataset YAML is already generated at:

• detector/dataset.yaml

Build the Value Classifier Dataset

Fill out:

• classifier/value_labels.csv

Columns:

• image_path: source image file
• split: train or val
• speed_limit_mph: posted value such as 25, 35, or 55
• bbox_index: which YOLO box to crop if an image has multiple labeled signs
• padding: optional crop padding ratio
• label_path: optional explicit path to the YOLO label file

Then generate the classifier crop folders:

.venv/bin/python scripts/speed_limit_vision/build_value_dataset.py

This writes cropped sign images into:

• classifier/train/<value>/...
• classifier/val/<value>/...

Train

Detector:

.venv/bin/python scripts/speed_limit_vision/train_detector.py --device mps

Classifier:

.venv/bin/python scripts/speed_limit_vision/train_value_classifier.py --device mps

Use --device cpu, --device mps, or a CUDA device string depending on the training host.

Rebalance Toward Real Comma Data

If the detector starts overfitting to synthetic/public data, build a lighter rebalanced dataset that keeps all real_* detector images and samples the rest:

.venv/bin/python scripts/speed_limit_vision/rebalance_detector_dataset.py \
  --workspace .tmp/speed_limit_training \
  --max-other-train 3200

Then point the detector trainer at the generated YAML:

.venv/bin/python scripts/speed_limit_vision/train_detector.py \
  --workspace .tmp/speed_limit_training \
  --data .tmp/speed_limit_training/detector_rebalanced/dataset.yaml \
  --device mps

This keeps validation unchanged while making retrains faster and more comma-biased.

Export ONNX

.venv/bin/python scripts/speed_limit_vision/export_models.py \
  --detector-weights .tmp/speed_limit_training/runs/detector/yolo11n-speed-limit-us/weights/best.pt \
  --classifier-weights .tmp/speed_limit_training/runs/classifier/yolo11n-cls-speed-limit-us/weights/best.pt \
  --install-repo-assets

That writes:

• .tmp/speed_limit_training/exports/speed_limit_us_detector.onnx
• .tmp/speed_limit_training/exports/speed_limit_us_value_classifier.onnx

And optionally copies them into:

• starpilot/assets/vision_models

Copy to the Device

.venv/bin/python scripts/speed_limit_vision/install_models.py --host comma@192.168.3.110

The runtime already prefers speed_limit_us_detector.onnx plus speed_limit_us_value_classifier.onnx when both files exist in starpilot/assets/vision_models.

Evaluate the Runtime Path

Run the real StarPilot runtime path, using the installed ONNX pair, against the known saved-frame cases:

.venv/bin/python scripts/speed_limit_vision/evaluate_runtime_cases.py --strict

For temporal behavior on a saved frame directory or route extract, replay the runtime directly:

.venv/bin/python scripts/replay_speed_limit_vision.py .tmp/vision_iter/seg10_5fps --frames-fps 5

The detector/classifier runtime is model-only by default. Use --crop-ocr with evaluate_runtime_manifest.py or replay_route_runtime.py only for an explicit legacy comparison. A model-only release must match reviewed-manifest accuracy and pass representative route replays at measured on-device cadence. Evaluate candidate recognition and temporal publish behavior separately: a correct single-frame candidate can still be suppressed by the history and speed-change confirmation policy.

Ignored review rows label the proposed crop, not the entire camera frame. Consequently, negative-window candidate and publish counts from evaluate_reviewed_route_events.py are an upper bound until the full frame is audited; another valid sign can be present outside the rejected crop. Use the per-row output and frame image to audit any regression delta before treating it as a runtime false positive.

Promotion Gate

Do not promote a checkpoint from classifier validation accuracy alone. Export it to an isolated model directory and run the complete runtime pipeline against the reviewed positive, hard-negative, and failed-drive manifests. A candidate must preserve exact-value recall, avoid new wrong-value reads, and remain within the accepted false-positive budget before route replay.

Mine detector proposals that fool an integrated-reject classifier into a new reject class before retraining:

.venv/bin/python scripts/speed_limit_vision/mine_classifier_reject_crops.py \
  --models-dir /path/to/candidate/models \
  --dataset /path/to/versioned/classifier \
  --manifest /path/to/reviewed-negative-manifest.csv

Keep the resulting dataset version separate from the current training set. If a hard-negative retrain lowers reviewed recall, reject the checkpoint even when it improves aggregate validation accuracy or removes a known false positive.

Active-Learning Review Pass

Keep parallel miners in separate directories and merge them only when their model and mining fingerprints match:

.venv/bin/python scripts/speed_limit_vision/merge_manual_review_queues.py \
  /path/to/shard0 /path/to/shard1 /path/to/shard2 /path/to/shard3 \
  --output-dir /path/to/merged

When rescanning with a new model, compare the fingerprinted queues before selecting another batch. The optional review output retains the full queue schema so it can be passed directly to the selector and review server:

.venv/bin/python scripts/speed_limit_vision/compare_manual_review_queues.py \
  --before /path/to/baseline/manual_review_queue.csv \
  --after /path/to/candidate/manual_review_queue.csv \
  --output-csv /path/to/comparison.csv \
  --review-output /path/to/disagreements/manual_review_queue.csv

.venv/bin/python scripts/speed_limit_vision/select_manual_review_queue.py \
  --input /path/to/disagreements/manual_review_queue.csv \
  --output /path/to/review/manual_review_queue.csv \
  --max-rows 1200 \
  --min-seconds-per-route-speed 3

The selector prioritizes value changes and gained/lost reads, balances routes and speed classes, and removes adjacent same-speed frames from one scene. Start the reviewer and import its labels without moving route media off the training volume:

.venv/bin/python scripts/speed_limit_vision/serve_manual_review_queue.py \
  --manifest /path/to/review/manual_review_queue.csv \
  --port 8765

.venv/bin/python scripts/speed_limit_vision/import_manual_review_queue.py \
  --queue /path/to/review/manual_review_queue.csv

Re-mine the Route Backlog

Re-run the backlog after a candidate passes the reviewed-manifest and route replay gates. Use a model fingerprinted run so new pseudo-labels are staged next to, rather than merged into, the original route-mining data:

.venv/bin/python scripts/speed_limit_vision/mine_route_training_samples.py \
  --workspace /Volumes/T5/starpilot_speed_limit/workspace/speed_limit_training_clean \
  --models-dir /path/to/promoted/models \
  --model-only \
  --run-id auto \
  --sample-every 2.0 \
  --transition-step 0.5 \
  --max-frames-per-route 720 \
  --max-positives-per-route 120 \
  --max-negatives-per-route 200

The output is written under staging/route_mining/model_<model-fingerprint>_run_<mining-fingerprint>/ with its own detector images, classifier labels, review manifest, and per-route completion state. The mining fingerprint includes the model-only mode, thresholds, sampling configuration, and relevant source code. Review and deduplicate that staged run before merging it into a training dataset. Never overwrite the canonical route samples or automatically train on every mined positive; map agreement and human review remain required because a stronger model can still reproduce its own mistakes at larger scale.