StarPilot/docs/how-to/train-speed-limit-vision.md

# Train Speed-Limit Vision

This flow is for replacing the current imported `ayoubsa_best` checkpoint with a U.S.-focused detector plus a separate posted-speed classifier.

The intended recipe is:

1. bootstrap from public traffic-sign data
2. fine-tune on comma-specific bookmarked drives
3. export ONNX models
4. copy them onto the device for runtime testing

## Why Two Models

The detector and the value reader solve different problems.

- detector: find the sign and decide whether it is a regulatory speed-limit sign
- classifier: read the posted value from the cropped sign

This is a better fit for U.S. roads than a single detector with baked-in classes like `Speed Limit 10`, `Speed Limit 20`, `Speed Limit 30`, and so on.

## Suggested Public Data

- `LISA` for U.S. roadside sign geometry and sign style
- `ARTS` for U.S. MUTCD-style sign annotations and values
- `GLARE` for U.S. glare/lighting failures

Use public data to get the model into the right regime, then fine-tune on comma bookmarks and replay clips.

## Install Training Deps

From the repo root:

```bash
uv sync --extra speedvision
```

The runtime device does not need these packages. Only the training machine does.

## Initialize a Workspace

```bash
.venv/bin/python scripts/speed_limit_vision/init_workspace.py
```

This creates `.tmp/speed_limit_training` with:

- detector image/label folders
- classifier crop folders
- review/bookmark manifests
- raw-source manifests
- export and run directories

To keep the raw datasets off the internal disk, point `--workspace` at the SSD-backed workspace, for example:

```bash
.venv/bin/python scripts/speed_limit_vision/init_workspace.py \
  --workspace /Volumes/T5/starpilot_speed_limit/workspace/speed_limit_training_clean
```

The workspace now also tracks:

- `manifests/raw_sources.csv`
- `manifests/public_detector_samples.csv`
- `manifests/public_classifier_samples.csv`

Those manifests are the provenance record for every imported public sample.

## Import Public Datasets

ARTS challenging subset:

```bash
.venv/bin/python scripts/speed_limit_vision/import_arts_challenging.py \
  --workspace /Volumes/T5/starpilot_speed_limit/workspace/speed_limit_training_clean
```

This imports mapped speed-limit signs from the raw `challenging-dev.tar.gz` archive and appends detector/classifier provenance rows into the manifest CSVs.

GLARE and LISA should be downloaded into the SSD raw tree first:

- `/Volumes/T5/starpilot_speed_limit/raw/glare_official`
- `/Volumes/T5/starpilot_speed_limit/raw/lisa_official`

Then import them into the same workspace so the detector/classifier datasets stay source-traceable.

For GLARE, do not pull the whole Drive tree blindly. Use the filtered raw downloader so only the `Images/` and optional `Tracks/` files land on disk:

```bash
.venv/bin/python scripts/speed_limit_vision/download_glare_raw.py \
  --workspace /Volumes/T5/starpilot_speed_limit/workspace/speed_limit_training_clean \
  --output-root /Volumes/T5/starpilot_speed_limit/raw/glare_raw \
  --prefix Images/ \
  --resume
```

Then import the completed image tree:

```bash
.venv/bin/python scripts/speed_limit_vision/import_glare_images.py \
  --workspace /Volumes/T5/starpilot_speed_limit/workspace/speed_limit_training_clean \
  --overwrite
```

## Import Bookmarked Debug Sessions

After a drive, copy or mount the debug session directory locally, then import it:

```bash
.venv/bin/python scripts/speed_limit_vision/import_debug_sessions.py
```

Or point at specific sessions:

```bash
.venv/bin/python scripts/speed_limit_vision/import_debug_sessions.py 20260330_220102 20260330_223355
```

This writes:

- `review/bookmarks.csv`
- snapshot images under `review/images`

That manifest is the shortlist for labeling.

If the route only lives on comma connect, fetch it directly into the same clip layout the bookmark tools expect:

```bash
.venv/bin/python scripts/speed_limit_vision/download_connect_routes.py \
  <dongle_id>/<route_log_id> \
  --streams fcamera,qlog
```

This reads the JWT from `~/.comma/auth.json`, downloads the requested route files from comma connect, and updates:

- `/Volumes/T5/starpilot_speed_limit/live_route_clips/bookmark_windows/data/media/0/realdata/...` when the SSD is mounted, otherwise the same `.tmp/...` paths under the repo
- `/Volumes/T5/starpilot_speed_limit/live_routes_meta/qlog_mtimes.txt` when the SSD is mounted, otherwise `.tmp/live_routes_meta/qlog_mtimes.txt`
- `/Volumes/T5/starpilot_speed_limit/live_routes_meta/files.txt` when the SSD is mounted, otherwise `.tmp/live_routes_meta/files.txt`

For multiple routes, either pass several ids on the command line or use `--routes-file`.

## Evaluate Real Sign Lead-Ins

Bookmark stills are often too late. To score what matters, replay the real `fcamera.hevc` footage from the `5` seconds before each bookmark:

```bash
.venv/bin/python scripts/speed_limit_vision/evaluate_bookmark_leadins.py \
  --json-out .tmp/live_route_clips/bookmark_windows_report.json
```

This produces a per-bookmark report of whether the current runtime saw anything in the usable sign approach window.

The evaluator now reads a local session-to-route map from `session_route_map.json` under the same `live_routes_meta` root. Keep that file local or on the SSD so personal route ids never need to live in the repo.

## Import Missed Lead-Ins for Labeling

Turn those lead-in misses into review frames and contact sheets:

```bash
.venv/bin/python scripts/speed_limit_vision/import_bookmark_leadins.py \
  --mode misses
```

This writes:

- `review/bookmark_leadins.csv`
- sampled frames under `review/leadins/frames`
- contact sheets under `review/leadins/contact_sheets`

That review set is the right source for labeling missed `55 mph`, night, and town-sequence failures.

The bookmark/lead-in importers also accept source metadata fields such as region, device, and driver. Use those when importing debug sessions from multiple users so the comma-specific fine-tune can be sliced by contributor or geography instead of becoming one opaque pool.

To shrink that review set to the most promising frames per missed sign window:

```bash
.venv/bin/python scripts/speed_limit_vision/rank_bookmark_leadin_frames.py
```

This writes `review/bookmark_leadin_shortlist.csv` with the top-ranked sampled frames per bookmark.

## Build the Detector Dataset

Take the imported review images and move or copy the ones you want into:

- `detector/images/train`
- `detector/images/val`

Label them in YOLO detect format into:

- `detector/labels/train`
- `detector/labels/val`

Recommended classes:

- `regulatory_speed_limit`
- `advisory_speed_limit`
- `school_zone_speed_limit`

The dataset YAML is already generated at:

- `detector/dataset.yaml`

## Build the Value Classifier Dataset

Fill out:

- `classifier/value_labels.csv`

Columns:

- `image_path`: source image file
- `split`: `train` or `val`
- `speed_limit_mph`: posted value such as `25`, `35`, or `55`
- `bbox_index`: which YOLO box to crop if an image has multiple labeled signs
- `padding`: optional crop padding ratio
- `label_path`: optional explicit path to the YOLO label file

Then generate the classifier crop folders:

```bash
.venv/bin/python scripts/speed_limit_vision/build_value_dataset.py
```

This writes cropped sign images into:

- `classifier/train/<value>/...`
- `classifier/val/<value>/...`

## Train

Detector:

```bash
.venv/bin/python scripts/speed_limit_vision/train_detector.py --device mps
```

Classifier:

```bash
.venv/bin/python scripts/speed_limit_vision/train_value_classifier.py --device mps
```

Use `--device cpu`, `--device mps`, or a CUDA device string depending on the training host.

## Rebalance Toward Real Comma Data

If the detector starts overfitting to synthetic/public data, build a lighter rebalanced dataset that keeps all `real_*` detector images and samples the rest:

```bash
.venv/bin/python scripts/speed_limit_vision/rebalance_detector_dataset.py \
  --workspace .tmp/speed_limit_training \
  --max-other-train 3200
```

Then point the detector trainer at the generated YAML:

```bash
.venv/bin/python scripts/speed_limit_vision/train_detector.py \
  --workspace .tmp/speed_limit_training \
  --data .tmp/speed_limit_training/detector_rebalanced/dataset.yaml \
  --device mps
```

This keeps validation unchanged while making retrains faster and more comma-biased.

## Export ONNX

```bash
.venv/bin/python scripts/speed_limit_vision/export_models.py \
  --detector-weights .tmp/speed_limit_training/runs/detector/yolo11n-speed-limit-us/weights/best.pt \
  --classifier-weights .tmp/speed_limit_training/runs/classifier/yolo11n-cls-speed-limit-us/weights/best.pt \
  --install-repo-assets
```

That writes:

- `.tmp/speed_limit_training/exports/speed_limit_us_detector.onnx`
- `.tmp/speed_limit_training/exports/speed_limit_us_value_classifier.onnx`

And optionally copies them into:

- `starpilot/assets/vision_models`

## Copy to the Device

```bash
.venv/bin/python scripts/speed_limit_vision/install_models.py --host comma@192.168.3.110
```

The runtime already prefers `speed_limit_us_detector.onnx` plus `speed_limit_us_value_classifier.onnx` when both files exist in `starpilot/assets/vision_models`.

## Evaluate the Runtime Path

Run the real StarPilot runtime path, using the installed ONNX pair, against the known saved-frame cases:

```bash
.venv/bin/python scripts/speed_limit_vision/evaluate_runtime_cases.py --strict
```

For temporal behavior on a saved frame directory or route extract, replay the runtime directly:

```bash
.venv/bin/python scripts/replay_speed_limit_vision.py .tmp/vision_iter/seg10_5fps --frames-fps 5
```

The detector/classifier runtime is model-only by default. Use `--crop-ocr` with
`evaluate_runtime_manifest.py` or `replay_route_runtime.py` only for an explicit
legacy comparison. A model-only release must match reviewed-manifest accuracy
and pass representative route replays at measured on-device cadence. Evaluate
candidate recognition and temporal publish behavior separately: a correct
single-frame candidate can still be suppressed by the history and speed-change
confirmation policy.

Ignored review rows label the proposed crop, not the entire camera frame.
Consequently, negative-window candidate and publish counts from
`evaluate_reviewed_route_events.py` are an upper bound until the full frame is
audited; another valid sign can be present outside the rejected crop. Use the
per-row output and frame image to audit any regression delta before treating it
as a runtime false positive.

## Promotion Gate

Do not promote a checkpoint from classifier validation accuracy alone. Export it
to an isolated model directory and run the complete runtime pipeline against the
reviewed positive, hard-negative, and failed-drive manifests. A candidate must
preserve exact-value recall, avoid new wrong-value reads, and remain within the
accepted false-positive budget before route replay.

Mine detector proposals that fool an integrated-reject classifier into a new
reject class before retraining:

```bash
.venv/bin/python scripts/speed_limit_vision/mine_classifier_reject_crops.py \
  --models-dir /path/to/candidate/models \
  --dataset /path/to/versioned/classifier \
  --manifest /path/to/reviewed-negative-manifest.csv
```

Keep the resulting dataset version separate from the current training set. If a
hard-negative retrain lowers reviewed recall, reject the checkpoint even when it
improves aggregate validation accuracy or removes a known false positive.

## Active-Learning Review Pass

Keep parallel miners in separate directories and merge them only when their
model and mining fingerprints match:

```bash
.venv/bin/python scripts/speed_limit_vision/merge_manual_review_queues.py \
  /path/to/shard0 /path/to/shard1 /path/to/shard2 /path/to/shard3 \
  --output-dir /path/to/merged
```

When rescanning with a new model, compare the fingerprinted queues before
selecting another batch. The optional review output retains the full queue
schema so it can be passed directly to the selector and review server:

```bash
.venv/bin/python scripts/speed_limit_vision/compare_manual_review_queues.py \
  --before /path/to/baseline/manual_review_queue.csv \
  --after /path/to/candidate/manual_review_queue.csv \
  --output-csv /path/to/comparison.csv \
  --review-output /path/to/disagreements/manual_review_queue.csv

.venv/bin/python scripts/speed_limit_vision/select_manual_review_queue.py \
  --input /path/to/disagreements/manual_review_queue.csv \
  --output /path/to/review/manual_review_queue.csv \
  --max-rows 1200 \
  --min-seconds-per-route-speed 3
```

The selector prioritizes value changes and gained/lost reads, balances routes
and speed classes, and removes adjacent same-speed frames from one scene. Start
the reviewer and import its labels without moving route media off the training
volume:

```bash
.venv/bin/python scripts/speed_limit_vision/serve_manual_review_queue.py \
  --manifest /path/to/review/manual_review_queue.csv \
  --port 8765

.venv/bin/python scripts/speed_limit_vision/import_manual_review_queue.py \
  --queue /path/to/review/manual_review_queue.csv
```

## Re-mine the Route Backlog

Re-run the backlog after a candidate passes the reviewed-manifest and route
replay gates. Use a model fingerprinted run so new pseudo-labels are staged next
to, rather than merged into, the original route-mining data:

```bash
.venv/bin/python scripts/speed_limit_vision/mine_route_training_samples.py \
  --workspace /Volumes/T5/starpilot_speed_limit/workspace/speed_limit_training_clean \
  --models-dir /path/to/promoted/models \
  --model-only \
  --run-id auto \
  --sample-every 2.0 \
  --transition-step 0.5 \
  --max-frames-per-route 720 \
  --max-positives-per-route 120 \
  --max-negatives-per-route 200
```

The output is written under
`staging/route_mining/model_<model-fingerprint>_run_<mining-fingerprint>/` with
its own detector images, classifier labels, review manifest, and per-route
completion state. The mining fingerprint includes the model-only mode,
thresholds, sampling configuration, and relevant source code. Review and
deduplicate that staged run before merging it into a training dataset. Never
overwrite the canonical route samples or automatically train on every mined
positive; map agreement and human review remain required because a stronger
model can still reproduce its own mistakes at larger scale.