StarPilot/tools/ci/FLEET_COVERAGE.md

Exact platform coverage gate

openpilot/starpilot/validation/required-platforms.json freezes 345 exact source platform IDs and seven distinct upstream additions. The source-inventory hash and revisions make the obligation set reviewable. A shared name or declaration does not mean that controller, firmware, settings, safety, or physical behavior is equivalent.

Run the reporter with an interface suite's two companion files:

python tools/ci/fleet_coverage.py \
  --interface-coverage ci-results/interfaces/coverage.json \
  --interface-results ci-results/interfaces/results.json \
  --output ci-results/fleet-coverage.json

The default --require complete exits zero only when both interface and independent mode coverage pass. CI and predeploy use --require interfaces to require every exact interface obligation and stable current execution provenance. That choice does not clear mode gaps: the full report remains uncovered when mode evidence is absent, and vehicle_qualification remains not_established. The separate interface_gate records only interface errors, failures and gaps. It compares the exact registry IDs and one concrete interface test per ID, checks every execution record, verifies the companion results hash, and compares the reported checkout, suite runner, test runner, dependency manifest, pinned opendbc commit, CAN sources, and native parser binary with the current checkout. An older report remains visible under reported_interface_counts, while current_interface_counts marks its passing tests as historical_pass_only. A missing port or upstream addition remains uncovered. A skip, missing test, failed test, duplicate, or malformed report never becomes a current pass. Upstream additions remain independent obligations even if they disappear from a later registry. The command refuses to overwrite an existing output file, including a failed or uncovered report. Use a fresh path for each run. A matching HEAD alone is not enough for current execution: the recorded before/after source snapshots must match the current snapshot, including relevant dirty source and native bytes. Unrelated UI source changes do not invalidate vehicle execution.

Recorded control evidence may be supplied through --mode-evidence path.json:

{
  "schema_version": 1,
  "enumeration_complete": true,
  "provenance": {
    "configuration_inventory_sha256": "<64 lowercase hex characters>",
    "trace_catalog_sha256": "<64 lowercase hex characters>"
  },
  "configurations": ["<exact configuration objects>"],
  "traces": ["<control-trace.schema.json documents>"]
}

The two SHA-256 values must be computed over their respective JSON arrays using UTF-8, sorted object keys, compact separators , and :, no ASCII escaping, and no trailing newline. For Python, this is sha256(json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")). The gate recomputes both digests; a digest-shaped label alone is insufficient.

Each configuration is the exact object accepted by openpilot.starpilot.validation.control_modes: platform, device, harness, policy, firmware, settings, schema, safety binary and adapter identities belong in it. The caller must enumerate every relevant configuration, including hardware and firmware variants. Platform names alone are not a configuration inventory. All 17 named scenarios must be covered for every configuration. Synthetic traces cannot satisfy the matrix; earlier failures remain failures when later traces pass. The off, lateral_only, longitudinal_only, and combined modes retain separate host-axis and stock ACC ownership checks. A missing mode file, provenance, configuration, recorded scenario or observation is uncovered.

This gate checks declared data and reported provenance; hashes and recording labels supplied by a caller do not authenticate a trace adapter or physical origin. Every output says vehicle_qualification: not_established, including a passing coverage contract. Recorded replay, adapter review, hardware, bus, firmware and release qualification remain separate work.

--require interfaces specifically gates the exact current production registry, all concrete mapped interface tests, execution failures and current source/native provenance. Inapplicable ancillary skips remain reported but do not count as skipped concrete interfaces. Missing Dom ports, changed addition inventory and independent mode gaps still keep full readiness uncovered. A green interface regression gate does not mean complete Dom parity, complete fleet readiness or vehicle qualification.