StarPilot/tools/ci

Local and release checks

Run ./test from the checkout for the focused pre-push checks. It uses the isolated developer runtime, retains failure logs, and leaves device artifacts unchanged. ./test --full wraps the complete Linux pre-deployment command below. Use ./test --help for scope and reporting options.

On Linux, use the managed host runtime to run the same source, unit, vehicle and pinned recording checks and process replay as GitHub Actions:

./dev python tools/ci/run_predeploy.py --output /tmp/starpilot-predeploy --download

The default run first builds the host native graph, then also runs the native Panda protocol executables and required Go/Python IPC checks. Use a new output directory per run. --download permits only the pinned public recordings; omit it when they are already cached. Node.js 24 must be on PATH (or set STARPILOT_NODE), and the pinned Go toolchain/dependencies are needed for map provider checks. A failure stops the run with a named log and preserves its exit status in results.json. --stage build, source, host, native, vehicles, recorded, or replay can select a stage while diagnosing a failure; a selected-stage pass is not a full release result.

The replay stage runs the full process suite with its existing coverage and reference comparisons. It can download replay logs independently of the pinned recording --download flag. Unaccounted missing routes and unqualified historical schema caches fail. Volvo C1/V40 is explicitly reported as lacking a recording, with its source and native suites required separately; see the process replay README for details. Passing the pinned recorded vehicle checks does not qualify this separate suite.

The full runner requires Linux: its USB/native transport and executable map IPC tests use Linux facilities. Run tools/setup_dependencies.sh first, provide the Node and Go versions above, and fetch the pinned map dependencies with (cd mapd_repo && go mod download).

On macOS, build host dependencies and run the source/host subset:

./dev python -m SCons -j4
./dev python tools/ci/run_predeploy.py --stage source --stage host --output /tmp/starpilot-host-checks

This subset does not qualify Linux native transport, vehicle/recorded suites, or device artifacts. The full runner rejects an unsupported native-stage host before executing checks.

Before shipping device artifacts, run ./build 4 in the source checkout after host validation. Host tests do not replace the device build, native firmware qualification, device startup checks, or physical vehicle testing. GitHub also runs platform-specific build, replay and rendering jobs.

Vehicle contract checks

Run the rebuilt feature and CI tests locally after building native dependencies:

python tools/ci/run_host_tests.py openpilot/starpilot tools/ci/tests --json-output feature-results.json

tools/op.sh test uses this wrapper too. It forwards test selection and reporting options to tools/test_runner.py. It creates disposable Params and messaging directories before any test imports, then removes them after the test process exits, including on failure. Existing developer settings and IPC namespaces are not used. It also sets SCALE=1 so UI imports do not query a desktop monitor. This is host logic testing; native visual captures and device rendering need their own graphics context. The hosted unit job uses the same wrapper for the complete openpilot test tree. Galaxy tests require Node.js 24 on PATH or selected with STARPILOT_NODE.

The source-integrity job also enforces cereal's custom-fork guidance with tools/ci/schema_policy.py. The pinned car.capnp and other upstream schemas must remain unchanged apart from the exact additive exceptions enumerated in schema-policy.json; those exceptions are checked against the pinned source. Fork fields belong inside the reserved custom.capnp structs; their type IDs stay fixed and log.capnp may rename only the associated reserved event aliases, keeping their ordinals and type associations. The check allows nested custom fields while rejecting edits to upstream-owned definitions, reused IDs and retargeted events. schema-policy.json records the official source pins and schema hashes; review it alongside upstream-sync.json when updating upstream. Every .capnp file under openpilot/cereal and opendbc_repo/opendbc/car requires explicit policy coverage; new schemas fail until reviewed. This guard prevents upstream schema collisions; it does not replace append-only evolution and compatibility testing of already shipped custom fields. Slot5 remains empty under its original name. Slot1 restores the full historical StarPilotCarParams fields0–8, nested SafetyConfig, and pcmCruiseSpeed=true default at type ID 0xaedffd8f31e7b55d and Event ordinal108. The optional autoSteeringLearning Event alias still refers to that outer type; its versioned learning child is appended at9 with a separately generated nested type ID. Historical messages have a default version0 child and cannot advertise a fit. The occupied-prefix guard protects both the historical and appended fields, identity and enum meanings. The transient direct learner layout in commit c204f0b165 requires that exact schema for replay; this repair does not claim compatibility with those incompatible development records. Keeping a type ID alone does not preserve field meanings in the other restored historical slots. Slot0 preserves the historical car-control HUD and steering-limit prefix and appends versioned nested navigation fields. Navigation readers require version2; old car-control records do not become navigation commands. Slot4 restores StarPilotModelDataV2 at Event111, including its original turn-direction field and enum, then appends a typed Vision observation. Slot9 retains all six historical model-status fields and appends SLC fields at6–30. Native historical-reader fixtures protect both layouts and prove that an old model-status record cannot advertise an active speed limit. This corrects the unreleased Domathon slot9 reuse; earlier development recordings need their exact source schema for replay. That one pre-release layout correction is explicit in the compiled baseline; it does not permit future incompatible regeneration. Slots 17/18/19 previously carried MapdExtendedOut/MapdIn/MapdOut at Event ordinals 143/144/145; none is available for an unrelated new envelope. Slot19 now restores MapdOut with all24 historical fields and three additive fields from the pinned provider. Seven further append-only fields carry sample status, source, GPS/computation times, generation and process session. V2 converts the Python GPS MONOTONIC observation to BOOTTIME only after a startup/resume fence; computation and output use BOOTTIME too. Older versions are unqualified for diagnostics, and no version grants map-based control authority. Its referenced enums live inside the reserved struct with their original explicit type IDs. Compiled-layout and real historical/Go wire fixtures protect the restored contract; core car.capnp is unchanged. Slots17/18 now restore the historical extended status and input layouts as well. The newer provider adds position and loop-rate status, a JSON path input field, and further input enum values. All older fields, helper type IDs and enum meanings are preserved; helper types are nested inside their reserved roots. Opaque pointer defaults are compared as serialized schema values in the compiled contract. This schema restoration alone does not start the provider, expose settings commands, or connect it to SLC control. Development AOL safety and intent previously collided with 143/144 and are retired from the compiled contract. Their transport now uses separate reserved Data ordinals 124/125 with bounded, versioned flat nested payload layouts protected by the compiled contract. The previous contract is archived in the external migration audit. Old development recordings need source-aware conversion; merely matching an Event ordinal must never select an AOL decoder.

Lane Change Assist uses the separate reserved Data ordinal 126. Its bounded typed payload is nested under AolAxisState as an independent schema declaration; it adds no fields to the published AOL state. Modeld owns this optional diagnostic service. Its frame-matched status changes pre-change alert wording and grants no control authority; missing status does not block ordinary engagement. The compiled contract protects the new payload and enums as well as every prior layout.

The native StarPilot test suite also runs test_custom_schema_evolution.py. custom-schema-contract.json records the compiled layouts of the first populated custom messages. This check allows additions while preserving their field names, types, offsets, defaults, union tags and enum ordinals. Field names are protected as an application API in addition to the wire layout. Extend the contract from a reviewed committed schema when a new message generation becomes a baseline; never regenerate it just to silence an incompatible change. This does not convert recordings made with older divergent vehicle schemas. The reviewed baseline includes the SLC and independent-axis message families. The unit job also executes the native Panda protocol test with fake transport; it exercises the parser and negotiation state used by pandad without a device. test_aol_wire.cc asserts the native safety encoder's bounded flat payload; the matching Python fixture decodes its exact bytes. A full Linux pandad build is still required for native call-site coverage on a Linux target.

The default tools/op.sh test discovers the openpilot/ tree. Vendored dependency tests need explicit jobs. The root workflow runs these host-only suites:

python tools/ci/run_vehicle_tests.py --suite interfaces --output ci-results/interfaces
python tools/ci/run_vehicle_tests.py --suite safety-debug --output ci-results/safety-debug
python tools/ci/run_vehicle_tests.py --suite safety-release --output ci-results/safety-release

Add --collect-only to inspect the selected IDs without running tests. Collection is explicitly marked not_run. Each run writes plan.json, complete per-test results.json, and coverage.json with source/dependency identity, native library hashes, actual safety compiler commands and version, outcomes and uncovered areas. The safety library used by behavioral tests is compiled once, loaded explicitly, and used in one test process; no worker can silently rebuild a different loaded variant. The runner also compares the selected vehicle/control source areas before and after execution, including uncommitted patches and untracked files. A changed snapshot fails the gate while retaining every individual test outcome. Keep results outside those source areas. These endpoint checks do not detect a change that is reverted during execution; qualification runs still require an isolated, stable checkout. Native binary provenance remains separate from source snapshots. The debug suite also retains the existing two compile-only variant tests. FUZZ_SEED defaults to 0 for reproducible input generation. An explicit MAX_EXAMPLES override is recorded.

The interface suite recursively includes test_*.py files throughout opendbc/car, including brand-specific tests. Only the exact route module car/tests/test_models.py is excluded. Each module's source hash and collected count are recorded; a module that silently collects zero tests fails the gate. Files containing module-level pytest tests or plain pytest classes run once through pytest, including any unittest classes in those mixed files; other files retain the unittest runner. Both runners execute serially in the same process with the pinned native library. AST checks reject declared test functions and methods without a collected counterpart, including when other tests in that module were collected. It asserts exact equality between the current platform registry and concrete generated interface tests: a missing, duplicate or extra platform fails collection. Abstract-class skips cannot satisfy platform coverage. Every mapped concrete interface test must pass: a class-level skip stays visible in the report and fails the interface coverage gate. Initialization and collection errors also produce error reports, including when a native import fails. The suite also runs the existing platform, limits, fingerprint, docs and route-declaration checks. A route exemption is reported as missing route coverage, not as a tested vehicle.

test_models.py generates its route-backed classes only when directly executed. Importing it with generic discovery would produce abstract skips, not a full vehicle replay. Historical route/schema qualification remains a separate gate; this workflow does not fetch those routes or regenerate expected outputs.

Recorded vehicle regressions

run_recorded_vehicle_tests.py runs the pinned public Model Y CAN fixture through current parser, controller and native safety tests. DEBUG includes stock and alpha-long configurations; RELEASE includes stock only. Alpha-long exclusion is recorded explicitly, and missing, skipped or errored selected cases fail the run. This checks recorded CAN and synthetic controller behavior, not the user's weekend failure, physical harness operation or complete vehicle identification.

./dev python tools/ci/run_recorded_vehicle_tests.py --variant debug --cache /absolute/cache/path --output /absolute/results/path --download

Use --variant release for the separate RELEASE library. --download permits only the manifest's public fixture URLs before execution; cached compressed bytes must match their pinned size and SHA-256. Omit it for fully offline use. Each process owns temporary Params and IPC, disables replay networking, records actual controller configuration and native library identity, and rejects source changes during the run. Use a new result directory to retain earlier evidence. The hosted recorded_vehicles matrix runs both variants and retains reports even on failure. A local pass does not establish the result of a later GitHub run.

The debug safety suite includes all top-level safety test files with ALLOW_DEBUG. The release safety suite compiles without that flag and checks the entire enum against the explicit release hook registry. Its behavioral cases are deliberately limited to no-output/silent, stock Tesla (including FSD14), Mazda, stock-longitudinal Toyota torque, GM camera/EV and four additional manual camera-ACC variants, Bolt Pedal, intercepted GM ASCM, stock SDGM, nine stock conventional-cruise gateway variants, Ford ownership/Transit steering contracts, six manual Honda/Acura Nidec variants, three manual radarless Honda/Acura Bosch variants, two manual Subaru angle-steering variants, four manual MRR35 Hyundai/Genesis angle variants, three manual Hyundai/Kia angle hybrids, two manual CCNC angle SUVs, and Carnival alternate-button resume. The Carnival tests use the runner's recorded safety library and include a real stock-cruise RX sequence with the exact frame emitted by the host controller. It also checks that the DEBUG-only Honda Bosch longitudinal path cannot grant AOL requests in RELEASE, including the classic Bosch family controller frames and brake-source variants. The Ioniq 6 tests enumerate every bit-15 safety parameter and require RELEASE to deny all dedicated longitudinal/AOL profiles. Their DEBUG controller and takeover cases remain explicit skips in the RELEASE result. The selected classes are listed in RELEASE_TARGETS; other release behaviors are uncovered, not silently skipped into a green result. The suite is not a whole-fleet release-mode safety qualification.

Two separate hosted jobs compile Panda firmware with DEBUG and RELEASE flags. Both use the checked-in development certificate. The recorder requires an actual main-source compile command with matching flags and captures ELF/binary hashes and the version stamp. These jobs do not use release credentials, flash hardware or qualify firmware compatibility. Build logs and provenance are retained on failure.

All jobs run on hosted Ubuntu without a device or self-hosted runner. A passing interface test or safety hook unit test does not cover additional downstream car ports, independent lateral/longitudinal modes, real CAN forwarding, hardware, timing, logs from another schema, AGNOS, or deployment/rollback behavior. Skips remain explicit in the JSON report; they do not count as passes.

After the interface suite, the workflow runs the exact fleet coverage gate and saves its report in the same artifact. This step fails while required ports, current interface results or recorded configuration/mode evidence are missing. It currently supplies no mode evidence, so the fleet gate remains uncovered even if every interface test passes. This is intentional during reconstruction: successful unit tests alone must not produce a fleet-wide green result. Recorded mode evidence needs its own reviewed producer before it can be connected here. The gate also runs after an interface failure; available collection and execution records remain in the artifact.

The small collection/provenance fixtures need only the standard library:

python -m unittest discover -s tools/ci/tests -v

The map-provider job runs tools/ci/run_map_tests.py with the Go version pinned in mapd_repo/go.mod. It records test outcomes, vet/build results, dependency pin, source hashes and embedded build identifiers. It does not start the provider or fetch maps. Runtime integration and a Linux ARM64/device build remain separate checks; see mapd_repo/INTEGRATION.md.

The unit job additionally requires the real Go-to-host map messaging test after building the host's native dependencies. It supplies pinned Go 1.25.1 and the repository Python environment, then runs this from mapd_repo:

STARPILOT_SHADOW_IPC_REQUIRED=1 STARPILOT_SHADOW_EXEC_REQUIRED=1 \
STARPILOT_PYTHON=/absolute/path/to/.venv/bin/python \
  go test -mod=readonly -race -run '^TestShadow(GoPythonIPC|ExecutableHostGpsIPC)$' -count=1 -v

Both tests use synthetic GPS/tiles and a disposable IPC namespace. The first uses a Go pump and publisher harness; the second builds and starts the actual shadow executable under isolated settings, then passes host GPS through the Go subscriber and reads its output in Python. Neither uses installed maps or the full provider mode. Missing host dependencies fail these required tests. The standalone provider suite reports explicit skips when their required targets are not selected; those skips are covered by the required unit-job step, not counted as passes.

The Ioniq 6 development branch retains custom AGNOS 19.8.1, matching the separately installed image. AGNOS_UPDATE_POLICY=retain prevents the launcher, background updater, and development branch helper from replacing that OS. A version mismatch blocks startup/update finalization and reports the required version. Other branches without this policy keep their explicit upstream update behavior. The retained version is an installation policy; it does not establish native ABI, camera, model inference or device compatibility with the upstream baseline. openpilot/system/updated/tests/test_retained_agnos.py exercises these paths using disposable OS markers, fake hardware and intercepted firmware calls.

For a read-only parked trial prerequisite report, run python3 tools/ci/run_device_preflight.py --font-dir /path/to/complete/reviewed/fonts --json-output /path/to/report.json. The standard-library driver reads the retained AGNOS and Python policy from launch_env.sh and pyproject.toml, checks the UI's existing bitmap-font and artwork manifests, and tries selected library imports in isolated, timed Python children. It reads /VERSION only on a Linux ARM64 host; on a desktop, target OS status is unavailable. It never starts services, accesses controls or qualifies graphics, camera, CAN, model inference or driving. Missing optional font input and absent device checks stay visible rather than becoming passes. The import probes put this source tree's vendored Python roots first and reject any loaded openpilot, msgq, opendbc or tinygrad module from another checkout. The JSON prerequisites field is pass, fail or incomplete; command exit codes are 0, 1 and 2 respectively. Even pass is a prerequisite report, never a device or driving qualification.

The separate safety-qualification workflow runs four nonmutation GitHub gates: C-line coverage, pinned opendbc MISRA, Panda MISRA, and Panda host tests. Mutation inventory and full mutation run locally in a disposable checkout: python tools/ci/run_safety_qualification.py --gate mutation-list --output /new/inventory/dir python tools/ci/run_safety_qualification.py --gate mutation-full --output /new/mutation/dir Both the workflow and local gates use the root locked safety extra; they do not call the nested opendbc or Panda setup scripts, which would sync or upgrade separate environments. A local disposable checkout can run one gate with python tools/ci/run_safety_qualification.py --gate coverage --output /new/evidence/dir (other gates: opendbc-misra, panda-misra, panda-host, mutation-list, mutation-full). Reports include source hashes, imported module origins, exact commands, full logs, test/skip counts, and an explicit pass/fail result. Coverage requires 100% of upstream checked C lines; full mutation requires zero surviving or infrastructure-error mutants and first runs the full unmutated test catalog. HITL is explicitly unset: Panda/Jungle/vehicle hardware tests remain a separate bench gate and their skips are recorded, never labeled as host passes.