StarPilot/tools/diagnostics

Runtime performance captures

performance.py reads existing local messages and Linux process statistics for a fixed duration, then writes one JSON report. It does not start publishers, change settings, restart processes, select models, or connect to another machine. The comparison command reads two saved files only.

From the source checkout, using the configured host runtime:

./dev python -m tools.diagnostics.performance capture --duration 30 --rate 20 --output /tmp/performance-before.json
./dev python -m tools.diagnostics.performance compare /tmp/performance-before.json /tmp/performance-after.json --output /tmp/performance-comparison.json

On a Linux runtime, use its existing Python environment. A privately staged standalone copy can identify the checkout explicitly:

python /path/to/performance.py capture --source-root /path/to/checkout --duration 5 --output /tmp/performance.json

No deployment or device connection is performed by these commands. Run from the runtime environment whose local messages you intend to observe. --source-root identifies source metadata; it does not change imports or select a running checkout. The report records the declared source root, imported openpilot root, whether they match, loaded messaging module path, checkout HEAD/dirty state, relevant source hashes, and a separate hash of the collector itself. These identify inspected source, not proof that existing processes were built from that HEAD.

--pid 1234 adds a local process to the manager-reported running PIDs and may be repeated. When no manager message is available, only explicit PIDs are sampled. No process is launched or scanned by name. Process names from manager are recorded as requestedName, alongside the kernel's comm, PID, and start ticks.

Limits and output

Defaults are 30 seconds at 20 polls per second. Duration is limited to 0.1–120 seconds, rate to 1–50 Hz, processes to 64 per poll, and retained process instances to 256. Samples stay in memory until one final report is written; there are no per-frame file writes. Slow polls skip schedule slots instead of creating catch-up bursts. The report includes actual duration, poll intervals, missed scheduled polls, and truncation counters. Existing output files are refused.

Each distribution contains sample count, missing counts by reason, min/mean/p50/p90/p95/p99/max. Percentiles use linear interpolation over the retained values. No samples produce null statistics, never invented zero values. Real zero measurements remain zero.

Message availability distinguishes an unavailable backend, a backend error, and a working subscription with no observed messages. Linux process availability separately distinguishes absent /proc, no process targets, unreadable processes, permission errors, and disappearing processes. A macOS capture can still collect messages; /proc CPU and RSS remain unavailable.

What the measurements mean

Measurement	Meaning and limit
observedReceiveRateHz	Fresh messages actually received divided by capture duration. Subscriptions conflate; the collector can miss intermediate messages. This is not publisher frequency, UI FPS, execution rate, or a dropped-frame count.
receiveAgeMs	Time since this collector last received a message, sampled every poll after first receipt. Includes periods without updates.
publicationAgeAtReceiveMs	Collector monotonic time minus Event.logMonoTime at receipt. Includes scheduling and transport delay. Assumes the local publisher uses the same monotonic clock; replay or other clock domains may violate that assumption. Zero and future timestamps are reported missing, with a reason.
reportedFrameIntervalMs	uiDebug.frameTimeMillis, the existing Raylib frame interval.
reportedPrePresentWallMs	uiDebug.cpuTimeMillis. Despite its schema name, the current publisher measures monotonic wall time for drawing/update before presentation. It is not thread CPU or presentation duration.
reportedModelExecutionMs	modelV2.modelExecutionTime converted from seconds to milliseconds. The publisher measures its model.run interval. This is not sensor-to-actuator latency.
reportedFrameDropPercent	The existing model publisher's frameDropPerc, not a drop rate inferred from collector message counts.
Device metrics	Reported utilization, maximum temperature, power, CPU-core mean/max; negative numeric values and missing fields have explicit missing counts. A schema's default zero cannot prove that a hardware sensor is implemented.
cpuPercentOneCore	Delta process user+system ticks divided by elapsed time and clock frequency. 100% means one logical CPU; a multithreaded process may exceed 100%. Includes no child-process CPU.
rssMiB	Resident pages from /proc/PID/stat, converted using the system page size. Not total allocation, GPU memory, or swapped memory.

Raw message validity counts are retained; reported metrics are not silently discarded when valid is false. In particular, the current uiDebug publisher does not set the envelope's default false validity flag. Distributions count fresh received values only, not repeated samples of a stale payload. Reported onroad, engagement, thermal, device and big-model states provide comparison context.

Processes are separated by (PID, start ticks). CPU requires a prior observation of that exact instance. Disappearance, counter regression, or PID reuse resets the baseline. observedInstanceCountsByName can reveal multiple observed instances across changed PIDs; pidReuseOrRestartEvents counts only identity changes directly observed at a reused PID. Neither is a complete system restart log. Retained names are manager labels, not proof a stale manager PID still belongs to that service.

Existing UI_FRAME_TIMING=1 instrumentation separately measures drawing, update, and presentation wall/thread CPU time. This tool does not enable it or parse its logs. collectorEnvironment.UI_FRAME_TIMING describes the collector's environment, not the settings of an already-running UI.

Compare captures

The comparison provides key UI/model/device measurements, per-process peak CPU/RSS, full scalar deltas, sample counts, source context, and environment/sampling differences. Missing values remain missing_data. Percentage change from a zero baseline is null. Process peaks group observed instances by requested name; per-instance percentiles are never averaged into a misleading pooled percentile.

Use the same duration, poll rate, workload, display state, model, and hardware when interpreting differences. Screen-off/offroad captures may legitimately lack UI or model messages. A lower observed message rate is not automatically slower execution. The report makes descriptive comparisons; it does not label a change as a performance regression or certify onroad reliability.

Focused tests (standard library compatible):

python -m unittest tools.diagnostics.tests.test_performance

Replay compositor trace (20 FPS)

replay_render.py is an opt-in wrapper for an existing owner-created replay worker. The normal worker never imports it. Keep the existing owner, private session, decoder, encoder and control procedure. In a separately authorized diagnostic owner's spawn callback, replace only -m openpilot.starpilot.replay.render with -m tools.diagnostics.replay_render --output /absolute/unused-trace.json --, preserving all existing worker arguments and environment. The wrapper refuses invalid private contexts and an existing output file; it does not connect to a device, create an owner, spoof parked authority, or start a session on its own. Do not run a private producer beside live replay.

Records stay in a 30,000-event ring until one mode-0600 JSON report is written. Overwritten records are counted. Tracing and picture capture affect frame timing; measure cadence in separate lightweight runs. Copy reports off the device before /tmp fills; log decompression needs the same space. After pausing through the owner and confirming readiness, send SIGUSR1 to this diagnostic worker PID to request a checkpoint. It writes at the next valid paused loop boundary with no borrowed readback pending; wait for a complete JSON report before stopping. The report declares report_boundary=paused_checkpoint and excludes subsequent teardown. Without a checkpoint it attempts a shutdown report, but the owner's cleanup deadline can kill it before that completes. Stop through the owning session and wait boundedly for owner idle; a successful stop response alone is insufficient. SIGKILL cannot flush. There are no pixel copies or per-frame file/console writes.

The report includes source revision/hashes, worker arguments, attempt IDs, clock and camera metadata before/after acquisition, early/late rejection reasons, inclusive and exclusive wall/thread-CPU timings, preparation, camera/model, HUD entry points, alerts/PiP, conversion, readback, encoding, muxing, publication, sleep and readback-pending observations. Deadline lateness mirrors the existing accepted-draw schedule; the initial draw has no previous deadline. Summary distributions cover only the retained event window; outcome counters cover the worker lifetime. GC intervals overlap other spans and have no exclusive/CPU measurement. The wrapper also accepts the unchanged worker before this patch: absent eligibility/preparation hooks are declared in capabilities, and its rejected draws remain unclassified_rejection rather than invented early/late classifications.

connect_import includes connection checks, possible FD exchange/import and texture setup. primary_acquisition includes receive/synchronization and matching; the camera span also includes its later receive and submission. These are nested boundaries, not isolated receive-only or pure GPU timings. submit_to_observed_ready_ns includes GPU queueing and polling delay. HUD methods not individually wrapped remain in composition's exclusive remainder. Buffer metadata is an observation, not an owned pixel snapshot. The wrapper retains no readback pixels beyond the original delivery context.

Use a fresh unchanged baseline and candidate at the same source bundle, scene, rate and profile; record GPU/browser/encoder conditions separately. Collect real CPU/load/temperature alongside this trace with the existing performance capture when authorized. Compare instrumentation on/off and reverse promising A/B order. Keep picture/decoded-content comparisons separate. This trace does not measure native upload/presentation, browser presentation, physical scanout or complete instrumentation overhead.

Local harness checks: ./dev pytest -q tools/diagnostics/tests/test_replay_render.py.

Replay compatibility checks:

./dev pytest -q openpilot/starpilot/replay/tests tools/diagnostics/tests/test_replay_render.py \
  openpilot/starpilot/ui/tests/test_replay_settings.py openpilot/starpilot/ui/tests/test_camera_display.py \
  openpilot/starpilot/ui/tests/test_camera_resources.py openpilot/starpilot/ui/tests/test_onroad_alerts.py \
  openpilot/starpilot/ui/tests/test_onroad_display_holds.py openpilot/selfdrive/ui/tests/test_camera_speed_gates.py \
  openpilot/starpilot/galaxy/tests/test_replay_http.py
node openpilot/starpilot/galaxy/tests/test_replay.mjs
node openpilot/starpilot/galaxy/tests/test_replay_media.mjs
node openpilot/starpilot/galaxy/tests/test_replay_export.mjs
node openpilot/starpilot/galaxy/tests/test_record_history.mjs

Offline longitudinal trace

longitudinal.py reads one existing local rlog or qlog segment and prints one JSON object. It never opens a device, starts a publisher, or writes a log. Example: ./dev python -m tools.diagnostics.longitudinal /absolute/path/2-rlog.zst (the ./dev wrapper prints its own build progress before the tool's JSON). The report records input and source hashes, limits, message counts, mode transitions, rejected pairs and distributions.

longitudinalPlan.aTarget is the planner's acceleration target; carControl.actuators.accel is the actuator command; carState.aEgo is the vehicle's speed-derived acceleration estimate, not an IMU measurement. The reported delta rates are sample-to-sample derivatives within continuous, same-source/experimental/long-active/pedal stretches. They are not physical jerk, publisher rates, proven road jitter, or evidence that a particular control tune is safe. qlog sampling is decimated. See python -m unittest tools.diagnostics.tests.test_longitudinal for synthetic gap/mode/invalid-message coverage.