mawei/StarPilot
1
0
Fork 0
mirror of https://github.com/firestar5683/StarPilot.git synced 2026-10-01 03:43:46 +08:00
Code Issues Packages Projects Releases Wiki Activity

attempt to get sim working

Browse Source
This commit is contained in:
Prabhaav Pillai
2026-08-28 18:35:24 -04:00
parent 5f95574f02
commit 4a3d96fbae
24 changed files with 812 additions and 165 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/sim/README.md
+152 -40
View File
@@ -1,50 +1,162 @@
openpilot in simulator
=====================
# StarPilot Live MetaDrive Simulator
sudo apt install libnvidia-gl-610 # match the KMD version shown by nvidia-smi (610.88)
cd /home/prabh/Projects/openpilot/.host_runtime/linux/worktree && uv pip install --python .venv/bin/python3 PyOpenGL cuda-python 2>&1 | tail -15
openpilot implements a [bridge](run_bridge.py) that allows it to run in the [MetaDrive simulator](https://github.com/metadriverse/metadrive).
Runs the **entire** openpilot stack (`manager.py` -> modeld/controlsd/plannerd/locationd)
on the host PC, bridged to a **straight-road MetaDrive world**, so you can drive the
various longitudinal modes live and watch them behave.
## Launching openpilot
First, start openpilot.
``` bash
# Run locally
./tools/sim/launch_openpilot.sh
This is a StarPilot-specific wrapper. Upstream docs: `tools/sim/launch_openpilot.sh` +
`run_bridge.py` (stock MetaDrive bridge).
---
## Quick start
```bash
# one command, pick a mode:
./tools/sim/starpilot_sim.sh hem # Hybrid Experimental Mode
./tools/sim/starpilot_sim.sh exp # standard Experimental mode
./tools/sim/starpilot_sim.sh chill # pure Chill / CCM mode
./tools/sim/starpilot_sim.sh cem # Conditional Experimental Mode
# extra flags (order doesn't matter):
./tools/sim/starpilot_sim.sh hem --headless # no UI window
./tools/sim/starpilot_sim.sh hem --joystick # use a game wheel instead of keyboard
./tools/sim/starpilot_sim.sh hem --cpu # force CPU backend for the model
./tools/sim/starpilot_sim.sh hem --pkl # use the precompiled pickle instead of tracing the ONNX live
```
## Bridge usage
```
$ ./run_bridge.py -h
usage: run_bridge.py [-h] [--joystick] [--high_quality] [--dual_camera]
Bridge between the simulator and openpilot.
Run it from a **real terminal with a display** (that's where the UI renders and the
keyboard controls work). Press `q` to exit.
options:
-h, --help show this help message and exit
--joystick
--high_quality
--dual_camera
### What each mode does
| command | toggle set |
|-----------|-----------------------------------------------------------------------------|
| `exp` | `ConditionalExperimental=off, ConditionalChill=off, HybridExperimental=off` (full-time experimental) |
| `chill` | `ConditionalChill=on` (full-time Chill / ACC) |
| `cem` | `ConditionalExperimental=on` (conditional experimental) |
| `hem` | `HybridExperimental=on` (hybrid experimental) |
These are mutually exclusive; the launcher sets exactly one.
---
## Prerequisites
- Host build already provisioned under `.host_runtime/linux/worktree` (the launcher
runs `./dev sync` to refresh it from this repo).
- `metadrive-simulator` installed in the host worktree venv (it is, as a dependency).
- The driving model in the model store: `~/.comma/starpilot/data/models/rdf43_driving_tinygrad.pkl`.
This is a PC (CUDA) build compiled from `driving_supercombo.onnx` (also kept in the model
store as `driving_supercombo.onnx`). The launcher copies the pkl into the worktree where
`modeld` expects it on every launch.
- An NVIDIA GPU with working tinygrad **CUDA** (default model device).
## How the model is built
**Default (Option 1, live-ONNX):** `modeld` loads `driving_supercombo.onnx` directly and
traces it in-memory on the RTX at startup — `OnnxRunner` + `TinyJit` captured live during the
first frames. There is **no** tinygrad-JIT pickle round-trip, so the JIT-unpickler bug (which
rewrites kernel targets to the comma-device default `QCOM`) is never triggered. This is the
reliable path on a PC host. Set `STARPIOT_LIVE_ONNX=<path-to>.onnx` (the sim launcher does
this for you) to enable it; `STARPIOT_MODEL_SIZE` defaults to `512x256`.
**Fallback (--pkl):** a precompiled tinygrad-JIT pickle, built for the PC CUDA backend:
```bash
./dev python selfdrive/modeld/compile_modeld.py \
--model-type supercombo --model-size 512x256 --camera-resolutions 1928x1208 \
--supercombo-onnx ~/.comma/starpilot/data/models/driving_supercombo.onnx \
--behavior-version v15 \
--output ~/.comma/starpilot/data/models/rdf43_driving_tinygrad.pkl
```
#### Bridge Controls:
- To engage openpilot press 2, then press 1 to increase the speed and 2 to decrease.
- To disengage, press "S" (simulates a user brake)
`compile_modeld.py` now skips its JIT pickle round-trip by default (`STARPIOT_DO_JIT_ROUNDTRIP`
re-enables it); that round-trip rewrites compiled device/target refs inside the JIT to the
comma-device default (`QCOM`), producing an artifact that cannot run on a PC CUDA host.
Because the pkl path still goes through this fork's buggy JIT unpickler at load time, prefer
the live-ONNX path.
#### All inputs:
## GPU notes
- **Model** runs on the RTX via tinygrad **CUDA** (`DEV=CUDA`, `STARPIOT_SIM_DEV=CUDA`).
The whole model — camera warp **and** policy — is compiled for CUDA (`WARP_DEV=CUDA`).
Pass `--cpu` to force the CPU backend instead.
### Low MetaDrive FPS / "not using the GPU" (read this)
Two separate things use the GPU in this sim:
1. **tinygrad model** — already CUDA on the RTX (works, verified). Unaffected by the below.
2. **MetaDrive world + camera rendering** (Panda3D). This is the part that was slow.
**Verified diagnosis on this laptop:** the RTX 4050 is **compute-only** — `nvidia-smi` shows
`Disp.A: Off`, 0 MiB — so it is *not* driving the display. MetaDrive renders through Panda3D's
`glxGraphicsPipe`, which runs on the display's GL context. On this host that resolves to
**Mesa llvmpipe software** rendering (checked with `glGetString(GL_RENDERER)` →
`llvmpipe (LLVM 20.1.2)`), i.e. **no GPU at all**, which is exactly why FPS is low.
- Installing `libnvidia-gl-610` adds the NVIDIA GL userspace libs (`10_nvidia.json`,
`libGLX_nvidia.so`), but it does **not** change the sim's FPS here, because NVIDIA is not
the display GPU and GLX still uses the iGPU/Mesa stack. (PRIME offload
`__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia` was tried and fails with
"Could not find a usable pixel format".)
- The real fix is making the *display's* GL hardware-accelerated (proper iGPU GLX on `:0`,
or an NVIDIA/EGL offscreen render setup) — a system/graphics-config task, not a repo change.
**CUDA image capture is off by default (and must stay off on non-NVIDIA GL).** Installing
`cupy`/`PyOpenGL`/`cuda-python` flips MetaDrive's `_cuda_enable` to `True`; if `image_on_cuda`
then follows it, `cudaGraphicsGLRegisterImage` fails with `cudaErrorUnknown` on a Mesa GL
context and **crashes MetaDrive at sensor init**. The bridge therefore defaults
`image_on_cuda` to `False` (safe CPU `RTMCopyRam` readback). To opt into CUDA images on a
machine where the GL context genuinely is NVIDIA-backed, set `STARPIOT_CUDA_IMAGES=1`.
The sim's `camerad` RGB→NV12 conversion likewise falls back to CPU numpy when no OpenCL ICD
is present; install `nvidia-opencl-icd` if you want that on-GPU too.
---
## Driving controls (keyboard)
| key | action |
|-----|--------|
| `r` | Reset simulation (back to the start point) |
| `i` | Toggle ignition (**starts ON** by default) |
| `2` | Cruise **Set** — engages openpilot (lateral + longitudinal control) |
| `1` | Cruise Resume / accel |
| `3` | Cruise Cancel |
| `q` | Quit everything |
| `w/a/s/d` | Manual throttle / steer / brake |
### "Go back to start, then turn on lateral + longitudinal control like in the car"
1. Press **`r`** — the world resets and the car respawns at the start of the straight road.
2. Press **`2`** (cruise set) — openpilot engages: **lateral** (steering) and
**longitudinal** (accel/brake) control turn on together, same as hitting the cruise
set button in the car. `1` bumps the set speed up, `3` cancels.
3. The bridge also auto-engages shortly after startup, so you usually just have to sit back
and let it drive the straight road.
> **Ignition is ON by default.** If the status line ever shows `Ignition: False`, press
> `i` once to turn it back on. Note the trap: pressing `i` toggles it *off*, so don't press
> it at startup unless you want to switch it off.
---
## Status / known blocker
With the **live-ONNX** path (the default), `modeld` traces the model in-memory on the RTX and
the old QCOM-rewrite blocker is bypassed entirely — there is no JIT pickle to unpickle, so
`CUDA_ERROR_INVALID_IMAGE` from a corrupted camera-warp kernel no longer applies. If you run
with `--pkl` instead, the pickle path still hits this fork's buggy JIT unpickler at load time
(rewrites kernel targets to `QCOM::a630`, then the warp recompiles as a QCOM image), which is
exactly why Option 1 is the default. Fixing the pkl path would require a small change in the
vendored tinygrad to preserve the target device across JIT unpickle.
For a working stop-sign reproduction today, use the replay forensics on a real route:
```bash
./dev python tools/replay/hem_forensic.py <dongleId>/<routeId> --segments 0,1
./dev python tools/replay/mode_sim.py <dongleId>/<routeId> --segment 0
```
| key | functionality |
|------|-----------------------|
| 1 | Cruise Resume / Accel |
| 2 | Cruise Set / Decel |
| 3 | Cruise Cancel |
| r | Reset Simulation |
| i | Toggle Ignition |
| q | Exit all |
| wasd | Control manually |
```
## MetaDrive
### Launching Metadrive
Start bridge processes located in tools/sim:
``` bash
./run_bridge.py
```
tools/sim/bridge/common.py
+63 -6
View File
@@ -58,6 +58,13 @@ class SimulatorBridge(ABC):
self.past_startup_engaged = False
self.startup_button_prev = True
self.startup_set_count = 0
self.auto_engage_speed_done = False
self.AUTO_CRUISE_KPH = 60.0
self.manual_throttle = 0.0
self.manual_brake = 0.0
self.manual_steer = 0.0
self.test_run = False
@@ -126,7 +133,9 @@ Ignition: {self.simulator_state.ignition} Engaged: {self.simulator_state.is_enga
self.simulator_state.left_blinker = False
self.simulator_state.right_blinker = False
throttle_manual = steer_manual = brake_manual = 0.
throttle_manual = self.manual_throttle
steer_manual = self.manual_steer
brake_manual = self.manual_brake
# Read manual controls
if not q.empty():
@@ -134,11 +143,11 @@ Ignition: {self.simulator_state.ignition} Engaged: {self.simulator_state.is_enga
if message.type == QueueMessageType.CONTROL_COMMAND:
m = message.info.split('_')
if m[0] == "steer":
steer_manual = float(m[1])
steer_manual = self.manual_steer = float(m[1])
elif m[0] == "throttle":
throttle_manual = float(m[1])
throttle_manual = self.manual_throttle = float(m[1])
elif m[0] == "brake":
brake_manual = float(m[1])
brake_manual = self.manual_brake = float(m[1])
elif m[0] == "cruise":
if m[1] == "down":
self.simulator_state.cruise_button = CruiseButtons.DECEL_SET
@@ -146,6 +155,7 @@ Ignition: {self.simulator_state.ignition} Engaged: {self.simulator_state.is_enga
self.simulator_state.cruise_button = CruiseButtons.RES_ACCEL
elif m[1] == "cancel":
self.simulator_state.cruise_button = CruiseButtons.CANCEL
self.manual_throttle = self.manual_brake = self.manual_steer = 0.0
elif m[1] == "main":
self.simulator_state.cruise_button = CruiseButtons.MAIN
elif m[0] == "blinker":
@@ -173,14 +183,26 @@ Ignition: {self.simulator_state.ignition} Engaged: {self.simulator_state.is_enga
self.simulator_state.is_engaged = self.simulated_car.sm['selfdriveState'].active
if self.simulator_state.is_engaged:
self.manual_throttle = self.manual_brake = self.manual_steer = 0.0
throttle_op = np.clip(self.simulated_car.sm['carControl'].actuators.accel / 1.6, 0.0, 1.0)
brake_op = np.clip(-self.simulated_car.sm['carControl'].actuators.accel / 4.0, 0.0, 1.0)
steer_op = self.simulated_car.sm['carControl'].actuators.steeringAngleDeg
# After auto-engaging at standstill the set speed is captured at ~floor
# (a crawl). Ramp it up to a real driving speed via cruise-accel presses.
if not self.auto_engage_speed_done and self.rk.frame % 5 == 0:
if self.simulated_car.sm['carState'].vCruise < self.AUTO_CRUISE_KPH:
self.simulator_state.cruise_button = CruiseButtons.RES_ACCEL
else:
self.auto_engage_speed_done = True
self.past_startup_engaged = True
elif not self.past_startup_engaged and self.simulated_car.sm['selfdriveState'].engageable:
self.simulator_state.cruise_button = CruiseButtons.DECEL_SET if self.startup_button_prev else CruiseButtons.MAIN # force engagement on startup
self.startup_button_prev = not self.startup_button_prev
# Auto-engage whenever the car is ready. Cruise-Set (DECEL_SET) engages from
# standstill; pressing periodically (not every frame) avoids ratcheting the set
# speed down while still catching the moment the car becomes engageable.
if self.rk.frame % 10 == 0:
self.simulator_state.cruise_button = CruiseButtons.DECEL_SET
throttle_out = throttle_op if self.simulator_state.is_engaged else throttle_manual
brake_out = brake_op if self.simulator_state.is_engaged else brake_manual
@@ -190,6 +212,41 @@ Ignition: {self.simulator_state.ignition} Engaged: {self.simulator_state.is_enga
self.world.read_state()
self.world.read_sensors(self.simulator_state)
if self.rk.frame % 300 == 0:
try:
from PIL import Image
cam = getattr(self.world, 'road_image', None)
if cam is not None and getattr(cam, 'any', None) and cam.any():
Image.fromarray(cam[:, :, ::-1]).save(f"/tmp/kilo/cam_{self.rk.frame}.jpg", quality=90)
except Exception as e:
print(f"CAM-ERR {e}", flush=True)
if self.rk.frame % 25 == 0 and not self.test_run:
try:
st = self.simulator_state
eng = st.is_engaged
engable = self.simulated_car.sm['selfdriveState'].engageable
active = self.simulated_car.sm['selfdriveState'].active
accel = self.simulated_car.sm['carControl'].actuators.accel if eng else float('nan')
cs = self.simulated_car.sm['carState']
sp = self.simulated_car.sm['starpilotPlan']
rd = self.simulated_car.sm['radarState']
m = self.simulated_car.sm['modelV2']
lead = rd.leadOne
print(f"BRIDGE engaged={eng} engable={engable} active={active} accel={accel:.2f} "
f"gas={throttle_out:.2f} brake={brake_out:.2f} steer={steer_out:.1f} "
f"v={st.speed if st.velocity is not None else -1:.2f} "
f"pos=({st.position[0]:.1f},{st.position[1]:.1f}) yaw={st.bearing:.1f} vCruiseK={cs.vCruise:.0f} "
f"forcStop={sp.forcingStop} apprStop={sp.approachStopLength:.1f} "
f"modelLen={m.position.x[-1]:.0f} mVel={m.velocity.x[-1]:.1f} "
f"mAcc={m.acceleration.x[-1]:.2f} sign={sp.stopSignConfirmed} "
f"leadD={lead.dRel:.1f} leadV={lead.vLead:.1f}", flush=True)
except Exception as e:
import traceback; traceback.print_exc()
print(f"BRIDGE-ERR {type(e).__name__}: {e}", flush=True)
if self.world.exit_event.is_set():
self.shutdown()
tools/sim/bridge/metadrive/metadrive_bridge.py
+10 -11
View File
@@ -1,4 +1,5 @@
import math
import os
from multiprocessing import Queue
from metadrive.component.sensors.base_camera import _cuda_enable
@@ -28,21 +29,13 @@ def curve_block(length, angle=45, direction=0):
}
def create_map(track_size=60):
curve_len = track_size * 2
return dict(
type=MapGenerateMethod.PG_MAP_FILE,
lane_num=2,
lane_width=4.5,
config=[
None,
straight_block(track_size),
curve_block(curve_len, 90),
straight_block(track_size),
curve_block(curve_len, 90),
straight_block(track_size),
curve_block(curve_len, 90),
straight_block(track_size),
curve_block(curve_len, 90),
straight_block(track_size * 10),
straight_block(track_size * 20),
]
)
@@ -73,7 +66,13 @@ class MetaDriveBridge(SimulatorBridge):
image_source="rgb_road",
),
sensors=sensors,
image_on_cuda=_cuda_enable,
# CUDA image capture (cudaGraphicsGLRegisterImage) only works when the
# Panda3D GL context is NVIDIA-backed. On hybrid/compute-only laptops the
# display GL runs on the iGPU via Mesa (often llvmpipe software), so CUDA-GL
# interop fails with cudaErrorUnknown and crashes MetaDrive at sensor init.
# Default to the safe CPU readback path; opt into CUDA explicitly only when
# the GL context is actually NVIDIA.
image_on_cuda=bool(os.getenv("STARPIOT_CUDA_IMAGES")) and _cuda_enable,
image_observation=True,
interface_panel=[],
out_of_route_done=False,
tools/sim/bridge/metadrive/metadrive_world.py
+1
View File
@@ -86,6 +86,7 @@ class MetaDriveWorld(World):
curr_pos = md_vehicle.position
state.velocity = md_vehicle.velocity
state.position = md_vehicle.position
state.bearing = md_vehicle.bearing
state.steering_angle = md_vehicle.steering_angle
state.gps.from_xy(curr_pos)
tools/sim/lib/camerad.py
+48 -18
View File
@@ -1,7 +1,5 @@
import numpy as np
import os
import pyopencl as cl
import pyopencl.array as cl_array
from msgq.visionipc import VisionIpcServer, VisionStreamType
from cereal import messaging
@@ -24,17 +22,25 @@ class Camerad:
self.vipc_server.start_listener()
# set up for pyopencl rgb to yuv conversion
self.ctx = cl.create_some_context()
self.queue = cl.CommandQueue(self.ctx)
cl_arg = f" -DHEIGHT={H} -DWIDTH={W} -DRGB_STRIDE={W * 3} -DUV_WIDTH={W // 2} -DUV_HEIGHT={H // 2} -DRGB_SIZE={W * H} -DCL_DEBUG "
kernel_fn = os.path.join(BASEDIR, "tools/sim/rgb_to_nv12.cl")
with open(kernel_fn) as f:
prg = cl.Program(self.ctx, f.read()).build(cl_arg)
self.krnl = prg.rgb_to_nv12
self.Wdiv4 = W // 4 if (W % 4 == 0) else (W + (4 - W % 4)) // 4
self.Hdiv4 = H // 4 if (H % 4 == 0) else (H + (4 - H % 4)) // 4
# GPU-accelerated rgb->nv12 via pyopencl when an OpenCL platform exists,
# otherwise fall back to a CPU numpy conversion (e.g. laptops without an
# OpenCL ICD, such as NVIDIA-only hosts).
self.ctx = None
try:
import pyopencl as cl
import pyopencl.array as cl_array
self._cl_array = cl_array
self.ctx = cl.create_some_context()
self.queue = cl.CommandQueue(self.ctx)
cl_arg = f" -DHEIGHT={H} -DWIDTH={W} -DRGB_STRIDE={W * 3} -DUV_WIDTH={W // 2} -DUV_HEIGHT={H // 2} -DRGB_SIZE={W * H} -DCL_DEBUG "
kernel_fn = os.path.join(BASEDIR, "tools/sim/rgb_to_nv12.cl")
with open(kernel_fn) as f:
prg = cl.Program(self.ctx, f.read()).build(cl_arg)
self.krnl = prg.rgb_to_nv12
self.Wdiv4 = W // 4 if (W % 4 == 0) else (W + (4 - W % 4)) // 4
self.Hdiv4 = H // 4 if (H % 4 == 0) else (H + (4 - H % 4)) // 4
except Exception:
self.ctx = None
def cam_send_yuv_road(self, yuv):
self._send_yuv(yuv, self.frame_road_id, 'roadCameraState', VisionStreamType.VISION_STREAM_ROAD)
@@ -49,11 +55,35 @@ class Camerad:
assert rgb.shape == (H, W, 3), f"{rgb.shape}"
assert rgb.dtype == np.uint8
rgb_cl = cl_array.to_device(self.queue, rgb)
yuv_cl = cl_array.empty_like(rgb_cl)
self.krnl(self.queue, (self.Wdiv4, self.Hdiv4), None, rgb_cl.data, yuv_cl.data).wait()
yuv = np.resize(yuv_cl.get(), rgb.size // 2)
return yuv.data.tobytes()
if self.ctx is not None:
rgb_cl = self._cl_array.to_device(self.queue, rgb)
yuv_cl = self._cl_array.empty_like(rgb_cl)
self.krnl(self.queue, (self.Wdiv4, self.Hdiv4), None, rgb_cl.data, yuv_cl.data).wait()
yuv = np.resize(yuv_cl.get(), rgb.size // 2)
return yuv.data.tobytes()
return self.rgb_to_yuv_cpu(rgb).tobytes()
@staticmethod
def rgb_to_yuv_cpu(rgb):
"""Numpy NV12 conversion mirroring tools/sim/rgb_to_nv12.cl (BT.601 limited)."""
r = rgb[:, :, 0].astype(np.int32)
g = rgb[:, :, 1].astype(np.int32)
b = rgb[:, :, 2].astype(np.int32)
y = ((b * 13 + g * 65 + r * 33 + 64) >> 7) + 16
y = np.clip(y, 0, 255).astype(np.uint8)
def avg2(ch):
return (ch[0::2, 0::2] + ch[0::2, 1::2] + ch[1::2, 0::2] + ch[1::2, 1::2] + 1) >> 1
r2, g2, b2 = avg2(r), avg2(g), avg2(b)
u = ((b2 * 56 - g2 * 37 - r2 * 19 + 0x8080) >> 8) & 0xFF
v = ((r2 * 56 - g2 * 47 - b2 * 9 + 0x8080) >> 8) & 0xFF
uv = np.empty((H // 2, W), dtype=np.uint8)
uv[:, 0::2] = u.astype(np.uint8)
uv[:, 1::2] = v.astype(np.uint8)
return np.concatenate([y.ravel(), uv.ravel()]).astype(np.uint8)
def _send_yuv(self, yuv, frame_id, pub_type, yuv_type):
eof = int(frame_id * 0.05 * 1e9)
tools/sim/lib/common.py
+2 -1
View File
@@ -5,7 +5,7 @@ import numpy as np
from abc import ABC, abstractmethod
from collections import namedtuple
W, H = 1928, 1208
W, H = 1164, 874
vec3 = namedtuple("vec3", ["x", "y", "z"])
@@ -41,6 +41,7 @@ class SimulatorState:
self.ignition = True
self.velocity: vec3 = None
self.position: tuple = (0, 0)
self.bearing: float = 0
self.gps = GPSState()
self.imu = IMUState()
tools/sim/lib/keyboard_ctrl.py
+15 -3
View File
@@ -34,7 +34,11 @@ KEYBOARD_HELP = """
def getch() -> str:
STDIN_FD = sys.stdin.fileno()
old_settings = termios.tcgetattr(STDIN_FD)
try:
old_settings = termios.tcgetattr(STDIN_FD)
except (termios.error, OSError):
# No controlling terminal (e.g. headless / background run). Idle instead of crashing.
return None
try:
# set
mode = old_settings.copy()
@@ -48,8 +52,13 @@ def getch() -> str:
termios.tcsetattr(STDIN_FD, termios.TCSAFLUSH, mode)
ch = sys.stdin.read(1)
except (termios.error, OSError):
ch = None
finally:
termios.tcsetattr(STDIN_FD, termios.TCSADRAIN, old_settings)
try:
termios.tcsetattr(STDIN_FD, termios.TCSADRAIN, old_settings)
except (termios.error, OSError):
pass
return ch
def print_keyboard_help():
@@ -60,7 +69,10 @@ def keyboard_poll_thread(q: 'Queue[QueueMessage]'):
while True:
c = getch()
if c == '1':
if c is None:
# No terminal input available (headless/background); avoid busy-looping.
time.sleep(0.05)
elif c == '1':
q.put(control_cmd_gen("cruise_up"))
elif c == '2':
q.put(control_cmd_gen("cruise_down"))
tools/sim/lib/simulated_car.py
+11 -1
View File
@@ -15,7 +15,7 @@ class SimulatedCar:
def __init__(self):
self.pm = messaging.PubMaster(['can', 'pandaStates'])
self.sm = messaging.SubMaster(['carControl', 'controlsState', 'carParams', 'selfdriveState'])
self.sm = messaging.SubMaster(['carControl', 'controlsState', 'carParams', 'selfdriveState', 'carState', 'starpilotPlan', 'radarState', 'modelV2'])
self.cp = self.get_car_can_parser()
self.idx = 0
self.params = Params()
@@ -76,6 +76,16 @@ class SimulatedCar:
msg.append(self.packer.make_can_msg("STEERING_CONTROL", 2, {}))
msg.append(self.packer.make_can_msg("ACC_HUD", 2, {}))
msg.append(self.packer.make_can_msg("LKAS_HUD", 2, {}))
# 0x35e CAMERA_MESSAGES: StarPilot's Honda carstate registers this message
# when it reads the speed-limit / stop-sign fields (HAS_CAMERA_MESSAGES
# flag), and can_valid requires *every* registered message to be valid.
# The real camera sends it on the cam bus, so publish it here too;
# CANPacker auto-fills the Honda COUNTER/CHECKSUM. SPEED_LIMIT_SIGN=0
# means no posted speed limit, so calculate_speed_limit() returns 0.0.
msg.append(self.packer.make_can_msg("CAMERA_MESSAGES", 2, {
"SPEED_LIMIT_SIGN": 0,
"ROAD_SIGN": 0,
}))
self.pm.send('can', can_list_to_can_capnp(msg))
tools/sim/starpilot_sim.sh
Executable
+183
View File
@@ -0,0 +1,183 @@
#!/usr/bin/env bash
# StarPilot live MetaDrive simulator launcher.
#
# Runs the ENTIRE openpilot stack (manager.py -> modeld/controlsd/plannerd/...) on
# the host RTX GPU, bridged to a straight-road MetaDrive world, in one of three
# longitudinal modes.
#
# Usage:
# ./tools/sim/starpilot_sim.sh {exp|chill|cem|hem} [--headless] [--joystick] [--cpu]
#
# exp : standard Experimental mode (ConditionalExperimental=off, ConditionalChill=off, HybridExperimental=off)
# chill : pure Chill / CCM mode (ConditionalChill=on)
# cem : Conditional Experimental mode (ConditionalExperimental=on)
# hem : Hybrid Experimental mode (HybridExperimental=on)
#
# The model runs on the RTX GPU (tinygrad CUDA) by default. Pass --cpu to force
# the CPU backend instead (fragile in this fork; not recommended).
#
# By default the model is traced LIVE from driving_supercombo.onnx inside modeld
# (Option 1: no tinygrad-JIT pickling, so no JIT-unpickle corruption). Pass --pkl
# to fall back to the precompiled pickle artifact instead.
#
# The stack runs from the isolated host worktree (.host_runtime/linux/worktree),
# which is synced from this repo by `./dev sync` (run automatically below).
set -euo pipefail
# Clean up any stale openpilot/sim processes and shared-memory sockets from a
# previous run. pkill on "manager.py" alone misses the the_galaxy/galaxy Flask
# processes (different proctitles) which otherwise hold port 8083 and crash-loop,
# and stale msgq sockets in /dev/shm collide with fresh publishers.
pkill -9 -f "system/manager/manager.py" 2>/dev/null || true
pkill -9 -f "run_bridge.py" 2>/dev/null || true
pkill -9 -f "metadrive" 2>/dev/null || true
pkill -9 -f "the_galaxy" 2>/dev/null || true
pkill -9 -f "galaxy.galaxy" 2>/dev/null || true
sleep 1
rm -rf /dev/shm/msgq* /dev/shm/visionipc* /tmp/openpilot* 2>/dev/null || true
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
MODE="${1:-hem}"
ARGS=("${@:2}")
HEADLESS=0
JOYSTICK=0
SIM_DEV="CUDA"
LIVE_ONNX=1
for a in "${ARGS[@]}"; do
case "$a" in
--headless) HEADLESS=1 ;;
--joystick) JOYSTICK=1 ;;
--cpu) SIM_DEV="CPU" ;;
--pkl) LIVE_ONNX=0 ;;
*) echo "unknown arg: $a" >&2; exit 1 ;;
esac
done
case "$MODE" in
exp|chill|cem|hem) : ;;
*) echo "usage: $0 {exp|chill|cem|hem} [--headless] [--joystick] [--cpu]" >&2; exit 1 ;;
esac
# 1. Sync main repo -> host worktree so our sim/mode changes land there.
"${ROOT_DIR}/dev" sync shared >/dev/null 2>&1 || true
HOST_WORKTREE="${ROOT_DIR}/.host_runtime/linux/worktree"
HOST_PY="${HOST_WORKTREE}/.venv/bin/python3"
MODEL_STORE="${HOME}/.comma/starpilot/data/models"
echo "==> StarPilot sim mode: ${MODE} (model device: ${SIM_DEV}) (host worktree: ${HOST_WORKTREE})"
# 2. Make sure the builtin model is present where modeld loads it. Sync wipes the
# worktree copy, so re-copy it from the model store on every launch. With the
# live-ONNX path (default) modeld ignores the pkl and traces the ONNX instead,
# but keep the pkl in place so --pkl still works.
MODEL_SRC="${MODEL_STORE}/rdf43_driving_tinygrad.pkl"
MODEL_DST="${HOST_WORKTREE}/selfdrive/modeld/models/driving_tinygrad.pkl"
if [[ -f "${MODEL_SRC}" ]]; then
if ! cmp -s "${MODEL_SRC}" "${MODEL_DST}"; then
cp -f "${MODEL_SRC}" "${MODEL_DST}"
echo "==> Copied builtin model to ${MODEL_DST}"
fi
else
echo "!! builtin model not found at ${MODEL_SRC}" >&2
fi
# Option 1 (live-ONNX): modeld traces driving_supercombo.onnx in-memory on the
# RTX, so the tinygrad-JIT pickle round-trip (and its QCOM-rewrite bug) never
# happens. Falls back to the pkl artifact when --pkl is passed or the onnx is
# missing.
# Locate the source ONNX: prefer the model store copy, fall back to the repo root.
ONNX_SRC="${MODEL_STORE}/driving_supercombo.onnx"
if [[ ! -f "${ONNX_SRC}" ]] && [[ -f "${ROOT_DIR}/driving_supercombo.onnx" ]]; then
ONNX_SRC="${ROOT_DIR}/driving_supercombo.onnx"
fi
if [[ "$LIVE_ONNX" == "1" ]]; then
if [[ -f "${ONNX_SRC}" ]]; then
export STARPIOT_LIVE_ONNX="${ONNX_SRC}"
echo "==> modeld will trace ${ONNX_SRC} live on ${SIM_DEV}"
else
echo "!! driving_supercombo.onnx NOT found (checked model store and repo root)." >&2
echo " Falling back to the precompiled pkl, which CRASHES with CUDA_ERROR_INVALID_IMAGE." >&2
echo " Place the onnx at ${MODEL_STORE}/driving_supercombo.onnx and re-run." >&2
fi
fi
# 3. Set the longitudinal mode params (mutually exclusive).
# ForceOnroad: on a PC host there is no real panda/ignition, so hardwared never
# transitions the device onroad by itself; force it so modeld/controlsd/plannerd
# come up and the sim can engage and drive.
"${HOST_PY}" - "${MODE}" <<'PY'
import sys
from openpilot.common.params import Params
p = Params()
mode = sys.argv[1]
p.put_bool_nonblocking("ConditionalExperimental", mode == "cem")
p.put_bool_nonblocking("ConditionalChill", mode == "chill")
p.put_bool_nonblocking("HybridExperimental", mode == "hem")
p.put_bool_nonblocking("ForceOnroad", True)
print(f"params: ConditionalExperimental={p.get_bool('ConditionalExperimental')} "
f"ConditionalChill={p.get_bool('ConditionalChill')} "
f"HybridExperimental={p.get_bool('HybridExperimental')} "
f"ForceOnroad={p.get_bool('ForceOnroad')}")
PY
# 4. Sim environment (mirrors tools/sim/launch_openpilot.sh) + GPU selection.
export PASSIVE="0"
export NOBOARD="1"
export SIMULATION="1"
export SKIP_FW_QUERY="1"
export FINGERPRINT="HONDA_CIVIC_2022"
export BLOCK="camerad,loggerd,encoderd,micd,logmessaged,soundd,mapd"
if [[ "$HEADLESS" == "1" ]]; then
export BLOCK="${BLOCK},ui"
fi
export DEV="${SIM_DEV}"
export STARPIOT_SIM_DEV="${SIM_DEV}"
# The supercombo artifact is compiled all-CUDA (warp + policy on the RTX). The
# warp device must be CUDA so the precompiled kernels match; modeld copies the
# host-memory camera frames onto the GPU before the warp.
if [[ "${SIM_DEV}" == "CPU" ]]; then
export WARP_DEV="CPU"
export QUEUE_DEV="CPU"
else
export WARP_DEV="${SIM_DEV}"
export QUEUE_DEV="${SIM_DEV}"
fi
# 5. Launch the full openpilot stack in the background, then the bridge in the foreground.
cat <<'HELP'
==> Controls (focus the terminal that launched this script):
i : toggle ignition (starts ON by default; if the status line shows
Ignition: False, press i once to turn it back on)
2 : cruise Set (engage lateral + longitudinal) 1 : cruise Resume / accel
3 : cruise Cancel r : reset simulation
w/a/s/d : manual throttle / steer / brake q : quit everything
z/x : blinker left / right
HELP
cd "${HOST_WORKTREE}"
MANAGER_LOG="${HOST_WORKTREE}/.host_sim_manager.log"
"${HOST_PY}" -c "from openpilot.selfdrive.test.helpers import set_params_enabled; set_params_enabled()"
echo "==> Starting openpilot stack (manager.py) ..."
"${HOST_PY}" system/manager/manager.py >"${MANAGER_LOG}" 2>&1 &
MANAGER_PID=$!
echo "==> manager.py pid ${MANAGER_PID} (log: ${MANAGER_LOG})"
trap 'echo "==> stopping manager (${MANAGER_PID})"; kill "${MANAGER_PID}" 2>/dev/null || true' EXIT
# modeld blocks on the camerad visionipc stream before it starts tracing the
# ONNX, and that stream is published by the bridge. The bridge MUST come up
# promptly or modeld never loads, so keep the startup delay short and let the
# bridge auto-engage once controls report engageable.
sleep 8
# Single-camera mode: MetaDrive must render only the road viewport. Dual-camera
# renders a second wide viewpoint every frame, roughly halving frame rate. The
# driving pipeline (modeld) consumes roadCameraState only, so the wide cam adds
# no control value in sim.
BRIDGE_ARGS=()
if [[ "$JOYSTICK" == "1" ]]; then
BRIDGE_ARGS+=(--joystick)
fi
echo "==> Starting MetaDrive bridge (${BRIDGE_ARGS[*]}) ..."
"${HOST_PY}" tools/sim/run_bridge.py "${BRIDGE_ARGS[@]}" || true
echo "==> Bridge exited."