mirror of
https://github.com/sunnypilot/sunnypilot.git
synced 2026-07-08 18:42:06 +08:00
Armand du Parc Locmaria
52e182611d
* zero ll patched big model * probe in a subprocess so usbgpu lock gets released * compiles * runs * num_jobs gets overwritten, use side effect * poll tg devices * make sure build crashes on missing gpu * fine not to rely on Device.default * seperate tg env for each model runner * comment * Revert "seperate tg env for each model runner" This reverts commit f6470cc4258eaeb3e8e37907ef370871c9af5aa4. * env is shared, gate on flag * no fallback warp dev must be set * build for current device only, unless pc/release * comment * list * listen for plug in * add icon to status bar, read params on every frame (?) * log available devices * try copy out when loading? * Revert "log available devices" This reverts commit e8c52a5d59456d4820ecb13b99a6c46ea1386a20. * Revert "try copy out when loading?" This reverts commit 518f403aa03faeda1950fe3dbce0d9e4c1584455. * don't trigger device probe/caching on modeld prepare * re-export with ll and road edges * dont cache devices in manager process * get USBGPU from params * no usbgpu env * missed one * sconscript don't poll * unconditional env * always explicitely set devices on input tensors * set DEV so amd uses right compiler and iface?? * fix flag * bump tg * rm xdg_cache_home * tg don't bump all the way * missing gmmu=0 at compile time * dm set dev * tg backend * update gitignore * missing import * unused imports * rely on Device.DEFAULT at compile time (already the case bc onnxrunner) * comments * dm warp needs DEV set too * build both smol and big * misc typos * set dev at compile time * don't need * DEV=CPU when getting metadata, ensure we don't grab gpu lock * this would also grab lock * put bool * warp compile always prepare only * missed one * poll ui * missing here * don't force usbgpu at build time * tmp patch fetch_fw * catch all, follow hardwared patterns * simpler * compile make input queues * revert this * group this more readable * rm empty line * make dummy frame using numpy * revert compile make input queues * no compiler at runtime * cleanup * fine to rebuild all on change to device node for now * fix usbgpu_present * fix sconscript * no size in header stream decompress * DEBUG=2 * minimal viable feedback * egpu gray * oops * gotta do this actually * modeld build only depends on modeld devices * don't ship onnx to release? or chunk * don't need * can only set compiler on dev= * none device works, will use default * make linter happy * chunk agnostic onnx input to compile_modeld * chunk big onnx * +x chunker * fix #! * and don't ship chunked onnx to release * firmware now in correct location * better err on missing onnx/chunk * SConscript also need to accept chunked onnx * metadata also need to load maybe chunked * dedupe cmd * this needs to be on cpu * devices are set in the tgflags, we already depend on them * rebuilding on changed order is fine * read file chunked can already load either chunked or not * chunk all big onnx * less confusing * unused import * python device to load onnx bytes * default device for runners, python for metadata * why not * chunked to shm
Neural networks in openpilot
To view the architecture of the ONNX networks, you can use netron
Driving Model (vision model + temporal policy model)
Vision inputs (Full size: 799906 x float32)
- image stream
- Two consecutive images (256 * 512 * 3 in RGB) recorded at 20 Hz : 393216 = 2 * 6 * 128 * 256
- Each 256 * 512 image is represented in YUV420 with 6 channels : 6 * 128 * 256
- Channels 0,1,2,3 represent the full-res Y channel and are represented in numpy as Y[::2, ::2], Y[::2, 1::2], Y[1::2, ::2], and Y[1::2, 1::2]
- Channel 4 represents the half-res U channel
- Channel 5 represents the half-res V channel
- Each 256 * 512 image is represented in YUV420 with 6 channels : 6 * 128 * 256
- Two consecutive images (256 * 512 * 3 in RGB) recorded at 20 Hz : 393216 = 2 * 6 * 128 * 256
- wide image stream
- Two consecutive images (256 * 512 * 3 in RGB) recorded at 20 Hz : 393216 = 2 * 6 * 128 * 256
- Each 256 * 512 image is represented in YUV420 with 6 channels : 6 * 128 * 256
- Channels 0,1,2,3 represent the full-res Y channel and are represented in numpy as Y[::2, ::2], Y[::2, 1::2], Y[1::2, ::2], and Y[1::2, 1::2]
- Channel 4 represents the half-res U channel
- Channel 5 represents the half-res V channel
- Each 256 * 512 image is represented in YUV420 with 6 channels : 6 * 128 * 256
- Two consecutive images (256 * 512 * 3 in RGB) recorded at 20 Hz : 393216 = 2 * 6 * 128 * 256
Policy inputs
- desire
- one-hot encoded buffer to command model to execute certain actions, bit needs to be sent for the past 5 seconds (at 20FPS) : 100 * 8
- traffic convention
- one-hot encoded vector to tell model whether traffic is right-hand or left-hand traffic : 2
- lateral control params
- speed and steering delay for predicting the desired curvature: 2
- previous desired curvatures
- vector of previously predicted desired curvatures: 100 * 1
- feature buffer
- a buffer of intermediate features including the current feature to form a 5 seconds temporal context (at 20FPS) : 100 * 512
Driving Model output format (Full size: XXX x float32)
Refer to slice_outputs and parse_vision_outputs/parse_policy_outputs in modeld.
Driver Monitoring Model
- .onnx model can be run with onnx runtimes
- .dlc file is a pre-quantized model and only runs on qualcomm DSPs
input format
- single image W = 1440 H = 960 luminance channel (Y) from the planar YUV420 format:
- full input size is 1440 * 960 = 1382400
- normalized ranging from 0.0 to 1.0 in float32 (onnx runner) or ranging from 0 to 255 in uint8 (snpe runner)
- camera calibration angles (roll, pitch, yaw) from liveCalibration: 3 x float32 inputs
output format
- 84 x float32 outputs = 2 + 41 * 2 (parsing example)
- for each person in the front seats (2 * 41)
- face pose: 12 = 6 + 6
- face orientation [pitch, yaw, roll] in camera frame: 3
- face position [dx, dy] relative to image center: 2
- normalized face size: 1
- standard deviations for above outputs: 6
- face visible probability: 1
- eyes: 20 = (8 + 1) + (8 + 1) + 1 + 1
- eye position and size, and their standard deviations: 8
- eye visible probability: 1
- eye closed probability: 1
- wearing sunglasses probability: 1
- face occluded probability: 1
- touching wheel probability: 1
- paying attention probability: 1
- (deprecated) distracted probabilities: 2
- using phone probability: 1
- distracted probability: 1
- face pose: 12 = 6 + 6
- common outputs 1
- left hand drive probability: 1
- for each person in the front seats (2 * 41)