Commit Graph

58 Commits

Author SHA1 Message Date
Cameron Clough 80da3aee14 mypy: use implicit-optional (#31590)
* mypy: set implicit-optional = true

* find and replace '| None = None' -> '= None' in function args
2024-02-25 13:29:18 -08:00
Cameron Clough 995250ae49 use pyupgrade to update to new typing syntax (#31580)
* add pyupgrade hook

* run pyupgrade (pre-commit run -a)

* ruff --fix

* Revert "add pyupgrade hook"

This reverts commit 56ec18bb6b8602a0b612f3803d96cdad14b52066.

* revert changes to third_party/

* manual type fixes

* explicit Optional wrapping capnp objects
2024-02-24 16:41:23 -08:00
Kacper Rączy 7a5d85232f regen: dummy dcamera support (#30323)
* DummyFrameReader

* dummy_dcam_if_missing flag

* Blank dcamera for all neo segments, as dmonitoringd does not support neo camera feeds

* Remove trailing whitespace

* zero_dcamera staticmethod

* enable dmonitoring in test_regen

* Simplify. User decides wether to use dummy or not

* Moving some stuff

* rename to dummy_driver_cam
2023-10-30 15:17:32 -07:00
Justin Newberry 1820b5f259 Proc Replay: cleanup iterable log typing (#30261)
* cleanup log typing

* use iterable

* fix those

* Cleanup

* cleanup

* missed that one
2023-10-17 12:58:28 -07:00
Kacper Rączy 18f41d1910 regen: skip fetching camera files when not needed (#30169)
* Fetch only camera files which are needed in current setup

* Refactor daemons to processes
2023-10-04 15:00:50 -07:00
Shane Smiskol 661edd7d8b regen: fix wide road camera msg name (#30147)
fixed
2023-10-02 21:21:49 -07:00
Adeeb Shihadeh a9626f95b6 add openpilot prefix to imports (#29498)
* add openpilot prefix to imports

* more

* more

* fix docs

* fix linter

* bump submodules

* fix patched tests

* update dynamic imports

* debug

* Revert "debug"

This reverts commit db5e13b9911cc74438bee123bc3430da6c31b24b.

* fix pm test
2023-08-20 20:49:55 -07:00
Justin Newberry 7fdd9fc37c CI: Pylint to ruff (#29294)
* pylint to riff

* pylint to riff

* pylint to riff

* revert more

* undo exclude removal

* exclude tinygrad

* set line length

* pylint exclusions to ruff

* same excludes as old linter

* fix tools QA

* remove unrequired check

* revert linting third_party

* ignore e402
2023-08-09 13:26:54 -07:00
Justin Newberry cf64b27168 Linter: Fix >160 line length test (#29287)
* test

* fix line length

* fix line length

* fix file size

* revert poetry

* cleanup for PR
2023-08-08 17:13:35 -07:00
Justin Newberry a91764954b Revert "CI: pylint to ruff (#29276)" (#29284)
This reverts commit cb31a53d8a.
2023-08-08 14:24:04 -07:00
Justin Newberry cb31a53d8a CI: pylint to ruff (#29276)
* pylint to riff

* pylint to riff

* pylint to riff

* revert more

* undo exclude removal

* exclude tinygrad
2023-08-08 13:41:30 -07:00
Kacper Rączy 1235110983 regen: blacklist-procs option (#29024)
* Add blacklist-procs option to regen

* Keep the original process order
2023-07-18 17:13:23 -07:00
Kacper Rączy b2d0f74ec3 process_replay: support for replaying multiple processes simultaneously (#28829)
* first working ProcessContainer and _replay_multi_process implementation

* Setup controlsd config_callback

* Add vision ipc support

* Remove old code

* add todo

* Update estimated processing times

* Use deepcopy when getting configs

* Rework the API to support passing multiple names or configs

* Add get_custom_params_from_lr

* Add more typing

* Enable simulation for controlsd

* Fix typing issues

* Regen refactor

* Fix linter issues

* Make wideRoadCameraState optional

* Move DummySocket to helpers

* Improve typing in regen

* Minor improvements

* Maintain sort order of internal_pub_queue using heapq

* Move ProcessContainer elsewhere

* Fix internal heap

* Change comment

* Remove subtest_name from ProcessConfig

* Update cereal

* Add newline
2023-07-18 14:56:24 -07:00
Kacper Rączy 15fcd0d9d4 regen: remove support for fake extra vision client (#28520)
* Remove extra client from camera replay

* Fix another device type check
2023-06-13 16:13:20 -07:00
Kacper Rączy 838346d8a3 process_replay: deprecated message migration (#28456)
* Move migration code to new file

* Migrate messages in process replay. Remove old code

* Fix old_logtime

* Keep old logtimes on migrated messages

* Update ref commit
2023-06-09 19:17:50 -07:00
Kacper Rączy 64377a88f9 process_replay: helpers (#28367)
* process_replay helper

* Refactor

* refactor check_enabled uses

* fix __init__

* noqa in __init__ imports

* Move compare_logs outside of process_replay

* Move save_logs to tools.lib.helpers

* Remove save_log from compare_logs

* fix lint issues
2023-06-06 10:31:13 -07:00
Kacper Rączy 1a0f482d4d regen: fix params (#28375)
Fix environment setup for regen
2023-06-02 15:39:23 -07:00
Kacper Rączy e6ac6320ac process_replay: universal replay mechanism using cereal fake sockets (#28115)
* replay_process_with_fake_sockets implementation

* add missing polled_pubs to configs

* drained_pubs field

* updated cereal

* Remove python/native variations of process replay. Replace with universal one using cereal fake sockets

* Replace old py FakeSocket with DummySocket

* Invalidate and deregister fake sockets after replay is done

* Remove unused import

* Set up new prefix for each replay

* Fixes for radard

* Refactor ReplayContext and ProcessConfig

* Minor fixes

* Reimplement controlsd fingerprinting callback

* time.sleep for sockets to safely reconnect

* Fix fingerprinting for controlsd

* Fixes for regen to work

* Fix replay loop to respect submaster frames

* Fix profiler to use new ProcessConfig fields

* Remove tqdm

* Refactor tests to use new ProcessConfig

* Add FrequencyBasedRcvCallback

* Make tolerance None by default

* Update cereal

* Add get_process_config utility func

* Update cereal. Simplify sync procedure

* Chain context managers

* New sub-socket reconnection procedure

* Fix linter issues

* Revert chaining of context managers

* Init controlsState only when replaying controlsd. Update cereal

* Update cereal

* Update process_replay to use new cereal API

* Update cereal

* Update cereal

* Update cereal

* Simplify radard recv callback

* Update release/files_common
2023-05-23 17:24:16 -07:00
Kacper Rączy ba4a1c1b87 regen: whitelist-proc (#27980)
* Add support for whitelist-procs to toggle on/off specific processes during regen

* Use process CONFIGS from process_replay to generate fake processes instead of hard-coded values

* Add rawgpsd to fake_daemons. Remove duplicates from replayable_messages.

* Remove duplicate block of code from setup_env

* Remove rawgpsd comment
2023-04-28 19:36:31 -07:00
Kacper Rączy eee7dd082f regen: ecamera support (#27999)
* Add wideRoadCameraState support in regen

* Fix for ecam FrameReader when use_route_meta

* Pylint fixes
2023-04-22 02:43:04 +00:00
Vivek Aithal 5f7d9a519e regen: Refactor log migrate functions to avoid needing azure keys (#26049)
* refactor migrate fns to avoid needing to use azure keys on import

* move azure key init behind a function

* resolve comments
2022-10-13 13:21:35 -07:00
Adeeb Shihadeh c782380fc1 Hyundai: share panda flags with CAN-FD platform (#26058)
* Hyundai: share panda flags with CAN-FD platform

* move that

* only set bit

* bump panda

* panda master

* regen + update refs for new param
2022-10-12 21:54:08 -07:00
Vivek Aithal dc72438be5 regen: add arg for migrating sensorEvents with old timestamps (#25980)
* add event logMonoTime

* add arg for old logtime
2022-10-05 21:16:41 -07:00
Kurt Nistelberger 29d3ed2ce6 Sensor events splitup (#25714)
* PoC of reading sensors via interrupts instead of polling

* add Gyro and draft for magn

* add more functionality to gpio.cc

* change LSM gyro to interrupt

* resolve rebase conflict

* update BMX accel interrupt impl

* add interrupt collector thread to fetch in parallel

* change get_event interface to return true on successful read

* update BMX gyro interrupt impl

* update gpio.h/.cc according to comments

* address comments, rename Edgetype enum

* Edgetype to EdgeType

* update sensor interrupt interface

* add error handling, and read fd on trigger

* avoid sending empty messages

* fix build

* use gpiochip

* less diff

* gpiochip on both edges, but skip falling edge if rising edge is detected

* init last_ts with 0

* update sensord testcases

* update sensord testsweet

* test for pipeline

* readd with_process

* add null check

* move tests update to seperate PR

* sensord: improve test coverage (#25683)

* update sensord-interrupt testsweet

* address review comments

* inc stddev threshold

* fix format string

* add version 0 check again

* relax strictness after c3 with bmx tests

* relax strictness after tests

Co-authored-by: Kurt Nistelberger <kurt.nistelberger@gmail.com>

* address PR comments

* fix typo

* remove 4ms limit, and skip first 0.5sec of data

* revert disable_interuppt change to destructor

* fix and remove timing skip

* make gpiochip generic

* sensord port

* change from sensorEvents to separated events

* fix gyro usage

* add splitted sensor tests

* modify debug script sensor_data_to_hist.py

* refactor get_event interface to remove sensorEvent message type

* update locationd to non sensorEvent usage

* tmp commit

* fix replay

* fix accelerometer type

* fix sensor to hist debug script

* update sensord tests to split events

* remove rebase artifacts

* port test_sensord.py

* small clean up

* change cereal to sensorEvents-splitup branch

* upate sensorEvents in regen

* fix route generation for splitted sensor events

* regen cleanUp from sensorEvents change

* .

* remove light and temp from locationd

* add generic init delay per sensor

* .

* update routes

* move bmx gyro/accel to its own channel

* adopt sensor tests to bmx channel

* remove rebase artifacts

* fix sensord test

* handle bmx not present

* add bmx sockets to regen

* .

* .

* code cleanUp

* .

* address PR comments

* address PR comments

* address PR comments

* lsm clean up

* readd sensorEvents

* rever regen.py

* .

* update replay refs

* move channels

* fix artifact

* bump cereal

* update refs

* fix timing issue

Co-authored-by: Bruce Wayne <batman@workstation-eu-intern2.eu.local>
Co-authored-by: gast04 <kurt.nistelberger@gmail.com>
Co-authored-by: Willem Melching <willem.melching@gmail.com>
Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-09-29 14:31:54 -07:00
Kurt Nistelberger 96ed5aa581 Sensor events regen (#25903) 2022-09-27 20:33:45 -07:00
Vivek Aithal 4fa62f1464 Live torque (#25456)
* wip torqued

* add basic logic

* setup in manager

* check sanity and publish msg

* add first order filter to outputs

* wire up controlsd, and update gains

* rename intercept to offset

* add cloudlog, live values are not updated

* fix bugs, do not reset points for now

* fix crashes

* rename to main

* fix bugs, works offline

* fix float in cereal bug

* add latacc filter

* randomly choose points, approx for iid

* add variable decay

* local param to capnp instead of dict

* verify works in replay

* use torqued output in controlsd

* use in controlsd; use points from past routes

* controlsd bugfix

* filter before updating gains, needs to be replaced

* save all points to ensure smooth transition across routes, revert friction factor to 1.5

* add filters to prevent noisy low-speed data points; improve fit sanity

* add engaged buffer

* revert lat_acc thresh

* use paramsd realtime process config

* make latacc-to-torque generic, and overrideable

* move freq to 4Hz, avoid storing in np.array, don't publish points in the message

* float instead of np

* remove constant while storing pts

* rename slope, offset to lat_accet_factor, offset

* resolve issues

* use camelcase in all capnp params

* use camelcase everywhere

* reduce latacc threshold or sanity, add car_sane todo, save points properly

* add and check tag

* write param to disk at end of route

* remove args

* rebase op, cereal

* save on exit

* restore default handler

* cpu usage check

* add to process replay

* handle reset better, reduce unnecessary computation

* always publish raw values - useful for debug

* regen routes

* update refs

* checks on cache restore

* check tuning vals too

* clean that up

* reduce cpu usage

* reduce cpu usage by 75%

* cleanup

* optimize further

* handle reset condition better, don't put points in init, use only in corolla

* bump cereal after rebasing

* update refs

* Update common/params.cc

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* remove unnecessary checks

* Update RELEASES.md

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-09-19 15:19:26 -07:00
Shane Smiskol 87ca42e993 process replay: initialize controlsState with original route (#25461)
* push

* do process replay

* commit

* update refs

* clean up

* clean up controlsd

* clean up controlsd

* Add assert

* debubuggier param name

* can be peristent

* Revert "can be peristent"

This reverts commit 6e6d3f6423c26a202623cef728e038259e9e46cd.

* Update selfdrive/test/process_replay/process_replay.py
2022-08-16 22:02:03 -07:00
Shane Smiskol b3dc7bb3cb regen: initialize once (#25458)
regen: init once
2022-08-16 18:40:30 -07:00
Adeeb Shihadeh 505bbce29d controlsd: handle radar missing + cleanup system error handling (#25179)
* controlsd: handle radar missing + cleanup system erorr handling

* fix carla

* update refs
2022-07-14 19:55:18 -07:00
Shane Smiskol ea241bf3dc FW fingerprinting: log all FW versions (#25042)
* get_fw_versions returns all fw versions with request's brand

* keep track of everything received

* debug

* need to regen or write a hack in build_fw_dict

* to be safe, still replace old responses within same brands (hyundai responds to two queries, can fix later)

to be safe, still replace old responses within same brands (hyundai responds to two queries, can fix later)

* update test_fw_query_on_routes

* clean up

* better name

* slightly cleaner

* fix test_startup unit test

del

* fix imports

* fix test_fw_fingerprint

fix test_fw_fingerprint

fix

* fingerprint on all FW_VERSIONS, not just brands with requests

* support old routes in test_fw_query_on_routes

* regen and update refs

* similar function style to before

* better comment

* space

switch name

* try to exact match first

* useless else

* fix debug script

* simpler dictionary

* bump cereal to master
2022-07-06 19:42:58 -07:00
Shane Smiskol f21b56f25a regen migration: use Panda safety parameters (#25043)
no magic numbers
2022-07-05 20:04:48 -07:00
Shane Smiskol 5dce006a82 regen: send wideRoadCameraState (#24863)
* fix camera malfunction

* revert

* send in one process to fix frames out of sync

* not used

* revert

* fix and add --no-upload for CI testing

* fingerprint if source is fw

* no FW versions
2022-06-16 22:41:13 -07:00
Shane Smiskol 170ed3d761 process replay: clean up common code (#24855)
* regen and process replay clean up

* test_fuzzy actually uses fingerprint hardcoding

fix

* revert

* revert

* this can be a url or path so just print full variable
2022-06-14 12:03:30 -07:00
Lukas Petersson 397da56c85 process replay: regen in parallel (#24628)
* regen in parallel

* prefixes

* clean regen

* clean output

* tqdm loc

* del swp file

* add routes back

* cleanup

* disable tqdm

* unique dirs

* unique dirs

* outdir in regen_all

* formatting when played from other dirs

* prefix dongle id

* local disable_tqdm

* formatting

* bug fix

* dont spam fakedata

* 16 char fake dongle ids

* formatting

* formatting

* more descriptive dongle

* fix azure path

* couple more fixes

* handle failures nicely

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-06-06 14:21:12 -07:00
George Hotz ea5b8cdfb1 nv12: encoderd avoids a full frame copy (#24519)
* rgb to nv12

* nv12 works (w memcpy)

* correct now

* no copy

* fix nv12 with fast debayer

* reverts of unused stuff

* ui use nv12

* comment out thumbnails for now

* rebase fix

* dm read nv12

* model read nv12

* fix ffmpeg encoder

* thumbnails from nv12

* replay to nv12

* python framereader support nv12

* remove hardcoded frame/buffer sizes

* fix build

* ffmpeg encoder fix buffers

* small cleanup

* reduce power usage test

* fix cpu usage test

* fix snapshot

* fix loggerd test

* bump cereal

Co-authored-by: Comma Device <device@comma.ai>
Co-authored-by: Joost Wooning <jwooning@gmail.com>
2022-06-01 17:18:28 +02:00
Adeeb Shihadeh 32a9ccfc37 visionipc python shim (#24660) 2022-05-25 20:49:56 -07:00
Lukas Petersson 7d06a366f7 process replay: fix regen routes with migration (#24651)
* hacks to make routes work

* clean up

Co-authored-by: Shane Smiskol <shane@smiskol.com>
2022-05-24 20:00:55 -07:00
Adeeb Shihadeh ab75c53792 replay: cleanup unused RGB buffers (#24537)
* replay: cleanup unused RGB buffers

* update replay cam

* little more
2022-05-14 19:55:12 -07:00
Shane Smiskol d6c6275163 Fix process replay regen (#24517)
* fix regen issues

* compress like uploader
2022-05-14 19:53:08 -07:00
Shane Smiskol d686f9e527 process replay: save logs to folder (#24509)
* save logs to folder for proc replay

* use fakedata dir

* fix

* only once and where needed
2022-05-12 19:19:47 -07:00
Willem Melching 77a6f3d034 delay bz2 compression from logging to uploading (#24392)
* remove log_name

* log without compression

* fix tests

* remove extension for bootlog

* another test fix

* uploader compresses

* also compress in athena

* only compress qlog

* more generic check in do_upload

* fix bootlog compression

* lower loggerd cpu usage

* dont link against bz2

* set core affinity to little cluster

* handle old files
2022-05-03 14:09:17 +02:00
Shane Smiskol 60b7114024 onnxruntime: initialize CUDA provider before predictions (#24330)
* initialize CUDA runtime before predictions

* dmond pauses modeld execution...

* done in other PR
2022-04-26 11:49:32 -07:00
Shane Smiskol a8e632bb98 regen: send vipc_client_extra frames (#24331)
* send extra frames to fix regen

* for sanity

* not used

* minimal changes
2022-04-26 00:15:09 -07:00
Shane Smiskol 6e62df65d7 Make safetyParam unsigned (#24160)
* bump panda & cereal

* make unsigned in boardd

* update health packet version

bump panda

* fix definition

* bump

* regen TOYOTA2 route

* upload route

* oh, ndog param wasn't being set

* regenerate routes with non-zero safety params

* update refs

* regen routes and update refs

* bump panda

* update refs

* update refs

* bump submodules to master
2022-04-21 00:01:05 -07:00
Shane Smiskol 1a5ddfaedd regen: check if regenerated route is enabled (#24207)
* check if regenerated route is enabled

* elif

* initialize CUDA runner before all other processes

* Revert "initialize CUDA runner before all other processes"

This reverts commit 124e3dd4d3408b6649f40b3e6e7f217ca347f59f.

* add back fixes

* fix typo
2022-04-13 18:01:16 -07:00
Shane Smiskol 5159878389 fix regen: specify frame timestamps for modeld (#24200)
* grab main frames when not using extra

* the actual fix
2022-04-12 23:07:33 -07:00
Shane Smiskol 9f3245fea7 regen: use setup_env from process replay (#24197)
* use setup_env from process replay

* rm line
2022-04-12 16:36:02 -07:00
Adeeb Shihadeh 8af20af66d pylint: enforce indentation (W0311) (#24039)
* pylint: enforce indentation (W0311)

* few more
2022-03-24 23:23:29 -07:00
Gijs Koning 57b6fdc17a Rename RGB vision streams to match YUV streams (#23961)
* Renaming VISION_STREAM_RGB_.. to match yuv names like VISION_STREAM_ROAD
VISION_STREAM_RGB_BACK -> VISION_STREAM_RGB_ROAD
VISION_STREAM_RGB_FRONT -> VISION_STREAM_RGB_DRIVER

* little more

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-03-14 14:03:54 -07:00
Adeeb Shihadeh 3e9cb7fa12 remove community features toggle (#23214)
* remove community features toggle

* bump cereal

* little more

* update refs
2022-01-24 14:49:47 -08:00