Commit Graph

134 Commits

Author SHA1 Message Date
Kacper Rączy a075905c4a process_replay: fix crash on segments without carParams (#28286)
* Fix crash on segments without carParams

* Add assertion for carParams
2023-05-27 02:10:10 +02:00
Kacper Rączy ac0272d0cc process_replay: use PARAMS_ROOT (#28282)
Use PARAMS_ROOT in process_replay
2023-05-24 13:16:22 -07:00
Kacper Rączy e6ac6320ac process_replay: universal replay mechanism using cereal fake sockets (#28115)
* replay_process_with_fake_sockets implementation

* add missing polled_pubs to configs

* drained_pubs field

* updated cereal

* Remove python/native variations of process replay. Replace with universal one using cereal fake sockets

* Replace old py FakeSocket with DummySocket

* Invalidate and deregister fake sockets after replay is done

* Remove unused import

* Set up new prefix for each replay

* Fixes for radard

* Refactor ReplayContext and ProcessConfig

* Minor fixes

* Reimplement controlsd fingerprinting callback

* time.sleep for sockets to safely reconnect

* Fix fingerprinting for controlsd

* Fixes for regen to work

* Fix replay loop to respect submaster frames

* Fix profiler to use new ProcessConfig fields

* Remove tqdm

* Refactor tests to use new ProcessConfig

* Add FrequencyBasedRcvCallback

* Make tolerance None by default

* Update cereal

* Add get_process_config utility func

* Update cereal. Simplify sync procedure

* Chain context managers

* New sub-socket reconnection procedure

* Fix linter issues

* Revert chaining of context managers

* Init controlsState only when replaying controlsd. Update cereal

* Update cereal

* Update process_replay to use new cereal API

* Update cereal

* Update cereal

* Update cereal

* Simplify radard recv callback

* Update release/files_common
2023-05-23 17:24:16 -07:00
Kacper Rączy 12b3ffcc49 process_replay: drain sockets instead recv_one (#28161)
* Drain the sockets until the next cycle in python process replay

* Add waiting mechanism to native process replay, which mimics the syncing

* Sort logs when comparing them. Drain all the sockets instead of only recv_socks.

* Reverse the retrieval order of FakeSocket

* Change pub order for torqued

* Update ref commit
2023-05-23 12:59:09 -07:00
Harald Schäfer bb3388f9d2 Laikad: blocking raw_gnss receive (#28083)
* Blocking receive

* redundant

* rm clocks

* timeout irrelevant now
2023-05-03 13:03:25 -07:00
Kacper Rączy ba4a1c1b87 regen: whitelist-proc (#27980)
* Add support for whitelist-procs to toggle on/off specific processes during regen

* Use process CONFIGS from process_replay to generate fake processes instead of hard-coded values

* Add rawgpsd to fake_daemons. Remove duplicates from replayable_messages.

* Remove duplicate block of code from setup_env

* Remove rawgpsd comment
2023-04-28 19:36:31 -07:00
Kacper Rączy eee7dd082f regen: ecamera support (#27999)
* Add wideRoadCameraState support in regen

* Fix for ecam FrameReader when use_route_meta

* Pylint fixes
2023-04-22 02:43:04 +00:00
Adeeb Shihadeh 85908f7e2b process replay: fix socket-ful replay flakiness (#27721)
* process replay: fix socket-ful replay flakiness

* bump up laikad timeout

* bump

* bump again

* revert that
2023-03-28 17:48:16 -07:00
Shane Smiskol 42449b482d boardd: ability to switch between ELM safety params (#27656)
* indecisive

* rename to generic FW query

* remove code and update comment

* we need this to start off, unless we set multiplexing immediately

* draft

* draft 2

* try that

* can't do this either, boardd might read Enabled after removing, but before setting new Request param

* this should work

* use one less param

* fix params

* match behavior (set all pandas to safety param of 1, disabling multiplexing for fingerprinting

* clean up (some tests may temp break)

* fix param name and sort

* time it

* yes it does matter

* add to hyundai's bus 5 query

* remove hyundai for now

* this should work

* clean up

* clean up

* flip argument around, clean up

* fix test_startup

* some clean up

* rm line

* comment makes more sense

* required typing

* clean up common type

* comments

* Update selfdrive/car/car_helpers.py

* line

* whoops, need to set before vin!

* fix debug

* annoying

* more debugging

* bug fix (needs both keys always)

* debuGG

debuGG

* Revert "debuGG"

This reverts commit 55b2f429324c0b92d5cfb2cabf8b20db1e166248.

* Revert "more debugging"

This reverts commit 02934c3403ad5270f03093508b704c151d1ccb2a.

* Revert "annoying"

This reverts commit 8b4e5e09989f9a0217e3ec1c0ba68735929b7366.

* clean that up

* bumpback

* bumpback

* every second write param

* flip

* stuff

* move up?

* fix timing out in CI

* rm
2023-03-23 00:14:31 -07:00
Adeeb Shihadeh 7319afbd51 process replay: prep for laikad subsock + QCOM GPS (#27632)
* process replay: prep for laikad subsock

* cleanup

* fix for qcomGnss

* detect ublox

* more debug info in err

* wip

* cleanup
2023-03-21 16:43:40 -07:00
Adeeb Shihadeh ee36c106af process replay: update after laika publish policy change (#27647)
* laika pubs all

* lr for locationd

* fix locationd qcom

* update refs
2023-03-21 16:14:22 -07:00
Shane Smiskol e729f5b763 Remove dashcam override param (#27509)
* remove dashcam override

* sort

* remove ford from segments
2023-03-06 15:36:39 -08:00
Harald Schäfer eb8bdc0026 MPC path in UI (#27380)
* 10s lat

* Full length MPC

* redfine N

* Leave controls the same for now

* Updates

* use long plan in lat plan

* interp plan

* simplergit add selfdrive/controls/plannerd.py selfdrive/controls/

* expand to 10s

* revert this

* fix linter

* vizualize

* fix long test

* typo

* cleanup

* compiles

* unused

* unused

* bump cereal

* bump cereal

* use model if no uiplanm

* update replay

* update ref commit

* bump cereal to master
2023-02-17 19:58:30 -08:00
Shane Smiskol 2bc2dbfb70 process replay: fix hang 2023-02-15 15:55:08 -08:00
Adeeb Shihadeh 6fa9b555a4 rename carState.radarOffCan to radarUnavailable (#27187)
* rename carState.radarOffCan to radarUnavailable

* bump cereal
2023-02-02 11:38:09 -08:00
Shane Smiskol 4f25bd05f2 process replay: test routes in dashcam (#27118)
* fix routes with dashcam override param

* move to bottom
2023-01-27 13:06:33 -08:00
Adeeb Shihadeh 0eed31f1a6 process replay: show process name in timeout error 2023-01-27 11:27:07 -08:00
Kurt Nistelberger 0756c7f374 make process replay more reliable 2023-01-25 10:46:06 -08:00
Adeeb Shihadeh c408b7e306 UI: show experimental long available to release users (#26910)
* UI: show experimental long available to release users

* more description

* cleanup

* hide toggle

* fix process replay

* clear param

* actually fix
2023-01-19 23:06:18 -08:00
Kurt Nistelberger 9131da9103 Revert "Add low Gnss laikad test (#26987)"
This reverts commit b7ce77b3aa.
2023-01-19 11:46:22 -08:00
Kurt Nistelberger b7ce77b3aa Add low Gnss laikad test (#26987)
* add low gnss test

* enable laikad

* temp fix for cache

* update replay

* save LaikadEphemeris for testing

* rem

* update refs

* add comment

Co-authored-by: Kurt Nistelberger <kurt.nistelberger@gmail.com>
2023-01-18 22:29:12 -08:00
Harald Schäfer 4c33d94d3f Revert locationd to gpsLocation (#26963)
* revert locationd laika change

* switch msg

* change proc replay

* back to gpslocation

* update ref

* no np floats in msg

* Ignore empty laikad messages in mapos

* more tolerance

* c++ doesnt work like that
2023-01-14 16:56:14 -08:00
Kurt Nistelberger 88423e25df Laikad: set active (#26850)
* laikad update, renaming

* update locationd

* fix naming

* address PR comments

* upsi

* .

* draft to fix replay

* fix process relay to allow no response for messages

* final fix for process replay

* .

* bump cereal

* update process replay ref commit

* reduce wait time

* .

* last ref change

* move laikad helpers to laika

* .

* fix ublox test

* update refs

* add proper qcom replay support

* fix gnss support if both is available

* update refs

* remove left over

* revert laikad msg

* move laika back to master

* init

* fix gps valid flag

* change time

* add gnss to ignore

* remove gps_valid flag

* .

* adopt orientation reset threshold

* .

* update laikad

* .

* fix stanstill KF resets

* test orienation reset count

* update laika

* bump cereal

* fix process replay

* update laika repo

* remove handle gps

* add extra logging for cache

* .

* add more log

* .

* .

* update laika

* dont remove gps code

* inc min satellite count

* update magic vals and add acc drop

* update laika

* upsi

* rem

* bump laika

* use nav and correct

* more fixes

* use sftp

* No more glonass

* Revert "No more glonass"

This reverts commit a76124da50a1e25f423ad1137c7a046e1d57811d.

* nump laika

* back support old ephemeris cache

* add health to ephemeris message

* bump laika

* remove print

* fix laikad tests

* clean

* remove extra log

* bump laika

* inc timeout for plotjuggler build

* rem cache clear

* .

* enable gps after checks

Co-authored-by: Kurt Nistelberger <kurt.nistelberger@gmail.com>
Co-authored-by: Bruce Wayne <harald.the.engineer@gmail.com>
Co-authored-by: Shane Smiskol <shane@smiskol.com>
2023-01-13 11:37:38 -08:00
Shane Smiskol b00bc4f57e Car interface: require fingerprint and FW versions to get params (#26932)
* require fingerprint and FW versions

* add get_non_essential_params()

* comment

* all required

* classmethod, need to allow subclasses to override _get_params

* fix that

* fix

* fix

* wrong fix 🤦
2023-01-12 12:25:24 -08:00
Adeeb Shihadeh f14deae475 Revert "Car interface: require fingerprint and FW versions to get params (#26766)"
This reverts commit b68dabb689.
2023-01-12 11:28:04 -08:00
Shane Smiskol b68dabb689 Car interface: require fingerprint and FW versions to get params (#26766)
* require fingerprint and FW versions

* add get_non_essential_params()

* comment

* all required

* classmethod, need to allow subclasses to override _get_params

* fix that

* fix
2023-01-11 23:44:50 -08:00
Kurt Nistelberger 981532f0c3 Laikad preperation (#26800)
* laikad update, renaming

* update locationd

* address PR comments

* draft to fix replay

* fix process relay to allow no response for messages

* bump cereal

* update process replay ref commit

* move laikad helpers to laika

* fix ublox test

* update refs

* add proper qcom replay support

* fix gnss support if both is available

* update refs

* move laika back to master

* move cereal back to master

Co-authored-by: Kurt Nistelberger <kurt.nistelberger@gmail.com>
2022-12-29 16:56:35 -08:00
Shane Smiskol c2326a4219 process replay: support experimental long (#26233)
Support experimental long
2022-10-27 16:44:22 -07:00
Adeeb Shihadeh 01d05f66fe auto-detect pigeon or quectel (#25991)
* auto-detect pigeon or quectel

* persistent

* fix sim

* fix process replay

* fix locationd unit tests

* fix that

Co-authored-by: Comma Device <device@comma.ai>
2022-10-06 16:46:15 -07:00
Adeeb Shihadeh 80c40048cb sensord: test fixups (#25935)
* sensord: test fixups

* fix linter

* check freq

* fp seems reliable

* clean that up

* update refs

Co-authored-by: Comma Device <device@comma.ai>
Co-authored-by: Bruce Wayne <batman@z840-provisioning.internal>
2022-09-30 12:25:11 -07:00
Kurt Nistelberger 29d3ed2ce6 Sensor events splitup (#25714)
* PoC of reading sensors via interrupts instead of polling

* add Gyro and draft for magn

* add more functionality to gpio.cc

* change LSM gyro to interrupt

* resolve rebase conflict

* update BMX accel interrupt impl

* add interrupt collector thread to fetch in parallel

* change get_event interface to return true on successful read

* update BMX gyro interrupt impl

* update gpio.h/.cc according to comments

* address comments, rename Edgetype enum

* Edgetype to EdgeType

* update sensor interrupt interface

* add error handling, and read fd on trigger

* avoid sending empty messages

* fix build

* use gpiochip

* less diff

* gpiochip on both edges, but skip falling edge if rising edge is detected

* init last_ts with 0

* update sensord testcases

* update sensord testsweet

* test for pipeline

* readd with_process

* add null check

* move tests update to seperate PR

* sensord: improve test coverage (#25683)

* update sensord-interrupt testsweet

* address review comments

* inc stddev threshold

* fix format string

* add version 0 check again

* relax strictness after c3 with bmx tests

* relax strictness after tests

Co-authored-by: Kurt Nistelberger <kurt.nistelberger@gmail.com>

* address PR comments

* fix typo

* remove 4ms limit, and skip first 0.5sec of data

* revert disable_interuppt change to destructor

* fix and remove timing skip

* make gpiochip generic

* sensord port

* change from sensorEvents to separated events

* fix gyro usage

* add splitted sensor tests

* modify debug script sensor_data_to_hist.py

* refactor get_event interface to remove sensorEvent message type

* update locationd to non sensorEvent usage

* tmp commit

* fix replay

* fix accelerometer type

* fix sensor to hist debug script

* update sensord tests to split events

* remove rebase artifacts

* port test_sensord.py

* small clean up

* change cereal to sensorEvents-splitup branch

* upate sensorEvents in regen

* fix route generation for splitted sensor events

* regen cleanUp from sensorEvents change

* .

* remove light and temp from locationd

* add generic init delay per sensor

* .

* update routes

* move bmx gyro/accel to its own channel

* adopt sensor tests to bmx channel

* remove rebase artifacts

* fix sensord test

* handle bmx not present

* add bmx sockets to regen

* .

* .

* code cleanUp

* .

* address PR comments

* address PR comments

* address PR comments

* lsm clean up

* readd sensorEvents

* rever regen.py

* .

* update replay refs

* move channels

* fix artifact

* bump cereal

* update refs

* fix timing issue

Co-authored-by: Bruce Wayne <batman@workstation-eu-intern2.eu.local>
Co-authored-by: gast04 <kurt.nistelberger@gmail.com>
Co-authored-by: Willem Melching <willem.melching@gmail.com>
Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-09-29 14:31:54 -07:00
Vivek Aithal 4fa62f1464 Live torque (#25456)
* wip torqued

* add basic logic

* setup in manager

* check sanity and publish msg

* add first order filter to outputs

* wire up controlsd, and update gains

* rename intercept to offset

* add cloudlog, live values are not updated

* fix bugs, do not reset points for now

* fix crashes

* rename to main

* fix bugs, works offline

* fix float in cereal bug

* add latacc filter

* randomly choose points, approx for iid

* add variable decay

* local param to capnp instead of dict

* verify works in replay

* use torqued output in controlsd

* use in controlsd; use points from past routes

* controlsd bugfix

* filter before updating gains, needs to be replaced

* save all points to ensure smooth transition across routes, revert friction factor to 1.5

* add filters to prevent noisy low-speed data points; improve fit sanity

* add engaged buffer

* revert lat_acc thresh

* use paramsd realtime process config

* make latacc-to-torque generic, and overrideable

* move freq to 4Hz, avoid storing in np.array, don't publish points in the message

* float instead of np

* remove constant while storing pts

* rename slope, offset to lat_accet_factor, offset

* resolve issues

* use camelcase in all capnp params

* use camelcase everywhere

* reduce latacc threshold or sanity, add car_sane todo, save points properly

* add and check tag

* write param to disk at end of route

* remove args

* rebase op, cereal

* save on exit

* restore default handler

* cpu usage check

* add to process replay

* handle reset better, reduce unnecessary computation

* always publish raw values - useful for debug

* regen routes

* update refs

* checks on cache restore

* check tuning vals too

* clean that up

* reduce cpu usage

* reduce cpu usage by 75%

* cleanup

* optimize further

* handle reset condition better, don't put points in init, use only in corolla

* bump cereal after rebasing

* update refs

* Update common/params.cc

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* remove unnecessary checks

* Update RELEASES.md

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-09-19 15:19:26 -07:00
Adeeb Shihadeh 90a4565eb2 params: make python and c++ API match (#25573)
* params: make python and c++ API match

* few more
2022-08-26 20:46:19 -07:00
Shane Smiskol 87ca42e993 process replay: initialize controlsState with original route (#25461)
* push

* do process replay

* commit

* update refs

* clean up

* clean up controlsd

* clean up controlsd

* Add assert

* debubuggier param name

* can be peristent

* Revert "can be peristent"

This reverts commit 6e6d3f6423c26a202623cef728e038259e9e46cd.

* Update selfdrive/test/process_replay/process_replay.py
2022-08-16 22:02:03 -07:00
Chris Souers 5b1ba4f135 Lateral planner: Don't prompt/initiate lane changes if lateral control is not active (#25436)
* Don't set lane change events if we're not able to steer.

* more specific variable name

* update process replay socket config

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-08-15 11:37:24 -07:00
Gijs Koning 4d7b7483d7 Replay: tolerances per field (#25116)
* tolerances per field in replay

* refactor

* Remove laikad parameters

* Small comment change
2022-07-12 03:36:57 -07:00
Adeeb Shihadeh 2ae52e9b22 process replay: ensure enabled for significant amount of time (#25121)
* process replay: ensure enabled for significant amount of time

* update refs

* 10s is reasonable
2022-07-11 17:20:56 -07:00
Gijs Koning eaf7eb4278 Laikad: Use clocks for faster fetching orbits (#25060)
* Use clocks msg to for first fetch of orbits. Which is sent earlier than ublox msgs

* refactor last_fetch_orbits

* Add comment.
Add test

* increase timeout

* Add clocks to process replay
2022-07-07 04:06:51 -07:00
Gijs Koning 12f8237bfb process replay: Fix setting environment vars (#25015)
Fix setting environments in process replay
2022-07-01 16:50:18 +02:00
Gijs Koning c49f997be5 Add laikadOffline subtest to process replay. (#24995)
* Add subtests to process replay.
Adds laikadOffline subtest

* Update cpp.

* Update ref

* Update ref again

* Update ref again

* update ref

* Fix disabling fetching orbits

* Add proc name to event exception

* update ref

* Update setup_env

* Fix offline test and update refs
2022-06-30 08:39:12 -07:00
Willem Melching 3823f55476 add laikad to process replay (#24889)
* merge

* Fix closing process executor after fetching orbits

* cleanup

* Add ref commit and revert test_processes hack

* Fix

* Fix ref

* Fix test

* Temp

* Temp

* Trying

* Trying

* Cleanup and change test

* add ref commit

* remove print

* fix test getting stuck

* cleanup fetch_orbits

Co-authored-by: Gijs Koning <gijs-koning@live.nl>
2022-06-28 13:43:15 +02:00
Willem Melching de0c12e5af calibrationd: start faster by not waiting for carParams (#24976)
* calibrationd: start faster by not waiting for carParams

* fix process replay

* update ref
2022-06-27 15:34:36 +02:00
ZwX1616 1f2f9ea9c9 fullframe DM model (#24860)
* Revert "put cereal on master"

This reverts commit a8ccd8f838.

* Revert "Revert fullframe DM model (#24812)"

This reverts commit c646eeee0a.

* revert revert cereal

* clip6

* 0.8 is fair

* Fiction compensation should be based on error

* Update refs

* Add deadzone

* not that

* good mg

* ref

* ref

* ee8f

* minor tweak

* ref

* recompile

* ref

* cereal

* match driverstatus

* new ref

* new ref

* pass token through jenkins credentials

* quote

* fix snpe dead weights

* final ref

Co-authored-by: Harald Schafer <harald.the.engineer@gmail.com>
Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2022-06-20 16:24:51 -07:00
Shane Smiskol 5dce006a82 regen: send wideRoadCameraState (#24863)
* fix camera malfunction

* revert

* send in one process to fix frames out of sync

* not used

* revert

* fix and add --no-upload for CI testing

* fingerprint if source is fw

* no FW versions
2022-06-16 22:41:13 -07:00
Shane Smiskol b86ef0b70e regen & process replay: support no disengage on accelerator (#24850)
* ACC on if enabled != 0

* small regen clean up and add HONDA3

* fixes

* revert unneeded changes

* not used

* just alt exp

Co-authored-by: redacid95 <jonathanc.olivier@gmail.com>
2022-06-14 23:30:35 -07:00
Shane Smiskol 170ed3d761 process replay: clean up common code (#24855)
* regen and process replay clean up

* test_fuzzy actually uses fingerprint hardcoding

fix

* revert

* revert

* this can be a url or path so just print full variable
2022-06-14 12:03:30 -07:00
ZwX1616 c646eeee0a Revert fullframe DM model (#24812)
* Revert "fullframe DM: flip RHD yaw to use matching thresholds"

This reverts commit 2ac6931003.

* Revert "fullframe DM model (#24762)"

This reverts commit d6c07a6b15.

* revert cereal
2022-06-10 16:16:46 -07:00
ZwX1616 d6c07a6b15 fullframe DM model (#24762)
* get log

* simplify two nonsense

* not needed

* libyuv is a joke

* clean up

* try small

* fast but not bad

* working

* clean up driverview

* simplified

* thats mirrored

* smol

* tweak

* ref is screen

* w/ ee

* update camera model

* no if TICI

* start

* update pose thresh

* less cpu more dsp

* new libyuv

* new snpe

* add files

* test

* should be fast

* update out len

* trigger test

* use master snpe

* add cereal

* update cereal

* refactor parsing

* missing ;

* get

* wrong type

* test model

* use driver data

* 10829278-72fe-4283-a118-2cef959ce174/1550

* no pf

* adapt driverview

* ;

* rhd learner

* update libyuv buildi x64

* ad4337ea

* remove blink slack

* test

* no

* use toggle

* b16

* fix for nv12

* 5b02cff5 both

* update test

* update cereal

* update cereal

* update cereal

* v2 packets

* revert libyuv

* no /

* update snpemodel

* ;

* memcpy

* fix test

* use toggle in driverview

* update power

* update replay

* Revert "update replay"

This reverts commit 1d0979ca59dbc89bc5890656e9501e83f0556d50.

* update model ref

* halve cpu

* fake 8bit onnx runner

* same thresh as report

* cereal master

Co-authored-by: Comma Device <device@comma.ai>
2022-06-08 20:13:46 -07:00
Willem Melching f49a9c9fd2 less TICI when not needed (#24698)
* less TICI when not needed

* fix process replay

* move reading voltages into hw abstraction layer

* Update selfdrive/hardware/tici/hardware.h

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Update selfdrive/hardware/hw.h

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Update selfdrive/hardware/base.h

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* rename init function

* Update selfdrive/athena/athenad.py

Co-authored-by: Robbe Derks <robbe.derks@gmail.com>

* Update selfdrive/boardd/boardd.cc

* Apply suggestions from code review

* Update selfdrive/thermald/thermald.py

* update ref

* fix alert width if all cameras are bad

* add ecam to test_loggerd

* bump cereal

* bump cereal

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
Co-authored-by: Robbe Derks <robbe.derks@gmail.com>
2022-06-02 15:20:51 +02:00
Gijs Koning b5aed2bf67 Simulator: Speed up process by removing second camera (#24679)
* simulator: support running wide camera only

* proper frame id

* use param name that makes more sense

* do some cleanup

* Update tools/sim/bridge.py

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2022-05-30 15:25:07 +02:00