Commit Graph

1374 Commits

Author SHA1 Message Date
Kacper Rączy 3aa74c28fc CI: add regen job (#30157)
* Ability to whitelist/blacklist cars in regen_all

* Add CI job for regen, running on 2 segments

* Run regen_all, not regen

* Use coverage run

* Add test_regen

* Use test_regen in ci test

* Add test case names

* ONNXCPU = 1

* Add mazda segment

* Use RUN_CL

* build cl image before running

* unset PYTHONWARNINGS

* Create regen cache

* Replace daemons with processes

* Skip ford

* Skip mazda

* Add comment about commented segments

* Update selfdrive/test/process_replay/test_regen.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Remove unset pythonwarnings

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2023-10-05 13:59:00 -07:00
Shane Smiskol a1ca5dc484 scripts: format diff when comparing logs (#30174)
* move format diff to compare_logs to pretty print

* smaller
2023-10-04 22:59:03 -07:00
Kacper Rączy 18f41d1910 regen: skip fetching camera files when not needed (#30169)
* Fetch only camera files which are needed in current setup

* Refactor daemons to processes
2023-10-04 15:00:50 -07:00
Kacper Rączy 898acb13f2 process_replay: migrate panda/peripheral/manager states (#30141)
* Migrate pandaStates

* Fix import

* Make pandaStates migration optional, so process_replay still works on logs without carParams

* peripheralState migration if its not available

* Do nothing if log file has peripheralStates

* Add migrate_managerState

---------

Co-authored-by: Shane Smiskol <shane@smiskol.com>
2023-10-04 13:41:50 -07:00
Kacper Rączy cf3549fe8a process_replay: migrate cameraState timestampSof if its missing (#30153)
Migrate timestampSof if its missing
2023-10-03 19:46:24 -07:00
Harald Schäfer 3c0a7d6986 Night Strike Model (#30148)
* baab2a46-f0c3-46eb-8756-00c55354945b/700

* Update model replay ref
2023-10-03 14:25:30 -07:00
Shane Smiskol 661edd7d8b regen: fix wide road camera msg name (#30147)
fixed
2023-10-02 21:21:49 -07:00
Kacper Rączy 9c5bbe1a91 process_replay: fix support for single cam segments (#30143)
* Fix assertion for vision_pubs

* Get camera states from meta

* Convert deviceType to string

* Use neo instead of eon for frame sizes dict

* update vision_pubs of process config when creating vipc
2023-10-02 19:36:47 -07:00
Adeeb Shihadeh a4aa801fbe simplify sensor alert (#29824)
* simplify

* update proc replay
2023-09-26 16:47:21 -07:00
Adeeb Shihadeh fa06a69101 add extra debug info to onroad test (#30067) 2023-09-26 16:11:26 -07:00
Vivek Aithal 6d5f4f149d locationd: Replace allAliveAndValid with allValid (#30064)
* replace allaliveandvalid check with allvalid, all necessary time checks are already used

* update refs
2023-09-26 15:31:40 -07:00
Justin Newberry 0251742079 CI: use "slow" marker for pytest (#30063)
* skip to mark

* fix quotes
2023-09-26 13:44:18 -07:00
Adeeb Shihadeh 5f318faf93 model replay: add cameraOdometry (#30007)
* model replay: add back cameraOdometry coverage

* update refs

* try again

* fix cmp log
2023-09-22 17:10:36 -07:00
Shane Smiskol 7f073483bb services: capitalize constant service list (#30005)
* rename

* bump

* Apply suggestions from code review
2023-09-21 18:25:15 -07:00
Harald Schäfer 3799fe46b3 Optimus Prime Model 🤖 (#29752)
* b449f295-7e40-4f97-96a6-d858815955e5/700

* make it compile b449f295-7e40-4f97-96a6-d858815955e5/700

* simpler - b449f295-7e40-4f97-96a6-d858815955e5/700

* feature_len changed

* 0a833ab4-9fc7-4e51-a686-9bce3eafcb91/700

* Revert "0a833ab4-9fc7-4e51-a686-9bce3eafcb91/700"

This reverts commit 41708b88447cfcf302b16f21f0caa69450bbd3df.

* 0a833ab4-9fc7-4e51-a686-9bce3eafcb91/700 again

* Update ref

* update ref

---------

Co-authored-by: Yassine <yassine.y10@gmail.com>
2023-09-21 13:51:12 -07:00
Justin Newberry f69a45faba Tests: power_monitoring disable nonblocking writes (#29994)
* better error logging

* fix power monitoring test

* revert that

* properly patch it
2023-09-20 16:51:46 -07:00
Vivek Aithal 00d1c682b0 Alerts: More understandable locationd, paramsd alerts (#29831)
* add locationd paramsd alerts

* remove more additions, just renames

* update refs

* Update selfdrive/controls/lib/events.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Update selfdrive/controls/lib/events.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* resolve comemnts

* update cereal

* update refs

* remove nosensor in new alerts

* bump cereal

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2023-09-11 16:28:42 -07:00
Adeeb Shihadeh d036148550 jenkins speedup (#29869)
* jenkins speedup

* cleanup
2023-09-10 16:47:07 -07:00
Lee Jong Mun 238f9d65db cleanup remnants of LQR and INDI tuning (#29860)
* not use 'indi' cleanup

* bump cereal

* rm more

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2023-09-10 16:30:13 -07:00
Kacper Rączy 1344a93a35 docker: multiarch images (amd64/aarch64) (#29691)
* arch specific tags

* merge multiarch script

* arm64 build and merge jobs

* update tag script

* casual ci test

* change image to 2204

* docker -v test

* arch specific naming

* Remove test_buildjet

* build -arch images only when CURRENT_ARCH_BUILD is set

* support new tags in tag_multiarch

* Toggle CURRENT_ARCH_BUILD on in ci

* Docker common

* Fix condition

* Fix path to docker_common

* Add more stuff to common

* Add build_arm job

* composite job for building

* Run checkout before build composite

* add shell arg

* move timeout to selfdrive_tests

* TARGET ARCHITECTURE var

* Support TARGET_ARCHITECTURE in workflow

* Rewrite to Single build job with matrix

* Remove shebang from docker_common

* Attempt for fix build matrix

* Remove setup arch

* build matrix for docker push

* Use 2vcpu for arm build

* 2vcpu for docker_push too

* temporarly unlock docker_push

* Remove requirement for target arch when pushing

* Unset target architecture in docker_push cl

* fix sha tags

* Rename action to compile-openpilot

* move push_image line to tag_multiarch step

* arch suffix for scons cache

* cache_key_prefix for setup-with-retry

* Re-disable docker_push on non-master

* Add newlines
2023-09-08 14:46:39 -07:00
Mitchell Goff 503fa121ee Rewrite dmonitoringmodeld in python (#29740)
* Added dmonitoringmodeld.py

* Removed dmonitoringmodeld.cc

* Use ModelRunner helpers from runners/__init__.py

* Fixed DriverStateResult field ordering

* Some bug fixes

* Set calib input

* Look ma, no loop!

* Bump dmonitoringmodeld cpu usage in test_onroad

* Fixed memory leak caused by np.ctypes.data_as

* Formatting fixes

* chmod +x

* remove USE_ONNX_MODEL

* Realtime priority 1, formatting fixes
2023-09-07 19:46:43 -07:00
Mitchell Goff 0c0af682a1 Rewrite navmodeld in python (#29579)
* Added navmodeld.py

* Deleted navmodeld.cc

* Write SConscript config flags to config.py

* Remove deleted files from release/files_common

* Some more bug fixes

* Added config.py to gitignore

* Get rid of config.py

* Use set_realtime_priority

* A tiny bit more cleanup

* set realtime priority 1

* Use ModelRunner helper class from runners/__init__.py

* Formatting fixes

* mama mia that's a SPICY memory leak
2023-09-07 14:39:37 -07:00
Justin Newberry 73eda51a11 System: use paths for download folders too (#29818)
* use paths for download folders too

* trailing slash

* reset that
2023-09-07 12:49:03 -07:00
Justin Newberry bfe990b112 System: cleanup paths to use a common class (#29816)
* use OP prefix for logmessage

* cleanup paths too

* cleanup the paths too

* add hw.py to release

* fix those issues

* fix unittests

* fix unittests

* fix unittests

* do swaglog_ipc properly across all the files

* fix that

* fix swaglog in c++

* review suggestions
2023-09-07 11:32:47 -07:00
Kacper Rączy 0267e0500c radard: expose radarTrackId for radar leads (#29809)
* Add radarState track identifier for debugging purposes

* update process replay ref
2023-09-07 11:24:54 -07:00
Vivek Aithal dbada885ac locationd: Add camera-IMU cross-checks (#29727)
* camera-gyro cross checks, but one way

* increase factor to account for gyro noise (potholes, bad roads etc

* increase factor to reduce FP with device taps, bad roads, etc

* factor to 30

* add inputsok to sensoir data invalid alert

* bugfix

* move the sensors check

* add localizer catchall alert

* update refcommit

* remove permanent alert

* revert sensorDataInvalid alert change, split into new PR
2023-09-06 21:01:01 -07:00
Justin Newberry 338288df6e Subaru: don't copy eyesight counters, calculate manually (#29533)
* subaru manual counters

* update ref

* that isn't used

* review suggestions
2023-09-06 16:08:59 -07:00
Adeeb Shihadeh 7b6afbc162 pytest: use a clean environment for all tests (#29788)
* pytest: use a clean environment for all tests

* rm that

* fix pj

* put build back

* fix params

* fix that

* handle no key

* that was removed

---------

Co-authored-by: Justin Newberry <justin@comma.ai>
2023-09-05 18:52:40 -07:00
Justin Newberry ca435d3447 CI: use pytest for unittesting (#29709)
* switch to pytest

* static analysis

* fix the helpers

* static analysis

* remove that

* more parallel test

* cleanup

* static analysis

* durations min and tolerance

* reduce those iterations

* sleep time could be decreased too

* in case it's negative

* try using caching

* remove parallelization for first PR

* PR Cleanup

* reduce diff

* fix that

* bump panda
2023-09-04 11:54:35 -07:00
Justin Newberry 4c039ab022 Tests: disable laika tests in CI (#29742)
skip laika
2023-08-31 17:17:07 -07:00
Justin Newberry 405d09714c Tests: expand temporary directory helpers for more directories (#29735)
* create helpers for testing

* document it

* fix other tests
2023-08-31 14:00:31 -07:00
Mitchell Goff 1eb60516cb Move update_calibration to modeld.py (#29688)
* Move update_calibration to modeld.py

* Updated ref commit
2023-08-30 14:12:52 -07:00
Justin Newberry f31b4ad25e Tests: remove sleep and prints from maneuver tests (#29711)
remove sleep and prints
2023-08-30 00:02:55 -07:00
Justin Newberry 5dc7028f91 Tests: logmessaged reduce global state (#29680)
* reduce global state

* use a decorator here too

* use that one

* use base imports

* ipchandler in setup

* move to common dir

* move to helpers

* wip

* fix the decorator
2023-08-29 13:36:26 -07:00
Kacper Rączy bb14ba4e5e process_replay: clean env vars after run (#29694)
Clean environ vars at the end of replay
2023-08-28 19:44:40 -07:00
Harald Schäfer 55b391509d Revert: Locationd: enable laikad (#29687)
* Revert: Locationd: enable laikad

* update ref
2023-08-28 16:27:11 -07:00
Adeeb Shihadeh d35beffbce CI: use buildjet for process replay (#29277)
* squash for pr

* backoff to 4vcpu

* review suggestions

* simplify

---------

Co-authored-by: Justin Newberry <jnewberry0502@gmail.com>
Co-authored-by: Justin Newberry <justin@comma.ai>
2023-08-25 18:50:21 -07:00
Justin Newberry c46965fc27 CI: use local scons cache instead of building a separate image (#29634)
* use local

* added ci_sim

* revert scons

* dont need these anymore

* same for sim

* fix dockerfiles
2023-08-25 16:06:17 -07:00
Justin Newberry 584239ceb5 CI: add comment about building locally (#29635)
add comment about build
2023-08-25 15:49:07 -07:00
Adeeb Shihadeh 4d6bb32df2 process replay: parallel by default (#29633)
* process replay: parallel by default

* more is helpful
2023-08-25 14:41:43 -07:00
Mitchell Goff 72a3c987c0 Rewrite modeld in python (#29230)
* Added modeld.py (WIP)

* No more VisionIpcBufExtra

* Started work on cython bindings for runmodel

* Got ONNXModel cython bindings mostly working, added ModelFrame bindings

* Got modeld main loop running without model eval

* Move everything into ModelState

* Doesn't crash!

* Moved ModelState into modeld.py

* Added driving_pyx

* Added cython bindings for message generation

* Moved CLContext definition to visionipc.pxd

* *facepalm*

* Move cl_pyx into commonmodel_pyx

* Split out ONNXModel into a subclass of RunModel

* Added snpemodel/thneedmodel bindings

* Removed modeld.cc

* Fixed scons for macOS

* Fixed sconscript

* Added flag for thneedmodel

* paths are now relative to openpilot root dir

* Set cl kernel paths in SConscript

* Set LD_PRELOAD=libthneed.so to fix ioctl interception

* Run from root dir

* A few more fixes

* A few more minor fixes

* Use C update_calibration for now to exactly match refs

* Add nav_instructions input

* Link driving_pyx.pyx with transformations

* Checked python FirstOrderFilter against C++ FirstOrderFilter

* Set process name to fix test_onroad

* Revert changes to onnxmodel.cc

* Fixed bad onnx_runner.py path in onnxmodel.cc

* Import all constants from driving.h

* logging -> cloudlog

* pylint import-error suppressions no longer needed?

* Loop in SConscript

* Added parens

* Bump modeld cpu usage in test_onroad

* Get rid of use_nav

* use config_realtime_process

* error message from ioctl sniffer was messing up pyenv

* cast distance_idx to int

* Removed cloudlog.infos in model.run

* Fixed rebase conflicts

* Clean up driving.pxd/pyx

* Fixed linter error
2023-08-25 14:36:26 -07:00
Adeeb Shihadeh c9e227a9c0 update azure-storage-blob (#29411)
Co-authored-by: Cameron Clough <cameronjclough@gmail.com>
2023-08-25 22:01:10 +01:00
Harald Schäfer f1b8a86464 K1FD: generate kalman gain at init (#29610)
* K1FD: generate kalman gain at init

* Update interfaces.py

* Update interfaces.py

* Update interfaces.py

* Update ref_commit
2023-08-25 13:44:03 -07:00
Justin Newberry 90cb417caa CI: use GITHUB_ENV for env variables (#29616)
export doesn't work across steps
2023-08-24 17:03:25 -07:00
Harald Schäfer 8205590624 Locationd: enable laikad (#29570)
* experiment

* llk back

* update refs

* update ref

* no internet

* Update ref
2023-08-24 16:51:45 -07:00
Justin Newberry 1ee6ed4b16 Linter: remove pylint (#29611)
remove pylint
2023-08-24 16:30:54 -07:00
Justin Newberry 3d9f238bf9 CI: use scripts for docker build (#29599)
* use scripts

* remove common

* do login in the workflow

* separate step

* cleanup

* one script is cleaner

* keep it consistent

* review suggestions

* move in ci too

* PR cleanup
2023-08-24 16:09:08 -07:00
Harald Schäfer 8a7281d285 Laikad upgrades: prep for laikad lld (#29545)
* Laikad upgrades: prep for laikad lld

* Update ref commit
2023-08-23 00:13:10 -07:00
Adeeb Shihadeh e0e139a233 soundd: update cpu usage 2023-08-22 15:16:38 -07:00
Harald Schäfer fda3106c28 Revert laikad llk (#29535)
* Revert "bump laika (#29522)"

This reverts commit 3523bef732.

* Revert "Locationd: switch to laikad (#29278)"

This reverts commit fd3b8a3d75.
2023-08-22 14:12:51 -07:00