Commit Graph

328 Commits

Author SHA1 Message Date
grekiki 7cd23af9b1 Mypy fixes for --check-untyped-defs (#24372)
more type fixes
old-commit-hash: 17e33978cdd9322918e9d5d388f7fa47aa30e9b7
2022-05-12 14:08:11 +02:00
Willem Melching c7e31ef974 encoderd: update deprecated ffmpeg functions (#24501)
old-commit-hash: 75f21492671ad34949061e57760b360e93e440f9
2022-05-12 12:43:07 +02:00
George Hotz eb9e57e070 encoderd: support remote encoder behind env var (#24490)
* remote encoder

* remote encoder on PC, fix type

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: bd0cc655fbbfdafc0ef081be8558b4c773e76eda
2022-05-10 16:53:47 -07:00
George Hotz 1f4628bad5 encoderd: make work on PC (#24483)
* don't use the codec in video_writer

* this produces broken videos for some reason

* bugfix

* refactor on the class

* works on device

* fix codec

* no codec enum

* fix pc

* move into dirs

* these includes also

* rename it ffmpegencoder

* add avcodec_close

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 2008db47c208f6f0fd661cc45f215bb80aef38d0
2022-05-10 15:20:51 -07:00
Willem Melching badbd3347c uploader: ensure requests.put gets file like object (#24462)
old-commit-hash: ba89faa5f77c76023f49b85888e6d6eb7f2e1a11
2022-05-07 15:42:47 +02:00
George Hotz 210508d3f1 sync qcam p frames (#24459)
Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: ac343433a22984109401a9de97eab28220e38e72
2022-05-06 21:47:58 -07:00
Joost Wooning 08da73721c uploader: size limit for automatic uploading of qlogs/qcams (#24403)
* uploader: size limit for automatic uploading of qlogs/qcams

* move check to add logging

* use constant

* mark as uploaded

Co-authored-by: Willem Melching <willem.melching@gmail.com>
old-commit-hash: df8f024e19de2d7a528b5921f73895f2078cf03a
2022-05-06 11:47:58 +02:00
Adeeb Shihadeh 2e45c3bd68 uploader: compress rlogs (#24447)
old-commit-hash: 66fbce638f1a6a2bc9c08c4a17d5efe47fb91651
2022-05-05 19:13:06 -07:00
Willem Melching be08693d5c delay bz2 compression from logging to uploading (#24392)
* remove log_name

* log without compression

* fix tests

* remove extension for bootlog

* another test fix

* uploader compresses

* also compress in athena

* only compress qlog

* more generic check in do_upload

* fix bootlog compression

* lower loggerd cpu usage

* dont link against bz2

* set core affinity to little cluster

* handle old files
old-commit-hash: 77a6f3d034a7b17d70b750b9ea454310f4b8edf1
2022-05-03 14:09:17 +02:00
George Hotz 7119a98414 loggerd: switch to v4l encoder try 2 (#24380)
* start v4l encoder

* v4l encoder starts

* start and stop

* fill in proper controls

* it dequeued a buffer

* getting bytes

* it made a video

* it does make files

* getting close

* ahh, so that's how dequeue works

* qcam works (no remuxing)

* remuxing works

* we just need to make shutdown and rollover graceful

* graceful destruction

* switch to polling

* should work now

* fix pc build

* refactors, stop properly

* touchups, remove a copy

* add v4l encoder to release

* inlcude file

* move writing to it's own thread

* fix minor memory leak

* block instead of dropping frames

* add counter, fix tests maybe

* better debugging and test print

* print file path in assert

* format string in test

* no more oversized qlogs

* match qcam

* touchups, remove omx encoder

* remove omx include files

* checked ioctl, better debugging, open by name

* unused import

* move linux includes to third_party/linux/include

* simple encoderd

* full packet

* encoderd should be complete

* lagging print

* updates

* name dq thread

* subset idx

* video file writing works

* debug

* potential bugfix

* rotation works

* iframe

* keep writing support

* ci should pass

* loggerd, not encoderd

* remote encoder code

* support remote encoder

* cereal to master, add encoderd

* header no longer required

* put that back there

* realtime

* lower decoder latency

* don't use queue for VisionIpcBufExtra, disable realtime again

* assert all written

* hmm simpler

* only push to to_write if we are writing

* assert timestamp is right

* use at and remove assert

* revert to queue

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 0baa4c3e2ad9ee6f8daba8267db44c2cd44caa62
2022-04-30 09:22:52 -07:00
George Hotz b872fe92ab loggerd: Separate writer from encoder (#24257)
* seperate writer from encoder

* video writer in release

* confirm loggerd on PC works, add YUV option to compressed_vipc

* make raw_logger use video_writer

* put this back to master

* close codec

* put this back

* use unique_ptr

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 8d2465578707022ca4a54defe171103955026a65
2022-04-20 14:51:24 -07:00
George Hotz d73af8d298 C2: remove some dead code (#24256)
* remove some dead code

* still needed

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 1795a2ac03d2707deeb6f17b4b5eb287284b9cef
2022-04-18 20:38:03 -07:00
George Hotz 6dde52671b loggerd: prereqs for deanlee loggerd omx encoder (try 2) (#24252)
* refactor encoders

* fix pc build

* buf_info

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 0ac35a656acad2ecb2a394ddce1358cb41dfb4ab
2022-04-18 18:21:46 -07:00
Adeeb Shihadeh 5a9644261f remove comma two support (#24248)
* remove comma two support

* cleanup release files

* little more

* more libs

* no more gralloc

* add snpe back
old-commit-hash: 5c48e7bc86a621dd35f4ff25da7a26ee506bac76
2022-04-18 17:55:23 -07:00
Comma Device d9d3f9a8ee encodeData: add flags support
old-commit-hash: df7275f1f77a98c66fcb16c287bafc5c4326328b
2022-04-16 15:54:22 -07:00
George Hotz 751615c99c loggerd: add qcam broadcast support, don't recreate pubmaster (#24226)
* add qcam broadcast support, don't recreate pubmaster

* fixed

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 35e776d2fd50f62af46ccfc598066fa75ef821e1
2022-04-16 14:42:36 -07:00
George Hotz 9720bb37b5 loggerd: broadcast encoder data (#24177)
* encode data try 1

* fix pc build

* low quality compressed streamer with pyav

* nvidia streamer

* timestamp support

* fix latency measurement

* refactor

* camerastream updates

* fix linter

* compressed to vipc, clean

* remove print

* touchups

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: dab978083b7014faa2aef2f43830e87323188578
2022-04-12 12:08:50 -07:00
George Hotz 0681a415e9 broadcast encode idx packets (#24176)
Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: cc8430b66866ba127ac0d753cee9168240d23102
2022-04-10 15:12:58 -07:00
Willem Melching 85aa5e2706 uploader: azure returns 401 for files already uploaded (#24141)
old-commit-hash: e7f23ceca252fb4c62819b29c38eaaa62d66ce39
2022-04-05 20:09:05 -07:00
Adeeb Shihadeh bd71c594a2 pylint: enforce indentation (W0311) (#24039)
* pylint: enforce indentation (W0311)

* few more
old-commit-hash: 8af20af66ddaa6bc06d7f72f9134aa9afeed3ed8
2022-03-24 23:23:29 -07:00
Adeeb Shihadeh e195b7de1e deleter: handle files (#24023)
old-commit-hash: 53909c175756fb0ea8e374ca01a41ecfb3d23466
2022-03-23 11:31:29 -07:00
Willem Melching 2eded18267 C3: detect metered Android hotspot (#23734)
* C3: detect metered networks

* show in ui

* fix text layout

* bump cereal

* revert ui changes

* set networkMetered

* add athena method

* add metered logging to uploader

* use in athena uploader

* remove param

* use networkmanager properties to set cell to unmetered

* fix indentation

* no need to check

* bump cereal

* review

* bump cereal
old-commit-hash: da5a0c41a00e4de076c2846ed54049c3dff3bc5a
2022-03-09 11:36:52 +01:00
Adeeb Shihadeh ef2106ebd2 move lock clearing to uploader (#23826)
* move lock clearing to uploader

* test case
old-commit-hash: 3243d1a81e5cfdf687ee406d92903ee44fe781d3
2022-02-22 22:30:11 -08:00
Willem Melching 5517800a59 omx encoder: move writing into thread (#23771)
* omx encoder: move writing to separate thread

* fix include

* pop

* log buffers sizes

* split copy and write
old-commit-hash: 2f00271ce611f8df3ed44e3a1b87b69327a5bc08
2022-02-22 13:02:50 +01:00
Willem Melching d4f372cfd7 Update raw_logger/framereader to new new ffmpeg api (#23799)
* Update raw_logger/framereader to new new ffmpeg api

* Can be removed now

* Handled by avcodec_free_context

* handle EAGAIN and clean up
old-commit-hash: 724bab8ef3aaad2c69b53fafc68f7afa0ebbe659
2022-02-21 16:39:32 +01:00
Willem Melching a7fa889605 improve logging around uploading (#23750)
* improve logging around uploading

* use int

* use raw

* duplicate
old-commit-hash: 7765bc216677026d7b8919525eff310716fd3e7d
2022-02-11 12:58:41 +01:00
Shane Smiskol 85b57ccbfc scons: fix duplicate environments with test flag (#23702)
* fix duplicate environments when built with --test

* lib

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: e5005b8f44a2eedeb88348d9f476b0f4d09db3f0
2022-02-06 14:08:02 -08:00
Adeeb Shihadeh 1c4b6e38e9 loggerd: less error logging
old-commit-hash: 652ef27a7270ad770d8331347cffb2e3a4c07b45
2022-01-24 21:15:14 -08:00
Dean Lee 5752972827 RawLogger: downscale qcamera before encoding (#23607)
old-commit-hash: a4fefdcb945656fba180825e01f8ea7f38e066a6
2022-01-24 13:41:44 -08:00
Ryan 0dbb4fb3b0 Add more type hinting (#23595)
* Add more type hinting.

* Revert joystick_alert changes.

* Add typing to statsd.

* Update selfdrive/statsd.py

* Update selfdrive/test/test_fingerprints.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: aa9e635311c4149b196f0c03afcb89d24106a62b
2022-01-21 15:11:16 -08:00
Willem Melching 962201fff1 Ensure controlsd can engage again in process replay (#23556)
* hacks to make process replay engage again

* dont change that

* enable engagement check again

* chmod +x

* first working regen

* proper logMonoTime

* fix video framerate consistency

* mpc is valid again

* proper alive checks

* revert loggerd change

* ensure engaged in regen, fix managerState, peripheralState

* ubloxRaw is unused

* add submaster config for gm

* regen all services we can run without HW

* fix loggerd

* loggerd: matroska without the extension

* update ref
old-commit-hash: d3d10e014a69f77d3f67c8d01100b03fde8ab034
2022-01-21 13:13:34 +01:00
Dean Lee 75a22388c7 loggerd: make test_loggerd.py run on PC (#23574)
* run on PC

* set frameId

* remove todo

* rawlogger: remove suffix mkv

* remove unused import PC

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: 22d43211d9bc828ea4e6bd6b84d865a597825eda
2022-01-19 13:55:02 -08:00
Robbe Derks 323febbf37 Statsd (#23413)
* device side of statsd

* need to start it

* enable in manager

* add sleep

* cleanup

* remove aggregates for now and standardize on industry terms

* manager needs main

* need to have a try/except

* atomic_write_on_fs_tmp does not work

* cleaner

* use dump

Co-authored-by: Willem Melching <willem.melching@gmail.com>

* one file at a time

* limit amount of files

* move to influx line protocol and cleanup

* needs to be a list

* fix timezone bug

* actually rate limit

* add to release

* normalized origin

* also log deviceType

* more stats

Co-authored-by: Willem Melching <willem.melching@gmail.com>
old-commit-hash: 1b49ce6ec401966b0c56f62ea2b71209e9d4075b
2022-01-10 15:21:48 +01:00
Ewout ter Hoeven f206ebd054 Pyupgrade 3.6: Update syntax with Python 3.6+ features (#23305)
Updated Python code with Python 3.6+ features:
- utf-8 encoding is now the default (PEP 3120)
- Replace list comprehensions by Generator Expressions (PEP 289)
- Replace yield loop by yield from (PEP 380)
- Remove the (object) subclass when defining a class
- Replace the IOError alias by OSError (PEP 3151)
- Define sets with curly braces {} instead of set()
- Remove "r" parameter from open function, which is default

Co-Authored-By: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-Authored-By: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>

Co-authored-by: Adeeb Shihadeh <8762862+adeebshihadeh@users.noreply.github.com>
Co-authored-by: GregorKikelj <96022003+GregorKikelj@users.noreply.github.com>
old-commit-hash: 332f568a8241fba9459cb70c76840b9670e6993a
2021-12-24 11:18:39 -08:00
Ewout ter Hoeven a962365292 Convert format strings strings to f-strings (#23241)
* Convert all text strings to f-strings

Reformats all the text from the old "%-formatted" and .format(...) format to the newer f-string format, as defined in PEP 498. This requires Python 3.6+.

Flynt 0.69 was used to reformat the strings. 120 f-strings were created in 51 files.

F-strings are in general more readable, concise and performant. See also: https://www.python.org/dev/peps/pep-0498/#rationale

* revert pyextra changes

* revert ublox.py

Co-authored-by: Willem Melching <willem.melching@gmail.com>
old-commit-hash: 55390d273f36f49da3896e687ac5530f40c1d150
2021-12-16 14:58:17 +01:00
Dean Lee 01eb405677 put logcat in bootlog (#23217)
* put logcat in boot log

* continue

* use std::unique_ptr

* Revert "use std::unique_ptr"

This reverts commit c671f8ca7160691791bade508c4a8eb5cdb73fb8.
old-commit-hash: f042962933837dbe612d4eb0f9bf5c0a0e0fb6f3
2021-12-14 11:54:54 -08:00
Dean Lee 3e72a81dac Use std::unique_ptr to manage memory (#23168)
* use unique_ptr

* trigger ci

* move down loggerdstate
old-commit-hash: 4aa291c406752169f4d836e952708eeccbb36c5d
2021-12-13 14:16:53 +01:00
Dean Lee a88eb06773 RawLogger: fix wrong pts (#23205)
old-commit-hash: cfad1f35e8a763a7ae15244966c925c751e684b9
2021-12-13 11:46:52 +01:00
Dean Lee dd7c226097 util: move all functions into util namespace (#23203)
old-commit-hash: 9decd3d8a20167de684322a458e28c8d1f746f83
2021-12-12 14:42:23 -08:00
Adeeb Shihadeh 916c2ed588 loggerd: fix delayed exit from main thread
old-commit-hash: 5f36b4819970e11d4ffc2757f1d6a990b00881f7
2021-12-07 18:22:00 -08:00
Willem Melching e7bf88f29f Remove Raw Upload toggle (#23143)
* Remove Raw Upload toggle

* fix tests
old-commit-hash: 2e0c73fc0cbac64e442085390dd22e6b408d0478
2021-12-07 11:25:40 +01:00
Adeeb Shihadeh 1ca1d7a673 increase loggerd per-cycle service limit to 200
old-commit-hash: f71a03f967a283991442eed69d1fa9857a54e463
2021-12-06 22:08:17 -08:00
Adeeb Shihadeh f8146db9b9 loggerd: don't let a single service clog up the main thread
old-commit-hash: baf1a77ca293add06ed0407425a6216cff163bae
2021-12-06 19:31:11 -08:00
Dean Lee 1473ec97da LogCameraInfo: remove unused variables (#23142)
old-commit-hash: ddc612e9f621f45e232af9e1a2329560bc4db418
2021-12-06 14:25:28 +01:00
Dean Lee 3ba7f99688 move struct LogCameraInfo into loggerd.h (#23129)
old-commit-hash: 480fb54926392b760944f5fafe7b4c96abee29ff
2021-12-04 21:08:55 -08:00
grekiki 9e4cc1f7ff Python: Replace more lists with generators (#23116)
* Replace lists with generators v2

* Replace set with {}

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Replace more set() with {}

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: 6951b3271d1770297cb6bd53878463035ec0bf43
2021-12-03 22:57:19 -08:00
Dean Lee 9b0fcaeb84 rename yuv streams (#23071)
* rename yuv streams

* bump cereal

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: 348d2d2b0d169b0d181a51eb0e23b1f9b8fc6793
2021-11-29 18:14:59 -08:00
Willem Melching 28dd415759 Do not run code on version.py import (#23063)
* Do not run code on version.py import

* fix athena
old-commit-hash: f7c46c694985456e569ce2313b29e43607c62a12
2021-11-29 19:38:55 +01:00
Willem Melching 3b8daea061 Clear loggerd locks once on boot (#23060)
* add test for clear_locks

* move to bootlog

* simplify test
old-commit-hash: a2f32fd3e18455ca2750f4a0a940ced2ca933163
2021-11-29 12:30:28 +01:00
Dean Lee 7f0beef092 loggerd: add test case for trigger_rotate (#23038)
* test rotate

* remove global LoggerdState
old-commit-hash: 1d323e0fd6e6050180bdf1b0a9793a4f52c47e9a
2021-11-26 14:53:09 +01:00