Commit Graph

350 Commits

Author SHA1 Message Date
Adeeb Shihadeh 33a85302e0 Fix master-ci dirty working tree + CI test (#25087)
* check

* test for dirty working tree

* swap order

* fix diff
old-commit-hash: df251ef50ebfe5c997e14a03fca4ec932bc4c5cf
2022-07-08 13:00:43 -07:00
Dean Lee e9f2954ee5 loggerd: remove 'enable' from struct LogCameraInfo (#25052)
remove enalbe
old-commit-hash: 568cc0f892d650bd5906b59e1ab169158a7a6bf7
2022-07-07 10:54:56 -07:00
Dean Lee 2156e71e7a move replay from selfdrive/ui/replay to tools/replay (#24971)
* mv to tools/replay

* change folder

* add .gitignore

* fix build doc

* disable warning

* enable warning after build

* build qt/util.cc qt/api.cc to library

* cleanup
old-commit-hash: fd5b3d76036b78864111790931a3abcb1d11ee0f
2022-06-28 16:12:42 +02:00
Dean Lee ae596241d1 FfmpegEncoder: free codec_ctx in encoder_close (#24967)
free context
old-commit-hash: 0aa9ae21d4068d1e8099ff4256d86a204ec31c1c
2022-06-27 11:20:13 +02:00
Dean Lee 1b725b5005 loggerd: remove rotate_lock (#24969)
remove lock
old-commit-hash: 748bbac3444cc30e6dafc2f0bda60b46f3fc8d5a
2022-06-27 11:19:46 +02:00
Dean Lee 16898dee68 logger.cc: remove unused function append_property (#24966)
remove append_property
old-commit-hash: 461f747247add7d4c50344042997427cdb2a90fa
2022-06-27 11:18:56 +02:00
Adeeb Shihadeh d9d83b0225 Move camerad to system/ (#24836)
* mv camerad

* add hardware symlink

* fix unit tests
old-commit-hash: 6123ab3d1c901ed3763e1a7cb8e1aac3f6b8fda3
2022-06-19 14:43:49 -07:00
Adeeb Shihadeh 3c48bc47b7 Move a bunch of stuff to system/ part 3 (#24829)
* move swaglog.py

* timezoned

* logmessaged

* version.py

* fix linter
old-commit-hash: 0fce5d90459b77bf2cfa70f55f322f0e1fb8d01c
2022-06-11 23:19:27 -07:00
Adeeb Shihadeh 9a45a3b99b Move selfdrive/hardware/ to system/ (#24725)
* move hardware to system/

* fix mypy
old-commit-hash: 1139fe507b01f34de9714c99228f411558b44231
2022-06-11 16:38:24 -07:00
Willem Melching 753045c73d less TICI when not needed (#24698)
* less TICI when not needed

* fix process replay

* move reading voltages into hw abstraction layer

* Update selfdrive/hardware/tici/hardware.h

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Update selfdrive/hardware/hw.h

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Update selfdrive/hardware/base.h

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* rename init function

* Update selfdrive/athena/athenad.py

Co-authored-by: Robbe Derks <robbe.derks@gmail.com>

* Update selfdrive/boardd/boardd.cc

* Apply suggestions from code review

* Update selfdrive/thermald/thermald.py

* update ref

* fix alert width if all cameras are bad

* add ecam to test_loggerd

* bump cereal

* bump cereal

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
Co-authored-by: Robbe Derks <robbe.derks@gmail.com>
old-commit-hash: f49a9c9fd2100a4f1401a93d88064ddbbbd935c1
2022-06-02 15:20:51 +02:00
George Hotz 2a6ffaf71d nv12: encoderd avoids a full frame copy (#24519)
* rgb to nv12

* nv12 works (w memcpy)

* correct now

* no copy

* fix nv12 with fast debayer

* reverts of unused stuff

* ui use nv12

* comment out thumbnails for now

* rebase fix

* dm read nv12

* model read nv12

* fix ffmpeg encoder

* thumbnails from nv12

* replay to nv12

* python framereader support nv12

* remove hardcoded frame/buffer sizes

* fix build

* ffmpeg encoder fix buffers

* small cleanup

* reduce power usage test

* fix cpu usage test

* fix snapshot

* fix loggerd test

* bump cereal

Co-authored-by: Comma Device <device@comma.ai>
Co-authored-by: Joost Wooning <jwooning@gmail.com>
old-commit-hash: ea5b8cdfb10686c972d8d2bc1778e6e6567e45a9
2022-06-01 17:18:28 +02:00
Adeeb Shihadeh 3e8625e50b visionipc python shim (#24660)
old-commit-hash: 32a9ccfc37d97e6692435153ac9f0fe72feb1e9f
2022-05-25 20:49:56 -07:00
Adeeb Shihadeh 8456b0e79c uploader: increase max uncompressed qlog size to 100MB
old-commit-hash: eca265e07b512831d473cd1cf6d6b3dea5a03327
2022-05-23 13:43:14 -07:00
Willem Melching f7e2813438 Small fixes for macOS build (#24620)
* Small fixes for macOS build

* remove gpu_libs
old-commit-hash: 1cc652602f008122cf82171f7705522d3e2457dd
2022-05-23 13:46:48 +02:00
George Hotz 109916df80 loggerd: Fix loggerd encode packet drops (#24599)
* hmm, try this

* hmm, try this

* rewrite handle_encoder_msg

* fix new logic

* add comments and an assert

* handle startup condition better

* handle restarts of encoderd

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 0000ad2ac0996075481841221e9bcc842f2812d7
2022-05-20 00:02:50 -07:00
Adeeb Shihadeh 3abe307374 encoder test speedup (#24601)
Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: d87f30ea069a4bbfc20c8caa32e9fdf89a498ed5
2022-05-19 22:33:17 -07:00
Adeeb Shihadeh f4675fc260 Merge common/ and selfdrive/common (#24556)
* Merge common/ and selfdrive/common

* fix that

* fix version

* fix unit tests
old-commit-hash: cb8885cffb313bea258c012c3026461a22bf8135
2022-05-18 14:11:57 -07:00
George Hotz fca47bb6a4 add unixTimestampNanos to encoderd (#24583)
Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: c2279ef39dd7a2cda88ec6949b24ff0f71c19983
2022-05-18 12:58:52 -07:00
George Hotz d0bcf17691 encoderd: fix bytesused (#24553)
fix bytesused

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 3718f9a6e0fdae601b1f90e1f88328609edea885
2022-05-16 16:58:19 -07:00
Adeeb Shihadeh fb1c3b0d5d common: replace custom xattr implementation with os module's (#24448)
* use os module's xattr function

* fix that

* handle in helper
old-commit-hash: 81dacbedcacaf9db43d19700e04e7361c0fbbbcc
2022-05-15 17:13:50 -07:00
Adeeb Shihadeh 6a8fefd5de loggerd: adjust qcamera threshold, seemed to be tuned for comma two
old-commit-hash: 5a58ebf183e1066f5d94163a05d9180a66447409
2022-05-14 01:04:46 -07:00
George Hotz a053d070be encoderd: enable (#24492)
* enable encoderd

* correct enable line

* fix loggerd tests

* fix power draw and cpu tests

* correct cpu for encoderd

* fix a bug, video_writer is shared

* fix issue with not recording dcam

* add recording state

* wooo tests pass. encode id keeps counting

* core 3

* loggerd then encoderd

* stop loggerd first

* core 3 always online

* see the camera when we see encoder packet

* encoderd on small core uses 37%

* remove encoder logic from loggerd

* delete unit test that doesn't really make sense anymore

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 94b9972eb798408a3c412716b8d9eb8ddb9354bc
2022-05-13 17:56:02 -07:00
grekiki 7cd23af9b1 Mypy fixes for --check-untyped-defs (#24372)
more type fixes
old-commit-hash: 17e33978cdd9322918e9d5d388f7fa47aa30e9b7
2022-05-12 14:08:11 +02:00
Willem Melching c7e31ef974 encoderd: update deprecated ffmpeg functions (#24501)
old-commit-hash: 75f21492671ad34949061e57760b360e93e440f9
2022-05-12 12:43:07 +02:00
George Hotz eb9e57e070 encoderd: support remote encoder behind env var (#24490)
* remote encoder

* remote encoder on PC, fix type

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: bd0cc655fbbfdafc0ef081be8558b4c773e76eda
2022-05-10 16:53:47 -07:00
George Hotz 1f4628bad5 encoderd: make work on PC (#24483)
* don't use the codec in video_writer

* this produces broken videos for some reason

* bugfix

* refactor on the class

* works on device

* fix codec

* no codec enum

* fix pc

* move into dirs

* these includes also

* rename it ffmpegencoder

* add avcodec_close

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 2008db47c208f6f0fd661cc45f215bb80aef38d0
2022-05-10 15:20:51 -07:00
Willem Melching badbd3347c uploader: ensure requests.put gets file like object (#24462)
old-commit-hash: ba89faa5f77c76023f49b85888e6d6eb7f2e1a11
2022-05-07 15:42:47 +02:00
George Hotz 210508d3f1 sync qcam p frames (#24459)
Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: ac343433a22984109401a9de97eab28220e38e72
2022-05-06 21:47:58 -07:00
Joost Wooning 08da73721c uploader: size limit for automatic uploading of qlogs/qcams (#24403)
* uploader: size limit for automatic uploading of qlogs/qcams

* move check to add logging

* use constant

* mark as uploaded

Co-authored-by: Willem Melching <willem.melching@gmail.com>
old-commit-hash: df8f024e19de2d7a528b5921f73895f2078cf03a
2022-05-06 11:47:58 +02:00
Adeeb Shihadeh 2e45c3bd68 uploader: compress rlogs (#24447)
old-commit-hash: 66fbce638f1a6a2bc9c08c4a17d5efe47fb91651
2022-05-05 19:13:06 -07:00
Willem Melching be08693d5c delay bz2 compression from logging to uploading (#24392)
* remove log_name

* log without compression

* fix tests

* remove extension for bootlog

* another test fix

* uploader compresses

* also compress in athena

* only compress qlog

* more generic check in do_upload

* fix bootlog compression

* lower loggerd cpu usage

* dont link against bz2

* set core affinity to little cluster

* handle old files
old-commit-hash: 77a6f3d034a7b17d70b750b9ea454310f4b8edf1
2022-05-03 14:09:17 +02:00
George Hotz 7119a98414 loggerd: switch to v4l encoder try 2 (#24380)
* start v4l encoder

* v4l encoder starts

* start and stop

* fill in proper controls

* it dequeued a buffer

* getting bytes

* it made a video

* it does make files

* getting close

* ahh, so that's how dequeue works

* qcam works (no remuxing)

* remuxing works

* we just need to make shutdown and rollover graceful

* graceful destruction

* switch to polling

* should work now

* fix pc build

* refactors, stop properly

* touchups, remove a copy

* add v4l encoder to release

* inlcude file

* move writing to it's own thread

* fix minor memory leak

* block instead of dropping frames

* add counter, fix tests maybe

* better debugging and test print

* print file path in assert

* format string in test

* no more oversized qlogs

* match qcam

* touchups, remove omx encoder

* remove omx include files

* checked ioctl, better debugging, open by name

* unused import

* move linux includes to third_party/linux/include

* simple encoderd

* full packet

* encoderd should be complete

* lagging print

* updates

* name dq thread

* subset idx

* video file writing works

* debug

* potential bugfix

* rotation works

* iframe

* keep writing support

* ci should pass

* loggerd, not encoderd

* remote encoder code

* support remote encoder

* cereal to master, add encoderd

* header no longer required

* put that back there

* realtime

* lower decoder latency

* don't use queue for VisionIpcBufExtra, disable realtime again

* assert all written

* hmm simpler

* only push to to_write if we are writing

* assert timestamp is right

* use at and remove assert

* revert to queue

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 0baa4c3e2ad9ee6f8daba8267db44c2cd44caa62
2022-04-30 09:22:52 -07:00
George Hotz b872fe92ab loggerd: Separate writer from encoder (#24257)
* seperate writer from encoder

* video writer in release

* confirm loggerd on PC works, add YUV option to compressed_vipc

* make raw_logger use video_writer

* put this back to master

* close codec

* put this back

* use unique_ptr

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 8d2465578707022ca4a54defe171103955026a65
2022-04-20 14:51:24 -07:00
George Hotz d73af8d298 C2: remove some dead code (#24256)
* remove some dead code

* still needed

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 1795a2ac03d2707deeb6f17b4b5eb287284b9cef
2022-04-18 20:38:03 -07:00
George Hotz 6dde52671b loggerd: prereqs for deanlee loggerd omx encoder (try 2) (#24252)
* refactor encoders

* fix pc build

* buf_info

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 0ac35a656acad2ecb2a394ddce1358cb41dfb4ab
2022-04-18 18:21:46 -07:00
Adeeb Shihadeh 5a9644261f remove comma two support (#24248)
* remove comma two support

* cleanup release files

* little more

* more libs

* no more gralloc

* add snpe back
old-commit-hash: 5c48e7bc86a621dd35f4ff25da7a26ee506bac76
2022-04-18 17:55:23 -07:00
Comma Device d9d3f9a8ee encodeData: add flags support
old-commit-hash: df7275f1f77a98c66fcb16c287bafc5c4326328b
2022-04-16 15:54:22 -07:00
George Hotz 751615c99c loggerd: add qcam broadcast support, don't recreate pubmaster (#24226)
* add qcam broadcast support, don't recreate pubmaster

* fixed

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 35e776d2fd50f62af46ccfc598066fa75ef821e1
2022-04-16 14:42:36 -07:00
George Hotz 9720bb37b5 loggerd: broadcast encoder data (#24177)
* encode data try 1

* fix pc build

* low quality compressed streamer with pyav

* nvidia streamer

* timestamp support

* fix latency measurement

* refactor

* camerastream updates

* fix linter

* compressed to vipc, clean

* remove print

* touchups

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: dab978083b7014faa2aef2f43830e87323188578
2022-04-12 12:08:50 -07:00
George Hotz 0681a415e9 broadcast encode idx packets (#24176)
Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: cc8430b66866ba127ac0d753cee9168240d23102
2022-04-10 15:12:58 -07:00
Willem Melching 85aa5e2706 uploader: azure returns 401 for files already uploaded (#24141)
old-commit-hash: e7f23ceca252fb4c62819b29c38eaaa62d66ce39
2022-04-05 20:09:05 -07:00
Adeeb Shihadeh bd71c594a2 pylint: enforce indentation (W0311) (#24039)
* pylint: enforce indentation (W0311)

* few more
old-commit-hash: 8af20af66ddaa6bc06d7f72f9134aa9afeed3ed8
2022-03-24 23:23:29 -07:00
Adeeb Shihadeh e195b7de1e deleter: handle files (#24023)
old-commit-hash: 53909c175756fb0ea8e374ca01a41ecfb3d23466
2022-03-23 11:31:29 -07:00
Willem Melching 2eded18267 C3: detect metered Android hotspot (#23734)
* C3: detect metered networks

* show in ui

* fix text layout

* bump cereal

* revert ui changes

* set networkMetered

* add athena method

* add metered logging to uploader

* use in athena uploader

* remove param

* use networkmanager properties to set cell to unmetered

* fix indentation

* no need to check

* bump cereal

* review

* bump cereal
old-commit-hash: da5a0c41a00e4de076c2846ed54049c3dff3bc5a
2022-03-09 11:36:52 +01:00
Adeeb Shihadeh ef2106ebd2 move lock clearing to uploader (#23826)
* move lock clearing to uploader

* test case
old-commit-hash: 3243d1a81e5cfdf687ee406d92903ee44fe781d3
2022-02-22 22:30:11 -08:00
Willem Melching 5517800a59 omx encoder: move writing into thread (#23771)
* omx encoder: move writing to separate thread

* fix include

* pop

* log buffers sizes

* split copy and write
old-commit-hash: 2f00271ce611f8df3ed44e3a1b87b69327a5bc08
2022-02-22 13:02:50 +01:00
Willem Melching d4f372cfd7 Update raw_logger/framereader to new new ffmpeg api (#23799)
* Update raw_logger/framereader to new new ffmpeg api

* Can be removed now

* Handled by avcodec_free_context

* handle EAGAIN and clean up
old-commit-hash: 724bab8ef3aaad2c69b53fafc68f7afa0ebbe659
2022-02-21 16:39:32 +01:00
Willem Melching a7fa889605 improve logging around uploading (#23750)
* improve logging around uploading

* use int

* use raw

* duplicate
old-commit-hash: 7765bc216677026d7b8919525eff310716fd3e7d
2022-02-11 12:58:41 +01:00
Shane Smiskol 85b57ccbfc scons: fix duplicate environments with test flag (#23702)
* fix duplicate environments when built with --test

* lib

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: e5005b8f44a2eedeb88348d9f476b0f4d09db3f0
2022-02-06 14:08:02 -08:00
Adeeb Shihadeh 1c4b6e38e9 loggerd: less error logging
old-commit-hash: 652ef27a7270ad770d8331347cffb2e3a4c07b45
2022-01-24 21:15:14 -08:00