Commit Graph

98 Commits

Author SHA1 Message Date
Dean Lee fb8ba34f31 replay: add qcam and YUV flags (#22788)
* common flags

* cleanup

* remove double semicolon

* camera

* apply reviews
2021-11-09 14:08:24 -08:00
Dean Lee ff33ca3413 util: remove function to_hex (#22792) 2021-11-04 16:18:01 +01:00
Mayfield 5246f0231e enable wshadow (#22756)
* enable wshadow (#22714)

* fix replay

* more build fixes

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2021-11-02 17:08:53 +01:00
Dean Lee 79651b6c49 replay: fix qcam download error (#22777)
* fix qcam download error

* test loading qlog/qcamera
2021-11-02 11:24:59 +01:00
Dean Lee 09858f7f6f replay: print colored text (#22771) 2021-11-02 11:24:45 +01:00
Dean Lee 0a5ca71603 replay: fix frame lag when replaying to the next segment (#22773) 2021-11-02 11:24:12 +01:00
Willem Melching f8d050298e framereader.cc: better avformat_open_input error printing 2021-11-02 09:56:49 +01:00
Dean Lee 6f78cc77fc replay: fix hanging on exit if stream thread has not been started yet (#22770) 2021-11-01 16:06:39 -07:00
Dean Lee 2b4a477fbc replay: improve segment download and merge (#22654)
* no-cache mode

* fix test cases build error

* space

* don't create cache dir in no-cache mode

* fix errors in test cases

* no_local_cache_

* set the number of connections by chunk_size

* use size_t instead of int64_t

* add test case for no-cache mode

* rename variables

* fix SIGSEGV

* cleanup

* faster decompressBZ2

* always decompress bz2

* add test cases

* prepare for python interface

* fix test cases build error

* continue

* camera_replay: cache remote file

* protected inheritance

* single option name

* TODO

* test_case for LogReader&FrameReader

* fix wrong require

* test case for FileReader

* cleanup test

* test:fix wrong filename

* check cached file's checksum

* fix mkdir permissions err

cleanup filereader

* remove initialize libav network libraries.

dd

* abort all loading if one failed

* cleanup tests

* use threadpool to limit concurrent downloads

* cache more segments

* merge 3 segments for replay

* one segment uses about 100M of memory

* use segments_need_merge.size()

* shutdown

* fix stuck if exit replay  before keyboard thread started

* load one segment at a time

* small cleanup

* cleanup filereader

* space

* tiny cleanup

* merge master

* cleanup test cases

* use util:create_directories

* cleanup framereader
2021-11-01 11:55:56 +01:00
Dean Lee 41e5e76596 replay: fixed the alignment issue when replaying from C3 qcamera (#22734)
* fix qcamera replay

* fix alignment in replay

* upscale yuv if needed

* remove space
2021-11-01 11:06:00 +01:00
Dean Lee d70e49dca5 replay: build on devices (#22632)
* there is no QThread::create on device

* add files to files_common

* modify SConscript

* no memory_resouce on device

* #define HAS_MEMORY_RESOURCE

* fix hangs on shutdown

* build on device with the --extra flag

* delete later

* setup -> extras

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-10-29 12:21:57 +02:00
Dean Lee 921600b95c framereader: use swscale to handle the YUV conversion (#22710) 2021-10-29 12:14:14 +02:00
Dean Lee af7dba9250 replay: use an enum for flags (#22635)
* enum flags

* continue

* remove yuv flag for now
2021-10-22 23:28:53 -07:00
Dean Lee 317deeae84 replay: improve seeking, queuing segments (#22625)
* improve seeking, queuing segments

* cleanup,init current_segment_ to 0

* use isSegmentLoaded

* remove to isSegmentMerged
2021-10-19 22:23:53 -07:00
Dean Lee 702fc5850c replay: fix replay looping (#22618) 2021-10-19 12:59:06 -07:00
Dean Lee 107ed35d1e replay: parse route string, replay from the specified segment (#22503)
* parse segment name

* return 0

* update regexp

* with |

* dd

* use struct

* move to class Route

* refactor loadFromLoacal

* simplify loadFromJson

* cleanup load from server

* const ref
2021-10-19 14:45:07 +02:00
Dean Lee 4678903a61 replay/camera: publish each camera in a separate thread (#22591)
* publish frames in a seperate thread for each camera

* cleanup

* cleanup

* prefetch next frame

* cleanup
2021-10-19 14:39:47 +02:00
Willem Melching e5e90aeae5 replay: improve download prints 2021-10-19 12:48:38 +02:00
Dean Lee 8d1d0c2cd7 replay: improve segment downloading (#22504)
* improve segment downloading

dd

* continue

* log retrying

* check aborting_ in loop

* std::endl

* log download information

* cleanup

* continue

* dd

* move download to seperate funciton

* simplify logging

* continue

* guard ts

* cleanup

* typo
2021-10-19 12:33:12 +02:00
Dean Lee 0189a19b8e replay: refactor FrameReader (#22438)
* decode from the previous keyframe after seek

* less memory

* some stream seems to contian no keyframes

* test random seek

* merge master

* continue

update test_cases

use fr

* merge master
2021-10-18 14:19:23 -07:00
Dean Lee ccc07db1ee replay: just load one segment to start replaying (#22598) 2021-10-18 12:03:34 -07:00
Dean Lee 94b3d4d267 replay: allocate events in contiguous memory pool (#22535)
* memory pool

* reserve events too

* remove dctor

* cleanup

* move pool into class LogReader

* cleanup include
2021-10-18 11:10:08 +02:00
Dean Lee f6de10b55a replay: start streaming after segment loaded (#22575)
* start streaming  after segment loaded

dd

* loop from beginning if reaches the end

* isSegmentLoaded

* one loop

* move to ctor

* delete stream_thread_ on exit

* pause streaming while testing seek

* Revert "one loop"

This reverts commit f029cd118f7ac876dee1dbf2b91478403211ad47.

* test:dummy stream thread

* cleanup

* start thread after vipcserver
2021-10-18 11:05:03 +02:00
Dean Lee c73d9ddaa6 replay: fix hanging on shutdown while downloading (#22592) 2021-10-18 11:03:30 +02:00
Dean Lee 085c424cf9 framereader: remove swscale (#22578) 2021-10-16 13:56:04 -07:00
Dean Lee ae5eedb0b1 Replay/framereader: fix c3 qcamera padding problem (#22572) 2021-10-16 12:24:34 -07:00
Dean Lee 47ce0b4cd2 replay: added a missing #include <array> on Mac (#22573) 2021-10-16 09:02:19 -07:00
Dean Lee 20b0ae0e65 replay: fix possible segfault in CameraServer (#22556) 2021-10-15 15:34:54 -07:00
Dean Lee acc52ece20 replay: refactor Route and Segment (#22531)
* new functions

* fix wrong call to qUrl::isLocalFile

* cleanup

* keep extension in cached files

* cleanup

* simplify segment

* delete thread

* add output

* pre-decompress

* remove suffix

* revert remove suffix

* 1 connection for log file

* cleanup

* segment may not be continuous,use map

* don't emit finish if aborting_

* use QFuture and thread pool

* cleanup

* fix segfault in LoadFromLocal

* cleanup

* handle segment failed to load

* output info

* continue error handling

* Remove redundant testSeekTo

* cleanup

* always return true

* keep time is ok now

change to 1s

write 1 byte at the end of the sparse file

* log loading segment

* merge #22476

* Update selfdrive/ui/replay/logreader.cc

* correct connect

* pub message in function

* typo

* Update selfdrive/ui/replay/replay.cc

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-10-15 14:35:17 -07:00
Dean Lee 43a846843f c++ replay: support local routes (#22288)
* support local route

* kind of works

* works

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-10-12 00:51:22 -07:00
Adeeb Shihadeh f1fcf17c4c replay: support old, pre-pandaStates routes (#22525)
* replay: support old, pre-pandaStates routes

* cleanup
2021-10-11 15:58:44 -07:00
Dean Lee c801c65b9d replay: get EncoderIndex from capnp::AnyStruct (#22518)
* get EncoderIndex from capnp::AnyStruct

* const
2021-10-11 09:57:35 -07:00
Dean Lee 537dff3ae0 replay/camera: fix incorrect console output (#22491) 2021-10-08 19:51:23 -07:00
Dean Lee 5527736df6 replay: simplify seek&merge (#22463)
* simplify seek&merge

* update test cases

update test cases

* cleanup test cases

* new function currentSeconds

* add TODO

* thread safe publishFrame

* cleanup

* fix 'at x S' not printed if seek back to old time

* exit replay if failed to load route

* move out setCurrentSegment from if statement

* cleanup

* use std::find

* const variables
2021-10-07 17:32:16 +02:00
Greg Hogan 188b634222 replay: print current time when paused (#22468) 2021-10-07 10:41:49 +02:00
Dean Lee f7cf4be995 repaly: stop publish socket on multiple publishers error (#22449) 2021-10-06 14:04:51 +02:00
Dean Lee 261d51de41 test_replay: temporarily comment out some unit tests (#22447) 2021-10-06 12:27:52 +02:00
Dean Lee b5988d5601 replay: cleanup output (#22434) 2021-10-05 15:15:46 +02:00
Dean Lee 60b0c43c90 replay: refresh segment window after current segment is changed by seek (#22436) 2021-10-05 15:15:33 +02:00
Dean Lee 97fc8b941e FrameReader: increase probesize to 10MB (#22437) 2021-10-05 15:15:11 +02:00
Willem Melching 94afd0ea0f replay: do not use OpenCL (#22431) 2021-10-04 19:43:52 +02:00
Dean Lee 2253cba98b c++ replay: graceful shutdown (#22280)
* graceful shutdown

* reset termios

* continue

* call clear in dctor

* unpause

* delete vipc_server

* relase context

* notify_one

* merge master

* merge master

* print vipc stop listen before shutdown down
2021-10-04 18:00:49 +02:00
Dean Lee 29b9c8f13b replay: cleanup console output (#22410) 2021-10-04 17:15:03 +02:00
Dean Lee 1eb79d7a59 c++ replay: publish all frames in CameraServer (#22378)
* cameraserver

* support yuv

* init camera server in start()

* trigger ci
2021-10-04 16:45:28 +02:00
Dean Lee fd801c454a c++ replay: get socket name from capnp::Schema (#22367)
* get socket name from canp::Schema

* use vector
2021-10-04 15:44:06 +02:00
Dean Lee d28b98c602 replay : move utility functions into separate file (#22414)
* move functions into util

* read bz2 into stream

* pre-decompress log in the download thread

* cleanup logreader

* cache sha256 path

* use readBZ2file in test_logger

* Revert "cache sha256 path"

This reverts commit 60459d3ea09a2c80f4560cf95b1ce7d6af59f06d.

* use macro

* use ostringstream

* cleanup readBZ2File

* move precise_nano_sleep into util
2021-10-04 15:39:59 +02:00
Dean Lee fa8ddd992d c++ replay: more robust seek (#22375)
* skip invalid segment

* test seek

* fix wrong cur_mono_time when route_start_ts is 0

* don't notify stream thread if current segment not merged

* continue merge next segment if current is invalid

* cleanup seekTo

* continue

* fix seek cross-boundary

* new demo route has 11 segments

* continue

* fix cross-boundary

* cleanup & better test cases

cleanup

remoe &

typo

update comment

* update test

* reserve space for new_events

* parallel sorting

* usleep(0)

* Revert "parallel sorting"

This reverts commit d7d1b42f49944aef0b6ca2962e45a7f5318a1152.
2021-10-04 15:00:57 +02:00
Dean Lee a6da064a93 replay/segment: std::atomic member variables (#22409) 2021-10-02 16:09:39 +02:00
grekiki 4c7acf680c C++ replay cache folder path / fix (#22396) 2021-10-01 17:32:41 +02:00
Adeeb Shihadeh dd67dc6b34 update demo route (#22381) 2021-09-29 18:51:30 -07:00