Commit Graph

21 Commits

Author SHA1 Message Date
Dean Lee dbec761941 logreader: support reading from corrupt log (#23050)
* catch exception outside loop

* print decompress error

* add test case for corrupt log

* fix decompressbz2 stuck if log is corrupt

* recovered from corrupt data

* add output

* ass space

* std::endl

* override load(), load from buffer

* override FrameReader::load to load from the buffer

* replace NULL with nullptr

* fix test case for corrupt log

* Trigger Build

* check bzerror too

Co-authored-by: Willem Melching <willem.melching@gmail.com>
2021-11-29 14:10:24 +01:00
Dean Lee 3fd0264960 refactor httprequest: emit single signals (#23039)
* refactor httprequest

* Trigger Build
2021-11-29 11:19:08 +01:00
Dean Lee f5e4867864 replay: refactor http download (#23052)
* refactor http download

* use chunk_size instead of parts

testcase:set chunksize to 5mb

* use template

space

* cleanup

* remove unused include

* check buffer overfllow

* simplify print download speed
2021-11-29 11:13:30 +01:00
Dean Lee 20d3cf1fcb framereader: support hardware decoding (#22906)
* support hardware accelerated video decoding

* support cuda only

* print decode info

* add yuv flag back

* fix height

* default is cuda

* fallback to CPU decoding

fix memory leak&cleanup

add output
2021-11-17 11:17:59 +01:00
Dean Lee fb8ba34f31 replay: add qcam and YUV flags (#22788)
* common flags

* cleanup

* remove double semicolon

* camera

* apply reviews
2021-11-09 14:08:24 -08:00
Dean Lee 2b4a477fbc replay: improve segment download and merge (#22654)
* no-cache mode

* fix test cases build error

* space

* don't create cache dir in no-cache mode

* fix errors in test cases

* no_local_cache_

* set the number of connections by chunk_size

* use size_t instead of int64_t

* add test case for no-cache mode

* rename variables

* fix SIGSEGV

* cleanup

* faster decompressBZ2

* always decompress bz2

* add test cases

* prepare for python interface

* fix test cases build error

* continue

* camera_replay: cache remote file

* protected inheritance

* single option name

* TODO

* test_case for LogReader&FrameReader

* fix wrong require

* test case for FileReader

* cleanup test

* test:fix wrong filename

* check cached file's checksum

* fix mkdir permissions err

cleanup filereader

* remove initialize libav network libraries.

dd

* abort all loading if one failed

* cleanup tests

* use threadpool to limit concurrent downloads

* cache more segments

* merge 3 segments for replay

* one segment uses about 100M of memory

* use segments_need_merge.size()

* shutdown

* fix stuck if exit replay  before keyboard thread started

* load one segment at a time

* small cleanup

* cleanup filereader

* space

* tiny cleanup

* merge master

* cleanup test cases

* use util:create_directories

* cleanup framereader
2021-11-01 11:55:56 +01:00
Dean Lee d70e49dca5 replay: build on devices (#22632)
* there is no QThread::create on device

* add files to files_common

* modify SConscript

* no memory_resouce on device

* #define HAS_MEMORY_RESOURCE

* fix hangs on shutdown

* build on device with the --extra flag

* delete later

* setup -> extras

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-10-29 12:21:57 +02:00
Dean Lee 107ed35d1e replay: parse route string, replay from the specified segment (#22503)
* parse segment name

* return 0

* update regexp

* with |

* dd

* use struct

* move to class Route

* refactor loadFromLoacal

* simplify loadFromJson

* cleanup load from server

* const ref
2021-10-19 14:45:07 +02:00
Dean Lee 8d1d0c2cd7 replay: improve segment downloading (#22504)
* improve segment downloading

dd

* continue

* log retrying

* check aborting_ in loop

* std::endl

* log download information

* cleanup

* continue

* dd

* move download to seperate funciton

* simplify logging

* continue

* guard ts

* cleanup

* typo
2021-10-19 12:33:12 +02:00
Dean Lee c73d9ddaa6 replay: fix hanging on shutdown while downloading (#22592) 2021-10-18 11:03:30 +02:00
Dean Lee acc52ece20 replay: refactor Route and Segment (#22531)
* new functions

* fix wrong call to qUrl::isLocalFile

* cleanup

* keep extension in cached files

* cleanup

* simplify segment

* delete thread

* add output

* pre-decompress

* remove suffix

* revert remove suffix

* 1 connection for log file

* cleanup

* segment may not be continuous,use map

* don't emit finish if aborting_

* use QFuture and thread pool

* cleanup

* fix segfault in LoadFromLocal

* cleanup

* handle segment failed to load

* output info

* continue error handling

* Remove redundant testSeekTo

* cleanup

* always return true

* keep time is ok now

change to 1s

write 1 byte at the end of the sparse file

* log loading segment

* merge #22476

* Update selfdrive/ui/replay/logreader.cc

* correct connect

* pub message in function

* typo

* Update selfdrive/ui/replay/replay.cc

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-10-15 14:35:17 -07:00
Dean Lee 43a846843f c++ replay: support local routes (#22288)
* support local route

* kind of works

* works

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2021-10-12 00:51:22 -07:00
Dean Lee 5527736df6 replay: simplify seek&merge (#22463)
* simplify seek&merge

* update test cases

update test cases

* cleanup test cases

* new function currentSeconds

* add TODO

* thread safe publishFrame

* cleanup

* fix 'at x S' not printed if seek back to old time

* exit replay if failed to load route

* move out setCurrentSegment from if statement

* cleanup

* use std::find

* const variables
2021-10-07 17:32:16 +02:00
Dean Lee 29b9c8f13b replay: cleanup console output (#22410) 2021-10-04 17:15:03 +02:00
Dean Lee d28b98c602 replay : move utility functions into separate file (#22414)
* move functions into util

* read bz2 into stream

* pre-decompress log in the download thread

* cleanup logreader

* cache sha256 path

* use readBZ2file in test_logger

* Revert "cache sha256 path"

This reverts commit 60459d3ea09a2c80f4560cf95b1ce7d6af59f06d.

* use macro

* use ostringstream

* cleanup readBZ2File

* move precise_nano_sleep into util
2021-10-04 15:39:59 +02:00
Dean Lee fa8ddd992d c++ replay: more robust seek (#22375)
* skip invalid segment

* test seek

* fix wrong cur_mono_time when route_start_ts is 0

* don't notify stream thread if current segment not merged

* continue merge next segment if current is invalid

* cleanup seekTo

* continue

* fix seek cross-boundary

* new demo route has 11 segments

* continue

* fix cross-boundary

* cleanup & better test cases

cleanup

remoe &

typo

update comment

* update test

* reserve space for new_events

* parallel sorting

* usleep(0)

* Revert "parallel sorting"

This reverts commit d7d1b42f49944aef0b6ca2962e45a7f5318a1152.
2021-10-04 15:00:57 +02:00
grekiki 4c7acf680c C++ replay cache folder path / fix (#22396) 2021-10-01 17:32:41 +02:00
Willem Melching b7c64542f4 replay: fall back to qlog (#22362) 2021-09-29 17:44:10 +02:00
Dean Lee 4e6ff308a8 c++ replay: chunking and concurrent downloads (#22308)
* download segment files by chunks in multiple threads

* remove easy_handl on aborting

* add test cases

* better error handling

* update test

* cleanup

* add CURLGlobalInitializer

* check http code

* finish
2021-09-28 12:24:48 +02:00
Dean Lee e888a7e939 load d/e camera by command line flags (#22341) 2021-09-28 11:25:18 +02:00
Dean Lee 50ae7dd6a9 replay improvements (#22203)
* refactor replay

* cleanup

small cleanup

* merge 22239

* cleanup

* add optional argument for start time

* small cleaup
2021-09-19 12:24:28 -07:00