Commit Graph

273 Commits

Author SHA1 Message Date
Adeeb Shihadeh 35bfdce477 loggerd: trigger rotate on frame id instead of frame count (#22848)
Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: ea761cbbd5103a589915ec3766b4d6056293cba6
2021-11-13 15:38:10 -08:00
Willem Melching 669ed5ff2b Put journal and nvme stats in bootlog (#22849)
* Run commands and put output in bootlog

* log nvme

* fix indent

* cereal without event

* use regular array
old-commit-hash: fa0cbe5d400aefaca17964100d2d16da33785d59
2021-11-10 16:07:22 +01:00
Dean Lee 48ed0e129e loggerd/RawLogger: fix memory leak in encode_frame (#22826)
old-commit-hash: 0bd9aed82c2dfbf2c3cfe5aba7cd6b980a5fc97e
2021-11-08 15:27:51 -08:00
Willem Melching 19deb98bc3 wshadow: fix comma three build
old-commit-hash: 45c506bb17e51d4f8592f74c75b9adcffa681201
2021-11-02 17:27:34 +01:00
Mayfield 43e1903073 enable wshadow (#22756)
* enable wshadow (#22714)

* fix replay

* more build fixes

Co-authored-by: Willem Melching <willem.melching@gmail.com>
old-commit-hash: 5246f0231e6bfa572c0dd89c60e538910817faa2
2021-11-02 17:08:53 +01:00
Dean Lee a031b938b0 replay: improve segment download and merge (#22654)
* no-cache mode

* fix test cases build error

* space

* don't create cache dir in no-cache mode

* fix errors in test cases

* no_local_cache_

* set the number of connections by chunk_size

* use size_t instead of int64_t

* add test case for no-cache mode

* rename variables

* fix SIGSEGV

* cleanup

* faster decompressBZ2

* always decompress bz2

* add test cases

* prepare for python interface

* fix test cases build error

* continue

* camera_replay: cache remote file

* protected inheritance

* single option name

* TODO

* test_case for LogReader&FrameReader

* fix wrong require

* test case for FileReader

* cleanup test

* test:fix wrong filename

* check cached file's checksum

* fix mkdir permissions err

cleanup filereader

* remove initialize libav network libraries.

dd

* abort all loading if one failed

* cleanup tests

* use threadpool to limit concurrent downloads

* cache more segments

* merge 3 segments for replay

* one segment uses about 100M of memory

* use segments_need_merge.size()

* shutdown

* fix stuck if exit replay  before keyboard thread started

* load one segment at a time

* small cleanup

* cleanup filereader

* space

* tiny cleanup

* merge master

* cleanup test cases

* use util:create_directories

* cleanup framereader
old-commit-hash: 2b4a477fbc8fc02da5876aee4f955194916f19a3
2021-11-01 11:55:56 +01:00
Dean Lee ebffd5f3ad util: add safe write functions (#22160)
* add safe write functions

* log error

* bzfile safe_fopen&safe_flush

* update test_case

* trigger ci

Co-authored-by: Willem Melching <willem.melching@gmail.com>
old-commit-hash: 652c42da2c6fa31168d5d299c959ab0c86799c4b
2021-10-29 13:23:31 +02:00
Dean Lee 0afe7d74bc util: add new function create_directories with unit tests (#21871)
* util::create_directories

* check bool ret

* don't assume mask

* rename with_umask to no_umask

* remove umask

* rebase master

* rebase master

* 0755
old-commit-hash: 1f39d8cee69cf28576cd9bd27ef52be57d44371e
2021-10-29 12:27:35 +02:00
Dean Lee c401d7aff3 loggerd: fix concurrency issues in sync encoders (#22590)
* sync_encoders

* add variable camera_synced

* cleanup
old-commit-hash: 18caf3535b2049a0809cb3b01fde19d196e34e32
2021-10-29 12:21:02 +02:00
Adeeb Shihadeh bb7751fdb0 don't set core affinity on PC (#22706)
old-commit-hash: 5b331fd6f5333e24a632839b64cdeb9129ef923b
2021-10-26 21:51:46 -07:00
Willem Melching 0f77dd4506 loggerd: set encodeIdx.valid if frame id is correct (#22634)
* loggerd: set encodeIdx.valid if frame id is correct

* check valid flag in test

* test cleanup

* bump cereal
old-commit-hash: 89f311714cf3606c386fd1de52b27e24780b8174
2021-10-26 16:00:43 +02:00
Adeeb Shihadeh b773f6711c loggerd: always run encoders (#22649)
* loggerd: always run encoders

* fix raw logger

* bump loggerd cpu usage
old-commit-hash: 76bd932cf555305a1e00d723bb5bb69d7fba39e9
2021-10-21 16:37:03 -07:00
Adeeb Shihadeh 25cfa2b65a tici: affine loggerd to efficiency cores (#22581)
old-commit-hash: 84de248fa7cbee37ecb2fd6a9d4df50ba60a1e04
2021-10-18 13:21:24 -07:00
Dean Lee 66ca3985c9 replay : move utility functions into separate file (#22414)
* move functions into util

* read bz2 into stream

* pre-decompress log in the download thread

* cleanup logreader

* cache sha256 path

* use readBZ2file in test_logger

* Revert "cache sha256 path"

This reverts commit 60459d3ea09a2c80f4560cf95b1ce7d6af59f06d.

* use macro

* use ostringstream

* cleanup readBZ2File

* move precise_nano_sleep into util
old-commit-hash: d28b98c60244e2bb9e32810e881ea445a912606c
2021-10-04 15:39:59 +02:00
Robbe Derks 0cf0485912 small typo
old-commit-hash: 3ccf712ac99ce8ef393d87ddd7241d1d823dd5a6
2021-09-24 10:23:41 +02:00
Adeeb Shihadeh c0356ffe9b add log docs
old-commit-hash: 0f8c6f130b5c9e023db136f64da1cbc1590a6994
2021-09-23 14:41:35 -07:00
ntegan1 d508125f35 Upload when on ethernet (#22188)
in addition to when on wifi

Co-authored-by: ntegan <nick@egan.gg>
old-commit-hash: 54b5972cf6f7220ba6af061c6dadc2dc80d813d0
2021-09-10 12:05:12 -07:00
Willem Melching 4844f8196d remove loggerd todo about encodeIdx decimation
old-commit-hash: 377fe84948c069c3f00824c06e0ae40f2aa2616e
2021-09-08 12:16:16 -07:00
Dean Lee 33b378419a loggerd: remove obsolete ethernetsniffer.py (#22139)
old-commit-hash: 8bca70878fbfeaa18ba7b0bcc17b1ba1baad1eb2
2021-09-06 12:11:19 -07:00
Dean Lee 2dac83c02b loggerd: remove obsolete testraw.cc (#22138)
old-commit-hash: 5a8ca8b449fb3418d31ca325fb38d393413d83b8
2021-09-06 12:10:12 -07:00
Willem Melching e49dd50f98 Disable encoder sync logic if only one rotate camera present (#22107)
* disable sync logic if only one rotate camera

* consistent brackets
old-commit-hash: 48035e9535ba362e382d01282a1475b4ace2778a
2021-09-01 23:08:06 -07:00
Willem Melching 60e7e09127 Ensure that first segment starts with all cameras on the same frameId (#22096)
* test that segment starts with the same frame id

* improve test

* add sync logic

* atomic max

* review comments
old-commit-hash: a93271a518b466cd83699acd8598a0361ee9a02b
2021-09-01 14:28:50 -07:00
Dean Lee 4c17c06799 Handle EINTR for all syscalls that can return it (#21948)
old-commit-hash: f02c8d0e2c04e45ceeea9f2fe9851319d724a637
2021-08-31 16:22:56 -07:00
Dean Lee ecf3ee7603 loggerd: create encoder threads from cameras_logged (#21875)
old-commit-hash: d9baaf7eb90f0815c4d635d33244975ad71876d2
2021-08-30 19:33:00 -07:00
Dean Lee 0fac38e34f LoggerState: test cases (#21855)
* test multiple threads writing to log

* fix typo

* use util::getenv

* check INIT_DATA and SENTINEL

* test END_OF_SEGMENT

* add comment

* test multiple threads logging and rotation

* cleanup

* update

* LoggerHandle:quick fix

* cleanup

* revert test_loggerd.py

* refactor test case

* check lock file

* check refcnt after close

* test_runner.cc

d

* int eixt_signal
old-commit-hash: 935cbd3139993eac84f7936dfdb191deb86114f1
2021-08-30 17:12:09 -07:00
Willem Melching 6f92abc815 Less open file permissions (#21922)
* Less open file permissions

* add test back

* remove params test for permissions

* remove umask

* bump cereal

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: dddab597bcee0a103895509b7909743f97cbdf83
2021-08-28 18:25:05 -07:00
Adeeb Shihadeh 277bd43b15 Revert "tici: give loggerd rt priority (#21503)"
This reverts commit 0a50cf851e79342cf3db01033bc91516c504cbea.

old-commit-hash: 54a4eeb0fccd92c3b2f872175cfedf136a1cfc71
2021-08-22 16:57:53 -07:00
Adeeb Shihadeh 5c1d1ebe76 tici: give loggerd rt priority (#21503)
* give loggerd rt priority

* assert

* little more

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 7e2752c878a9e8c38639e4caa234bcabb979db2e
2021-08-20 21:27:31 -07:00
Dean Lee f65c7e7418 capnp:Text::Reader: use implicit conversion from string (#21980)
old-commit-hash: c6a34dd9f7ea1798a54883070eb548c17addbb12
2021-08-19 13:49:08 -07:00
Dean Lee 7d09945de3 BZFile: handle EINTR on BZ2_bzWrite (#21949)
old-commit-hash: b66e8adbefc3c8a360917fb0b8b8023374a63320
2021-08-18 12:08:07 +02:00
Dean Lee aff7f0b1f3 enclose macro arguments in parentheses (#21947)
* macro argument should be enclosed in parentheses

* CL_CHECK
old-commit-hash: 4625098a7953308cc76c6c9b81d28e6bb89f671c
2021-08-18 10:56:37 +02:00
Adeeb Shihadeh 9b1a7a6f01 use same env variable for loggerd, deleter, and uploader
old-commit-hash: 1d4b19f739270bd6b075c7ea9b48e7460ab867c0
2021-08-07 18:58:11 -07:00
Willem Melching ae4f3c98d5 refactor loggerd: trigger rotate in encoder thread (#21860)
* trigger rotate in eoncode thread

* rotate in time

* lgtm

* dcam trigger rotate on C3

* check trigger rotate field

* Use >=

* add rotator thread

* set cnt to 0

* test encodeIdx is increasing across segments

* test both segmentId and encodeId

* fix encodeIdx

* no thread needed

* add log for failed to encode

* stricter C3 test, dont check start encodeId on C2 dcam

* only update last_camera_seen_tms when considered for rotate

Co-authored-by: deanlee <deanlee3@gmail.com>
old-commit-hash: a39873872de5b2239e4b89119974e5aa961eba48
2021-08-06 11:10:33 +02:00
Dean Lee 94d534afea hw.h: move Hardware-related paths to namespace Path (#21826)
* refactor getenv and path

* apply review
old-commit-hash: fefc70b4cab71a11b3c5ed1b8680606ee2013a48
2021-08-04 13:29:03 +02:00
George Hotz 17867a1122 loggerd: implement DONT_LOG flag (#21832)
* implement DONT_LOG flag

* better typing
old-commit-hash: 8354cfc53c023bfd20b7887c7ff4560fb9615b4f
2021-08-02 20:16:38 -07:00
Adeeb Shihadeh db62739c00 fix loggerd test failure when camerad is slow to start
old-commit-hash: d748b44f48f6d8d4e2b1ce6195363bded06ba377
2021-08-02 09:10:29 -07:00
Dean Lee 7fed9b0c0c logger.cc/logger_build_init_data(): reuse params_map (#21811)
* faster log params

* Update selfdrive/loggerd/logger.cc

* fix build

Co-authored-by: Willem Melching <willem.melching@gmail.com>
old-commit-hash: fecdf16f6bd729dadf35053e4dcde39cae9cb73e
2021-08-02 11:21:12 +02:00
Dean Lee 31ea9413bb util/read_files_in_dir: return map by value (#21815)
old-commit-hash: 11ffbc693694780c612d9d449f49a2feabe8a0a2
2021-08-02 10:59:54 +02:00
Adeeb Shihadeh dc0c4bd99b debug msg for loggerd tests
old-commit-hash: 48652db706c7070bb336fc40be7acdc5c2a6e97a
2021-07-28 14:01:27 -07:00
Adeeb Shihadeh 14a441a314 publish uploader state (#21580)
* uploader state

* add time

* bump

* populate whole packet

* bump cereal

* revert

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 6b6903280719da6cf764baa9417766cf785c0101
2021-07-13 17:18:10 -07:00
Willem Melching f8edcd64c4 loggerd: power failure print sync done
old-commit-hash: dd128f591689464801d98f8251416af7f5ea6909
2021-07-12 12:28:07 +02:00
Adeeb Shihadeh 255dfffd7f fix qlog decimation calculation in loggerd tests
old-commit-hash: d8411e7a62a8e3c3df9fe8a7406849b83e927a3d
2021-07-10 23:20:15 -07:00
Greg Hogan a637623fa0 upload can only create (#21513)
old-commit-hash: d066b8f7e813894338b2747c74ad089968ff70e9
2021-07-07 18:28:06 -07:00
Adeeb Shihadeh 19bc1b7397 fast jenkins (#21472)
* speed up setup

* speedup onroad test

* .git doesn't like inplace

* tici2

* no inplace

* revert

* fix

* that one too
old-commit-hash: 0bfd994f73a73b321f0ac45e1f3f5d1e907e6962
2021-07-02 16:29:57 -07:00
Adeeb Shihadeh d31397b84b segment regen (#21148)
* start seg regen

* send vipc

* fix up some paths

* set fp

* fix no entries

* engages

* seperate camera procs

* send real frames

* regen test routes

* nice argparser

* fix valgrind test

* move that

* fix that
old-commit-hash: 4a1aec27efb48bdba1d267c86ef66c8a044c4986
2021-06-28 21:44:40 -07:00
Dean Lee 147b355dde add space after function name and if statement (#21225)
old-commit-hash: e4e669bb0fc1a919dfb38800099f8e3ad52346f1
2021-06-11 10:17:52 +02:00
Dean Lee f70a79b838 Use C++ header files instead of C header files. (#21192)
* use cstring instead of string.h

* use cstdio instead of stdio.h

* remove inttypes.h

* use cstdlib instead of stdlib.h

* use cstdint instead of stdint.h

* #include <cstddef>

* cstdlib

* use cmath

* remove stddef.h

* use cassert

* use csignal

* use ctime

* use cerror

* rebase master
old-commit-hash: c53cb5d570adb82bc454fcabab458d7e7f09738c
2021-06-08 13:46:22 -07:00
Adeeb Shihadeh 61d3b1f177 disable uploading raw by default (#21128)
old-commit-hash: c76c30fa2d37d471d065e908739c69f06b43ccfe
2021-06-03 15:24:30 -07:00
Willem Melching 724e0f6e11 deleter.py: delete boot and crash folders last (#21028)
old-commit-hash: 129c760e4fad558bf3c5d03fb5fb8edfb76bfc8b
2021-06-01 14:08:08 -07:00
Dean Lee 497472932a Cleanup util.h (#20849)
* cleanup util.h

* fix build error:omx_encoder.cc:408: undefined reference to do_exit

* fix build error

* rebase master

* move helper functions back to util

* ExitHandleHelper->ExitHandlerHelper

* std::clamp

* struct ExitHandlerHelper to class

* rebase master

* cleanup

* restore ExitHandler

* prefer cassert
old-commit-hash: 65e0b1e699092355bfcb1f588a5a2ec949a34b77
2021-06-01 16:53:12 +02:00