Commit Graph

79 Commits

Author SHA1 Message Date
Adeeb Shihadeh 3c48bc47b7 Move a bunch of stuff to system/ part 3 (#24829)
* move swaglog.py

* timezoned

* logmessaged

* version.py

* fix linter
old-commit-hash: 0fce5d90459b77bf2cfa70f55f322f0e1fb8d01c
2022-06-11 23:19:27 -07:00
Adeeb Shihadeh 9a45a3b99b Move selfdrive/hardware/ to system/ (#24725)
* move hardware to system/

* fix mypy
old-commit-hash: 1139fe507b01f34de9714c99228f411558b44231
2022-06-11 16:38:24 -07:00
Adeeb Shihadeh 8456b0e79c uploader: increase max uncompressed qlog size to 100MB
old-commit-hash: eca265e07b512831d473cd1cf6d6b3dea5a03327
2022-05-23 13:43:14 -07:00
grekiki 7cd23af9b1 Mypy fixes for --check-untyped-defs (#24372)
more type fixes
old-commit-hash: 17e33978cdd9322918e9d5d388f7fa47aa30e9b7
2022-05-12 14:08:11 +02:00
Willem Melching badbd3347c uploader: ensure requests.put gets file like object (#24462)
old-commit-hash: ba89faa5f77c76023f49b85888e6d6eb7f2e1a11
2022-05-07 15:42:47 +02:00
Joost Wooning 08da73721c uploader: size limit for automatic uploading of qlogs/qcams (#24403)
* uploader: size limit for automatic uploading of qlogs/qcams

* move check to add logging

* use constant

* mark as uploaded

Co-authored-by: Willem Melching <willem.melching@gmail.com>
old-commit-hash: df8f024e19de2d7a528b5921f73895f2078cf03a
2022-05-06 11:47:58 +02:00
Adeeb Shihadeh 2e45c3bd68 uploader: compress rlogs (#24447)
old-commit-hash: 66fbce638f1a6a2bc9c08c4a17d5efe47fb91651
2022-05-05 19:13:06 -07:00
Willem Melching be08693d5c delay bz2 compression from logging to uploading (#24392)
* remove log_name

* log without compression

* fix tests

* remove extension for bootlog

* another test fix

* uploader compresses

* also compress in athena

* only compress qlog

* more generic check in do_upload

* fix bootlog compression

* lower loggerd cpu usage

* dont link against bz2

* set core affinity to little cluster

* handle old files
old-commit-hash: 77a6f3d034a7b17d70b750b9ea454310f4b8edf1
2022-05-03 14:09:17 +02:00
Willem Melching 85aa5e2706 uploader: azure returns 401 for files already uploaded (#24141)
old-commit-hash: e7f23ceca252fb4c62819b29c38eaaa62d66ce39
2022-04-05 20:09:05 -07:00
Willem Melching 2eded18267 C3: detect metered Android hotspot (#23734)
* C3: detect metered networks

* show in ui

* fix text layout

* bump cereal

* revert ui changes

* set networkMetered

* add athena method

* add metered logging to uploader

* use in athena uploader

* remove param

* use networkmanager properties to set cell to unmetered

* fix indentation

* no need to check

* bump cereal

* review

* bump cereal
old-commit-hash: da5a0c41a00e4de076c2846ed54049c3dff3bc5a
2022-03-09 11:36:52 +01:00
Adeeb Shihadeh ef2106ebd2 move lock clearing to uploader (#23826)
* move lock clearing to uploader

* test case
old-commit-hash: 3243d1a81e5cfdf687ee406d92903ee44fe781d3
2022-02-22 22:30:11 -08:00
Willem Melching a7fa889605 improve logging around uploading (#23750)
* improve logging around uploading

* use int

* use raw

* duplicate
old-commit-hash: 7765bc216677026d7b8919525eff310716fd3e7d
2022-02-11 12:58:41 +01:00
Ewout ter Hoeven a962365292 Convert format strings strings to f-strings (#23241)
* Convert all text strings to f-strings

Reformats all the text from the old "%-formatted" and .format(...) format to the newer f-string format, as defined in PEP 498. This requires Python 3.6+.

Flynt 0.69 was used to reformat the strings. 120 f-strings were created in 51 files.

F-strings are in general more readable, concise and performant. See also: https://www.python.org/dev/peps/pep-0498/#rationale

* revert pyextra changes

* revert ublox.py

Co-authored-by: Willem Melching <willem.melching@gmail.com>
old-commit-hash: 55390d273f36f49da3896e687ac5530f40c1d150
2021-12-16 14:58:17 +01:00
Willem Melching e7bf88f29f Remove Raw Upload toggle (#23143)
* Remove Raw Upload toggle

* fix tests
old-commit-hash: 2e0c73fc0cbac64e442085390dd22e6b408d0478
2021-12-07 11:25:40 +01:00
Willem Melching 9399b6ae61 uploader: switch to upload_url v1.4 (#22966)
old-commit-hash: 23a1f478174ca443f760a5db910bc235c05bfffb
2021-11-18 16:23:50 +01:00
ntegan1 d508125f35 Upload when on ethernet (#22188)
in addition to when on wifi

Co-authored-by: ntegan <nick@egan.gg>
old-commit-hash: 54b5972cf6f7220ba6af061c6dadc2dc80d813d0
2021-09-10 12:05:12 -07:00
Adeeb Shihadeh 14a441a314 publish uploader state (#21580)
* uploader state

* add time

* bump

* populate whole packet

* bump cereal

* revert

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 6b6903280719da6cf764baa9417766cf785c0101
2021-07-13 17:18:10 -07:00
Greg Hogan a637623fa0 upload can only create (#21513)
old-commit-hash: d066b8f7e813894338b2747c74ad089968ff70e9
2021-07-07 18:28:06 -07:00
Adeeb Shihadeh 61d3b1f177 disable uploading raw by default (#21128)
old-commit-hash: c76c30fa2d37d471d065e908739c69f06b43ccfe
2021-06-03 15:24:30 -07:00
Willem Melching 8f4a445171 add putBool/getBool wrappers to cython params class (#20611)
* add putBool/getBool wrappers to cython class

* use new API

* some more puts

* fix mockparams arguments

* add get_bool to MockParams

* typo
old-commit-hash: ae094042ade44815fdba50196c15ba552bbf33d7
2021-04-07 15:36:37 +02:00
Adeeb Shihadeh 893946ae0c increase log level of NVME cloudlog
old-commit-hash: 538df2596dd365382f5ecf41ffa17f31216cd02f
2021-04-02 21:43:58 -07:00
Greg Hogan f0b9176bd7 logging cleanup (#20502)
* switch some uploader messages to debug log

* use allow_sleep

* log when not request or response

* test both with params and without params
old-commit-hash: 1e3f8b014ef63f5e3d0a3f7622a03892dd8958b2
2021-03-26 17:36:01 -07:00
Adeeb Shihadeh ed23f0778a log when nvme isn't mounted (#20428)
Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 33500bf23fcda9c4404f1c09177f7e971d65fc56
2021-03-21 23:12:09 -07:00
Adeeb Shihadeh 05eb44115e cereal cleanup part 2 (#20092)
* car stuff

* thermal

* Revert "car stuff"

This reverts commit 77fd1c65ebd01abfa8493ae12c9e6b14f7ada976.

* panda state

* camera stuff

* start deg

* most is building

* builds

* planner + controls run

* fix up paramsd

* cleanup

* process replay passes

* fix webcam build

* camerad

* no more frame

* thermald

* ui

* paramsd

* camera replay

* fix long tests

* fix camerad tests

* maxSteeringAngle

* bump cereal

* more frame

* cereal master
old-commit-hash: 312b681a46b8153314a8420611b6479dd6f70dfc
2021-02-16 21:39:32 -08:00
Willem Melching 4fd576e7f7 Put bootlog in own folder (#19939)
* put bootlog in own folder

* fix uploader

* remove print statements

* update test

* remove comment

* also write initddata

* remove sentinels from test
old-commit-hash: 87119aebe2e7051df49f6f2be9c29d2af4737656
2021-01-27 13:36:40 -08:00
Willem Melching 8a187ec406 Linux tombstones: add stacktrace and upload (#19737)
* Add stacktrace to tombstones

* make sentry output prettier

* Refactor

* Generate upload filename

* Actually move file

* Fix spaces

* copy and upload

* dont delete just yet

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: f3db94a0c480bc12df98ec679a2f84833e134ee3
2021-01-18 13:29:01 +01:00
Adeeb Shihadeh c64a8356fe move to github container registry + CI speedup (#19621)
old-commit-hash: 59d7c5e64ac3c8666c54c3443f912e4e0db92eb9
2020-12-31 14:57:01 -08:00
Adeeb Shihadeh 5bbe4f3d52 manager tests + make all processes exit cleanly (#19595)
* manager tests

* logcatd exits cleanly

* sigint

* boardd

* multiple dbus connections hangs for some reason

* clocksd proclogd

* network type from thermal

* fix tests

* fix android logcatd

* fix mac

* fix mac proclogd

* move on device athena tests

* build first

* build first

Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: ffa7e0cbdb0859751685caae1321d3ccf732fed9
2020-12-29 22:32:03 -08:00
Adeeb Shihadeh b950cab874 HW abstraction layer (#19530)
* start hw refactor

* move that

* pins

* put that back
old-commit-hash: 8674b023bab2b44405139ceec5785a389f837682
2020-12-16 21:30:23 -08:00
Adeeb Shihadeh d5ad6237cf Revert "Handle python-dbus not installed"
This reverts commit fd22f01b1f21005846bef211851d17f602a7c600.

old-commit-hash: a2c6fdd14ce9ef1dfa59db6de91ebf863920d381
2020-12-15 20:29:24 -08:00
Willem Melching d47396f4da Handle python-dbus not installed
old-commit-hash: 25054cdc0538a0324b7f7c1bcc9d6166291b4e10
2020-12-14 15:04:17 +01:00
Adeeb Shihadeh de4c543e63 remove hotspot exceptions in uploader (#2319)
* remove hotspot exceptions

* skip net check while onroad
old-commit-hash: ee37d03c1ffa76d91d7085c2eb9ca2ce762ee647
2020-10-12 13:12:01 -07:00
grekiki 079f409e58 Uploader speedup (#2214)
* use caching for getxattr

* fix some git issues

* Scheduled network checks

* attempt optimization

* Delete speed_test.py

* Style fixes

* Fix styling

* fix spaces

* fix spaces

* fix naming

* Update uploader.py

* Update mark_all_uploaded.py

* Add file to release

* Update selfdrive/loggerd/tools/mark_all_uploaded.py

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: cf46de13d2934c52e9c87691c671fe9c78156ec9
2020-09-22 11:37:24 +02:00
Adeeb Shihadeh 6919e06a9c small uploader cleanup
old-commit-hash: 82f40f6ff5f4677fed6c1aa84a3229a29d212356
2020-08-27 14:22:59 -07:00
robbederks 7671d18467 Tici camerad (#2048)
* fix camera config

* typos

* oops

* more typo

* lambless

* forget to send

* visualize hist

* more typo

* 0xC000

* simple

* data loss prevention, clean up later

* loggerd

* back up code

* backup

* fixed memory leak

* fix vsync

* upload ecam

* WB

* 3stream

* fix OMX crash on loggerd rotation

* rewritten debayer kernel

* update viewer

* improved AE

* no artifact lines/grids

* standard trigger

* cleanups

* CCM

* cleanups

* slight tweak

* upd push sock

* build all these

* update tele fl

* update cereal

* upd viewer

* DualCameraState -> MultiCameraState

* cameras_open

* disable frame zmq push by default

* more cleanup

* no apks

* fix submodule error

* wat

* clean up trash

* remove junk

* only build on qcom2

* no need to check these

* update cereal

* some more minor cleanup

* bump panda

* add todo

* minor typo

* Revert "minor typo"

This reverts commit 9233a1df7cac214fae6827cdae3a10cb3bd060d9.

* not care

* use consistent hdr

* some cleanup

* Update selfdrive/camerad/main.cc

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* more cleanups

* remove irrelevant stuff

* this too

* cleanup

* rerun ci

Co-authored-by: ZwX1616 <zwx1616@gmail.com>
Co-authored-by: Tici <robbe@comma.ai>
Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
old-commit-hash: 37d6472bfa1c0ef9f1f8353162b97444ca66e45c
2020-08-26 19:59:56 -07:00
Willem Melching 16fe1bb2ad Hardware abstraction class (#2080)
* hardware abstraction class

* De Morgan

* Rename pc hardware class

* Fix sound card in controlsd

* Pc get sim info

* fix hardware in test

* two more

* No more random imei on android

* no randomness on android

* Need to return something that looks like imei for registration to work

* Return proper network strength

* Unused import

* Bug fixes + gpsd is only android
old-commit-hash: c7152d54192605c833912b69cf85514ac2e1b1fc
2020-08-26 14:57:17 +02:00
Adeeb Shihadeh e5637bc5de Disable raw upload while onroad (#1894)
* disable raw upload while onroad

* fix tests
old-commit-hash: c8f880665a6331aaed7799b37b12ea639e899c77
2020-07-20 15:21:39 +02:00
Willem Melching 55c3a89421 slow down upload check when offroad (#1634)
old-commit-hash: f4e8df7ee6de2998968bfa93f09f2359ca21b89e
2020-06-03 19:28:41 -07:00
Willem Melching a56e2b01f8 Make pylint more strict (#1626)
* make pylint more strict

* cleanup in progress

* done cleaning up

* no opendbc
old-commit-hash: 843a64c72fb94397a1edf814fa66044c8d92acbf
2020-06-02 16:29:32 -07:00
Adeeb Shihadeh 68a3061c9d enable E261 in flake8: two spaces before inline comment
old-commit-hash: 27754a277c36b82ef4040e536cc918ba8ad77bec
2020-05-31 14:07:29 -07:00
Adeeb 5e857427ba Enable more flake8 checks (#1602)
* enable some more flake8 checks

* some more quick ones

* bump opendbc

* e401

* e711 e712

* e115 e116

* e222

* e301

* remove that

* e129

* e701 e702

* e125 e131

* e227

* e306

* e262

* W503

* e713

* e704

* e731

* bump opendbc

* fix some e722
old-commit-hash: d9bf9f0a4036f55411f6dfbb438990a5eb7f4930
2020-05-30 20:14:58 -07:00
Greg Hogan 8be9cddc62 handle ignored uploads more gracefully
old-commit-hash: 85a09b3113fe45b5feee7c1c421e2c43cb2ac03a
2020-04-09 21:59:26 -07:00
Greg Hogan 41ff9244c0 uploader: allow ctrl+c
old-commit-hash: 13ec384ea0bce744a8fffaa1d7a977b91f06cfb7
2020-03-30 12:27:39 -07:00
Greg Hogan 004c181167 uploader: do not delete files after uploading (#1253)
* cffi xattr function wrapper

* xattr wrapper error handling

* xattr tests

* use xattr for tracking files uploaded

* uploader xattr exception handling

* update uploader tests

* remove unused import

* update release build

* xattrs.py -> xattr.py
old-commit-hash: f21d0f325e1304b08a4e05ccc6b0fe439a7e3928
2020-03-19 18:09:26 -07:00
Willem Melching 1d39b5a646 more generic catch in uploader and thermald
old-commit-hash: 891c4cabb8c1909e981d902b36b0f3bfb529ad16
2020-02-21 15:02:24 -08:00
Adeeb 65a2c62114 Abstract common events + event cleanup (#1129)
* too many if

* unused

* whitespace

* key

* sefldrive/car/*

* no more gctx

* lower

* start abstracting common events

* all cars

* start small

* all cars

* reverse gear

* wrongCarMode

* wrongGear

* espDisabled

* steerUnvailable

* make linter happy

* c isn't used

* fix esp_disabled in VW

* update ref

* more red

* more cleanup

* fix subaru

* update ref

old-commit-hash: e8cb6ea06a2470dd89a6ad8fd17bd1d1b62034cc
2020-02-20 16:22:25 -08:00
Willem Melching 0e5e604412 Add cloudlog for is_on_wifi fail
old-commit-hash: f6835e9490
2020-01-20 11:25:16 -08:00
Willem Melching 26d634d6c0 catch CalledProcessError in uploader wifi check
old-commit-hash: 6adbe24d4c
2020-01-20 10:57:08 -08:00
George Hotz 510b718f0f logcatd, loggerd, mapd, modeld, proclogd
old-commit-hash: da079d47d7
2020-01-17 11:20:17 -08:00
George Hotz 81dd5a50cf root commit
old-commit-hash: 6c33a5c1f3
2020-01-17 10:02:52 -08:00