Today, Firehose uploads and user-requested uploads from Connect are treated the same. This is not ideal behavior if the user wants to upload routes immediately for a bug report and the queue is full of uploads for Firehose. The workaround is to clear the queue and retry the upload from Connect.
This PR adds an optional `priority` to requested file uploads in `athenad`. By default, all requests are marked `Low`. However, if the caller wishes to mark their uploads as "more important," then the upload queue will prioritize those requests when uploading.
The only caveat to this PR is that we won't reorder files currently being uploaded. Most connections are quick enough to finish uploading max 4 files before polling the new high-priority items in the queue.
Closes https://github.com/commaai/openpilot/issues/34836
**Verification**
Added test case to insert upload tasks with differing priorities. Polling the queue produces items in the correct order (smallest # to largest).
PR to mark Connect uploads as high priority: https://github.com/commaai/connect/pull/557
---------
Co-authored-by: Cameron Clough <cameronjclough@gmail.com>
* switch logging from raw file format to zstd compressed files
* more zst suffix
* compress bootlog
* remove class RawFile
* Optimize ZstdFileWriter by adding input caching
* use ZSTD_compressStream2
* cleanup
* LOG_COMPRESSION_LEVEL=10
* space
* add zst suffix to LOGS_SIZE_RATE
* zstd uploader
* fix that
* fix name of function
* comment
* log failed
* fix comma_api_source for routes with both bz2 and zst rlogs
* TODO
* 10-14 achieves almost no benefit on qlogs in a few cases, but takes 2x the time
* these aren't written out
* regen: specify any list of sources
ooh this is pretty nice
* regen and process replay
* damn, actually we don't need all this (cool tho)
Revert "regen: specify any list of sources"
This reverts commit ceb0b4abed9ad463a9fe98d9b98a05875a52806f.
* just let it auto resolve
* fix athenad/uploader tests
* zst here too
* TODOs
* yes
* Revert "TODOs"
This reverts commit 8c7da1dbd0340c72290b5eb5563b642080ddc131.
* Revert "zst here too"
This reverts commit 23b0023ddfd22c8090be7a7caa09e7026a12aa5c.
* Revert "just let it auto resolve"
This reverts commit f296d62424227ad05facc62abc18a6f81b474e84.
* Revert "regen and process replay"
This reverts commit 0768330e96974a42616d229d159780619d049cd0.
* revert readme
* not in save_log either
* lfg
* Revert "lfg"
This reverts commit 3718559c6c4de7d1f0c80dc9f1a1d335fe679a89.
old-commit-hash: 7dec7c39bec0b7015c9cc12ee78f07a07ed47ee6
Trey Moen