mawei/StarPilot
1
0
Fork 0
mirror of https://github.com/firestar5683/StarPilot.git synced 2026-09-30 19:33:45 +08:00
Code Issues Packages Projects Releases Wiki Activity

uploader: compress with zstd (#32736)

Browse Source 
* zstd uploader

* fix that

* fix name of function

* comment

* log failed

* fix comma_api_source for routes with both bz2 and zst rlogs

* TODO

* 10-14 achieves almost no benefit on qlogs in a few cases, but takes 2x the time

* these aren't written out

* regen: specify any list of sources

ooh this is pretty nice

* regen and process replay

* damn, actually we don't need all this (cool tho)

Revert "regen: specify any list of sources"

This reverts commit ceb0b4abed9ad463a9fe98d9b98a05875a52806f.

* just let it auto resolve

* fix athenad/uploader tests

* zst here too

* TODOs

* yes

* Revert "TODOs"

This reverts commit 8c7da1dbd0340c72290b5eb5563b642080ddc131.

* Revert "zst here too"

This reverts commit 23b0023ddfd22c8090be7a7caa09e7026a12aa5c.

* Revert "just let it auto resolve"

This reverts commit f296d62424227ad05facc62abc18a6f81b474e84.

* Revert "regen and process replay"

This reverts commit 0768330e96974a42616d229d159780619d049cd0.

* revert readme

* not in save_log either

* lfg

* Revert "lfg"

This reverts commit 3718559c6c4de7d1f0c80dc9f1a1d335fe679a89.
old-commit-hash: 7dec7c39bec0b7015c9cc12ee78f07a07ed47ee6
This commit is contained in:
Shane Smiskol
2024-07-26 19:33:35 -07:00
committed by GitHub
parent 4ea781b062
commit b0a8b3bc9d
4 changed files with 53 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
system/loggerd/tests/test_uploader.py
+5 -5
View File
@@ -62,10 +62,10 @@ class TestUploader(UploaderTestCase):
def gen_order(self, seg1: list[int], seg2: list[int], boot=True) -> list[str]:
keys = []
if boot:
keys += [f"boot/{self.seg_format.format(i)}.bz2" for i in seg1]
keys += [f"boot/{self.seg_format2.format(i)}.bz2" for i in seg2]
keys += [f"{self.seg_format.format(i)}/qlog.bz2" for i in seg1]
keys += [f"{self.seg_format2.format(i)}/qlog.bz2" for i in seg2]
keys += [f"boot/{self.seg_format.format(i)}.zst" for i in seg1]
keys += [f"boot/{self.seg_format2.format(i)}.zst" for i in seg2]
keys += [f"{self.seg_format.format(i)}/qlog.zst" for i in seg1]
keys += [f"{self.seg_format2.format(i)}/qlog.zst" for i in seg2]
return keys
def test_upload(self):
@@ -159,7 +159,7 @@ class TestUploader(UploaderTestCase):
self.join_thread()
for f_path in f_paths:
fn = f_path.with_suffix(f_path.suffix.replace(".bz2", ""))
fn = f_path.with_suffix(f_path.suffix.replace(".zst", ""))
uploaded = UPLOAD_ATTR_NAME in os.listxattr(fn) and os.getxattr(fn, UPLOAD_ATTR_NAME) == UPLOAD_ATTR_VALUE
assert not uploaded, "File upload when locked"