stats: sunnylink integration (#1454)

* sunnylink: add statsd process and related telemetry logging infrastructure

- Introduced `statsd_sp` process for handling Sunnylink-specific stats.
- Enhanced metrics logging with improved directory structure and data handling.

* sunnylink: re-enable and refine stat_handler for telemetry processing

- Reactivated `stat_handler` thread with improved path handling.
- Made `stat_handler` more flexible by allowing directory injection.

* statsd: fix formatting issue in telemetry string generation

- Corrected missing comma between `sunnylink_dongle_id` and `comma_dongle_id`.

* update statsd_sp process configuration for enhanced readiness logic

- Modified `statsd_sp` initialization to include `always_run` alongside `sunnylink_ready_shim`.
- Ensures robust process activation conditions.

* refactor(statsd): enhance and unify StatLogSP implementation

- Replaced custom `StatLogSP` in sunnylink with centralized implementation from `system.statsd`.
- Ensures consistent logic for StatLogSP handling across modules.

* fix

* refactor(statsd): add intercept parameter to StatLogSP for configurable logging

- Introduced optional `intercept` parameter to `StatLogSP` to manage `comma_statlog` initialization.
- Updated usage in `sunnylink` to disable interception where unnecessary.

* Dont complain

* feat(statsd): add raw metric type and SunnyPilot-specific stats collection

- Introduced `METRIC_TYPE.RAW` for base64-encoded raw data metrics.
- Added `sp_stats` thread to export SunnyPilot params as raw metrics.
- Enhanced telemetry handling with decoding and serialization updates.

* refactor(statsd): improve `sp_stats` error handling and param processing

- Enhanced exception handling for `params.get` to prevent crashes.
- Added support for nested dict values to be included in stats.

* refactor(statsd): adjust imports and minor code formatting updates

- Updated `Ratekeeper` import path for consistency with the `openpilot` module structure.
- Fixed minor formatting for improved readability.

* refactor(statsd): update typings and remove unused NoReturn annotation

- Removed unnecessary `NoReturn` typing for `stats_main` to simplify function definition.
- Adjusted `get_influxdb_line_raw` to refine typing for `value` parameter.

* cleanup

* init

* init

* slightly more

* staticmethod

* handle them all

* get them models

* log with route

* more

* car

* Revert "car"

This reverts commit fe1c90cf4d422864388e4a73d5cd25c9e585a9b3.

* handle capnp

* Revert "handle capnp"

This reverts commit c5aea6880333863838a7c258541fb7a2812f2e2b.

* 1 more time

* Revert "1 more time"

This reverts commit a364474fa580d9ef9760a8896f9db76d526abead.

* Cleaning to expose wider

* feat(interfaces, statsd): log car params to stats system

- Added `STATSLOGSP` import and logging to capture `carFingerprint` in metrics.
- Improved error handling in `get_influxdb_line_raw` for robust metric generation.

* refactor(interfaces): streamline car params logging to stats

- Simplified logging by directly converting `CP` to a dictionary.
- Removed legacy stats aggregation for clarity.

* feat(sunnylink): enable compression for stats in SunnyLink

- Added optional compression for stats payload to support large data.
- Updated `stat_handler` to handle compression and base64 encoding.

* fix(statsd): filter complex types in `get_influxdb_line_raw`

- Skips unsupported types (dict, list, bytes) to prevent formatting errors.
- Simplifies type annotation for `value` parameter.

* fix(statsd): use `json.dumps` for string conversion in `get_influxdb_line_raw`

- Ensures proper handling of special characters in values.
- Prevents potential formatting issues with raw `str()` conversion.

* refactor(interfaces, statsd): update parameter keys for stats logging

- Renamed logged keys for better clarity (`sunnypilot_params` → `sunnypilot.car_params`, `device_params`).
- Ensures consistency across data logs.

* bet

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>

system/athena/athenad.py

@@ -744,26 +744,40 @@ def log_handler(end_event: threading.Event, log_attr_name=LOG_ATTR_NAME) -> None
       cloudlog.exception("athena.log_handler.exception")
 
 
-def stat_handler(end_event: threading.Event) -> None:
-  STATS_DIR = Paths.stats_root()
+def stat_handler(end_event: threading.Event, stats_dir=None, is_sunnylink=False) -> None:
+  stats_dir = stats_dir or Paths.stats_root()
   last_scan = 0.0
 
   while not end_event.is_set():
     curr_scan = time.monotonic()
     try:
       if curr_scan - last_scan > 10:
-        stat_filenames = list(filter(lambda name: not name.startswith(tempfile.gettempprefix()), os.listdir(STATS_DIR)))
+        stat_filenames = list(filter(lambda name: not name.startswith(tempfile.gettempprefix()), os.listdir(stats_dir)))
         if len(stat_filenames) > 0:
-          stat_path = os.path.join(STATS_DIR, stat_filenames[0])
+          stat_path = os.path.join(stats_dir, stat_filenames[0])
           with open(stat_path) as f:
+            payload = f.read()
+            is_compressed = False
+
+            # Log the current size of the file
+            if is_sunnylink:
+              # Compress and encode the data if it exceeds the maximum size
+              compressed_data = gzip.compress(payload.encode())
+              payload = base64.b64encode(compressed_data).decode()
+              is_compressed = True
+
             jsonrpc = {
               "method": "storeStats",
               "params": {
-                "stats": f.read()
+                "stats": payload
               },
               "jsonrpc": "2.0",
               "id": stat_filenames[0]
             }
+
+            if is_sunnylink and is_compressed:
+              jsonrpc["params"]["compressed"] = is_compressed
+
             low_priority_send_queue.put_nowait(json.dumps(jsonrpc))
           os.remove(stat_path)
         last_scan = curr_scan