* Introduce Model Manager to handle downloads and verification
This commit introduces a new Model Manager responsible for handling model downloads, including driving and navigation application models. The manager also verifies file hashes and communicates download progress for an improved user experience.
The Model Manager is asynchronous and utilizes asyncio and aiohttp for enhanced performance, including robust error handling.
Impacted files in the 'cereal', 'common', 'sunnypilot', and 'system' directories have been updated accordingly. The 'ModelsFetcher' process configuration has been modified to run only when off-road, ensuring optimum resource management.
This update aims to enhance code clarity, improve performance, and streamline the handling of model downloads.
* "Update model management and fetching for SunnyPilot"
This update refactors the model management, downloading and cache verification mechanisms of SunnyPilot. New functionalities, such as smart cache handling, have been implemented in ModelFetcher in sunnypilot/models/model_fetcher.py. Also, the model downloading process has been moved to a separate async function _download_bundle in ModelManagerSP in sunnypilot/models/model_manager.py. Hash verification of files is now performed in an async function verify_file in sunnypilot/models/model_helper.py. Changes in system parameters related to model management have been reflected in system/manager/manager.py.
* Integrate download ETA calculations in model manager
This commit introduces a new feature that calculates and tracks the Estimated Time of Arrival (ETA) for downloading models in the model manager component. The 'eta' property in the 'DownloadProgress' structure in 'custom.capnp' is changed from 'Float32' to 'UInt32'.
In the 'model_manager.py' file, a new method '_calculate_eta' has been added to perform ETA calculations. An additional dictionary '_download_start_times' has been created to keep track of the start time of each model download. The ETA is calculated every time a portion of the model file is downloaded, and it gets updated in the 'DownloadProgress' structure. Finally, the start time is cleared after the download completes.
In 'model_manager_audit.py', an additional check is added to only print downloadProgress for the downloads currently in progress.
* format
* no default model cache {} because it can be considered a valid json, we do not want that
* Refactor typing annotations to use PEP 604 syntax.
Updated type hints to adopt PEP 604 union syntax (`X | None`) and replaced `List` and `Dict` with modern built-in `list` and `dict`. This change improves consistency and readability while aligning with Python 3.10+ standards.
* Simplify logging messages and remove unused imports.
Removed an unused import from `model_manager.py` to improve clarity and maintainability. Also refined a log message in `model_fetcher.py` by removing unnecessary formatting for consistency.
* Refactor model handling and simplify cache fallback logic.
Updated type annotations for `selected_bundle` in `model_manager.py` for clarity. Streamlined cache fallback logic in `model_fetcher.py` by removing redundant conditionals while preserving functionality. These changes improve code readability and maintainability.
* "Fix formatting for ModelManager_DownloadIndex retrieval
Condensed parameter alignment in the get method for improved readability and adherence to style guidelines. This change does not affect functionality but ensures consistent code formatting."
* Need to have main defined for process_config to be able to run it
* Refactor model management to support active bundle tracking
Introduce the concept of an active model bundle with a new persistent parameter and API updates. Added fields for `generation` and `environment` in model metadata, improved caching, and updated methods to manage active model states efficiently.
* UI commit (#515)
* Refactor model management to support active bundle tracking
Introduce the concept of an active model bundle with a new persistent parameter and API updates. Added fields for `generation` and `environment` in model metadata, improved caching, and updated methods to manage active model states efficiently.
* Add new driving model selection feature to settings
This commit introduces a new feature to the settings that allows users to select different driving models. It fetches available models and displays their download progress. The created UI also suggests a calibration reset after model download. The changes include the creation of 'SoftwarePanelSP' within 'settings.' Additionally, 'sunnypilot/SConscript' has been updated to include 'settings.cc' and 'software_panel.cc'. Changes also include localization for this feature.
* Show model description during download status
This update ensures the model description is displayed when a model is in the downloading state. It improves the user interface by providing real-time feedback during the download process.
* Update translations for multiple languages
Added new and updated translation strings in several language files, including Spanish, Arabic, Chinese (Simplified and Traditional), Turkish, Korean, Thai, Japanese, and Brazilian Portuguese. These updates include placeholder translations for new UI elements and features.
* Refactor model name handling and add generation check.
Replaced `bundleName` with `model_name` for better clarity in status messages. Added a generation mismatch check before showing the reset parameters dialog to avoid unnecessary prompts.
* Update model handling in SoftwarePanelSP
Remove unused "common/model.h" and replace "CURRENT_MODEL" with "..." as the default return value in GetModelName. Adjust logic to check for active bundles instead of selected bundles for improved accuracy. Minor text change for clarity in UI label.
* Rename `GetModelName` to `GetActiveModelName` for clarity.
The new name better reflects the function's purpose of retrieving the active model name, improving code readability. All relevant calls and references have been updated to ensure consistency across the codebase.
* Update sunnypilot/models/model_helper.py
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
* Refactor download status handling and add 'cached' state
Introduce 'cached' as a new download status and adjust relevant logic across components to support it. Simplify and streamline model status handling in the software panel for better readability and maintainability. Ensure consistent status reporting for all model types.
* Update translations for multiple languages
Refined and expanded translations across various languages, replacing placeholders with meaningful text. This improves clarity and user experience in the multilingual interface.
* Update terminology from 'bundle' to 'model' in UI texts
Replaced occurrences of 'bundle' with 'model' in button labels, dialog titles, and messages in the SoftwarePanelSP code. This improves clarity and aligns terminology with current functionality.
* Update translation placeholders for model fetching texts
Replaced "Fetching bundles" with "Fetching models" across multiple languages to align text placeholders with the updated functionality. Adjusted related background download messages for clarity and consistency.
* cleanup
* not used, and likely not needed
---------
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
* cleaning up
* Update system/manager/process_config.py
* Simplify model parsing and index handling logic.
Refactored `ModelManager_DownloadIndex` retrieval to use the walrus operator, streamlining the conditional logic. Additionally, restructured model list initialization in `_parse_bundle` for improved readability and maintainability. These changes enhance code clarity and reduce redundancy.
* `Improve error handling in model cache retrieval`
Revised the `get` method to ensure it returns an empty dictionary on errors or missing data, avoiding potential `None`-related issues. Added logging for clearer diagnostics when cached model data is unavailable or retrieval fails. This improves reliability and debuggability of the model fetching process.
* Fix cached model data handling by parsing JSON response
Previously, cached model data was returned as a raw string, causing potential issues when using the data. The change ensures the cached data is properly parsed into JSON format before returning, improving reliability and consistency.
* Adjust modelManagerSP rate and Ratekeeper frequency
Reduced the rate for modelManagerSP in services and aligned the Ratekeeper frequency in model_manager.py to 0.1.
* Update model fetcher URL and adjust modelManagerSP rate
Updated the model fetcher URL to point to the correct resource for driving models. Adjusted the rate of modelManagerSP in both its service definition and the corresponding Ratekeeper initialization to 1 Hz for improved consistency.
* Refactor model download logic for clarity and efficiency
Simplify the logic for finding the model to download by combining redundant constructs into a single line. This improves code readability and reduces unnecessary variable assignments.
* Fix cache keys for manual prebuilt actions because they were missing the cache when manually built
* no need to log
* formatting
* revert ci changes
* Refactor and restructure `modeld` to `models` module.
Renamed `modeld` directory to `models` for clarity and consistency. Updated all references and imports to reflect the new structure. This improves maintainability and aligns with naming conventions.
---------
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* Add support for GitHub Actions runner management
Introduce a new process and configuration to manage a GitHub Actions runner. Added a persistent "EnableGithubRunner" parameter and a script to control the runner service. Integrated the feature into the system's process manager logic.
* Restrict GitHub runner usage on metered networks.
This update modifies the `use_github_runner` function to include a check for metered networks using `HARDWARE.get_network_metered()`. This ensures the GitHub runner is not enabled when the network is metered, improving network usage efficiency.
* Add GitHub runner service toggle to developer panel
Introduces a new toggle in the developer panel to enable or disable the GitHub runner service. This provides developers with a convenient way to control the service from the UI.
* translations
* Refactor network condition check for GitHub runner.
Updated `use_github_runner` to include network type in metered network checks. This ensures more accurate validation by considering specific network types when determining metered status.
* Mark as executable
* Update paths and shebangs for consistency across scripts
Modified file paths to align with the new directory structure under `/data/media/0/github` and updated the shebang in `github_runner.sh` for better environment compatibility. Adjusted the `BUILD_DIR` in the GitHub workflow to reflect the new path.
* Fix string translation for GitHub runner parameter text
Added translation support for the GitHub runner service description text. This ensures consistent localization across the UI.
* Remove gitlab_runner.sh from Sunnypilot blacklist.
The script is no longer required to be blacklisted, likely due to updates or changes in its usage. This improves the maintainability of the blacklist by removing unnecessary entries.
* lang stuff
* Update BASE_DIR determination based on mount point
Refactored scripts to dynamically set BASE_DIR depending on whether /data/media is a valid mount point. This ensures compatibility with different environments and improves robustness of path resolution.
* Refactor GitHub runner logic in process_config.
Simplified enabling conditions for the GitHub runner by removing dependency on hardware network checks and adding a logical combination of offroad and runner-related functions. This improves code readability and reduces hardware coupling.
* needs cleanup
* only if tici
* bump tinygrad
* check width
* base modelframe
* .
* need to be args
* more cleanup
* no _frame in base
* tici only
* its DrivingModelFrame
* .6 is fair
---------
Co-authored-by: Comma Device <device@comma.ai>
* No /dev/shm on MacOS so we go to tmp instead given that the /dev/shm makes tests not really work fine on mac
* Cleanups
Fix cereal messaging tests
* keep msgq stock
More shm updates for macos
* typo
* fix return
* Semicolon...
* add longitudinal profiles
* stash
* unfortunately even longitudinalPlan causes circle
* add to process config
* reach target speed smoothly
* stash
* works
* clean up
* debug alert
* rename
* fix
* better text
* toggle via exp button
* try coming to a stop better, smoother target reaching
* closer to target
* revert controlsd migration
* add description to alert
* generate report from local logs
* hide bad maneuvers
* pdflike
* Revert "pdflike"
This reverts commit 6d4af1bf9be2e9e0798eaecf026a53d860da7613.
* try this
* use alert manager
* fix that check
* wat
* Revert "wat"
This reverts commit 93d0d27ab838d3f580d06ff212f380e0b912d599.
* some clean up
* rm
* cleanup
* move
* fix test
* more fix
* clean up
* fix that
* Move vCruise to card
* cleanup drive_helpers
* works in sim
* update refs
* update that
* too slow :(
old-commit-hash: 2f3256ed8baa3d7a5c4d1befe9148dbfecb37246
* GPS helpers
* Use GPS socket in timed
* Use helper in controlsd
* Use updated and logMonoTime
* Helper type annot
* Not updated
* Increase time threashold
old-commit-hash: cf355ecf20b546e6c6caca40936fe7505d5f8b7f
* ban all of common & copy numpy_fast
* add numpy_fast
* these are okay
* and ban controls
* better name
* Conversions
* do utils, kalman
* clean up
* sorting
* don't forget
old-commit-hash: 8d961a12e5b033855f2abf5d8bd9c51374f74c6e
* car logger
* try this
* fix double printing!
* clean up
* need to import where we actually use it (card won't be in submodule)
* NullHandler does not forward everything properly, need propagate=False to stop root logger from printing
* card is staying
* set up
* forward
old-commit-hash: 12a7b7f7fb1d3a275314b0232795942a7ff3b955
* all
* right sz
* what is this
* Revert "what is this"
This reverts commit 3ff771926f109452a055dfb61cee06b34e498164.
* ok
* big
* update intrs
* fl
* more specifuc
* fish
* revert
* revert
---------
Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 3ccc63deec85f454521d465c4e0e3438e76cb547
* use timezone from tilequery mapbox api
* add timezone into cereal
* better clean
* more
* one line
* poetry lock
old-commit-hash: 7013eed50cd350a9b329280b0eccfc9db051a72b
* remove navigation
* lil more
* fix cabana
* revert for now
* fix docs
* move that over
* lock
* rm assets
old-commit-hash: e491c9d964a2aef07eeaa7b47f4aa9f992130482
* Custom setproctitle (#32667)
* add custom setproctitle
* add test
* Update poetry.lock
* fix lint
* support only Linux
* test only Linux
* final lint
* Update test_setproctitle.py
* Update setproctitle.py
* convert to threadnames
* delete proctitles
* Check str len and use PR_GET_NAME
* fix poetry.lock
* lint fix
* Update common/threadname.py
---------
Co-authored-by: reddyn12 <nikhilr.ssm@gmail.com>
Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
* revert that for now
* use last 15
* fix
* use name
* update those
* and modeld
* rm
---------
Co-authored-by: schlimeszn <138847413+schlimeszn@users.noreply.github.com>
Co-authored-by: reddyn12 <nikhilr.ssm@gmail.com>
Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 83ac80c6c899dbf4b5b2d422947b02b57d83bccd
* Replace markdown-it-py with small function in common/
* simple test
* unused
* lock
* linting fixes
old-commit-hash: 5265d13fe087972311592febd4071c16cece65e0
* it's something
* backup
* 16:10
* cleanup
* this is fine
* close
* remove some junk
* no heck
* disos
* real 10
* for some reason this is flipped
* 20hz
* no return
* ae
* tear
* need curve laster
* correct real gains
* fix time
* cleanup
* why the scam
* disable for now
* 0.7
* hdr
* that doesnt work
* what
* hugeoof
* clean up
* cleanup
* fix regs
* welp cant
* is this corrent
* it is sq
* remove
* back
* stg10bit
* back2ten
* Revert "remove"
This reverts commit 18712ab7e103c12621c929cd0f772ecb9b348247.
* 20hz and swb
* correct height
* 10bit
* ui hack for now
* slight
* perfect
* blk64
* ccm
* fix page faults
* template
* set 4x
* is this fine
* try
* this seems to work
* Revert "this seems to work"
This reverts commit d3c9023d3f14bd9394fed2d6276dba777ed0e606.
* needs to be static
* close
* 64 is optimal
* 2
* take
* not 1
* offset
* whats going on
* i have no idea
* less resistence
* box defs
* no
* reduce blur artifacts
* simplify
* fix
* fake short is too much for bright
* can be subzero
* should not use lsvc
* no wasted bit
* cont no slow
* no less than 10bit
* it is based
* wrong
* right
* quart
* shift
* raise noise floor
* 4.5/4.7
* same ballpark
* int is fine
* shane owes me m4a4
* Revert "shane owes me m4a4"
This reverts commit b4283fee18efebedae628a6cfd926ff1416dcfe5.
* back
* Revert "4.5/4.7"
This reverts commit e38f96e90cb5370bd378f6b66def9e7e3ed0ce5d.
* default
* oof
* clean up
* simpilfy
* from sensorinfo
* no div
* better name
* not the wrong one
* not anymore relevant
* too
* not call it debayer
* cl headers
* arg is 2nd
* gone is is_bggr
* define
* no is hdr
* rgb_tmp
* p1
* clean up
* 4
* cant for
* fix somewhre else
* const
* ap
* rects
* just set staruc
* nnew tmp
* pull it for now
* 12
* common rect
* Revert "not anymore relevant"
This reverts commit 1d574673a16cc31b7a255609e07775c3579eef15.
* Revert "too"
This reverts commit c2d4dcc52a859fe799362f9fcc2ffda99b264e50.
* Revert "Revert "too""
This reverts commit 0abbabe1fde51592f1619058638b4ac6a6dee4b3.
* no tol is fine
* rename
* sensor id
* unsgin
* flag
* some linalg
* cast
* should be h ref
* cap
---------
Co-authored-by: Comma Device <device@comma.ai>
old-commit-hash: 331c7b103a7e4349c75be373589fc68161e69500
* permanent
* param
* correct behavior
* toggle
* need trans
* ref_commit
* translate for chs/t
* disable on P and R
* read
old-commit-hash: 1d05704d27dbe4f73c0eba7e8dad2ae84d4e205e
DevTekVE