* Introduce Model Manager to handle downloads and verification
This commit introduces a new Model Manager responsible for handling model downloads, including driving and navigation application models. The manager also verifies file hashes and communicates download progress for an improved user experience.
The Model Manager is asynchronous and utilizes asyncio and aiohttp for enhanced performance, including robust error handling.
Impacted files in the 'cereal', 'common', 'sunnypilot', and 'system' directories have been updated accordingly. The 'ModelsFetcher' process configuration has been modified to run only when off-road, ensuring optimum resource management.
This update aims to enhance code clarity, improve performance, and streamline the handling of model downloads.
* "Update model management and fetching for SunnyPilot"
This update refactors the model management, downloading and cache verification mechanisms of SunnyPilot. New functionalities, such as smart cache handling, have been implemented in ModelFetcher in sunnypilot/models/model_fetcher.py. Also, the model downloading process has been moved to a separate async function _download_bundle in ModelManagerSP in sunnypilot/models/model_manager.py. Hash verification of files is now performed in an async function verify_file in sunnypilot/models/model_helper.py. Changes in system parameters related to model management have been reflected in system/manager/manager.py.
* Integrate download ETA calculations in model manager
This commit introduces a new feature that calculates and tracks the Estimated Time of Arrival (ETA) for downloading models in the model manager component. The 'eta' property in the 'DownloadProgress' structure in 'custom.capnp' is changed from 'Float32' to 'UInt32'.
In the 'model_manager.py' file, a new method '_calculate_eta' has been added to perform ETA calculations. An additional dictionary '_download_start_times' has been created to keep track of the start time of each model download. The ETA is calculated every time a portion of the model file is downloaded, and it gets updated in the 'DownloadProgress' structure. Finally, the start time is cleared after the download completes.
In 'model_manager_audit.py', an additional check is added to only print downloadProgress for the downloads currently in progress.
* format
* no default model cache {} because it can be considered a valid json, we do not want that
* Refactor typing annotations to use PEP 604 syntax.
Updated type hints to adopt PEP 604 union syntax (`X | None`) and replaced `List` and `Dict` with modern built-in `list` and `dict`. This change improves consistency and readability while aligning with Python 3.10+ standards.
* Simplify logging messages and remove unused imports.
Removed an unused import from `model_manager.py` to improve clarity and maintainability. Also refined a log message in `model_fetcher.py` by removing unnecessary formatting for consistency.
* Refactor model handling and simplify cache fallback logic.
Updated type annotations for `selected_bundle` in `model_manager.py` for clarity. Streamlined cache fallback logic in `model_fetcher.py` by removing redundant conditionals while preserving functionality. These changes improve code readability and maintainability.
* "Fix formatting for ModelManager_DownloadIndex retrieval
Condensed parameter alignment in the get method for improved readability and adherence to style guidelines. This change does not affect functionality but ensures consistent code formatting."
* Need to have main defined for process_config to be able to run it
* Refactor model management to support active bundle tracking
Introduce the concept of an active model bundle with a new persistent parameter and API updates. Added fields for `generation` and `environment` in model metadata, improved caching, and updated methods to manage active model states efficiently.
* UI commit (#515)
* Refactor model management to support active bundle tracking
Introduce the concept of an active model bundle with a new persistent parameter and API updates. Added fields for `generation` and `environment` in model metadata, improved caching, and updated methods to manage active model states efficiently.
* Add new driving model selection feature to settings
This commit introduces a new feature to the settings that allows users to select different driving models. It fetches available models and displays their download progress. The created UI also suggests a calibration reset after model download. The changes include the creation of 'SoftwarePanelSP' within 'settings.' Additionally, 'sunnypilot/SConscript' has been updated to include 'settings.cc' and 'software_panel.cc'. Changes also include localization for this feature.
* Show model description during download status
This update ensures the model description is displayed when a model is in the downloading state. It improves the user interface by providing real-time feedback during the download process.
* Update translations for multiple languages
Added new and updated translation strings in several language files, including Spanish, Arabic, Chinese (Simplified and Traditional), Turkish, Korean, Thai, Japanese, and Brazilian Portuguese. These updates include placeholder translations for new UI elements and features.
* Refactor model name handling and add generation check.
Replaced `bundleName` with `model_name` for better clarity in status messages. Added a generation mismatch check before showing the reset parameters dialog to avoid unnecessary prompts.
* Update model handling in SoftwarePanelSP
Remove unused "common/model.h" and replace "CURRENT_MODEL" with "..." as the default return value in GetModelName. Adjust logic to check for active bundles instead of selected bundles for improved accuracy. Minor text change for clarity in UI label.
* Rename `GetModelName` to `GetActiveModelName` for clarity.
The new name better reflects the function's purpose of retrieving the active model name, improving code readability. All relevant calls and references have been updated to ensure consistency across the codebase.
* Update sunnypilot/models/model_helper.py
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
* Refactor download status handling and add 'cached' state
Introduce 'cached' as a new download status and adjust relevant logic across components to support it. Simplify and streamline model status handling in the software panel for better readability and maintainability. Ensure consistent status reporting for all model types.
* Update translations for multiple languages
Refined and expanded translations across various languages, replacing placeholders with meaningful text. This improves clarity and user experience in the multilingual interface.
* Update terminology from 'bundle' to 'model' in UI texts
Replaced occurrences of 'bundle' with 'model' in button labels, dialog titles, and messages in the SoftwarePanelSP code. This improves clarity and aligns terminology with current functionality.
* Update translation placeholders for model fetching texts
Replaced "Fetching bundles" with "Fetching models" across multiple languages to align text placeholders with the updated functionality. Adjusted related background download messages for clarity and consistency.
* cleanup
* not used, and likely not needed
---------
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
* cleaning up
* Update system/manager/process_config.py
* Simplify model parsing and index handling logic.
Refactored `ModelManager_DownloadIndex` retrieval to use the walrus operator, streamlining the conditional logic. Additionally, restructured model list initialization in `_parse_bundle` for improved readability and maintainability. These changes enhance code clarity and reduce redundancy.
* `Improve error handling in model cache retrieval`
Revised the `get` method to ensure it returns an empty dictionary on errors or missing data, avoiding potential `None`-related issues. Added logging for clearer diagnostics when cached model data is unavailable or retrieval fails. This improves reliability and debuggability of the model fetching process.
* Fix cached model data handling by parsing JSON response
Previously, cached model data was returned as a raw string, causing potential issues when using the data. The change ensures the cached data is properly parsed into JSON format before returning, improving reliability and consistency.
* Adjust modelManagerSP rate and Ratekeeper frequency
Reduced the rate for modelManagerSP in services and aligned the Ratekeeper frequency in model_manager.py to 0.1.
* Update model fetcher URL and adjust modelManagerSP rate
Updated the model fetcher URL to point to the correct resource for driving models. Adjusted the rate of modelManagerSP in both its service definition and the corresponding Ratekeeper initialization to 1 Hz for improved consistency.
* Refactor model download logic for clarity and efficiency
Simplify the logic for finding the model to download by combining redundant constructs into a single line. This improves code readability and reduces unnecessary variable assignments.
* Fix cache keys for manual prebuilt actions because they were missing the cache when manually built
* no need to log
* formatting
* revert ci changes
* Refactor and restructure `modeld` to `models` module.
Renamed `modeld` directory to `models` for clarity and consistency. Updated all references and imports to reflect the new structure. This improves maintainability and aligns with naming conventions.
---------
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* Add script for squash and merge workflow automation
Introduced `squash_and_merge.py` to automate the process of squashing a source branch and merging it into a target branch. The script handles backups, commit message creation, stashing, cleanup, and includes optional push functionality. Provides user prompts and error handling to simplify complex merge workflows.
* temp1
* Switch to PEP 604 style for optional and generic type hints
Replaces `Optional` and `List` with the simpler `X | None` and `list[X]` syntax introduced in Python 3.10. Updated all function signatures accordingly for consistency and modernity. Also made minor string formatting consistency adjustments.
* Fix type annotation for temp_branch variable
Updated the temp_branch variable to include a proper type annotation (`str | None`). This change improves code clarity and aligns with Python typing standards.
* more hints
* add sp flag
* return if sp macro
* controls
* drive stats
* some more
* more
* skip the whole thing
* missed
* more
* more main
* not ready to push yet but sure
* get them icons
* later
* own icons
* Revert "own icons"
This reverts commit e07bd8e670.
* blank icon
* render differently on mac
* static
* set toggle and title space properly
* format
* remove prior objects
* good spacing for ButtonParamControlSP
* more space formatting
* on device fix
* update home wifi prompt
* handle AbstractControlSP_SELECTOR hide events properly
* use sp
* ignore name
* more
* keep spacing
* better flag
* settings button moved
* small cleanup
* move files to sp dir
* remove for now
* developer different icon
* add scrolling to ui test
* Revert "add scrolling to ui test"
This reverts commit c8d1c65d89.
* format
* make them virtual
* do we still need this?
* Apply suggestions from code review
Co-authored-by: DevTekVE <devtekve@gmail.com>
* revert for now
* shorter license
---------
Co-authored-by: DevTekVE <devtekve@gmail.com>
* Add CLion run configs and external tools setup
Introduced CLion configuration files for "Build Debug" and "Build Release," along with external tool definitions for Poetry SCons commands. Additionally, commented out `.idea` in `.gitignore` to track necessary project-specific settings.
* rename
* only exclude specific files from gitignore
* wildcard
---------
Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
* Add support for GitHub Actions runner management
Introduce a new process and configuration to manage a GitHub Actions runner. Added a persistent "EnableGithubRunner" parameter and a script to control the runner service. Integrated the feature into the system's process manager logic.
* Restrict GitHub runner usage on metered networks.
This update modifies the `use_github_runner` function to include a check for metered networks using `HARDWARE.get_network_metered()`. This ensures the GitHub runner is not enabled when the network is metered, improving network usage efficiency.
* Add GitHub runner service toggle to developer panel
Introduces a new toggle in the developer panel to enable or disable the GitHub runner service. This provides developers with a convenient way to control the service from the UI.
* translations
* Refactor network condition check for GitHub runner.
Updated `use_github_runner` to include network type in metered network checks. This ensures more accurate validation by considering specific network types when determining metered status.
* Mark as executable
* Update paths and shebangs for consistency across scripts
Modified file paths to align with the new directory structure under `/data/media/0/github` and updated the shebang in `github_runner.sh` for better environment compatibility. Adjusted the `BUILD_DIR` in the GitHub workflow to reflect the new path.
* Fix string translation for GitHub runner parameter text
Added translation support for the GitHub runner service description text. This ensures consistent localization across the UI.
* Remove gitlab_runner.sh from Sunnypilot blacklist.
The script is no longer required to be blacklisted, likely due to updates or changes in its usage. This improves the maintainability of the blacklist by removing unnecessary entries.
* lang stuff
* Update BASE_DIR determination based on mount point
Refactored scripts to dynamically set BASE_DIR depending on whether /data/media is a valid mount point. This ensures compatibility with different environments and improves robustness of path resolution.
* Refactor GitHub runner logic in process_config.
Simplified enabling conditions for the GitHub runner by removing dependency on hardware network checks and adding a logical combination of offroad and runner-related functions. This improves code readability and reduces hardware coupling.
Update version identifier in workflow for staging and release
Appended "-staging" and "-release" to the VERSION variable in the GitHub Actions workflow to clearly distinguish build types. This ensures clarity in versioning for staging and release environments.
* Add support for custom model naming in workflow
This update introduces a new optional input `custom_name` to the `sunnypilot-build-model` GitHub workflow. If provided, it customizes the artifact name; otherwise, the default naming scheme is used. This enhances flexibility for build artifact identification.
* fix name
* Update cache key logic in build workflow
Replaced `github.ref_name` with a fallback to `github.head_ref || github.ref_name` in cache keys to handle branches and pull requests more reliably. This ensures consistent cache retrieval and improves workflow efficiency.
* Simplify artifact upload logic in CI workflow
Consolidated redundant artifact upload steps into a single action, leveraging a fallback mechanism for naming. This streamlines the workflow, reducing duplication and improving maintainability.
* Make publish step resilient and add conditional notify
Added `continue-on-error` to the publish step to allow workflow progression even if it fails. Introduced a condition to notify step that depends on publish outcome being successful. These changes improve workflow reliability and error handling.
* Fix conditional syntax in GitHub Actions workflow
Updated the conditional statement in the notify job to use the correct GitHub Actions expression syntax. This ensures proper evaluation of the publish step's success outcome.
* Update condition for notify step in GitHub Actions workflow
Replaced the deprecated syntax `${{ steps.publish.outcome == 'success' }}` with the recommended `success()` function. This ensures compatibility with newer GitHub Actions features and improves maintainability.
* Update build workflow to handle PR drafts and improve concurrency
Added support for pull request draft events to trigger the workflow, ensuring drafts are not missed. Removed `continue-on-error` in the publish step to improve reliability and prevent silent failures. Simplifies and enhances workflow robustness.
* Update pull request workflows and add draft check logic
Removed unnecessary pull request event types and drafts settings, consolidating logic. Added a condition to skip publishing for non-draft pull requests, ensuring better control over the deployment process.
* Create a GitHub Actions workflow for synchronizing the repository to GitLab and enhance GitLab CI settings
A new GitHub Actions workflow for mirroring the current repository to GitLab is added. This workflow is triggered by both push and delete events in addition to manual triggering. The role of GitHub actions runner is defined through a series of steps. A new '.gitlab-ci.yml' build configuration file is also introduced, with a more comprehensive definition of variables, jobs, and pipeline rules for its utilization in the GitLab CI/CD pipeline.
Further, other changes include the addition of scripts for installing and uninstalling GitLab CI runner, along with modifying the SCons build system configuration file to include custom cache directory. Moreover, 'release_files.py' has been revised to include additional blacklisted and whitelisted files specific to Sunnypilot, ensuring suitable settings for the CI flow.
The improvements facilitate a smoother integration between GitHub and GitLab, powerfully harnessing the capabilities of both platforms for more efficient and effective CI/CD pipelines and version control management.
* not needed for this
* Update workflow to build model from upstream repository
Revised the CI workflow to build directly from the upstream `commaai/openpilot` repository. Simplified configuration, removed unused steps, and added support for specifying the upstream branch dynamically via inputs.
* Update SConstruct to allow passing arbitrary cache_dir
Modified the SConstruct file to enable setting a custom cache directory via arguments. This enhances flexibility in configuring cache paths during the build process.
* Refactor build workflow to improve branch handling logic.
Reorganized conditions for setting environment variables, replacing repository_dispatch with workflow_dispatch for prebuilt builds. Added a fallback error message for unsupported triggers to improve robustness. This enhances clarity and ensures compatibility with the intended workflow triggers.
* test test test
* test test test
* test test test
* Enable publication flag during build configurations
Added `SHOULD_PUBLISH=true` to all relevant build configuration steps to ensure proper handling of publishing logic. Updated environment variables to include this flag for downstream usage. Removed the error-check step for unsupported configurations.
* Simplify publish condition in workflow logic
Replaced the previous condition for publishing with a single output variable, `should_publish`, to streamline logic and improve maintainability. This change reduces redundancy and makes the workflow more adaptable to future updates.
* Simplify restore key patterns in build workflow.
Removed unnecessary trailing dashes in SCons cache restore keys to streamline and slightly improve key matching logic. This ensures better consistency with the current workflow setup.
* Update cache key usage in build workflow
Replaced `github.ref_name` with `github.head_ref` for cache keys to ensure accurate branch-specific caching. Added fallback restore-keys for master branches to improve cache efficiency and reduce redundant builds.
* Improved debug logging in GitHub actions workflow
This commit refines the debug logging in our GitHub Actions workflow for the Sunnypilot build. This change provides more granularity, enabling logging only during debug mode, which helps to keep the runtime logs less cluttered during normal operations. This includes the conditionally displaying of environmental variables, GitHub output contents, and directory listings.
Additionally, debug mode verbosity was added to rsync commands to aid troubleshooting file transfers during the build process.
Blank lines were also reduced for better readability and cleaner code presentation.
* test diff path
* Refactor SCons cache handling in build workflow
Replaced hardcoded cache directory paths with an environment variable (`SCONS_CACHE_DIR`) for better maintainability and flexibility. Updated related workflow steps to utilize the new variable and adjusted cache key usage. Removed unused `BASE_BUILD_NUMBER` variable to clean up the configuration.
* Update cache key to include commit SHA in workflow
This change adds the commit SHA to the cache key in the GitHub Actions workflow. It ensures more precise caching by differentiating builds based on the specific commit, reducing potential conflicts.
* clean
Update GitHub runner service to set environment variables
Updated the ExecStart command to explicitly set HOME, USER, LOGNAME, and MAIL environment variables. This ensures the runner operates with the correct environment configuration, improving reliability and compatibility.
Refactor GitHub runner service ExecStart command.
Replaced direct command execution with running the service as a specific user using `su`. This improves compatibility and aligns with best practices for user-based execution. No functional changes are expected.
Update GitHub runner service to set environment variables
The ExecStart command now sets HOME, USER, LOGNAME, and MAIL environment variables for the runner process. This ensures proper environment initialization for the designated user, improving compatibility and reliability during execution.
Refactor GitHub runner service template handling.
Revised the `modify_service_template` function to create a properly structured service template for the GitHub runner. Updated service permissions, execution parameters, and enabled the function call to ensure usage during runner setup.
* Refactor GitHub Runner installer to improve argument parsing.
Reworked the script to implement flexible and explicit command-line argument parsing using flags like `--token`, `--repo`, and `--start-at-boot`. Added support for setting default values and enabling/disabling auto-start based on the `--start-at-boot` flag. Improved error handling and usage messaging for better user experience.
* Update cache keys in sunnypilot build workflow
Modified the cache keys to include `github.ref_name` for more precise caching and restore behavior. This improves build consistency by better differentiating between branches and refs. No changes to the overall workflow logic.
* Remove 'tinygrad/*' from release file exclusions
This change modifies the release file exclusions by removing 'tinygrad/*'. The adjustment ensures that files in the 'tinygrad' directory are now included in the release process, aligning with updated packaging requirements.
* Simplify release file exclusions list.
Removed redundant and unnecessary file patterns from the exclusions list in `release_files.py`. This streamlines the file handling process and reduces maintenance overhead.
* Refactor SCons cache path and key structure.
Updated the SCons cache directory path to `SCONS_CACHE_DIR` for clarity and consistency. Improved the caching key structure to include `github.head_ref` for better cache differentiation and restore hierarchy. Adjusted related build instructions to reflect the new environment variable.
* Refactor branch configuration in CI workflow
Standardize branch name handling by replacing hardcoded values with environment variables. This improves maintainability and simplifies updates to branch names across the workflow. Updated references to use the new dynamic environment variable approach.
set python upper bound to 3.13 (#34286)
* try
* test
* ...
* wow
* y
* docker
* <
(cherry picked from commit 9c9b273a3e)
Co-authored-by: Maxime Desroches <desroches.maxime@gmail.com>
Allows us to automatically keep our LFS in sync with comma's and also to manually perform a sync if we need to. Even able to sync the LFS from a given commit hash or a given branch. Useful for model stuff.
Jason Wen