mirror of
https://github.com/sunnypilot/sunnypilot.git
synced 2026-08-06 02:25:45 +08:00
e2bc0996ef868e94de608dd410010a5380e2d0e6
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 James Vecellio-Grant
|
6d41ce2032 |
ci: Refactor model building workflows (#1096)
* Tinygrad bump from sync-20250627 * bump tinygrad_repo * Reformat metadata generator to match driving_models.json * bump tinygrad * Revert "bump tinygrad" This reverts commit |
||
|
James Vecellio-Grant
|
c47aadee19 |
sunnypilot modeld: Sync Tinygrad (#1081)
* Tinygrad bump from sync-20250627 * bump tinygrad_repo * Reformat metadata generator to match driving_models.json * bump tinygrad * Revert "bump tinygrad" This reverts commit |
||
|
DevTekVE
|
3c36374bed |
ci: build upstream (tinnygrad) models from our pipeline (#498)
* Add job to fetch and upload model artifacts
Introduced a new `get_model` job to retrieve and upload model artifacts as part of the workflow. Updated the `build_model` job to download and clean up model files before building, ensuring an isolated and streamlined process. These changes improve modularity and artifact handling in the CI pipeline.
* Refine model file handling in build workflow
Restrict deletion and download commands to ".onnx" files to prevent unintended removal or overwriting of non-ONNX files. This ensures a more targeted and safer workflow for handling model artifacts.
* Update model artifact handling in build workflow
Changed artifact upload path to include only .onnx files and commented out the deletion of existing models. Also added a dependency for the build stage to ensure proper execution sequence.
* Update build workflow to refine model artifact handling
Removed unused code for deleting models and fixed the artifact path. Added a reference file logging custom name and upstream branch for better traceability. These changes streamline the workflow and improve clarity.
* Set dynamic run-name for sunnypilot build workflow
Added a customizable `run-name` field in the sunnypilot build model workflow. This allows more informative and dynamic naming based on input parameters such as `custom_name` or `upstream_branch`. Improves clarity and traceability in workflow runs.
* Add customization options and metadata generation to build
Introduce new inputs for file name prefix and 20Hz model specification in the workflow. Enable renaming of model files, generation of file hashes, and creation of a `metadata.json` file containing build details. This improves flexibility and enhances artifact information management.
* remove EOF
* No need for this hack anymore
* Add support for compiling ONNX models to Tinygrad format
* Add model directory environment variable and metadata generation script
* Fix quoting in model generator script command
* Refactor conditional flag for 20Hz input in build model script
* meh
* Add powersave management to build model workflow
* Add environment variable setup to build model workflow
* make it path call it a day
* kill me
* kill me x2
* mmm
* mmm
* Generate metadata.pkl
* Generate metadata.pkl
* ignore failure on getting model metadata
* Cleanup the existing onnxs
* Fix rm command syntax in build model script
* Cleaning a little
* Add optional file name parameter to generate_metadata function
* update
* stg
* linter
* more formatting
* Update sunnypilot model build workflow: improve custom naming, handle 20Hz default, and add commit date as output.
* Update commit date format in workflow script
Changed the commit date format in sunnypilot-build-model.yaml to a more human-readable format ('Month DD, YYYY'). This improves readability and consistency in the workflow outputs.
* Fix output naming in sunnypilot-build-model workflow
Corrected the output name from 'model_date' to 'commit_date' to match its usage. This ensures consistency and prevents potential errors in the workflow.
Add debug echo for model date in GitHub workflow
This change adds a debug echo statement to display the model date in the sunnypilot-build-model workflow. It helps verify the correct commit date is being set during the workflow process.
* Update BUILD_DIR path in sunnypilot workflow
Replaced hardcoded BUILD_DIR with a dynamic path using `github.workspace`. This enhances flexibility and ensures compatibility with different environments or runners.
* Revert "Update BUILD_DIR path in sunnypilot workflow"
This reverts commit
|
||
|
DevTekVE
|
898b7f28a8 |
Better model builder (#559)
* Add job to fetch and upload model artifacts
Introduced a new `get_model` job to retrieve and upload model artifacts as part of the workflow. Updated the `build_model` job to download and clean up model files before building, ensuring an isolated and streamlined process. These changes improve modularity and artifact handling in the CI pipeline.
* Refine model file handling in build workflow
Restrict deletion and download commands to ".onnx" files to prevent unintended removal or overwriting of non-ONNX files. This ensures a more targeted and safer workflow for handling model artifacts.
* Update model artifact handling in build workflow
Changed artifact upload path to include only .onnx files and commented out the deletion of existing models. Also added a dependency for the build stage to ensure proper execution sequence.
* Update build workflow to refine model artifact handling
Removed unused code for deleting models and fixed the artifact path. Added a reference file logging custom name and upstream branch for better traceability. These changes streamline the workflow and improve clarity.
* Set dynamic run-name for sunnypilot build workflow
Added a customizable `run-name` field in the sunnypilot build model workflow. This allows more informative and dynamic naming based on input parameters such as `custom_name` or `upstream_branch`. Improves clarity and traceability in workflow runs.
* Add customization options and metadata generation to build
Introduce new inputs for file name prefix and 20Hz model specification in the workflow. Enable renaming of model files, generation of file hashes, and creation of a `metadata.json` file containing build details. This improves flexibility and enhances artifact information management.
* Refactor CI workflow to remove redundant get_model job
The `get_model` job has been removed, and its relevant steps have been merged into the `build_model` job to streamline the workflow. Additionally, a patch for `SConstruct` has been added to support an arbitrary cache directory configuration. This simplifies the pipeline while maintaining functionality.
* powersave ok ok
* Revert "powersave ok ok"
This reverts commit
|
||
|
DevTekVE
|
d5f5887830 |
CI: Increase test report creation timeout and improve build times (#545)
* Increase timeout for test report creation by 1 minute Adjusted the `timeout-minutes` value to allow more time for the test report creation step. * Add script to disable power save in build workflow This commit integrates the `disable-powersave.py` script into the sunnypilot build workflow. It ensures power save mode is disabled during the build process, improving reliability and consistency. * Set PYTHONPATH before running disable-powersave.py The change ensures the correct Python module path is set when executing the disable-powersave script. This update prevents potential import issues by including the GitHub workspace in the PYTHONPATH. * Refactor powersave handling with a dedicated script Replaced inline powersave disable logic with a new script `manage-powersave.py` to handle enabling and disabling power saving mode. Updated the CI workflow to use this script and added a step to re-enable powersave after builds. This improves clarity and modularity for power management operations. * Enable CPU core count display in power save script Introduce multiprocessing to show the number of CPU cores available. Added messages to indicate power save mode state and core count, improving script feedback and user clarity. * Refine power save mode logging output. Improve clarity of logging messages in `manage-powersave.py` by indicating CPU core counts before and after applying changes. Simplified the power save state message for better readability. * Updated scons cache key restore logic in GitHub workflows This commit updates the restore key logic in two GitHub workflow config files (sunnypilot-build-model.yaml and sunnypilot-build-prebuilt.yaml). The restore key sequence has been revised for improved accuracy and consistency. This should optimize the cache hit rate and speed up the subsequent builds. * Add comments on GitHub Actions cache isolation behavior Clarifies the cache isolation enforced by GitHub Actions for security reasons. Notes that only caches from the default branch are shared across all builds, and this behavior cannot be changed. This improves maintainability and understanding of the workflow configuration. |
||
|
DevTekVE
|
0bfa9fca88 |
SP: GitHub runner as service and able to toggle via developer panel (#494)
* Add support for GitHub Actions runner management Introduce a new process and configuration to manage a GitHub Actions runner. Added a persistent "EnableGithubRunner" parameter and a script to control the runner service. Integrated the feature into the system's process manager logic. * Restrict GitHub runner usage on metered networks. This update modifies the `use_github_runner` function to include a check for metered networks using `HARDWARE.get_network_metered()`. This ensures the GitHub runner is not enabled when the network is metered, improving network usage efficiency. * Add GitHub runner service toggle to developer panel Introduces a new toggle in the developer panel to enable or disable the GitHub runner service. This provides developers with a convenient way to control the service from the UI. * translations * Refactor network condition check for GitHub runner. Updated `use_github_runner` to include network type in metered network checks. This ensures more accurate validation by considering specific network types when determining metered status. * Mark as executable * Update paths and shebangs for consistency across scripts Modified file paths to align with the new directory structure under `/data/media/0/github` and updated the shebang in `github_runner.sh` for better environment compatibility. Adjusted the `BUILD_DIR` in the GitHub workflow to reflect the new path. * Fix string translation for GitHub runner parameter text Added translation support for the GitHub runner service description text. This ensures consistent localization across the UI. * Remove gitlab_runner.sh from Sunnypilot blacklist. The script is no longer required to be blacklisted, likely due to updates or changes in its usage. This improves the maintainability of the blacklist by removing unnecessary entries. * lang stuff * Update BASE_DIR determination based on mount point Refactored scripts to dynamically set BASE_DIR depending on whether /data/media is a valid mount point. This ensures compatibility with different environments and improves robustness of path resolution. * Refactor GitHub runner logic in process_config. Simplified enabling conditions for the GitHub runner by removing dependency on hardware network checks and adding a logical combination of offroad and runner-related functions. This improves code readability and reduces hardware coupling. |
||
|
DevTekVE
|
37e493ce0d |
CI: Add GitHub Action to build models from upstream (#491)
* Add support for custom model naming in workflow
This update introduces a new optional input `custom_name` to the `sunnypilot-build-model` GitHub workflow. If provided, it customizes the artifact name; otherwise, the default naming scheme is used. This enhances flexibility for build artifact identification.
* fix name
* Update cache key logic in build workflow
Replaced `github.ref_name` with a fallback to `github.head_ref || github.ref_name` in cache keys to handle branches and pull requests more reliably. This ensures consistent cache retrieval and improves workflow efficiency.
* Simplify artifact upload logic in CI workflow
Consolidated redundant artifact upload steps into a single action, leveraging a fallback mechanism for naming. This streamlines the workflow, reducing duplication and improving maintainability.
* Make publish step resilient and add conditional notify
Added `continue-on-error` to the publish step to allow workflow progression even if it fails. Introduced a condition to notify step that depends on publish outcome being successful. These changes improve workflow reliability and error handling.
* Fix conditional syntax in GitHub Actions workflow
Updated the conditional statement in the notify job to use the correct GitHub Actions expression syntax. This ensures proper evaluation of the publish step's success outcome.
* Update condition for notify step in GitHub Actions workflow
Replaced the deprecated syntax `${{ steps.publish.outcome == 'success' }}` with the recommended `success()` function. This ensures compatibility with newer GitHub Actions features and improves maintainability.
* Update build workflow to handle PR drafts and improve concurrency
Added support for pull request draft events to trigger the workflow, ensuring drafts are not missed. Removed `continue-on-error` in the publish step to improve reliability and prevent silent failures. Simplifies and enhances workflow robustness.
* Update pull request workflows and add draft check logic
Removed unnecessary pull request event types and drafts settings, consolidating logic. Added a condition to skip publishing for non-draft pull requests, ensuring better control over the deployment process.
|
||
|
DevTekVE
|
ad537fcb89 |
CI: Build prebuilts and models from github and self hosted runners (#476)
* Create a GitHub Actions workflow for synchronizing the repository to GitLab and enhance GitLab CI settings A new GitHub Actions workflow for mirroring the current repository to GitLab is added. This workflow is triggered by both push and delete events in addition to manual triggering. The role of GitHub actions runner is defined through a series of steps. A new '.gitlab-ci.yml' build configuration file is also introduced, with a more comprehensive definition of variables, jobs, and pipeline rules for its utilization in the GitLab CI/CD pipeline. Further, other changes include the addition of scripts for installing and uninstalling GitLab CI runner, along with modifying the SCons build system configuration file to include custom cache directory. Moreover, 'release_files.py' has been revised to include additional blacklisted and whitelisted files specific to Sunnypilot, ensuring suitable settings for the CI flow. The improvements facilitate a smoother integration between GitHub and GitLab, powerfully harnessing the capabilities of both platforms for more efficient and effective CI/CD pipelines and version control management. * not needed for this * Update workflow to build model from upstream repository Revised the CI workflow to build directly from the upstream `commaai/openpilot` repository. Simplified configuration, removed unused steps, and added support for specifying the upstream branch dynamically via inputs. * Update SConstruct to allow passing arbitrary cache_dir Modified the SConstruct file to enable setting a custom cache directory via arguments. This enhances flexibility in configuring cache paths during the build process. * Refactor build workflow to improve branch handling logic. Reorganized conditions for setting environment variables, replacing repository_dispatch with workflow_dispatch for prebuilt builds. Added a fallback error message for unsupported triggers to improve robustness. This enhances clarity and ensures compatibility with the intended workflow triggers. * test test test * test test test * test test test * Enable publication flag during build configurations Added `SHOULD_PUBLISH=true` to all relevant build configuration steps to ensure proper handling of publishing logic. Updated environment variables to include this flag for downstream usage. Removed the error-check step for unsupported configurations. * Simplify publish condition in workflow logic Replaced the previous condition for publishing with a single output variable, `should_publish`, to streamline logic and improve maintainability. This change reduces redundancy and makes the workflow more adaptable to future updates. * Simplify restore key patterns in build workflow. Removed unnecessary trailing dashes in SCons cache restore keys to streamline and slightly improve key matching logic. This ensures better consistency with the current workflow setup. * Update cache key usage in build workflow Replaced `github.ref_name` with `github.head_ref` for cache keys to ensure accurate branch-specific caching. Added fallback restore-keys for master branches to improve cache efficiency and reduce redundant builds. * Improved debug logging in GitHub actions workflow This commit refines the debug logging in our GitHub Actions workflow for the Sunnypilot build. This change provides more granularity, enabling logging only during debug mode, which helps to keep the runtime logs less cluttered during normal operations. This includes the conditionally displaying of environmental variables, GitHub output contents, and directory listings. Additionally, debug mode verbosity was added to rsync commands to aid troubleshooting file transfers during the build process. Blank lines were also reduced for better readability and cleaner code presentation. * test diff path * Refactor SCons cache handling in build workflow Replaced hardcoded cache directory paths with an environment variable (`SCONS_CACHE_DIR`) for better maintainability and flexibility. Updated related workflow steps to utilize the new variable and adjusted cache key usage. Removed unused `BASE_BUILD_NUMBER` variable to clean up the configuration. * Update cache key to include commit SHA in workflow This change adds the commit SHA to the cache key in the GitHub Actions workflow. It ensures more precise caching by differentiating builds based on the specific commit, reducing potential conflicts. * clean Update GitHub runner service to set environment variables Updated the ExecStart command to explicitly set HOME, USER, LOGNAME, and MAIL environment variables. This ensures the runner operates with the correct environment configuration, improving reliability and compatibility. Refactor GitHub runner service ExecStart command. Replaced direct command execution with running the service as a specific user using `su`. This improves compatibility and aligns with best practices for user-based execution. No functional changes are expected. Update GitHub runner service to set environment variables The ExecStart command now sets HOME, USER, LOGNAME, and MAIL environment variables for the runner process. This ensures proper environment initialization for the designated user, improving compatibility and reliability during execution. Refactor GitHub runner service template handling. Revised the `modify_service_template` function to create a properly structured service template for the GitHub runner. Updated service permissions, execution parameters, and enabled the function call to ensure usage during runner setup. * Refactor GitHub Runner installer to improve argument parsing. Reworked the script to implement flexible and explicit command-line argument parsing using flags like `--token`, `--repo`, and `--start-at-boot`. Added support for setting default values and enabling/disabling auto-start based on the `--start-at-boot` flag. Improved error handling and usage messaging for better user experience. * Update cache keys in sunnypilot build workflow Modified the cache keys to include `github.ref_name` for more precise caching and restore behavior. This improves build consistency by better differentiating between branches and refs. No changes to the overall workflow logic. * Remove 'tinygrad/*' from release file exclusions This change modifies the release file exclusions by removing 'tinygrad/*'. The adjustment ensures that files in the 'tinygrad' directory are now included in the release process, aligning with updated packaging requirements. * Simplify release file exclusions list. Removed redundant and unnecessary file patterns from the exclusions list in `release_files.py`. This streamlines the file handling process and reduces maintenance overhead. * Refactor SCons cache path and key structure. Updated the SCons cache directory path to `SCONS_CACHE_DIR` for clarity and consistency. Improved the caching key structure to include `github.head_ref` for better cache differentiation and restore hierarchy. Adjusted related build instructions to reflect the new environment variable. * Refactor branch configuration in CI workflow Standardize branch name handling by replacing hardcoded values with environment variables. This improves maintainability and simplifies updates to branch names across the workflow. Updated references to use the new dynamic environment variable approach. |