Compare commits

...

20 Commits

Author SHA1 Message Date
DevTekVE 40d9e092b6 Refactor type annotations and return types in model_runner
Removed specific type hints and return annotations for `self.inputs` and `run_model` methods to enhance flexibility and maintain consistency. These changes streamline the code and improve compatibility with varying input/output types during model inference.
2024-12-29 18:04:05 +01:00
DevTekVE 6318aa52e3 Refactor type hints for improved code consistency.
Replaced specific type hints like `dict[str, np.ndarray]` with generic `dict` in several method signatures to simplify annotations. This improves overall readability and maintains functionality, while aligning with existing code practices.
2024-12-29 17:59:50 +01:00
DevTekVE 7852fa66b1 Clean 2024-12-29 17:51:30 +01:00
DevTekVE 8d9c1e2035 Add type annotations to inputs and prepare_inputs method
This commit adds explicit type annotations for the `inputs` dictionary and the return value of the `prepare_inputs` method in `model_runner.py`. These changes improve code readability and ensure type consistency, enhancing maintainability and reducing potential errors.
2024-12-29 17:50:58 +01:00
DevTekVE c39f722f7b Rename TinyGradRunner to TinygradRunner for consistency.
The class name and references were updated to maintain naming consistency across the codebase. This aligns with naming conventions and improves code clarity. No functional changes were introduced.
2024-12-29 17:48:00 +01:00
DevTekVE 550c08ac4c Refactor prepare_inputs to use explicit CLMem type.
This update replaces the generic `any` type with the more explicit `CLMem` type for better type safety and clarity. It ensures consistency across the `prepare_inputs` method implementations in derived classes, improving code readability and robustness.
2024-12-29 17:44:23 +01:00
DevTekVE 8c838af5fa Refactor model inference to use internal state for inputs
Simplified the `run_model` method by removing the requirement to pass inputs as arguments, and instead leveraging an internal `inputs` state. Adjusted `prepare_inputs` methods across model runners to populate this internal state. This refactor improves code clarity and reduces redundancy in managing input data.
2024-12-29 17:40:45 +01:00
DevTekVE 2521d60b1b Fix model_runner output to ensure tensor conversion to NumPy.
Updated the `run_model` method to explicitly convert tensor outputs to NumPy arrays using `.numpy()`. This ensures compatibility with downstream processes relying on NumPy array inputs.
2024-12-29 17:31:27 +01:00
DevTekVE 206398d3b5 Remove unused imports and redundant pass statements.
This change cleans up the code by removing unused imports and redundant `pass` statements in abstract methods. It improves code readability and adheres to cleaner coding practices.
2024-12-29 17:22:13 +01:00
DevTekVE a9e99615cd Remove redundant tensor assignment in model runner.
The `assign` operation was unnecessary as new tensors are realized for updated inputs. This simplifies the code and avoids redundant updates, improving clarity and maintainability.
2024-12-29 17:20:15 +01:00
DevTekVE ec44b78ad8 Refactor model runner initialization logic.
Removed the `create_model_runner` factory function and replaced it with direct initialization of `TinyGradRunner` or `ONNXRunner`. Simplified the `__init__` methods by standardizing paths as constants within `model_runner.py` for cleaner and more maintainable code.
2024-12-29 17:14:08 +01:00
DevTekVE dcd3e09294 Refactor formatting in model_runner.py for readability.
Consolidated multiline function declarations and calls into single lines where appropriate to improve code readability and maintainability. No changes were made to the functionality.
2024-12-29 17:01:26 +01:00
DevTekVE 839a7a58e0 Refactoring model handling in modeld.py with ModelRunner abstraction
A significant refactoring of `modeld.py` was performed to enhance the handling of model logic. A new abstraction called `ModelRunner` has been introduced which encapsulates the model-running logic. This refactor simplifies the `modeld.py` script and provides easier management across different hardware configurations. Using this segregation, varying processing methods for models can be handled distinctly ensuring cleaner and more maintainable code. An instance of the appropriate model runner is now created during initialization based on whether a TICI hardware or a different type is used.
2024-12-29 16:59:10 +01:00
DevTekVE 262d17ead2 Tools: Setup CLion run configs and external tools (#486)
* Add CLion run configs and external tools setup

Introduced CLion configuration files for "Build Debug" and "Build Release," along with external tool definitions for Poetry SCons commands. Additionally, commented out `.idea` in `.gitignore` to track necessary project-specific settings.

* rename

* only exclude specific files from gitignore

* wildcard

---------

Co-authored-by: Jason Wen <haibin.wen3@gmail.com>
2024-12-24 01:14:06 -05:00
DevTekVE 0bfa9fca88 SP: GitHub runner as service and able to toggle via developer panel (#494)
* Add support for GitHub Actions runner management

Introduce a new process and configuration to manage a GitHub Actions runner. Added a persistent "EnableGithubRunner" parameter and a script to control the runner service. Integrated the feature into the system's process manager logic.

* Restrict GitHub runner usage on metered networks.

This update modifies the `use_github_runner` function to include a check for metered networks using `HARDWARE.get_network_metered()`. This ensures the GitHub runner is not enabled when the network is metered, improving network usage efficiency.

* Add GitHub runner service toggle to developer panel

Introduces a new toggle in the developer panel to enable or disable the GitHub runner service. This provides developers with a convenient way to control the service from the UI.

* translations

* Refactor network condition check for GitHub runner.

Updated `use_github_runner` to include network type in metered network checks. This ensures more accurate validation by considering specific network types when determining metered status.

* Mark as executable

* Update paths and shebangs for consistency across scripts

Modified file paths to align with the new directory structure under `/data/media/0/github` and updated the shebang in `github_runner.sh` for better environment compatibility. Adjusted the `BUILD_DIR` in the GitHub workflow to reflect the new path.

* Fix string translation for GitHub runner parameter text

Added translation support for the GitHub runner service description text. This ensures consistent localization across the UI.

* Remove gitlab_runner.sh from Sunnypilot blacklist.

The script is no longer required to be blacklisted, likely due to updates or changes in its usage. This improves the maintainability of the blacklist by removing unnecessary entries.

* lang stuff

* Update BASE_DIR determination based on mount point

Refactored scripts to dynamically set BASE_DIR depending on whether /data/media is a valid mount point. This ensures compatibility with different environments and improves robustness of path resolution.

* Refactor GitHub runner logic in process_config.

Simplified enabling conditions for the GitHub runner by removing dependency on hardware network checks and adding a logical combination of offroad and runner-related functions. This improves code readability and reduces hardware coupling.
2024-12-23 09:19:26 +01:00
DevTekVE cd73feec57 ci: Update version identifier in workflow for staging and release (#496)
Update version identifier in workflow for staging and release

Appended "-staging" and "-release" to the VERSION variable in the GitHub Actions workflow to clearly distinguish build types. This ensures clarity in versioning for staging and release environments.
2024-12-23 01:47:41 +01:00
DevTekVE bd49be185f ci: enable macos runners again (#495)
no need to block macos anymore. Turns out runners are free on github public repos
2024-12-23 01:03:21 +01:00
DevTekVE 37e493ce0d CI: Add GitHub Action to build models from upstream (#491)
* Add support for custom model naming in workflow

This update introduces a new optional input `custom_name` to the `sunnypilot-build-model` GitHub workflow. If provided, it customizes the artifact name; otherwise, the default naming scheme is used. This enhances flexibility for build artifact identification.

* fix name

* Update cache key logic in build workflow

Replaced `github.ref_name` with a fallback to `github.head_ref || github.ref_name` in cache keys to handle branches and pull requests more reliably. This ensures consistent cache retrieval and improves workflow efficiency.

* Simplify artifact upload logic in CI workflow

Consolidated redundant artifact upload steps into a single action, leveraging a fallback mechanism for naming. This streamlines the workflow, reducing duplication and improving maintainability.

* Make publish step resilient and add conditional notify

Added `continue-on-error` to the publish step to allow workflow progression even if it fails. Introduced a condition to notify step that depends on publish outcome being successful. These changes improve workflow reliability and error handling.

* Fix conditional syntax in GitHub Actions workflow

Updated the conditional statement in the notify job to use the correct GitHub Actions expression syntax. This ensures proper evaluation of the publish step's success outcome.

* Update condition for notify step in GitHub Actions workflow

Replaced the deprecated syntax `${{ steps.publish.outcome == 'success' }}` with the recommended `success()` function. This ensures compatibility with newer GitHub Actions features and improves maintainability.

* Update build workflow to handle PR drafts and improve concurrency

Added support for pull request draft events to trigger the workflow, ensuring drafts are not missed. Removed `continue-on-error` in the publish step to improve reliability and prevent silent failures. Simplifies and enhances workflow robustness.

* Update pull request workflows and add draft check logic

Removed unnecessary pull request event types and drafts settings, consolidating logic. Added a condition to skip publishing for non-draft pull requests, ensuring better control over the deployment process.
2024-12-23 00:37:02 +01:00
DevTekVE e114dc5a6f CI: Use environments for manual approvals on GH Actions for feature branches (#493)
* We no longer need to dispatch manually a workflow, it will wait for the publish to be approved for feature branches.
* The notification template has been extracted to a variable for the repo, meaning no need to do commit push to update it! (https://github.com/sunnypilot/sunnypilot/settings/variables/actions/DISCORD_GENERAL_UPDATE_NOTICE)
* The configuration for what can be auto deployed is now also a variable, no need to commit a change to update that (https://github.com/sunnypilot/sunnypilot/settings/variables/actions/AUTO_DEPLOY_PREBUILT_BRANCHES)
* The configuration for what sends a message on `#dev-feedback` channel on discord is also configurable (https://github.com/sunnypilot/sunnypilot/settings/variables/actions/DEV_FEEDBACK_NOTIFICATION_BRANCHES)
2024-12-22 18:18:21 +01:00
DevTekVE ad537fcb89 CI: Build prebuilts and models from github and self hosted runners (#476)
* Create a GitHub Actions workflow for synchronizing the repository to GitLab and enhance GitLab CI settings

A new GitHub Actions workflow for mirroring the current repository to GitLab is added. This workflow is triggered by both push and delete events in addition to manual triggering. The role of GitHub actions runner is defined through a series of steps. A new '.gitlab-ci.yml' build configuration file is also introduced, with a more comprehensive definition of variables, jobs, and pipeline rules for its utilization in the GitLab CI/CD pipeline.

Further, other changes include the addition of scripts for installing and uninstalling GitLab CI runner, along with modifying the SCons build system configuration file to include custom cache directory. Moreover, 'release_files.py' has been revised to include additional blacklisted and whitelisted files specific to Sunnypilot, ensuring suitable settings for the CI flow.

The improvements facilitate a smoother integration between GitHub and GitLab, powerfully harnessing the capabilities of both platforms for more efficient and effective CI/CD pipelines and version control management.

* not needed for this

* Update workflow to build model from upstream repository

Revised the CI workflow to build directly from the upstream `commaai/openpilot` repository. Simplified configuration, removed unused steps, and added support for specifying the upstream branch dynamically via inputs.

* Update SConstruct to allow passing arbitrary cache_dir

Modified the SConstruct file to enable setting a custom cache directory via arguments. This enhances flexibility in configuring cache paths during the build process.

* Refactor build workflow to improve branch handling logic.

Reorganized conditions for setting environment variables, replacing repository_dispatch with workflow_dispatch for prebuilt builds. Added a fallback error message for unsupported triggers to improve robustness. This enhances clarity and ensures compatibility with the intended workflow triggers.

* test test test

* test test test

* test test test

* Enable publication flag during build configurations

Added `SHOULD_PUBLISH=true` to all relevant build configuration steps to ensure proper handling of publishing logic. Updated environment variables to include this flag for downstream usage. Removed the error-check step for unsupported configurations.

* Simplify publish condition in workflow logic

Replaced the previous condition for publishing with a single output variable, `should_publish`, to streamline logic and improve maintainability. This change reduces redundancy and makes the workflow more adaptable to future updates.

* Simplify restore key patterns in build workflow.

Removed unnecessary trailing dashes in SCons cache restore keys to streamline and slightly improve key matching logic. This ensures better consistency with the current workflow setup.

* Update cache key usage in build workflow

Replaced `github.ref_name` with `github.head_ref` for cache keys to ensure accurate branch-specific caching. Added fallback restore-keys for master branches to improve cache efficiency and reduce redundant builds.

* Improved debug logging in GitHub actions workflow

This commit refines the debug logging in our GitHub Actions workflow for the Sunnypilot build. This change provides more granularity, enabling logging only during debug mode, which helps to keep the runtime logs less cluttered during normal operations. This includes the conditionally displaying of environmental variables, GitHub output contents, and directory listings.

Additionally, debug mode verbosity was added to rsync commands to aid troubleshooting file transfers during the build process.

Blank lines were also reduced for better readability and cleaner code presentation.

* test diff path

* Refactor SCons cache handling in build workflow

Replaced hardcoded cache directory paths with an environment variable (`SCONS_CACHE_DIR`) for better maintainability and flexibility. Updated related workflow steps to utilize the new variable and adjusted cache key usage. Removed unused `BASE_BUILD_NUMBER` variable to clean up the configuration.

* Update cache key to include commit SHA in workflow

This change adds the commit SHA to the cache key in the GitHub Actions workflow. It ensures more precise caching by differentiating builds based on the specific commit, reducing potential conflicts.

* clean

Update GitHub runner service to set environment variables

Updated the ExecStart command to explicitly set HOME, USER, LOGNAME, and MAIL environment variables. This ensures the runner operates with the correct environment configuration, improving reliability and compatibility.

Refactor GitHub runner service ExecStart command.

Replaced direct command execution with running the service as a specific user using `su`. This improves compatibility and aligns with best practices for user-based execution. No functional changes are expected.

Update GitHub runner service to set environment variables

The ExecStart command now sets HOME, USER, LOGNAME, and MAIL environment variables for the runner process. This ensures proper environment initialization for the designated user, improving compatibility and reliability during execution.

Refactor GitHub runner service template handling.

Revised the `modify_service_template` function to create a properly structured service template for the GitHub runner. Updated service permissions, execution parameters, and enabled the function call to ensure usage during runner setup.

* Refactor GitHub Runner installer to improve argument parsing.

Reworked the script to implement flexible and explicit command-line argument parsing using flags like `--token`, `--repo`, and `--start-at-boot`. Added support for setting default values and enabling/disabling auto-start based on the `--start-at-boot` flag. Improved error handling and usage messaging for better user experience.

* Update cache keys in sunnypilot build workflow

Modified the cache keys to include `github.ref_name` for more precise caching and restore behavior. This improves build consistency by better differentiating between branches and refs. No changes to the overall workflow logic.

* Remove 'tinygrad/*' from release file exclusions

This change modifies the release file exclusions by removing 'tinygrad/*'. The adjustment ensures that files in the 'tinygrad' directory are now included in the release process, aligning with updated packaging requirements.

* Simplify release file exclusions list.

Removed redundant and unnecessary file patterns from the exclusions list in `release_files.py`. This streamlines the file handling process and reduces maintenance overhead.

* Refactor SCons cache path and key structure.

Updated the SCons cache directory path to `SCONS_CACHE_DIR` for clarity and consistency. Improved the caching key structure to include `github.head_ref` for better cache differentiation and restore hierarchy. Adjusted related build instructions to reflect the new environment variable.

* Refactor branch configuration in CI workflow

Standardize branch name handling by replacing hardcoded values with environment variables. This improves maintainability and simplifies updates to branch names across the workflow. Updated references to use the new dynamic environment variable approach.
2024-12-21 20:19:48 +01:00
31 changed files with 1035 additions and 51 deletions
-1
View File
@@ -95,7 +95,6 @@ jobs:
timeout-minutes: 30
build_mac:
if: github.repository == 'commaai/openpilot' # Blocking macos builds as well since they have a 10x miltiplier for GH action minutes, waaaay too much!
name: build macOS
runs-on: ${{ ((github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
@@ -0,0 +1,86 @@
name: Build Model from Upstream
env:
BUILD_DIR: "/data/openpilot"
OUTPUT_DIR: ${{ github.workspace }}/output
SCONS_CACHE_DIR: ${{ github.workspace }}/release/ci/scons_cache
UPSTREAM_REPO: "commaai/openpilot"
on:
workflow_dispatch:
inputs:
upstream_branch:
description: 'Upstream branch to build from'
required: true
default: 'master'
type: string
custom_name:
description: 'Custom name for the model'
required: false
type: string
jobs:
build_model:
runs-on: self-hosted
steps:
- uses: actions/checkout@v4
with:
repository: ${{ env.UPSTREAM_REPO }}
ref: ${{ github.event.inputs.upstream_branch }}
submodules: recursive
- run: git lfs pull
- name: Cache SCons
uses: actions/cache@v4
with:
path: ${{env.SCONS_CACHE_DIR}}
key: scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-model-${{ github.sha }}
restore-keys: |
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-model-${{ github.sha }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-model
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}
scons-${{ runner.os }}-${{ runner.arch }}-master-new
scons-${{ runner.os }}-${{ runner.arch }}-master
scons-${{ runner.os }}-${{ runner.arch }}
- name: Setup build environment
run: |
mkdir -p "${BUILD_DIR}/"
sudo find $BUILD_DIR/ -mindepth 1 -delete
echo "Starting build stage..."
echo "Building from: ${{ env.UPSTREAM_REPO }} branch: ${{ github.event.inputs.upstream_branch }}"
- name: Patch SConstruct to pass arbitrary cache
run: |
sed -i.bak 's#cache_dir =#default_cache_dir =#' ${{ github.workspace }}/SConstruct
printf '/default_cache_dir/a\\\ncache_dir = ARGUMENTS.get("cache_dir", default_cache_dir)\n' | sed -i.bak -f - ${{ github.workspace }}/SConstruct
cat ${{ github.workspace }}/SConstruct
- name: Build Model
run: |
source /etc/profile
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
scons -j$(nproc) cache_dir=${{ env.SCONS_CACHE_DIR }} ${{ github.workspace }}/selfdrive/modeld
- name: Prepare Output
run: |
sudo rm -rf ${OUTPUT_DIR}
mkdir -p ${OUTPUT_DIR}
rsync -avm \
--include='*.dlc' \
--include='*.thneed' \
--include='*.pkl' \
--include='*.onnx' \
--exclude='*' \
--delete-excluded \
--chown=comma:comma \
./selfdrive/modeld/models/ ${OUTPUT_DIR}/
- name: Upload Build Artifacts
uses: actions/upload-artifact@v4
with:
name: model-${{ github.event.inputs.custom_name || github.event.inputs.upstream_branch }}-${{ github.run_number }}
path: ${{ env.OUTPUT_DIR }}
@@ -0,0 +1,268 @@
name: sunnypilot prebuilt action
env:
BUILD_DIR: "/data/openpilot"
OUTPUT_DIR: ${{ github.workspace }}/output
CI_DIR: ${{ github.workspace }}/release/ci
SCONS_CACHE_DIR: ${{ github.workspace }}/release/ci/scons_cache
PUBLIC_REPO_URL: "https://github.com/sunnypilot/sunnypilot"
# Branch configurations
MASTER_BRANCH: "master"
MASTER_NEW_BRANCH: "master-new"
DEV_C3_SOURCE_BRANCH: "master-dev-c3-new"
# Target branch configurations
STAGING_TARGET_BRANCH: "staging-c3-new"
DEV_TARGET_BRANCH: "dev-c3-new"
RELEASE_TARGET_BRANCH: "release-c3-new"
on:
push:
branches: [ master, master-new, master-dev-c3-new ]
tags: [ '*' ]
pull_request:
branches: [ master, master-new ]
workflow_dispatch:
inputs:
extra_version:
description: 'Extra version identifier'
required: false
default: ''
jobs:
build:
concurrency:
group: build-${{ github.head_ref || github.ref_name }}
cancel-in-progress: false
runs-on: self-hosted
outputs:
new_branch: ${{ steps.set-env.outputs.new_branch }}
version: ${{ steps.set-env.outputs.version }}
extra_version_identifier: ${{ steps.set-env.outputs.extra_version_identifier }}
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- run: git lfs pull
- name: Cache SCons
uses: actions/cache@v4
with:
path: ${{env.SCONS_CACHE_DIR}}
key: scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-${{ github.sha }}
restore-keys: |
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_NEW_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}
- name: Set Configuration
run: |
if [[ "${{ github.ref_name }}" == "${{ env.DEV_C3_SOURCE_BRANCH }}" ]]; then
# Dev configuration
echo "BRANCH_TYPE=dev" >> $GITHUB_ENV
echo "NEW_BRANCH=${{ env.DEV_TARGET_BRANCH }}" >> $GITHUB_ENV
echo "VERSION=$(date '+%Y.%m.%d')-${{ github.run_number }}" >> $GITHUB_ENV
echo "EXTRA_VERSION_IDENTIFIER=${{ github.run_number }}" >> $GITHUB_ENV
elif [[ "${{ github.ref_name }}" == "${{ env.MASTER_BRANCH }}" || "${{ github.ref_name }}" == "${{ env.MASTER_NEW_BRANCH }}" ]]; then
# Master configuration
echo "BRANCH_TYPE=master" >> $GITHUB_ENV
echo "NEW_BRANCH=${{ env.STAGING_TARGET_BRANCH }}" >> $GITHUB_ENV
echo "EXTRA_VERSION_IDENTIFIER=staging" >> $GITHUB_ENV
echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')-staging" >> $GITHUB_ENV
elif [[ "${{ github.ref }}" == refs/tags/* ]]; then
# Tag configuration
echo "BRANCH_TYPE=tag" >> $GITHUB_ENV
echo "NEW_BRANCH=${{ env.RELEASE_TARGET_BRANCH }}" >> $GITHUB_ENV
echo "EXTRA_VERSION_IDENTIFIER=release" >> $GITHUB_ENV
echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')-release" >> $GITHUB_ENV
else
# Feature branch configuration
echo "BRANCH_TYPE=dispatch" >> $GITHUB_ENV
echo "NEW_BRANCH=${{ github.head_ref || github.ref_name }}-prebuilt" >> $GITHUB_ENV
echo "VERSION=$(date '+%Y.%m.%d')-${{ github.run_number }}" >> $GITHUB_ENV
fi
- name: Set environment variables
id: set-env
run: |
# Write to GITHUB_OUTPUT from environment variables
echo "new_branch=$NEW_BRANCH" >> $GITHUB_OUTPUT
[[ ! -z "$EXTRA_VERSION_IDENTIFIER" ]] && echo "extra_version_identifier=$EXTRA_VERSION_IDENTIFIER" >> $GITHUB_OUTPUT
[[ ! -z "$VERSION" ]] && echo "version=$VERSION" >> $GITHUB_OUTPUT
# Set up common environment
source /etc/profile;
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
printenv >> $GITHUB_ENV
if [[ "${{ runner.debug }}" == "1" ]]; then
cat $GITHUB_OUTPUT
fi
- name: Setup build environment
run: |
mkdir -p "${BUILD_DIR}/"
sudo find $BUILD_DIR/ -mindepth 1 -delete
echo "Starting build stage..."
echo "BUILD_DIR: ${BUILD_DIR}"
echo "CI_DIR: ${CI_DIR}"
echo "VERSION: ${{ steps.set-env.outputs.version }}"
echo "UV_PROJECT_ENVIRONMENT: ${UV_PROJECT_ENVIRONMENT}"
echo "VIRTUAL_ENV: ${VIRTUAL_ENV}"
echo "-------"
if [[ "${{ runner.debug }}" == "1" ]]; then
printenv
fi
- name: Build Panda
run: |
scons -j$(nproc) cache_dir=${{env.SCONS_CACHE_DIR}} ${{ github.workspace }}/panda
- name: Build Main Project
run: |
export PYTHONPATH="$BUILD_DIR"
./release/release_files.py | sort | uniq | rsync -rRl${RUNNER_DEBUG:+v} --files-from=- . $BUILD_DIR/
cd $BUILD_DIR
sed -i '/from .board.jungle import PandaJungle, PandaJungleDFU/s/^/#/' panda/__init__.py
scons -j$(nproc) cache_dir=${{env.SCONS_CACHE_DIR}} --minimal
touch ${BUILD_DIR}/prebuilt
if [[ "${{ runner.debug }}" == "1" ]]; then
ls -la ${BUILD_DIR}
fi
- name: Prepare Output
run: |
sudo rm -rf ${OUTPUT_DIR}
mkdir -p ${OUTPUT_DIR}
rsync -am${RUNNER_DEBUG:+v} \
--include='**/panda/board/' \
--include='**/panda/board/obj' \
--include='**/panda/board/obj/panda.bin.signed' \
--include='**/panda/board/obj/panda_h7.bin.signed' \
--include='**/panda/board/obj/bootstub.panda.bin' \
--include='**/panda/board/obj/bootstub.panda_h7.bin' \
--exclude='.sconsign.dblite' \
--exclude='*.a' \
--exclude='*.o' \
--exclude='*.os' \
--exclude='*.pyc' \
--exclude='moc_*' \
--exclude='*.cc' \
--exclude='Jenkinsfile' \
--exclude='supercombo.onnx' \
--exclude='**/panda/board/*' \
--exclude='**/panda/board/obj/**' \
--exclude='**/panda/certs/' \
--exclude='**/panda/crypto/' \
--exclude='**/release/' \
--exclude='**/.github/' \
--exclude='**/selfdrive/ui/replay/' \
--exclude='**/__pycache__/' \
--exclude='**/selfdrive/ui/*.h' \
--exclude='**/selfdrive/ui/**/*.h' \
--exclude='**/selfdrive/ui/qt/offroad/sunnypilot/' \
--exclude='${{env.SCONS_CACHE_DIR}}' \
--exclude='**/.git/' \
--exclude='**/SConstruct' \
--exclude='**/SConscript' \
--exclude='**/.venv/' \
--delete-excluded \
--chown=comma:comma \
${BUILD_DIR}/ ${OUTPUT_DIR}/
- name: 'Tar.gz files'
run: |
tar czf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }} .
ls -la prebuilt.tar.gz
- name: 'Upload Artifact'
uses: actions/upload-artifact@v4
with:
name: prebuilt
path: prebuilt.tar.gz
publish:
concurrency:
group: publish-${{ github.head_ref || github.ref_name }}
cancel-in-progress: true
if: ${{ github.event_name != 'pull_request' || github.event_name == 'pull_request' && github.event.pull_request.draft }}
needs: build
runs-on: ubuntu-24.04
environment: ${{ contains(fromJSON(vars.AUTO_DEPLOY_PREBUILT_BRANCHES), github.head_ref || github.ref_name) && 'auto-deploy' || 'feature-branch' }}
steps:
- uses: actions/checkout@v4
- name: Download build artifacts
uses: actions/download-artifact@v4
with:
name: prebuilt
- name: Untar prebuilt
run: |
mkdir -p ${{ env.OUTPUT_DIR }}
tar xzf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }}
- name: Configure Git
run: |
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"
- name: Publish to Public Repository
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
echo '${{ toJSON(needs.build.outputs) }}'
ls -la ${{ env.OUTPUT_DIR }}
${{ env.CI_DIR }}/publish.sh \
"${{ github.workspace }}" \
"${{ env.OUTPUT_DIR }}" \
"${{ needs.build.outputs.new_branch }}" \
"${{ needs.build.outputs.version }}" \
"https://x-access-token:${{github.token}}@github.com/sunnypilot/sunnypilot.git" \
"-${{ needs.build.outputs.extra_version_identifier }}"
echo ""
echo "---- ️ To update the list of branches that auto deploy prebuilts -----"
echo ""
echo "1. Go to: ${{ github.server_url }}/${{ github.repository }}/settings/variables/actions/AUTO_DEPLOY_PREBUILT_BRANCHES"
echo "2. Current value: ${{ vars.AUTO_DEPLOY_PREBUILT_BRANCHES }}"
echo "3. Update as needed (JSON array with no spaces)"
notify:
needs: [ build, publish ]
runs-on: ubuntu-24.04
if: success()
steps:
- uses: actions/checkout@v4
- name: Setup Alpine Linux environment
uses: jirutka/setup-alpine@v1.2.0
with:
packages: 'jq gettext curl'
- name: Send Discord Notification
env:
DISCORD_WEBHOOK: ${{ contains(fromJSON(vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES), github.head_ref || github.ref_name) && secrets.DISCORD_DEV_FEEDBACK_CHANNEL_WEBHOOK || secrets.DISCORD_DEV_PRIVATE_CHANNEL_WEBHOOK }}
run: |
TEMPLATE='${{ vars.DISCORD_GENERAL_UPDATE_NOTICE }}'
export EXTRA_VERSION_IDENTIFIER="${{ needs.build.outputs.extra_version_identifier }}"
export VERSION="${{ needs.build.outputs.version }}"
export branch_name=${{ github.head_ref || github.ref_name }}
export new_branch=${{ needs.build.outputs.new_branch }}
export extra_version_identifier=${{ needs.build.outputs.extra_version_identifier || github.run_number}}
echo ${TEMPLATE} | envsubst | jq -c '.' | tee payload.json
curl -X POST -H "Content-Type: application/json" -d @payload.json $DISCORD_WEBHOOK
echo ""
echo "---- ️ To update the list of branches that notify to dev-feedback -----"
echo ""
echo "1. Go to: ${{ github.server_url }}/${{ github.repository }}/settings/variables/actions/DEV_FEEDBACK_NOTIFICATION_BRANCHES"
echo "2. Current value: ${{ vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES }}"
echo "3. Update as needed (JSON array with no spaces)"
shell: alpine.sh {0}
+5
View File
@@ -103,3 +103,8 @@ Pipfile
# Ignore all local history of files
.history
.ionide
### JetBrains ###
!.idea/customTargets.xml
!.idea/tools/*
!.run/*
+25
View File
@@ -0,0 +1,25 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="CLionExternalBuildManager">
<target id="a62f99e8-5ec4-434c-8122-49efed5af108" name="uv Scons Build Debug" defaultType="TOOL">
<configuration id="b93ec964-16e5-4962-a12e-3ed360ce8f02" name="uv Scons Build Debug">
<build type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Build Debug" />
</build>
<clean type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Clean" />
</clean>
</configuration>
</target>
<target id="edd8ad9d-183b-467c-a355-0d9a0ecab026" name="uv Scons Build Release" defaultType="TOOL">
<configuration id="09523339-5ce3-4223-ab9e-904f38ad7752" name="uv Scons Build Release">
<build type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Build Release" />
</build>
<clean type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Clean" />
</clean>
</configuration>
</target>
</component>
</project>
+23
View File
@@ -0,0 +1,23 @@
<toolSet name="External Tools">
<tool name="uv Scons Build Debug" showInMainMenu="false" showInEditor="false" showInProject="false" showInSearchPopup="false" disabled="false" useConsole="true" showConsoleOnStdOut="false" showConsoleOnStdErr="false" synchronizeAfterRun="true">
<exec>
<option name="COMMAND" value="bash" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -u -j$(nproc) --compile_db --ccflags=\&quot;-fno-inline\&quot;&quot;" />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
<tool name="uv Scons Clean" showInMainMenu="false" showInEditor="false" showInProject="false" showInSearchPopup="false" disabled="false" useConsole="true" showConsoleOnStdOut="false" showConsoleOnStdErr="false" synchronizeAfterRun="true">
<exec>
<option name="COMMAND" value="bash" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -c&quot; " />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
<tool name="uv Scons Build Release" showInMainMenu="false" showInEditor="false" showInProject="false" showInSearchPopup="false" disabled="false" useConsole="true" showConsoleOnStdOut="false" showConsoleOnStdErr="false" synchronizeAfterRun="true">
<exec>
<option name="COMMAND" value="bash" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -u -j$(nproc) --compile_db&quot; " />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
</toolSet>
+10
View File
@@ -0,0 +1,10 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Build Debug" type="CLionExternalRunConfiguration" factoryName="Application" REDIRECT_INPUT="false" ELEVATE="false" USE_EXTERNAL_CONSOLE="false" EMULATE_TERMINAL="false" WORKING_DIR="file://$ProjectFileDir$/selfdrive/ui" PASS_PARENT_ENVS_2="true" PROJECT_NAME="sunnypilot" TARGET_NAME="uv Scons Build Debug" CONFIG_NAME="uv Scons Build Debug" RUN_PATH="ui">
<envs>
<env name="QT_DBL_CLICK_DIST" value="150" />
</envs>
<method v="2">
<option name="CLION.EXTERNAL.BUILD" enabled="true" />
</method>
</configuration>
</component>
+10
View File
@@ -0,0 +1,10 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Build Release" type="CLionExternalRunConfiguration" factoryName="Application" REDIRECT_INPUT="false" ELEVATE="false" USE_EXTERNAL_CONSOLE="false" EMULATE_TERMINAL="false" WORKING_DIR="file://$ProjectFileDir$/selfdrive/ui" PASS_PARENT_ENVS_2="true" PROJECT_NAME="sunnypilot" TARGET_NAME="uv Scons Build Release" CONFIG_NAME="uv Scons Build Release" RUN_PATH="ui">
<envs>
<env name="QT_DBL_CLICK_DIST" value="150" />
</envs>
<method v="2">
<option name="CLION.EXTERNAL.BUILD" enabled="true" />
</method>
</configuration>
</component>
+2 -1
View File
@@ -237,7 +237,8 @@ if GetOption('compile_db'):
env.CompilationDatabase('compile_commands.json')
# Setup cache dir
cache_dir = '/data/scons_cache' if AGNOS else '/tmp/scons_cache'
default_cache_dir = '/data/scons_cache' if AGNOS else '/tmp/scons_cache'
cache_dir = ARGUMENTS.get('cache_dir', default_cache_dir)
CacheDir(cache_dir)
Clean(["."], cache_dir)
+1
View File
@@ -200,6 +200,7 @@ std::unordered_map<std::string, uint32_t> keys = {
{"UpdaterTargetBranch", CLEAR_ON_MANAGER_START},
{"UpdaterLastFetchTime", PERSISTENT},
{"Version", PERSISTENT},
{"EnableGithubRunner", PERSISTENT},
};
} // namespace
+147
View File
@@ -0,0 +1,147 @@
#!/usr/bin/env bash
set -e
# Default values
DEFAULT_REPO_URL="https://github.com/sunnypilot"
START_AT_BOOT=false
# Parse command line arguments
while [[ $# -gt 0 ]]; do
case $1 in
--start-at-boot)
START_AT_BOOT=true
shift
;;
--token)
GITHUB_TOKEN="$2"
shift 2
;;
--repo)
REPO_URL="$2"
shift 2
;;
*)
if [ -z "$GITHUB_TOKEN" ]; then
GITHUB_TOKEN="$1"
elif [ -z "$REPO_URL" ]; then
REPO_URL="$1"
fi
shift
;;
esac
done
# Check required arguments
if [ -z "$GITHUB_TOKEN" ]; then
echo "Usage: $0 [--start-at-boot] [--token <github_token>] [--repo <repository_url>]"
echo "Required argument: github_token"
echo "Optional arguments:"
echo " --start-at-boot Enable auto-start at boot (default: false)"
echo " --repo Repository URL (default: ${DEFAULT_REPO_URL})"
exit 1
fi
# Set repository URL if not provided
REPO_URL="${REPO_URL:-$DEFAULT_REPO_URL}"
# Determine BASE_DIR based on mount point
if mountpoint -q /data/media; then
BASE_DIR="/data/media/0/github"
else
BASE_DIR="/data/github"
fi
# Constants
RUNNER_USER="github-runner"
USER_GROUPS="comma,gpu,gpio,sudo"
RUNNER_DIR="${BASE_DIR}/runner"
BUILDS_DIR="${BASE_DIR}/builds"
LOGS_DIR="${BASE_DIR}/logs"
CACHE_DIR="${BASE_DIR}/cache"
OPENPILOT_DIR="${BASE_DIR}/openpilot"
create_directories() {
sudo mkdir -p "$RUNNER_DIR" "$BUILDS_DIR" "$LOGS_DIR" "$CACHE_DIR" "$OPENPILOT_DIR"
mkdir -p "/data/openpilot"
sudo chown -R comma:comma "/data/openpilot"
}
download_and_setup_runner() {
cd "$RUNNER_DIR"
curl -o actions-runner-linux-arm64-2.321.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.321.0/actions-runner-linux-arm64-2.321.0.tar.gz
tar xzf ./actions-runner-linux-arm64-2.321.0.tar.gz
rm ./actions-runner-linux-arm64-2.321.0.tar.gz
chmod +x ./config.sh
}
setup_runner_user() {
sudo useradd --comment 'GitHub Runner' --create-home --home-dir ${BASE_DIR} ${RUNNER_USER} --shell /bin/bash -G ${USER_GROUPS} || sudo usermod -aG ${USER_GROUPS} ${RUNNER_USER}
export BASE_DIR
sudo -u ${RUNNER_USER} bash -c "truncate -s 0 '${BASE_DIR}/.bash_logout'"
}
create_sudoers_entry() {
sudo grep -qxF "${RUNNER_USER} ALL=(ALL) NOPASSWD: ALL" /etc/sudoers || echo "${RUNNER_USER} ALL=(ALL) NOPASSWD: ALL" | sudo tee -a /etc/sudoers
}
configure_runner() {
cd "$RUNNER_DIR"
sudo -u ${RUNNER_USER} ./config.sh --url "$REPO_URL" --token "$GITHUB_TOKEN" --name $(hostname) --runnergroup "tici-tizi" --labels "tici" --work "$BUILDS_DIR" --unattended
}
set_directory_permissions() {
sudo chown -R ${RUNNER_USER}:comma "$BASE_DIR"
sudo chmod g+rwx "$BASE_DIR"
sudo chmod g+s "$BASE_DIR"
}
modify_service_template() {
cat <<EOL > "$RUNNER_DIR/bin/actions.runner.service.template"
[Unit]
Description={{Description}}
After=network-online.target nss-lookup.target time-sync.target
Wants=network-online.target nss-lookup.target time-sync.target
StartLimitInterval=5
StartLimitBurst=10
[Service]
Type=simple
User=root
ExecStart=/usr/bin/unshare -m -- /bin/bash -c 'mount --bind ${OPENPILOT_DIR} /data/openpilot && setpriv --reuid={{User}} --regid={{User}} --init-groups env HOME=${BASE_DIR} USER={{User}} LOGNAME={{User}} MAIL=/var/mail/{{User}} {{RunnerRoot}}/runsvc.sh'
WorkingDirectory={{RunnerRoot}}
KillMode=process
KillSignal=SIGTERM
TimeoutStopSec=5min
Restart=always
RestartSec=120
[Install]
WantedBy=multi-user.target
EOL
}
# Make filesystem writable
sudo mount -o remount,rw /
# Ensure filesystem is remounted as read-only on script exit
trap "sudo mount -o remount,ro /" EXIT
# Execute installation steps
setup_runner_user
create_sudoers_entry
create_directories
download_and_setup_runner
modify_service_template
configure_runner
set_directory_permissions
# Install and start service using built-in installer
cd "$RUNNER_DIR"
sudo ./svc.sh install $RUNNER_USER
# Handle auto-start configuration
if [ "$START_AT_BOOT" = false ]; then
sudo systemctl disable actions.runner.sunnypilot.$(uname -n)
fi
sudo ./svc.sh start
+78
View File
@@ -0,0 +1,78 @@
#!/usr/bin/env bash
set -e
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
cd $DIR
# Take parameters as arguments
SOURCE_DIR=$1
OUTPUT_DIR=$2
DEV_BRANCH=$3
VERSION=$4
GIT_ORIGIN=$5
EXTRA_VERSION_IDENTIFIER=$6
# Check parameters
if [ -z "$SOURCE_DIR" ] || [ -z "$OUTPUT_DIR" ]; then
echo "Error: No source or output directory provided."
exit 1
fi
if [ -z "$DEV_BRANCH" ] || [ -z "$VERSION" ]; then
echo "Error: No dev branch or version provided."
exit 1
fi
if [ -z "$GIT_ORIGIN" ]; then
echo "Error: No GIT_ORIGIN provided"
exit 1
fi
# "Tagging"
echo "#define COMMA_VERSION \"$VERSION\"" > ${OUTPUT_DIR}/common/version.h
## set git identity
#source $DIR/identity.sh
#export GIT_SSH_COMMAND="ssh -i /data/gitkey"
echo "[-] Setting up repo T=$SECONDS"
cd $OUTPUT_DIR
git init
# set git username/password
#source /data/identity.sh
git rm -rf $OUTPUT_DIR/.git || true # Doing cleanup, but it might fail if the .git doesn't exist or not allowed to delete
git remote remove origin || true # ensure cleanup
git remote add origin $GIT_ORIGIN
#git push origin -d $DEV_BRANCH || true # Ensuring we delete the remote branch if it exists as we are wiping it out
git fetch origin $DEV_BRANCH || (git checkout -b $DEV_BRANCH && git commit --allow-empty -m "sunnypilot v$VERSION release" && git push -u origin $DEV_BRANCH)
echo "[-] committing version $VERSION T=$SECONDS"
git add -f .
git commit -a -m "sunnypilot v$VERSION release"
git branch --set-upstream-to=origin/$DEV_BRANCH
# include source commit hash and build date in commit
GIT_HASH=$(git --git-dir=$SOURCE_DIR/.git rev-parse HEAD)
DATETIME=$(date '+%Y-%m-%dT%H:%M:%S')
SP_VERSION=$(cat $SOURCE_DIR/common/version.h | awk -F\" '{print $2}')
# Add built files to git
git add -f .
if [ "$EXTRA_VERSION_IDENTIFIER" = "-release" ] || [ "$EXTRA_VERSION_IDENTIFIER" = "-staging" ]; then
export VERSION=${VERSION%"$EXTRA_VERSION_IDENTIFIER"}
git commit --amend -m "sunnypilot v$VERSION"
else
git commit --amend -m "sunnypilot v$VERSION
version: sunnypilot v$SP_VERSION release
date: $DATETIME
master commit: $GIT_HASH
"
fi
git branch -m $DEV_BRANCH
# Push!
echo "[-] pushing T=$SECONDS"
git push -f origin $DEV_BRANCH
+66
View File
@@ -0,0 +1,66 @@
#!/usr/bin/env bash
# Determine BASE_DIR based on mount point
if mountpoint -q /data/media; then
GITHUB_BASE_DIR="/data/media/0/github"
else
GITHUB_BASE_DIR="/data/github"
fi
# Define directories and user
BIN_DIR="$GITHUB_BASE_DIR/bin"
BUILDS_DIR="$GITHUB_BASE_DIR/builds"
OPENPILOT_DIR="$GITHUB_BASE_DIR/openpilot"
LOGS_DIR="$GITHUB_BASE_DIR/logs"
CACHE_DIR="$GITHUB_BASE_DIR/cache"
RUNNER_USERNAME="github-runner"
# Define the systemd service name
SERVICE_NAME="github-runner"
USER_GROUPS="comma,gpu,gpio,sudo"
# Function to stop and disable the systemd service
stop_and_uninstall_service() {
cd $GITHUB_BASE_DIR/runner
sudo ./svc.sh stop
sudo ./svc.sh uninstall
}
# Function to remove the systemd service file
remove_runner() {
cd $GITHUB_BASE_DIR/runner
sudo rm .runner
sudo su -c './config.sh remove' github-runner
}
# Function to delete the Github Runner directories
delete_directories() {
sudo rm -rf "$BIN_DIR/github-runner"
sudo rm -rf "$GITHUB_BASE_DIR" "$BIN_DIR" "$BUILDS_DIR" "$LOGS_DIR" "$CACHE_DIR" "$OPENPILOT_DIR"
}
# Function to remove the Github Runner user
delete_user() {
for group in ${USER_GROUPS//,/ }
do
sudo gpasswd -d ${RUNNER_USERNAME} ${group}
done
sudo userdel -r ${RUNNER_USERNAME}
}
# Function to remove sudoers entry
remove_sudoers_entry() {
sudo sed -i.bak "/${RUNNER_USERNAME} ALL=(ALL) NOPASSWD: ALL/d" /etc/sudoers
}
# Make filesystem writable
sudo mount -o remount rw /
# Ensure filesystem is remounted as read-only on script exit
trap "sudo mount -o remount ro /" EXIT
# Call functions
stop_and_uninstall_service
remove_runner
delete_directories
delete_user
remove_sudoers_entry
# End of uninstall script
+74
View File
@@ -32,6 +32,7 @@ blacklist = [
".git/",
".github/",
".devcontainer/",
"Darwin/",
".vscode",
@@ -47,6 +48,42 @@ blacklist = [
".gitmodules",
]
# Sunnypilot blacklist
sunnypilot_blacklist = [
"system/loggerd/sunnylink_uploader.py", # Temporarily, until we are ready to roll it out widely
".idea/",
".run/",
".*__pycache__/.*",
".*\\.pyc",
"teleoprtc/*",
"third_party/snpe/x86_64/*",
"body/board/canloader.py",
"body/board/flash_base.sh",
"body/board/flash_knee.sh",
"body/board/recover.sh",
".*/test/",
".*/tests/",
".*tinygrad_repo/tinygrad/renderer/",
"README.md",
".*internal/",
"docs/.*",
".sconsign.dblite",
"release/ci/scons_cache/",
".gitlab-ci.yml",
".clang-tidy",
".dockerignore",
".editorconfig",
".python-version",
"SECURITY.md",
"codecov.yml",
"conftest.py",
"poetry.lock",
".venv/",
]
# Merge the blacklists
blacklist += sunnypilot_blacklist
# gets you through the blacklist
whitelist = [
"tools/lib/",
@@ -119,8 +156,45 @@ whitelist = [
"opendbc_repo/dbc/toyota_tss2_adas.dbc",
"opendbc_repo/dbc/vw_golf_mk4.dbc",
"opendbc_repo/dbc/vw_mqb_2010.dbc",
"opendbc_repo/dbc/tesla_can.dbc",
"opendbc_repo/dbc/tesla_radar_bosch_generated.dbc",
"opendbc_repo/dbc/tesla_radar_continental_generated.dbc",
"opendbc_repo/dbc/tesla_powertrain.dbc",
]
# Sunnypilot whitelist
sunnypilot_whitelist = [
"^README.md",
".*selfdrive/test/fuzzy_generation.py",
".*selfdrive/test/helpers.py",
".*selfdrive/test/__init__.py",
".*selfdrive/test/setup_device_ci.sh",
".*selfdrive/test/test_time_to_onroad.py",
".*selfdrive/test/test_onroad.py",
".*system/manager/test/test_manager.py",
".*system/manager/test/__init__.py",
".*system/qcomgpsd/tests/test_qcomgpsd.py",
".*system/updated/casync/tests/test_casync.py",
".*system/updated/tests/test_git.py",
".*system/updated/tests/test_base.py",
".*selfdrive/ui/tests/test_translations.py",
".*selfdrive/car/tests/__init__.py",
".*selfdrive/car/tests/test_car_interfaces.py",
".*selfdrive/navd/tests/test_navd.py",
".*selfdrive/navd/tests/test_map_renderer.py",
".*selfdrive/boardd/tests/test_boardd_loopback.py",
".*INTEGRATION.md",
".*HOW-TOS.md",
".*CARS.md",
".*LIMITATIONS.md",
".*CONTRIBUTING.md",
".*sunnyhaibin0850_qrcode_paypal.me.png",
"opendbc/.*.dbc",
]
# Merge the whitelists
whitelist += sunnypilot_whitelist
if __name__ == "__main__":
for f in Path(ROOT).rglob("**/*"):
+11 -49
View File
@@ -1,21 +1,13 @@
#!/usr/bin/env python3
import os
from openpilot.system.hardware import TICI
from openpilot.selfdrive.modeld.runners.model_runner import ONNXRunner, TinygradRunner
#
if TICI:
from tinygrad.tensor import Tensor
from tinygrad.dtype import dtypes
from openpilot.selfdrive.modeld.runners.tinygrad_helpers import qcom_tensor_from_opencl_address
os.environ['QCOM'] = '1'
else:
from openpilot.selfdrive.modeld.runners.ort_helpers import make_onnx_cpu_runner
import time
import pickle
import numpy as np
import cereal.messaging as messaging
from cereal import car, log
from pathlib import Path
from setproctitle import setproctitle
from cereal.messaging import PubMaster, SubMaster
from msgq.visionipc import VisionIpcClient, VisionStreamType, VisionBuf
@@ -33,13 +25,8 @@ from openpilot.selfdrive.modeld.fill_model_msg import fill_model_msg, fill_pose_
from openpilot.selfdrive.modeld.constants import ModelConstants
from openpilot.selfdrive.modeld.models.commonmodel_pyx import DrivingModelFrame, CLContext
PROCESS_NAME = "selfdrive.modeld.modeld"
SEND_RAW_PRED = os.getenv('SEND_RAW_PRED')
MODEL_PATH = Path(__file__).parent / 'models/supercombo.onnx'
MODEL_PKL_PATH = Path(__file__).parent / 'models/supercombo_tinygrad.pkl'
METADATA_PATH = Path(__file__).parent / 'models/supercombo_metadata.pkl'
class FrameMeta:
frame_id: int = 0
@@ -69,27 +56,12 @@ class ModelState:
'features_buffer': np.zeros((1, ModelConstants.HISTORY_BUFFER_LEN, ModelConstants.FEATURE_LEN), dtype=np.float32),
}
with open(METADATA_PATH, 'rb') as f:
model_metadata = pickle.load(f)
self.input_shapes = model_metadata['input_shapes']
self.output_slices = model_metadata['output_slices']
net_output_size = model_metadata['output_shapes']['outputs'][1]
self.output = np.zeros(net_output_size, dtype=np.float32)
# Initialize model runner
self.model_runner = TinygradRunner() if TICI else ONNXRunner(self.frames)
self.parser = Parser()
if TICI:
self.tensor_inputs = {k: Tensor(v, device='NPY').realize() for k,v in self.numpy_inputs.items()}
with open(MODEL_PKL_PATH, "rb") as f:
self.model_run = pickle.load(f)
else:
self.onnx_cpu_runner = make_onnx_cpu_runner(MODEL_PATH)
def slice_outputs(self, model_outputs: np.ndarray) -> dict[str, np.ndarray]:
parsed_model_outputs = {k: model_outputs[np.newaxis, v] for k,v in self.output_slices.items()}
if SEND_RAW_PRED:
parsed_model_outputs['raw_pred'] = model_outputs.copy()
return parsed_model_outputs
net_output_size = self.model_runner.model_metadata['output_shapes']['outputs'][1]
self.output = np.zeros(net_output_size, dtype=np.float32)
def run(self, buf: VisionBuf, wbuf: VisionBuf, transform: np.ndarray, transform_wide: np.ndarray,
inputs: dict[str, np.ndarray], prepare_only: bool) -> dict[str, np.ndarray] | None:
@@ -106,24 +78,15 @@ class ModelState:
imgs_cl = {'input_imgs': self.frames['input_imgs'].prepare(buf, transform.flatten()),
'big_input_imgs': self.frames['big_input_imgs'].prepare(wbuf, transform_wide.flatten())}
if TICI:
# The imgs tensors are backed by opencl memory, only need init once
for key in imgs_cl:
if key not in self.tensor_inputs:
self.tensor_inputs[key] = qcom_tensor_from_opencl_address(imgs_cl[key].mem_address, self.input_shapes[key], dtype=dtypes.uint8)
else:
for key in imgs_cl:
self.numpy_inputs[key] = self.frames[key].buffer_from_cl(imgs_cl[key]).reshape(self.input_shapes[key])
# Prepare inputs using the model runner
self.model_runner.prepare_inputs(imgs_cl, self.numpy_inputs)
if prepare_only:
return None
if TICI:
self.output = self.model_run(**self.tensor_inputs).numpy().flatten()
else:
self.output = self.onnx_cpu_runner.run(None, self.numpy_inputs)[0].flatten()
outputs = self.parser.parse_outputs(self.slice_outputs(self.output))
# Run model inference
self.output = self.model_runner.run_model()
outputs = self.parser.parse_outputs(self.model_runner.slice_outputs(self.output))
self.full_features_20Hz[:-1] = self.full_features_20Hz[1:]
self.full_features_20Hz[-1] = outputs['hidden_state'][0, :]
@@ -190,7 +153,6 @@ def main(demo=False):
meta_main = FrameMeta()
meta_extra = FrameMeta()
if demo:
CP = get_demo_car_params()
else:
+92
View File
@@ -0,0 +1,92 @@
import os
from openpilot.system.hardware import TICI
#
if TICI:
from tinygrad.tensor import Tensor
from tinygrad.dtype import dtypes
from openpilot.selfdrive.modeld.runners.tinygrad_helpers import qcom_tensor_from_opencl_address
os.environ['QCOM'] = '1'
else:
from openpilot.selfdrive.modeld.runners.ort_helpers import make_onnx_cpu_runner
import pickle
import numpy as np
from pathlib import Path
from abc import ABC, abstractmethod
from openpilot.selfdrive.modeld.models.commonmodel_pyx import DrivingModelFrame, CLMem
SEND_RAW_PRED = os.getenv('SEND_RAW_PRED')
MODEL_PATH = Path(__file__).parent / '../models/supercombo.onnx'
MODEL_PKL_PATH = Path(__file__).parent / '../models/supercombo_tinygrad.pkl'
METADATA_PATH = Path(__file__).parent / '../models/supercombo_metadata.pkl'
class ModelRunner(ABC):
"""Abstract base class for model runners that defines the interface for running ML models."""
def __init__(self):
"""Initialize the model runner with paths to model and metadata files."""
with open(METADATA_PATH, 'rb') as f:
self.model_metadata = pickle.load(f)
self.input_shapes = self.model_metadata['input_shapes']
self.output_slices = self.model_metadata['output_slices']
self.inputs: dict = {}
@abstractmethod
def prepare_inputs(self, imgs_cl: dict[str, CLMem], numpy_inputs: dict[str, np.ndarray])-> dict:
"""Prepare inputs for model inference."""
@abstractmethod
def run_model(self):
"""Run model inference with prepared inputs."""
def slice_outputs(self, model_outputs: np.ndarray) -> dict:
"""Slice model outputs according to metadata configuration."""
parsed_outputs = {k: model_outputs[np.newaxis, v] for k, v in self.output_slices.items()}
if SEND_RAW_PRED:
parsed_outputs['raw_pred'] = model_outputs.copy()
return parsed_outputs
class TinygradRunner(ModelRunner):
"""Tinygrad implementation of model runner for TICI hardware."""
def __init__(self):
super().__init__()
# Load Tinygrad model
with open(MODEL_PKL_PATH, "rb") as f:
self.model_run = pickle.load(f)
def prepare_inputs(self, imgs_cl: dict[str, CLMem], numpy_inputs: dict[str, np.ndarray]) -> dict:
# Initialize image tensors if not already done
for key in imgs_cl:
if key not in self.inputs:
self.inputs[key] = qcom_tensor_from_opencl_address(imgs_cl[key].mem_address, self.input_shapes[key], dtype=dtypes.uint8)
# Update numpy inputs
for k, v in numpy_inputs.items():
if k not in self.inputs:
self.inputs[k] = Tensor(v, device='NPY').realize()
return self.inputs
def run_model(self):
return self.model_run(**self.inputs).numpy().flatten()
class ONNXRunner(ModelRunner):
"""ONNX implementation of model runner for non-TICI hardware."""
def __init__(self, frames: dict[str, DrivingModelFrame]):
super().__init__()
self.runner = make_onnx_cpu_runner(MODEL_PATH)
self.frames = frames
def prepare_inputs(self, imgs_cl: dict[str, CLMem], numpy_inputs: dict[str, np.ndarray]) -> dict:
self.inputs = numpy_inputs.copy()
for key in imgs_cl:
self.inputs[key] = self.frames[key].buffer_from_cl(imgs_cl[key]).reshape(self.input_shapes[key])
return self.inputs
def run_model(self):
return self.runner.run(None, self.inputs)[0].flatten()
@@ -24,6 +24,9 @@ DeveloperPanel::DeveloperPanel(SettingsWindow *parent) : ListWidget(parent) {
});
addItem(longManeuverToggle);
auto enableGithubRunner = new ParamControl("EnableGithubRunner", tr("Enable GitHub runner service"), tr("Enables or disables the github runner service."), "");
addItem(enableGithubRunner);
// Joystick and longitudinal maneuvers should be hidden on release branches
is_release = params.getBool("IsReleaseBranch");
+8
View File
@@ -123,6 +123,14 @@
<source>Longitudinal Maneuver Mode</source>
<translation>وضع المناورة الطولية</translation>
</message>
<message>
<source>Enable GitHub runner service</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enables or disables the github runner service.</source>
<translation type="unfinished"></translation>
</message>
</context>
<context>
<name>DevicePanel</name>
+8
View File
@@ -123,6 +123,14 @@
<source>Longitudinal Maneuver Mode</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enable GitHub runner service</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enables or disables the github runner service.</source>
<translation type="unfinished"></translation>
</message>
</context>
<context>
<name>DevicePanel</name>
+8
View File
@@ -123,6 +123,14 @@
<source>Longitudinal Maneuver Mode</source>
<translation>Modo de maniobra longitudinal</translation>
</message>
<message>
<source>Enable GitHub runner service</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enables or disables the github runner service.</source>
<translation type="unfinished"></translation>
</message>
</context>
<context>
<name>DevicePanel</name>
+8
View File
@@ -123,6 +123,14 @@
<source>Longitudinal Maneuver Mode</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enable GitHub runner service</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enables or disables the github runner service.</source>
<translation type="unfinished"></translation>
</message>
</context>
<context>
<name>DevicePanel</name>
+8
View File
@@ -123,6 +123,14 @@
<source>Longitudinal Maneuver Mode</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enable GitHub runner service</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enables or disables the github runner service.</source>
<translation type="unfinished"></translation>
</message>
</context>
<context>
<name>DevicePanel</name>
+8
View File
@@ -123,6 +123,14 @@
<source>Longitudinal Maneuver Mode</source>
<translation> </translation>
</message>
<message>
<source>Enable GitHub runner service</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enables or disables the github runner service.</source>
<translation type="unfinished"></translation>
</message>
</context>
<context>
<name>DevicePanel</name>
+8
View File
@@ -123,6 +123,14 @@
<source>Longitudinal Maneuver Mode</source>
<translation>Modo Longitudinal Maneuver</translation>
</message>
<message>
<source>Enable GitHub runner service</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enables or disables the github runner service.</source>
<translation type="unfinished"></translation>
</message>
</context>
<context>
<name>DevicePanel</name>
+8
View File
@@ -123,6 +123,14 @@
<source>Longitudinal Maneuver Mode</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enable GitHub runner service</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enables or disables the github runner service.</source>
<translation type="unfinished"></translation>
</message>
</context>
<context>
<name>DevicePanel</name>
+8
View File
@@ -123,6 +123,14 @@
<source>Longitudinal Maneuver Mode</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enable GitHub runner service</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enables or disables the github runner service.</source>
<translation type="unfinished"></translation>
</message>
</context>
<context>
<name>DevicePanel</name>
+8
View File
@@ -123,6 +123,14 @@
<source>Longitudinal Maneuver Mode</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enable GitHub runner service</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enables or disables the github runner service.</source>
<translation type="unfinished"></translation>
</message>
</context>
<context>
<name>DevicePanel</name>
+8
View File
@@ -123,6 +123,14 @@
<source>Longitudinal Maneuver Mode</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enable GitHub runner service</source>
<translation type="unfinished"></translation>
</message>
<message>
<source>Enables or disables the github runner service.</source>
<translation type="unfinished"></translation>
</message>
</context>
<context>
<name>DevicePanel</name>
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env bash
# Define the service name
SERVICE_NAME="actions.runner.sunnypilot.$(uname -n)"
# Function to control the service
control_service() {
local action=$1 # Store the function argument in a local variable
sudo systemctl $action ${SERVICE_NAME}
}
service_exists_and_is_loaded() {
sudo systemctl status ${SERVICE_NAME} &>/dev/null
if [[ $? -ne 4 ]]; then
return 0 # Service is known to systemd (i.e., loaded)
else
return 1 # Service is unknown to systemd (i.e., not loaded)
fi
}
# Check for required argument
if [[ -z $1 ]] || { [[ $1 != "start" ]] && [[ $1 != "stop" ]]; }; then
echo "Usage: $0 {start|stop}"
exit 1
fi
# Store the script argument in a descriptive variable
ACTION=$1
# Trap EXIT signal (Ctrl+C) and stop the service
trap 'control_service stop ; exit' SIGINT SIGKILL EXIT
# Enter the main loop
while true; do
# Check if the service is actually present on the system
if service_exists_and_is_loaded; then
control_service $ACTION # Call the function with the specified action
fi
sleep 1 # Pause before the next iteration
done
+6
View File
@@ -54,6 +54,9 @@ def only_onroad(started: bool, params: Params, CP: car.CarParams) -> bool:
def only_offroad(started: bool, params: Params, CP: car.CarParams) -> bool:
return not started
def use_github_runner(started, params, CP: car.CarParams) -> bool:
return not PC and params.get_bool("EnableGithubRunner") and not params.get_bool("NetworkMetered")
def or_(*fns):
return lambda *args: operator.or_(*(fn(*args) for fn in fns))
@@ -111,4 +114,7 @@ procs = [
PythonProcess("joystick", "tools.joystick.joystick_control", and_(joystick, iscar)),
]
if os.path.exists("./github_runner.sh"):
procs += [NativeProcess("github_runner_start", "system/manager", ["./github_runner.sh", "start"], and_(only_offroad, use_github_runner), sigkill=False)]
managed_processes = {p.name: p for p in procs}