From ad537fcb89f4ab850c7cb6600d88453f68818a5f Mon Sep 17 00:00:00 2001 From: DevTekVE Date: Sat, 21 Dec 2024 20:19:48 +0100 Subject: [PATCH 1/6] CI: Build prebuilts and models from github and self hosted runners (#476) * Create a GitHub Actions workflow for synchronizing the repository to GitLab and enhance GitLab CI settings A new GitHub Actions workflow for mirroring the current repository to GitLab is added. This workflow is triggered by both push and delete events in addition to manual triggering. The role of GitHub actions runner is defined through a series of steps. A new '.gitlab-ci.yml' build configuration file is also introduced, with a more comprehensive definition of variables, jobs, and pipeline rules for its utilization in the GitLab CI/CD pipeline. Further, other changes include the addition of scripts for installing and uninstalling GitLab CI runner, along with modifying the SCons build system configuration file to include custom cache directory. Moreover, 'release_files.py' has been revised to include additional blacklisted and whitelisted files specific to Sunnypilot, ensuring suitable settings for the CI flow. The improvements facilitate a smoother integration between GitHub and GitLab, powerfully harnessing the capabilities of both platforms for more efficient and effective CI/CD pipelines and version control management. * not needed for this * Update workflow to build model from upstream repository Revised the CI workflow to build directly from the upstream `commaai/openpilot` repository. Simplified configuration, removed unused steps, and added support for specifying the upstream branch dynamically via inputs. * Update SConstruct to allow passing arbitrary cache_dir Modified the SConstruct file to enable setting a custom cache directory via arguments. This enhances flexibility in configuring cache paths during the build process. * Refactor build workflow to improve branch handling logic. Reorganized conditions for setting environment variables, replacing repository_dispatch with workflow_dispatch for prebuilt builds. Added a fallback error message for unsupported triggers to improve robustness. This enhances clarity and ensures compatibility with the intended workflow triggers. * test test test * test test test * test test test * Enable publication flag during build configurations Added `SHOULD_PUBLISH=true` to all relevant build configuration steps to ensure proper handling of publishing logic. Updated environment variables to include this flag for downstream usage. Removed the error-check step for unsupported configurations. * Simplify publish condition in workflow logic Replaced the previous condition for publishing with a single output variable, `should_publish`, to streamline logic and improve maintainability. This change reduces redundancy and makes the workflow more adaptable to future updates. * Simplify restore key patterns in build workflow. Removed unnecessary trailing dashes in SCons cache restore keys to streamline and slightly improve key matching logic. This ensures better consistency with the current workflow setup. * Update cache key usage in build workflow Replaced `github.ref_name` with `github.head_ref` for cache keys to ensure accurate branch-specific caching. Added fallback restore-keys for master branches to improve cache efficiency and reduce redundant builds. * Improved debug logging in GitHub actions workflow This commit refines the debug logging in our GitHub Actions workflow for the Sunnypilot build. This change provides more granularity, enabling logging only during debug mode, which helps to keep the runtime logs less cluttered during normal operations. This includes the conditionally displaying of environmental variables, GitHub output contents, and directory listings. Additionally, debug mode verbosity was added to rsync commands to aid troubleshooting file transfers during the build process. Blank lines were also reduced for better readability and cleaner code presentation. * test diff path * Refactor SCons cache handling in build workflow Replaced hardcoded cache directory paths with an environment variable (`SCONS_CACHE_DIR`) for better maintainability and flexibility. Updated related workflow steps to utilize the new variable and adjusted cache key usage. Removed unused `BASE_BUILD_NUMBER` variable to clean up the configuration. * Update cache key to include commit SHA in workflow This change adds the commit SHA to the cache key in the GitHub Actions workflow. It ensures more precise caching by differentiating builds based on the specific commit, reducing potential conflicts. * clean Update GitHub runner service to set environment variables Updated the ExecStart command to explicitly set HOME, USER, LOGNAME, and MAIL environment variables. This ensures the runner operates with the correct environment configuration, improving reliability and compatibility. Refactor GitHub runner service ExecStart command. Replaced direct command execution with running the service as a specific user using `su`. This improves compatibility and aligns with best practices for user-based execution. No functional changes are expected. Update GitHub runner service to set environment variables The ExecStart command now sets HOME, USER, LOGNAME, and MAIL environment variables for the runner process. This ensures proper environment initialization for the designated user, improving compatibility and reliability during execution. Refactor GitHub runner service template handling. Revised the `modify_service_template` function to create a properly structured service template for the GitHub runner. Updated service permissions, execution parameters, and enabled the function call to ensure usage during runner setup. * Refactor GitHub Runner installer to improve argument parsing. Reworked the script to implement flexible and explicit command-line argument parsing using flags like `--token`, `--repo`, and `--start-at-boot`. Added support for setting default values and enabling/disabling auto-start based on the `--start-at-boot` flag. Improved error handling and usage messaging for better user experience. * Update cache keys in sunnypilot build workflow Modified the cache keys to include `github.ref_name` for more precise caching and restore behavior. This improves build consistency by better differentiating between branches and refs. No changes to the overall workflow logic. * Remove 'tinygrad/*' from release file exclusions This change modifies the release file exclusions by removing 'tinygrad/*'. The adjustment ensures that files in the 'tinygrad' directory are now included in the release process, aligning with updated packaging requirements. * Simplify release file exclusions list. Removed redundant and unnecessary file patterns from the exclusions list in `release_files.py`. This streamlines the file handling process and reduces maintenance overhead. * Refactor SCons cache path and key structure. Updated the SCons cache directory path to `SCONS_CACHE_DIR` for clarity and consistency. Improved the caching key structure to include `github.head_ref` for better cache differentiation and restore hierarchy. Adjusted related build instructions to reflect the new environment variable. * Refactor branch configuration in CI workflow Standardize branch name handling by replacing hardcoded values with environment variables. This improves maintainability and simplifies updates to branch names across the workflow. Updated references to use the new dynamic environment variable approach. --- .github/workflows/sunnypilot-build-model.yaml | 82 ++++++ .../workflows/sunnypilot-build-prebuilt.yaml | 251 ++++++++++++++++++ SConstruct | 3 +- .../discord_template_notify_dev_private.json | 13 + .../discord_template_notify_dev_public.json | 9 + release/ci/install_github_runner.sh | 141 ++++++++++ release/ci/publish.sh | 78 ++++++ release/ci/uninstall_github_runner.sh | 60 +++++ release/release_files.py | 75 ++++++ 9 files changed, 711 insertions(+), 1 deletion(-) create mode 100644 .github/workflows/sunnypilot-build-model.yaml create mode 100644 .github/workflows/sunnypilot-build-prebuilt.yaml create mode 100644 release/ci/discord_template_notify_dev_private.json create mode 100644 release/ci/discord_template_notify_dev_public.json create mode 100755 release/ci/install_github_runner.sh create mode 100755 release/ci/publish.sh create mode 100755 release/ci/uninstall_github_runner.sh diff --git a/.github/workflows/sunnypilot-build-model.yaml b/.github/workflows/sunnypilot-build-model.yaml new file mode 100644 index 0000000000..bc4bd4f4a1 --- /dev/null +++ b/.github/workflows/sunnypilot-build-model.yaml @@ -0,0 +1,82 @@ +name: Build Model from Upstream + +env: + BUILD_DIR: "/data/github/openpilot" + OUTPUT_DIR: ${{ github.workspace }}/output + SCONS_CACHE_DIR: ${{ github.workspace }}/release/ci/scons_cache + UPSTREAM_REPO: "commaai/openpilot" + +on: + workflow_dispatch: + inputs: + upstream_branch: + description: 'Upstream branch to build from' + required: true + default: 'master' + type: string + +jobs: + build_model: + runs-on: self-hosted + + steps: + - uses: actions/checkout@v4 + with: + repository: ${{ env.UPSTREAM_REPO }} + ref: ${{ github.event.inputs.upstream_branch }} + submodules: recursive + + - run: git lfs pull + + - name: Cache SCons + uses: actions/cache@v4 + with: + path: ${{env.SCONS_CACHE_DIR}} + key: scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }}-model-${{ github.head_ref }}-${{ github.sha }} + restore-keys: | + scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }}-model-${{ github.head_ref }} + scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }}-model + scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }} + scons-${{ runner.os }}-${{ runner.arch }}-master-new + scons-${{ runner.os }}-${{ runner.arch }}-master + scons-${{ runner.os }}-${{ runner.arch }} + + - name: Setup build environment + run: | + mkdir -p "${BUILD_DIR}/" + sudo find $BUILD_DIR/ -mindepth 1 -delete + echo "Starting build stage..." + echo "Building from: ${{ env.UPSTREAM_REPO }} branch: ${{ github.event.inputs.upstream_branch }}" + + - name: Patch SConstruct to pass arbitrary cache + run: | + sed -i.bak 's#cache_dir =#default_cache_dir =#' ${{ github.workspace }}/SConstruct + printf '/default_cache_dir/a\\\ncache_dir = ARGUMENTS.get("cache_dir", default_cache_dir)\n' | sed -i.bak -f - ${{ github.workspace }}/SConstruct + cat ${{ github.workspace }}/SConstruct + + - name: Build Model + run: | + source /etc/profile + export UV_PROJECT_ENVIRONMENT=${HOME}/venv + export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT + scons -j$(nproc) cache_dir=${{ env.SCONS_CACHE_DIR }} ${{ github.workspace }}/selfdrive/modeld + + - name: Prepare Output + run: | + sudo rm -rf ${OUTPUT_DIR} + mkdir -p ${OUTPUT_DIR} + rsync -avm \ + --include='*.dlc' \ + --include='*.thneed' \ + --include='*.pkl' \ + --include='*.onnx' \ + --exclude='*' \ + --delete-excluded \ + --chown=comma:comma \ + ./selfdrive/modeld/models/ ${OUTPUT_DIR}/ + + - name: Upload Build Artifacts + uses: actions/upload-artifact@v4 + with: + name: model-${{ github.event.inputs.upstream_branch }}-${{ github.run_number }} + path: ${{ env.OUTPUT_DIR }} \ No newline at end of file diff --git a/.github/workflows/sunnypilot-build-prebuilt.yaml b/.github/workflows/sunnypilot-build-prebuilt.yaml new file mode 100644 index 0000000000..147a87d6d1 --- /dev/null +++ b/.github/workflows/sunnypilot-build-prebuilt.yaml @@ -0,0 +1,251 @@ +name: sunnypilot prebuilt action + +env: + BUILD_DIR: "/data/openpilot" + OUTPUT_DIR: ${{ github.workspace }}/output + CI_DIR: ${{ github.workspace }}/release/ci + SCONS_CACHE_DIR: ${{ github.workspace }}/release/ci/scons_cache + PUBLIC_REPO_URL: "https://github.com/sunnyhaibin/sunnypilot" + + # Branch configurations + MASTER_BRANCH: "master" + MASTER_NEW_BRANCH: "master-new" + DEV_C3_SOURCE_BRANCH: "master-dev-c3-new" + + # Target branch configurations + STAGING_TARGET_BRANCH: "staging-c3-new" + DEV_TARGET_BRANCH: "dev-c3-new" + RELEASE_TARGET_BRANCH: "release-c3-new" + +on: + push: + branches: [ master, master-new, master-dev-c3-new ] + tags: [ '*' ] + pull_request: + branches: [ master, master-new ] + workflow_dispatch: + inputs: + extra_version: + description: 'Extra version identifier' + required: false + default: '' + +jobs: + build: + runs-on: self-hosted + outputs: + new_branch: ${{ steps.set-env.outputs.new_branch }} + version: ${{ steps.set-env.outputs.version }} + extra_version_identifier: ${{ steps.set-env.outputs.extra_version_identifier }} + should_publish: ${{ steps.set-env.outputs.should_publish }} + steps: + - uses: actions/checkout@v4 + with: + submodules: recursive + - run: git lfs pull + + - name: Cache SCons + uses: actions/cache@v4 + with: + path: ${{env.SCONS_CACHE_DIR}} + key: scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }}-${{ github.head_ref }}-${{ github.sha }} + restore-keys: | + scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }}-${{ github.head_ref }} + scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }} + scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_NEW_BRANCH }} + scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_BRANCH }} + scons-${{ runner.os }}-${{ runner.arch }} + + - name: Configure for dev branch + if: github.ref_name == env.DEV_C3_SOURCE_BRANCH + run: | + echo "BRANCH_TYPE=dev" >> $GITHUB_ENV + echo "NEW_BRANCH=${{ env.DEV_TARGET_BRANCH }}" >> $GITHUB_ENV + echo "EXTRA_VERSION_IDENTIFIER=${{ github.run_number }}" >> $GITHUB_ENV + echo "SHOULD_PUBLISH=true" >> $GITHUB_ENV + + - name: Configure for master branches + if: github.ref_name == env.MASTER_BRANCH || github.ref_name == env.MASTER_NEW_BRANCH + run: | + echo "BRANCH_TYPE=master" >> $GITHUB_ENV + echo "NEW_BRANCH=${{ env.STAGING_TARGET_BRANCH }}" >> $GITHUB_ENV + echo "EXTRA_VERSION_IDENTIFIER=staging" >> $GITHUB_ENV + echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')" >> $GITHUB_ENV + echo "SHOULD_PUBLISH=true" >> $GITHUB_ENV + + - name: Configure for tags + if: startsWith(github.ref, 'refs/tags/') + run: | + echo "BRANCH_TYPE=tag" >> $GITHUB_ENV + echo "NEW_BRANCH=${{ env.RELEASE_TARGET_BRANCH }}" >> $GITHUB_ENV + echo "EXTRA_VERSION_IDENTIFIER=release" >> $GITHUB_ENV + echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')" >> $GITHUB_ENV + echo "SHOULD_PUBLISH=true" >> $GITHUB_ENV + + - name: Configure for manual build + if: github.event_name == 'workflow_dispatch' + run: | + echo "BRANCH_TYPE=dispatch" >> $GITHUB_ENV + echo "NEW_BRANCH=${{ github.ref_name }}-prebuilt" >> $GITHUB_ENV + echo "VERSION=$(date '+%Y.%m.%d')-${{ github.run_number }}" >> $GITHUB_ENV + echo "SHOULD_PUBLISH=true" >> $GITHUB_ENV + + - name: Set environment variables + id: set-env + run: | + # Write to GITHUB_OUTPUT from environment variables + echo "new_branch=$NEW_BRANCH" >> $GITHUB_OUTPUT + [[ ! -z "$EXTRA_VERSION_IDENTIFIER" ]] && echo "extra_version_identifier=$EXTRA_VERSION_IDENTIFIER" >> $GITHUB_OUTPUT + [[ ! -z "$VERSION" ]] && echo "version=$VERSION" >> $GITHUB_OUTPUT + echo "should_publish=${SHOULD_PUBLISH:-false}" >> $GITHUB_OUTPUT + + # Set up common environment + source /etc/profile; + export UV_PROJECT_ENVIRONMENT=${HOME}/venv + export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT + printenv >> $GITHUB_ENV + if [[ "${{ runner.debug }}" == "1" ]]; then + cat $GITHUB_OUTPUT + fi + + - name: Setup build environment + run: | + mkdir -p "${BUILD_DIR}/" + sudo find $BUILD_DIR/ -mindepth 1 -delete + echo "Starting build stage..." + echo "BUILD_DIR: ${BUILD_DIR}" + echo "CI_DIR: ${CI_DIR}" + echo "VERSION: ${{ steps.set-env.outputs.version }}" + echo "UV_PROJECT_ENVIRONMENT: ${UV_PROJECT_ENVIRONMENT}" + echo "VIRTUAL_ENV: ${VIRTUAL_ENV}" + echo "-------" + if [[ "${{ runner.debug }}" == "1" ]]; then + printenv + fi + + - name: Build Panda + run: | + scons -j$(nproc) cache_dir=${{env.SCONS_CACHE_DIR}} ${{ github.workspace }}/panda + + - name: Build Main Project + run: | + export PYTHONPATH="$BUILD_DIR" + ./release/release_files.py | sort | uniq | rsync -rRl${RUNNER_DEBUG:+v} --files-from=- . $BUILD_DIR/ + cd $BUILD_DIR + sed -i '/from .board.jungle import PandaJungle, PandaJungleDFU/s/^/#/' panda/__init__.py + scons -j$(nproc) cache_dir=${{env.SCONS_CACHE_DIR}} --minimal + touch ${BUILD_DIR}/prebuilt + if [[ "${{ runner.debug }}" == "1" ]]; then + ls -la ${BUILD_DIR} + fi + + - name: Prepare Output + run: | + sudo rm -rf ${OUTPUT_DIR} + mkdir -p ${OUTPUT_DIR} + rsync -am${RUNNER_DEBUG:+v} \ + --include='**/panda/board/' \ + --include='**/panda/board/obj' \ + --include='**/panda/board/obj/panda.bin.signed' \ + --include='**/panda/board/obj/panda_h7.bin.signed' \ + --include='**/panda/board/obj/bootstub.panda.bin' \ + --include='**/panda/board/obj/bootstub.panda_h7.bin' \ + --exclude='.sconsign.dblite' \ + --exclude='*.a' \ + --exclude='*.o' \ + --exclude='*.os' \ + --exclude='*.pyc' \ + --exclude='moc_*' \ + --exclude='*.cc' \ + --exclude='Jenkinsfile' \ + --exclude='supercombo.onnx' \ + --exclude='**/panda/board/*' \ + --exclude='**/panda/board/obj/**' \ + --exclude='**/panda/certs/' \ + --exclude='**/panda/crypto/' \ + --exclude='**/release/' \ + --exclude='**/.github/' \ + --exclude='**/selfdrive/ui/replay/' \ + --exclude='**/__pycache__/' \ + --exclude='**/selfdrive/ui/*.h' \ + --exclude='**/selfdrive/ui/**/*.h' \ + --exclude='**/selfdrive/ui/qt/offroad/sunnypilot/' \ + --exclude='${{env.SCONS_CACHE_DIR}}' \ + --exclude='**/.git/' \ + --exclude='**/SConstruct' \ + --exclude='**/SConscript' \ + --exclude='**/.venv/' \ + --delete-excluded \ + --chown=comma:comma \ + ${BUILD_DIR}/ ${OUTPUT_DIR}/ + + - name: 'Tar.gz files' + run: | + tar czf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }} . + ls -la prebuilt.tar.gz + + - name: 'Upload Artifact' + uses: actions/upload-artifact@v4 + with: + name: prebuilt + path: prebuilt.tar.gz + + publish-public: + needs: build + runs-on: ubuntu-latest + if: needs.build.outputs.should_publish == 'true' + steps: + - uses: actions/checkout@v4 + + - name: Download build artifacts + uses: actions/download-artifact@v4 + with: + name: prebuilt + + - name: Untar prebuilt + run: | + mkdir -p ${{ env.OUTPUT_DIR }} + tar xzf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }} + + - name: Configure Git + run: | + git config --global user.email "github-actions[bot]@users.noreply.github.com" + git config --global user.name "github-actions[bot]" + + - name: Publish to Public Repository + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + echo '${{ toJSON(needs.build.outputs) }}' + ls -la ${{ env.OUTPUT_DIR }} + + ${{ env.CI_DIR }}/publish.sh \ + "${{ github.workspace }}" \ + "${{ env.OUTPUT_DIR }}" \ + "${{ needs.build.outputs.new_branch }}" \ + "${{ needs.build.outputs.version }}" \ + "https://x-access-token:${{github.token}}@github.com/sunnypilot/sunnypilot.git" \ + "-${{ needs.build.outputs.extra_version_identifier }}" + + notify: + needs: [ build, publish-public ] + runs-on: ubuntu-latest + if: needs.build.outputs.new_branch != '' + + steps: + - uses: actions/checkout@v4 + + - name: Install dependencies + run: | + sudo apt-get update + sudo apt-get install -y curl jq gettext-base + + - name: Send Discord Notification + env: + DISCORD_WEBHOOK: ${{ needs.build.outputs.new_branch == 'dev-c3' && secrets.DISCORD_NEW_BUILD_WEBHOOK_URL || secrets.DISCORD_MANUAL_BUILD_WEBHOOK_URL }} + run: | + TEMPLATE="${{ needs.build.outputs.new_branch == 'dev-c3' && 'discord_template_notify_dev_public.json' || 'discord_template_notify_dev_private.json' }}" + export EXTRA_VERSION_IDENTIFIER="${{ needs.build.outputs.extra_version_identifier }}" + export VERSION="${{ needs.build.outputs.version }}" + cat release/ci/${TEMPLATE} | envsubst > payload.json + curl -X POST -H "Content-Type: application/json" -d @payload.json $DISCORD_WEBHOOK \ No newline at end of file diff --git a/SConstruct b/SConstruct index d1661e7b6c..934143fa08 100644 --- a/SConstruct +++ b/SConstruct @@ -237,7 +237,8 @@ if GetOption('compile_db'): env.CompilationDatabase('compile_commands.json') # Setup cache dir -cache_dir = '/data/scons_cache' if AGNOS else '/tmp/scons_cache' +default_cache_dir = '/data/scons_cache' if AGNOS else '/tmp/scons_cache' +cache_dir = ARGUMENTS.get('cache_dir', default_cache_dir) CacheDir(cache_dir) Clean(["."], cache_dir) diff --git a/release/ci/discord_template_notify_dev_private.json b/release/ci/discord_template_notify_dev_private.json new file mode 100644 index 0000000000..5da1e4d405 --- /dev/null +++ b/release/ci/discord_template_notify_dev_private.json @@ -0,0 +1,13 @@ +{ + "embeds": [ + { + "title": "❗️ Action Required for `${CI_COMMIT_REF_NAME}` ❗️", + "description": "[${CI_PROJECT_NAME}](${CI_PROJECT_URL}): Pipeline [#${CI_PIPELINE_ID}](${CI_PROJECT_URL}/-/pipelines/${CI_PIPELINE_ID}) of branch [${CI_COMMIT_REF_NAME}](${CI_PROJECT_URL}/-/commits/${CI_COMMIT_REF_NAME}) by ${GITLAB_USER_NAME} (${GITLAB_USER_LOGIN}) is ready to publish manually as [${NEW_BRANCH}](${PUBLIC_REPO_URL}/tree/${NEW_BRANCH})", + "color": 16763904, + "author": { + "name": "${GITLAB_USER_LOGIN}", + "icon_url": "${AVATAR_URL}" + } + } + ] +} diff --git a/release/ci/discord_template_notify_dev_public.json b/release/ci/discord_template_notify_dev_public.json new file mode 100644 index 0000000000..c933ea10b4 --- /dev/null +++ b/release/ci/discord_template_notify_dev_public.json @@ -0,0 +1,9 @@ +{ + "embeds": [ + { + "title": "🎉 sunnypilot `${NEW_BRANCH}` New Update 🎉", + "description": "[sunnypilot](${PUBLIC_REPO_URL}): Build #${EXTRA_VERSION_IDENTIFIER} of branch [${NEW_BRANCH}](${PUBLIC_REPO_URL}/tree/${NEW_BRANCH}) has been published.\n\nDrive safe! 🚗💨", + "color": 4321431 + } + ] +} diff --git a/release/ci/install_github_runner.sh b/release/ci/install_github_runner.sh new file mode 100755 index 0000000000..3aa6b79dd3 --- /dev/null +++ b/release/ci/install_github_runner.sh @@ -0,0 +1,141 @@ +#!/usr/bin/env bash +set -e + +# Default values +DEFAULT_REPO_URL="https://github.com/sunnypilot" +START_AT_BOOT=false + +# Parse command line arguments +while [[ $# -gt 0 ]]; do + case $1 in + --start-at-boot) + START_AT_BOOT=true + shift + ;; + --token) + GITHUB_TOKEN="$2" + shift 2 + ;; + --repo) + REPO_URL="$2" + shift 2 + ;; + *) + if [ -z "$GITHUB_TOKEN" ]; then + GITHUB_TOKEN="$1" + elif [ -z "$REPO_URL" ]; then + REPO_URL="$1" + fi + shift + ;; + esac +done + +# Check required arguments +if [ -z "$GITHUB_TOKEN" ]; then + echo "Usage: $0 [--start-at-boot] [--token ] [--repo ]" + echo "Required argument: github_token" + echo "Optional arguments:" + echo " --start-at-boot Enable auto-start at boot (default: false)" + echo " --repo Repository URL (default: ${DEFAULT_REPO_URL})" + exit 1 +fi + +# Set repository URL if not provided +REPO_URL="${REPO_URL:-$DEFAULT_REPO_URL}" + +# Constants +RUNNER_USER="github-runner" +USER_GROUPS="comma,gpu,gpio,sudo" +BASE_DIR="/data/github" +RUNNER_DIR="${BASE_DIR}/runner" +BUILDS_DIR="${BASE_DIR}/builds" +LOGS_DIR="${BASE_DIR}/logs" +CACHE_DIR="${BASE_DIR}/cache" +OPENPILOT_DIR="${BASE_DIR}/openpilot" + +create_directories() { + sudo mkdir -p "$RUNNER_DIR" "$BUILDS_DIR" "$LOGS_DIR" "$CACHE_DIR" "$OPENPILOT_DIR" + mkdir -p "/data/openpilot" + sudo chown -R comma:comma "/data/openpilot" +} + +download_and_setup_runner() { + cd "$RUNNER_DIR" + curl -o actions-runner-linux-arm64-2.321.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.321.0/actions-runner-linux-arm64-2.321.0.tar.gz + tar xzf ./actions-runner-linux-arm64-2.321.0.tar.gz + rm ./actions-runner-linux-arm64-2.321.0.tar.gz + chmod +x ./config.sh +} + +setup_runner_user() { + sudo useradd --comment 'GitHub Runner' --create-home --home-dir ${BASE_DIR} ${RUNNER_USER} --shell /bin/bash -G ${USER_GROUPS} || sudo usermod -aG ${USER_GROUPS} ${RUNNER_USER} + export BASE_DIR + sudo -u ${RUNNER_USER} bash -c "truncate -s 0 '${BASE_DIR}/.bash_logout'" +} + +create_sudoers_entry() { + sudo grep -qxF "${RUNNER_USER} ALL=(ALL) NOPASSWD: ALL" /etc/sudoers || echo "${RUNNER_USER} ALL=(ALL) NOPASSWD: ALL" | sudo tee -a /etc/sudoers +} + +configure_runner() { + cd "$RUNNER_DIR" + sudo -u ${RUNNER_USER} ./config.sh --url "$REPO_URL" --token "$GITHUB_TOKEN" --name $(hostname) --runnergroup "tici-tizi" --labels "tici" --work "$BUILDS_DIR" --unattended +} + +set_directory_permissions() { + sudo chown -R ${RUNNER_USER}:comma "$BASE_DIR" + sudo chmod g+rwx "$BASE_DIR" + sudo chmod g+s "$BASE_DIR" +} + +modify_service_template() { + cat < "$RUNNER_DIR/bin/actions.runner.service.template" +[Unit] +Description={{Description}} +After=network-online.target nss-lookup.target time-sync.target +Wants=network-online.target nss-lookup.target time-sync.target +StartLimitInterval=5 +StartLimitBurst=10 + +[Service] +Type=simple +User=root +ExecStart=/usr/bin/unshare -m -- /bin/bash -c 'mount --bind ${OPENPILOT_DIR} /data/openpilot && setpriv --reuid={{User}} --regid={{User}} --init-groups env HOME=${BASE_DIR} USER={{User}} LOGNAME={{User}} MAIL=/var/mail/{{User}} {{RunnerRoot}}/runsvc.sh' +WorkingDirectory={{RunnerRoot}} +KillMode=process +KillSignal=SIGTERM +TimeoutStopSec=5min +Restart=always +RestartSec=120 + +[Install] +WantedBy=multi-user.target +EOL +} + +# Make filesystem writable +sudo mount -o remount,rw / + +# Ensure filesystem is remounted as read-only on script exit +trap "sudo mount -o remount,ro /" EXIT + +# Execute installation steps +setup_runner_user +create_sudoers_entry +create_directories +download_and_setup_runner +modify_service_template +configure_runner +set_directory_permissions + +# Install and start service using built-in installer +cd "$RUNNER_DIR" +sudo ./svc.sh install $RUNNER_USER + +# Handle auto-start configuration +if [ "$START_AT_BOOT" = false ]; then + sudo systemctl disable actions.runner.sunnypilot.$(uname -n) +fi + +sudo ./svc.sh start \ No newline at end of file diff --git a/release/ci/publish.sh b/release/ci/publish.sh new file mode 100755 index 0000000000..d723782934 --- /dev/null +++ b/release/ci/publish.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash + +set -e + +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)" +cd $DIR + +# Take parameters as arguments +SOURCE_DIR=$1 +OUTPUT_DIR=$2 +DEV_BRANCH=$3 +VERSION=$4 +GIT_ORIGIN=$5 +EXTRA_VERSION_IDENTIFIER=$6 + +# Check parameters +if [ -z "$SOURCE_DIR" ] || [ -z "$OUTPUT_DIR" ]; then + echo "Error: No source or output directory provided." + exit 1 +fi + +if [ -z "$DEV_BRANCH" ] || [ -z "$VERSION" ]; then + echo "Error: No dev branch or version provided." + exit 1 +fi + +if [ -z "$GIT_ORIGIN" ]; then + echo "Error: No GIT_ORIGIN provided" + exit 1 +fi + +# "Tagging" +echo "#define COMMA_VERSION \"$VERSION\"" > ${OUTPUT_DIR}/common/version.h + +## set git identity +#source $DIR/identity.sh +#export GIT_SSH_COMMAND="ssh -i /data/gitkey" + +echo "[-] Setting up repo T=$SECONDS" +cd $OUTPUT_DIR +git init + +# set git username/password +#source /data/identity.sh + +git rm -rf $OUTPUT_DIR/.git || true # Doing cleanup, but it might fail if the .git doesn't exist or not allowed to delete +git remote remove origin || true # ensure cleanup +git remote add origin $GIT_ORIGIN +#git push origin -d $DEV_BRANCH || true # Ensuring we delete the remote branch if it exists as we are wiping it out +git fetch origin $DEV_BRANCH || (git checkout -b $DEV_BRANCH && git commit --allow-empty -m "sunnypilot v$VERSION release" && git push -u origin $DEV_BRANCH) + +echo "[-] committing version $VERSION T=$SECONDS" +git add -f . +git commit -a -m "sunnypilot v$VERSION release" +git branch --set-upstream-to=origin/$DEV_BRANCH + +# include source commit hash and build date in commit +GIT_HASH=$(git --git-dir=$SOURCE_DIR/.git rev-parse HEAD) +DATETIME=$(date '+%Y-%m-%dT%H:%M:%S') +SP_VERSION=$(cat $SOURCE_DIR/common/version.h | awk -F\" '{print $2}') + +# Add built files to git +git add -f . +if [ "$EXTRA_VERSION_IDENTIFIER" = "-release" ] || [ "$EXTRA_VERSION_IDENTIFIER" = "-staging" ]; then + export VERSION=${VERSION%"$EXTRA_VERSION_IDENTIFIER"} + git commit --amend -m "sunnypilot v$VERSION" +else + git commit --amend -m "sunnypilot v$VERSION + version: sunnypilot v$SP_VERSION release + date: $DATETIME + master commit: $GIT_HASH + " +fi +git branch -m $DEV_BRANCH + +# Push! +echo "[-] pushing T=$SECONDS" +git push -f origin $DEV_BRANCH diff --git a/release/ci/uninstall_github_runner.sh b/release/ci/uninstall_github_runner.sh new file mode 100755 index 0000000000..a7b0ca3c0e --- /dev/null +++ b/release/ci/uninstall_github_runner.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# Define directories and user +GITHUB_BASE_DIR="/data/github" +BIN_DIR="$GITHUB_BASE_DIR/bin" +BUILDS_DIR="$GITHUB_BASE_DIR/builds" +OPENPILOT_DIR="$GITHUB_BASE_DIR/openpilot" +LOGS_DIR="$GITHUB_BASE_DIR/logs" +CACHE_DIR="$GITHUB_BASE_DIR/cache" +RUNNER_USERNAME="github-runner" +# Define the systemd service name +SERVICE_NAME="github-runner" +USER_GROUPS="comma,gpu,gpio,sudo" + +# Function to stop and disable the systemd service +stop_and_uninstall_service() { + cd $GITHUB_BASE_DIR/runner + sudo ./svc.sh stop + sudo ./svc.sh uninstall +} + +# Function to remove the systemd service file +remove_runner() { + cd $GITHUB_BASE_DIR/runner + sudo rm .runner + sudo su -c './config.sh remove' github-runner +} + +# Function to delete the Github Runner directories +delete_directories() { + sudo rm -rf "$BIN_DIR/github-runner" + sudo rm -rf "$GITHUB_BASE_DIR" "$BIN_DIR" "$BUILDS_DIR" "$LOGS_DIR" "$CACHE_DIR" "$OPENPILOT_DIR" +} + +# Function to remove the Github Runner user +delete_user() { + for group in ${USER_GROUPS//,/ } + do + sudo gpasswd -d ${RUNNER_USERNAME} ${group} + done + sudo userdel -r ${RUNNER_USERNAME} +} + +# Function to remove sudoers entry +remove_sudoers_entry() { + sudo sed -i.bak "/${RUNNER_USERNAME} ALL=(ALL) NOPASSWD: ALL/d" /etc/sudoers +} + +# Make filesystem writable +sudo mount -o remount rw / + +# Ensure filesystem is remounted as read-only on script exit +trap "sudo mount -o remount ro /" EXIT + +# Call functions +stop_and_uninstall_service +remove_runner +delete_directories +delete_user +remove_sudoers_entry +# End of uninstall script diff --git a/release/release_files.py b/release/release_files.py index 0e1ed852a2..702255f965 100755 --- a/release/release_files.py +++ b/release/release_files.py @@ -32,6 +32,7 @@ blacklist = [ ".git/", ".github/", + ".devcontainer/", "Darwin/", ".vscode", @@ -47,6 +48,43 @@ blacklist = [ ".gitmodules", ] +# Sunnypilot blacklist +sunnypilot_blacklist = [ + "system/loggerd/sunnylink_uploader.py", # Temporarily, until we are ready to roll it out widely + "system/manager/gitlab_runner.sh", + ".idea/", + ".run/", + ".*__pycache__/.*", + ".*\\.pyc", + "teleoprtc/*", + "third_party/snpe/x86_64/*", + "body/board/canloader.py", + "body/board/flash_base.sh", + "body/board/flash_knee.sh", + "body/board/recover.sh", + ".*/test/", + ".*/tests/", + ".*tinygrad_repo/tinygrad/renderer/", + "README.md", + ".*internal/", + "docs/.*", + ".sconsign.dblite", + "release/ci/scons_cache/", + ".gitlab-ci.yml", + ".clang-tidy", + ".dockerignore", + ".editorconfig", + ".python-version", + "SECURITY.md", + "codecov.yml", + "conftest.py", + "poetry.lock", + ".venv/", +] + +# Merge the blacklists +blacklist += sunnypilot_blacklist + # gets you through the blacklist whitelist = [ "tools/lib/", @@ -119,8 +157,45 @@ whitelist = [ "opendbc_repo/dbc/toyota_tss2_adas.dbc", "opendbc_repo/dbc/vw_golf_mk4.dbc", "opendbc_repo/dbc/vw_mqb_2010.dbc", + "opendbc_repo/dbc/tesla_can.dbc", + "opendbc_repo/dbc/tesla_radar_bosch_generated.dbc", + "opendbc_repo/dbc/tesla_radar_continental_generated.dbc", + "opendbc_repo/dbc/tesla_powertrain.dbc", ] +# Sunnypilot whitelist +sunnypilot_whitelist = [ + "^README.md", + ".*selfdrive/test/fuzzy_generation.py", + ".*selfdrive/test/helpers.py", + ".*selfdrive/test/__init__.py", + ".*selfdrive/test/setup_device_ci.sh", + ".*selfdrive/test/test_time_to_onroad.py", + ".*selfdrive/test/test_onroad.py", + ".*system/manager/test/test_manager.py", + ".*system/manager/test/__init__.py", + ".*system/qcomgpsd/tests/test_qcomgpsd.py", + ".*system/updated/casync/tests/test_casync.py", + ".*system/updated/tests/test_git.py", + ".*system/updated/tests/test_base.py", + ".*selfdrive/ui/tests/test_translations.py", + ".*selfdrive/car/tests/__init__.py", + ".*selfdrive/car/tests/test_car_interfaces.py", + ".*selfdrive/navd/tests/test_navd.py", + ".*selfdrive/navd/tests/test_map_renderer.py", + ".*selfdrive/boardd/tests/test_boardd_loopback.py", + ".*INTEGRATION.md", + ".*HOW-TOS.md", + ".*CARS.md", + ".*LIMITATIONS.md", + ".*CONTRIBUTING.md", + ".*sunnyhaibin0850_qrcode_paypal.me.png", + "opendbc/.*.dbc", +] + +# Merge the whitelists +whitelist += sunnypilot_whitelist + if __name__ == "__main__": for f in Path(ROOT).rglob("**/*"): From e114dc5a6f34aa218dfded19209786379ebbd0b4 Mon Sep 17 00:00:00 2001 From: DevTekVE Date: Sun, 22 Dec 2024 18:18:21 +0100 Subject: [PATCH 2/6] CI: Use environments for manual approvals on GH Actions for feature branches (#493) * We no longer need to dispatch manually a workflow, it will wait for the publish to be approved for feature branches. * The notification template has been extracted to a variable for the repo, meaning no need to do commit push to update it! (https://github.com/sunnypilot/sunnypilot/settings/variables/actions/DISCORD_GENERAL_UPDATE_NOTICE) * The configuration for what can be auto deployed is now also a variable, no need to commit a change to update that (https://github.com/sunnypilot/sunnypilot/settings/variables/actions/AUTO_DEPLOY_PREBUILT_BRANCHES) * The configuration for what sends a message on `#dev-feedback` channel on discord is also configurable (https://github.com/sunnypilot/sunnypilot/settings/variables/actions/DEV_FEEDBACK_NOTIFICATION_BRANCHES) --- .../workflows/sunnypilot-build-prebuilt.yaml | 121 ++++++++++-------- .../discord_template_notify_dev_private.json | 13 -- .../discord_template_notify_dev_public.json | 9 -- 3 files changed, 68 insertions(+), 75 deletions(-) delete mode 100644 release/ci/discord_template_notify_dev_private.json delete mode 100644 release/ci/discord_template_notify_dev_public.json diff --git a/.github/workflows/sunnypilot-build-prebuilt.yaml b/.github/workflows/sunnypilot-build-prebuilt.yaml index 147a87d6d1..04556e3a41 100644 --- a/.github/workflows/sunnypilot-build-prebuilt.yaml +++ b/.github/workflows/sunnypilot-build-prebuilt.yaml @@ -5,7 +5,7 @@ env: OUTPUT_DIR: ${{ github.workspace }}/output CI_DIR: ${{ github.workspace }}/release/ci SCONS_CACHE_DIR: ${{ github.workspace }}/release/ci/scons_cache - PUBLIC_REPO_URL: "https://github.com/sunnyhaibin/sunnypilot" + PUBLIC_REPO_URL: "https://github.com/sunnypilot/sunnypilot" # Branch configurations MASTER_BRANCH: "master" @@ -32,12 +32,14 @@ on: jobs: build: + concurrency: + group: build-${{ github.head_ref || github.ref_name }} + cancel-in-progress: false runs-on: self-hosted outputs: new_branch: ${{ steps.set-env.outputs.new_branch }} version: ${{ steps.set-env.outputs.version }} extra_version_identifier: ${{ steps.set-env.outputs.extra_version_identifier }} - should_publish: ${{ steps.set-env.outputs.should_publish }} steps: - uses: actions/checkout@v4 with: @@ -48,47 +50,43 @@ jobs: uses: actions/cache@v4 with: path: ${{env.SCONS_CACHE_DIR}} - key: scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }}-${{ github.head_ref }}-${{ github.sha }} + key: scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-${{ github.sha }} restore-keys: | - scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }}-${{ github.head_ref }} + scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref }} scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }} scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_NEW_BRANCH }} scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_BRANCH }} scons-${{ runner.os }}-${{ runner.arch }} - - name: Configure for dev branch - if: github.ref_name == env.DEV_C3_SOURCE_BRANCH + - name: Set Configuration run: | - echo "BRANCH_TYPE=dev" >> $GITHUB_ENV - echo "NEW_BRANCH=${{ env.DEV_TARGET_BRANCH }}" >> $GITHUB_ENV - echo "EXTRA_VERSION_IDENTIFIER=${{ github.run_number }}" >> $GITHUB_ENV - echo "SHOULD_PUBLISH=true" >> $GITHUB_ENV - - - name: Configure for master branches - if: github.ref_name == env.MASTER_BRANCH || github.ref_name == env.MASTER_NEW_BRANCH - run: | - echo "BRANCH_TYPE=master" >> $GITHUB_ENV - echo "NEW_BRANCH=${{ env.STAGING_TARGET_BRANCH }}" >> $GITHUB_ENV - echo "EXTRA_VERSION_IDENTIFIER=staging" >> $GITHUB_ENV - echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')" >> $GITHUB_ENV - echo "SHOULD_PUBLISH=true" >> $GITHUB_ENV - - - name: Configure for tags - if: startsWith(github.ref, 'refs/tags/') - run: | - echo "BRANCH_TYPE=tag" >> $GITHUB_ENV - echo "NEW_BRANCH=${{ env.RELEASE_TARGET_BRANCH }}" >> $GITHUB_ENV - echo "EXTRA_VERSION_IDENTIFIER=release" >> $GITHUB_ENV - echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')" >> $GITHUB_ENV - echo "SHOULD_PUBLISH=true" >> $GITHUB_ENV - - - name: Configure for manual build - if: github.event_name == 'workflow_dispatch' - run: | - echo "BRANCH_TYPE=dispatch" >> $GITHUB_ENV - echo "NEW_BRANCH=${{ github.ref_name }}-prebuilt" >> $GITHUB_ENV - echo "VERSION=$(date '+%Y.%m.%d')-${{ github.run_number }}" >> $GITHUB_ENV - echo "SHOULD_PUBLISH=true" >> $GITHUB_ENV + if [[ "${{ github.ref_name }}" == "${{ env.DEV_C3_SOURCE_BRANCH }}" ]]; then + # Dev configuration + echo "BRANCH_TYPE=dev" >> $GITHUB_ENV + echo "NEW_BRANCH=${{ env.DEV_TARGET_BRANCH }}" >> $GITHUB_ENV + echo "VERSION=$(date '+%Y.%m.%d')-${{ github.run_number }}" >> $GITHUB_ENV + echo "EXTRA_VERSION_IDENTIFIER=${{ github.run_number }}" >> $GITHUB_ENV + + elif [[ "${{ github.ref_name }}" == "${{ env.MASTER_BRANCH }}" || "${{ github.ref_name }}" == "${{ env.MASTER_NEW_BRANCH }}" ]]; then + # Master configuration + echo "BRANCH_TYPE=master" >> $GITHUB_ENV + echo "NEW_BRANCH=${{ env.STAGING_TARGET_BRANCH }}" >> $GITHUB_ENV + echo "EXTRA_VERSION_IDENTIFIER=staging" >> $GITHUB_ENV + echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')" >> $GITHUB_ENV + + elif [[ "${{ github.ref }}" == refs/tags/* ]]; then + # Tag configuration + echo "BRANCH_TYPE=tag" >> $GITHUB_ENV + echo "NEW_BRANCH=${{ env.RELEASE_TARGET_BRANCH }}" >> $GITHUB_ENV + echo "EXTRA_VERSION_IDENTIFIER=release" >> $GITHUB_ENV + echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')" >> $GITHUB_ENV + + else + # Feature branch configuration + echo "BRANCH_TYPE=dispatch" >> $GITHUB_ENV + echo "NEW_BRANCH=${{ github.head_ref || github.ref_name }}-prebuilt" >> $GITHUB_ENV + echo "VERSION=$(date '+%Y.%m.%d')-${{ github.run_number }}" >> $GITHUB_ENV + fi - name: Set environment variables id: set-env @@ -97,7 +95,6 @@ jobs: echo "new_branch=$NEW_BRANCH" >> $GITHUB_OUTPUT [[ ! -z "$EXTRA_VERSION_IDENTIFIER" ]] && echo "extra_version_identifier=$EXTRA_VERSION_IDENTIFIER" >> $GITHUB_OUTPUT [[ ! -z "$VERSION" ]] && echo "version=$VERSION" >> $GITHUB_OUTPUT - echo "should_publish=${SHOULD_PUBLISH:-false}" >> $GITHUB_OUTPUT # Set up common environment source /etc/profile; @@ -190,10 +187,13 @@ jobs: name: prebuilt path: prebuilt.tar.gz - publish-public: + publish: + concurrency: + group: publish-${{ github.head_ref || github.ref_name }} + cancel-in-progress: true needs: build - runs-on: ubuntu-latest - if: needs.build.outputs.should_publish == 'true' + runs-on: ubuntu-24.04 + environment: ${{ contains(fromJSON(vars.AUTO_DEPLOY_PREBUILT_BRANCHES), github.head_ref || github.ref_name) && 'auto-deploy' || 'feature-branch' }} steps: - uses: actions/checkout@v4 @@ -226,26 +226,41 @@ jobs: "${{ needs.build.outputs.version }}" \ "https://x-access-token:${{github.token}}@github.com/sunnypilot/sunnypilot.git" \ "-${{ needs.build.outputs.extra_version_identifier }}" + + echo "" + echo "---- ℹ️ To update the list of branches that auto deploy prebuilts -----" + echo "" + echo "1. Go to: ${{ github.server_url }}/${{ github.repository }}/settings/variables/actions/AUTO_DEPLOY_PREBUILT_BRANCHES" + echo "2. Current value: ${{ vars.AUTO_DEPLOY_PREBUILT_BRANCHES }}" + echo "3. Update as needed (JSON array with no spaces)" notify: - needs: [ build, publish-public ] - runs-on: ubuntu-latest - if: needs.build.outputs.new_branch != '' - + needs: [ build, publish ] + runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v4 - - - name: Install dependencies - run: | - sudo apt-get update - sudo apt-get install -y curl jq gettext-base + - name: Setup Alpine Linux environment + uses: jirutka/setup-alpine@v1.2.0 + with: + packages: 'jq gettext curl' - name: Send Discord Notification env: - DISCORD_WEBHOOK: ${{ needs.build.outputs.new_branch == 'dev-c3' && secrets.DISCORD_NEW_BUILD_WEBHOOK_URL || secrets.DISCORD_MANUAL_BUILD_WEBHOOK_URL }} + DISCORD_WEBHOOK: ${{ contains(fromJSON(vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES), github.head_ref || github.ref_name) && secrets.DISCORD_DEV_FEEDBACK_CHANNEL_WEBHOOK || secrets.DISCORD_DEV_PRIVATE_CHANNEL_WEBHOOK }} run: | - TEMPLATE="${{ needs.build.outputs.new_branch == 'dev-c3' && 'discord_template_notify_dev_public.json' || 'discord_template_notify_dev_private.json' }}" + TEMPLATE='${{ vars.DISCORD_GENERAL_UPDATE_NOTICE }}' export EXTRA_VERSION_IDENTIFIER="${{ needs.build.outputs.extra_version_identifier }}" export VERSION="${{ needs.build.outputs.version }}" - cat release/ci/${TEMPLATE} | envsubst > payload.json - curl -X POST -H "Content-Type: application/json" -d @payload.json $DISCORD_WEBHOOK \ No newline at end of file + export branch_name=${{ github.head_ref || github.ref_name }} + export new_branch=${{ needs.build.outputs.new_branch }} + export extra_version_identifier=${{ needs.build.outputs.extra_version_identifier || github.run_number}} + echo ${TEMPLATE} | envsubst | jq -c '.' | tee payload.json + curl -X POST -H "Content-Type: application/json" -d @payload.json $DISCORD_WEBHOOK + + echo "" + echo "---- ℹ️ To update the list of branches that notify to dev-feedback -----" + echo "" + echo "1. Go to: ${{ github.server_url }}/${{ github.repository }}/settings/variables/actions/DEV_FEEDBACK_NOTIFICATION_BRANCHES" + echo "2. Current value: ${{ vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES }}" + echo "3. Update as needed (JSON array with no spaces)" + shell: alpine.sh {0} \ No newline at end of file diff --git a/release/ci/discord_template_notify_dev_private.json b/release/ci/discord_template_notify_dev_private.json deleted file mode 100644 index 5da1e4d405..0000000000 --- a/release/ci/discord_template_notify_dev_private.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "embeds": [ - { - "title": "❗️ Action Required for `${CI_COMMIT_REF_NAME}` ❗️", - "description": "[${CI_PROJECT_NAME}](${CI_PROJECT_URL}): Pipeline [#${CI_PIPELINE_ID}](${CI_PROJECT_URL}/-/pipelines/${CI_PIPELINE_ID}) of branch [${CI_COMMIT_REF_NAME}](${CI_PROJECT_URL}/-/commits/${CI_COMMIT_REF_NAME}) by ${GITLAB_USER_NAME} (${GITLAB_USER_LOGIN}) is ready to publish manually as [${NEW_BRANCH}](${PUBLIC_REPO_URL}/tree/${NEW_BRANCH})", - "color": 16763904, - "author": { - "name": "${GITLAB_USER_LOGIN}", - "icon_url": "${AVATAR_URL}" - } - } - ] -} diff --git a/release/ci/discord_template_notify_dev_public.json b/release/ci/discord_template_notify_dev_public.json deleted file mode 100644 index c933ea10b4..0000000000 --- a/release/ci/discord_template_notify_dev_public.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "embeds": [ - { - "title": "🎉 sunnypilot `${NEW_BRANCH}` New Update 🎉", - "description": "[sunnypilot](${PUBLIC_REPO_URL}): Build #${EXTRA_VERSION_IDENTIFIER} of branch [${NEW_BRANCH}](${PUBLIC_REPO_URL}/tree/${NEW_BRANCH}) has been published.\n\nDrive safe! 🚗💨", - "color": 4321431 - } - ] -} From 37e493ce0d180f994109651bab55f8b5fdb1b215 Mon Sep 17 00:00:00 2001 From: DevTekVE Date: Mon, 23 Dec 2024 00:37:02 +0100 Subject: [PATCH 3/6] CI: Add GitHub Action to build models from upstream (#491) * Add support for custom model naming in workflow This update introduces a new optional input `custom_name` to the `sunnypilot-build-model` GitHub workflow. If provided, it customizes the artifact name; otherwise, the default naming scheme is used. This enhances flexibility for build artifact identification. * fix name * Update cache key logic in build workflow Replaced `github.ref_name` with a fallback to `github.head_ref || github.ref_name` in cache keys to handle branches and pull requests more reliably. This ensures consistent cache retrieval and improves workflow efficiency. * Simplify artifact upload logic in CI workflow Consolidated redundant artifact upload steps into a single action, leveraging a fallback mechanism for naming. This streamlines the workflow, reducing duplication and improving maintainability. * Make publish step resilient and add conditional notify Added `continue-on-error` to the publish step to allow workflow progression even if it fails. Introduced a condition to notify step that depends on publish outcome being successful. These changes improve workflow reliability and error handling. * Fix conditional syntax in GitHub Actions workflow Updated the conditional statement in the notify job to use the correct GitHub Actions expression syntax. This ensures proper evaluation of the publish step's success outcome. * Update condition for notify step in GitHub Actions workflow Replaced the deprecated syntax `${{ steps.publish.outcome == 'success' }}` with the recommended `success()` function. This ensures compatibility with newer GitHub Actions features and improves maintainability. * Update build workflow to handle PR drafts and improve concurrency Added support for pull request draft events to trigger the workflow, ensuring drafts are not missed. Removed `continue-on-error` in the publish step to improve reliability and prevent silent failures. Simplifies and enhances workflow robustness. * Update pull request workflows and add draft check logic Removed unnecessary pull request event types and drafts settings, consolidating logic. Added a condition to skip publishing for non-draft pull requests, ensuring better control over the deployment process. --- .github/workflows/sunnypilot-build-model.yaml | 16 ++++++++++------ .github/workflows/sunnypilot-build-prebuilt.yaml | 2 ++ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/.github/workflows/sunnypilot-build-model.yaml b/.github/workflows/sunnypilot-build-model.yaml index bc4bd4f4a1..b01a7a0624 100644 --- a/.github/workflows/sunnypilot-build-model.yaml +++ b/.github/workflows/sunnypilot-build-model.yaml @@ -14,6 +14,10 @@ on: required: true default: 'master' type: string + custom_name: + description: 'Custom name for the model' + required: false + type: string jobs: build_model: @@ -32,11 +36,11 @@ jobs: uses: actions/cache@v4 with: path: ${{env.SCONS_CACHE_DIR}} - key: scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }}-model-${{ github.head_ref }}-${{ github.sha }} + key: scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-model-${{ github.sha }} restore-keys: | - scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }}-model-${{ github.head_ref }} - scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }}-model - scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }} + scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-model-${{ github.sha }} + scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-model + scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }} scons-${{ runner.os }}-${{ runner.arch }}-master-new scons-${{ runner.os }}-${{ runner.arch }}-master scons-${{ runner.os }}-${{ runner.arch }} @@ -78,5 +82,5 @@ jobs: - name: Upload Build Artifacts uses: actions/upload-artifact@v4 with: - name: model-${{ github.event.inputs.upstream_branch }}-${{ github.run_number }} - path: ${{ env.OUTPUT_DIR }} \ No newline at end of file + name: model-${{ github.event.inputs.custom_name || github.event.inputs.upstream_branch }}-${{ github.run_number }} + path: ${{ env.OUTPUT_DIR }} diff --git a/.github/workflows/sunnypilot-build-prebuilt.yaml b/.github/workflows/sunnypilot-build-prebuilt.yaml index 04556e3a41..367f292e0c 100644 --- a/.github/workflows/sunnypilot-build-prebuilt.yaml +++ b/.github/workflows/sunnypilot-build-prebuilt.yaml @@ -191,6 +191,7 @@ jobs: concurrency: group: publish-${{ github.head_ref || github.ref_name }} cancel-in-progress: true + if: ${{ github.event_name != 'pull_request' || github.event_name == 'pull_request' && github.event.pull_request.draft }} needs: build runs-on: ubuntu-24.04 environment: ${{ contains(fromJSON(vars.AUTO_DEPLOY_PREBUILT_BRANCHES), github.head_ref || github.ref_name) && 'auto-deploy' || 'feature-branch' }} @@ -237,6 +238,7 @@ jobs: notify: needs: [ build, publish ] runs-on: ubuntu-24.04 + if: success() steps: - uses: actions/checkout@v4 - name: Setup Alpine Linux environment From bd49be185fa327d28bcdb62ec509d3d3270b3317 Mon Sep 17 00:00:00 2001 From: DevTekVE Date: Mon, 23 Dec 2024 01:03:21 +0100 Subject: [PATCH 4/6] ci: enable macos runners again (#495) no need to block macos anymore. Turns out runners are free on github public repos --- .github/workflows/selfdrive_tests.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/selfdrive_tests.yaml b/.github/workflows/selfdrive_tests.yaml index 6bb55716a2..d0e3486020 100644 --- a/.github/workflows/selfdrive_tests.yaml +++ b/.github/workflows/selfdrive_tests.yaml @@ -95,7 +95,6 @@ jobs: timeout-minutes: 30 build_mac: - if: github.repository == 'commaai/openpilot' # Blocking macos builds as well since they have a 10x miltiplier for GH action minutes, waaaay too much! name: build macOS runs-on: ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || From cd73feec574a355bb0ff5aaff5786573c6e144a3 Mon Sep 17 00:00:00 2001 From: DevTekVE Date: Mon, 23 Dec 2024 01:47:41 +0100 Subject: [PATCH 5/6] ci: Update version identifier in workflow for staging and release (#496) Update version identifier in workflow for staging and release Appended "-staging" and "-release" to the VERSION variable in the GitHub Actions workflow to clearly distinguish build types. This ensures clarity in versioning for staging and release environments. --- .github/workflows/sunnypilot-build-prebuilt.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/sunnypilot-build-prebuilt.yaml b/.github/workflows/sunnypilot-build-prebuilt.yaml index 367f292e0c..1c450425d7 100644 --- a/.github/workflows/sunnypilot-build-prebuilt.yaml +++ b/.github/workflows/sunnypilot-build-prebuilt.yaml @@ -72,14 +72,14 @@ jobs: echo "BRANCH_TYPE=master" >> $GITHUB_ENV echo "NEW_BRANCH=${{ env.STAGING_TARGET_BRANCH }}" >> $GITHUB_ENV echo "EXTRA_VERSION_IDENTIFIER=staging" >> $GITHUB_ENV - echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')" >> $GITHUB_ENV + echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')-staging" >> $GITHUB_ENV elif [[ "${{ github.ref }}" == refs/tags/* ]]; then # Tag configuration echo "BRANCH_TYPE=tag" >> $GITHUB_ENV echo "NEW_BRANCH=${{ env.RELEASE_TARGET_BRANCH }}" >> $GITHUB_ENV echo "EXTRA_VERSION_IDENTIFIER=release" >> $GITHUB_ENV - echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')" >> $GITHUB_ENV + echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')-release" >> $GITHUB_ENV else # Feature branch configuration From 0bfa9fca88f75194183f5780a854c6c8429d4d25 Mon Sep 17 00:00:00 2001 From: DevTekVE Date: Mon, 23 Dec 2024 09:19:26 +0100 Subject: [PATCH 6/6] SP: GitHub runner as service and able to toggle via developer panel (#494) * Add support for GitHub Actions runner management Introduce a new process and configuration to manage a GitHub Actions runner. Added a persistent "EnableGithubRunner" parameter and a script to control the runner service. Integrated the feature into the system's process manager logic. * Restrict GitHub runner usage on metered networks. This update modifies the `use_github_runner` function to include a check for metered networks using `HARDWARE.get_network_metered()`. This ensures the GitHub runner is not enabled when the network is metered, improving network usage efficiency. * Add GitHub runner service toggle to developer panel Introduces a new toggle in the developer panel to enable or disable the GitHub runner service. This provides developers with a convenient way to control the service from the UI. * translations * Refactor network condition check for GitHub runner. Updated `use_github_runner` to include network type in metered network checks. This ensures more accurate validation by considering specific network types when determining metered status. * Mark as executable * Update paths and shebangs for consistency across scripts Modified file paths to align with the new directory structure under `/data/media/0/github` and updated the shebang in `github_runner.sh` for better environment compatibility. Adjusted the `BUILD_DIR` in the GitHub workflow to reflect the new path. * Fix string translation for GitHub runner parameter text Added translation support for the GitHub runner service description text. This ensures consistent localization across the UI. * Remove gitlab_runner.sh from Sunnypilot blacklist. The script is no longer required to be blacklisted, likely due to updates or changes in its usage. This improves the maintainability of the blacklist by removing unnecessary entries. * lang stuff * Update BASE_DIR determination based on mount point Refactored scripts to dynamically set BASE_DIR depending on whether /data/media is a valid mount point. This ensures compatibility with different environments and improves robustness of path resolution. * Refactor GitHub runner logic in process_config. Simplified enabling conditions for the GitHub runner by removing dependency on hardware network checks and adding a logical combination of offroad and runner-related functions. This improves code readability and reduces hardware coupling. --- .github/workflows/sunnypilot-build-model.yaml | 2 +- common/params.cc | 1 + release/ci/install_github_runner.sh | 8 +++- release/ci/uninstall_github_runner.sh | 8 +++- release/release_files.py | 1 - selfdrive/ui/qt/offroad/developer_panel.cc | 3 ++ selfdrive/ui/translations/main_ar.ts | 8 ++++ selfdrive/ui/translations/main_de.ts | 8 ++++ selfdrive/ui/translations/main_es.ts | 8 ++++ selfdrive/ui/translations/main_fr.ts | 8 ++++ selfdrive/ui/translations/main_ja.ts | 8 ++++ selfdrive/ui/translations/main_ko.ts | 8 ++++ selfdrive/ui/translations/main_pt-BR.ts | 8 ++++ selfdrive/ui/translations/main_th.ts | 8 ++++ selfdrive/ui/translations/main_tr.ts | 8 ++++ selfdrive/ui/translations/main_zh-CHS.ts | 8 ++++ selfdrive/ui/translations/main_zh-CHT.ts | 8 ++++ system/manager/github_runner.sh | 40 +++++++++++++++++++ system/manager/process_config.py | 6 +++ 19 files changed, 153 insertions(+), 4 deletions(-) create mode 100755 system/manager/github_runner.sh diff --git a/.github/workflows/sunnypilot-build-model.yaml b/.github/workflows/sunnypilot-build-model.yaml index b01a7a0624..9b700e186f 100644 --- a/.github/workflows/sunnypilot-build-model.yaml +++ b/.github/workflows/sunnypilot-build-model.yaml @@ -1,7 +1,7 @@ name: Build Model from Upstream env: - BUILD_DIR: "/data/github/openpilot" + BUILD_DIR: "/data/openpilot" OUTPUT_DIR: ${{ github.workspace }}/output SCONS_CACHE_DIR: ${{ github.workspace }}/release/ci/scons_cache UPSTREAM_REPO: "commaai/openpilot" diff --git a/common/params.cc b/common/params.cc index 2011b52131..1ab37ea84c 100644 --- a/common/params.cc +++ b/common/params.cc @@ -200,6 +200,7 @@ std::unordered_map keys = { {"UpdaterTargetBranch", CLEAR_ON_MANAGER_START}, {"UpdaterLastFetchTime", PERSISTENT}, {"Version", PERSISTENT}, + {"EnableGithubRunner", PERSISTENT}, }; } // namespace diff --git a/release/ci/install_github_runner.sh b/release/ci/install_github_runner.sh index 3aa6b79dd3..1b138996d6 100755 --- a/release/ci/install_github_runner.sh +++ b/release/ci/install_github_runner.sh @@ -44,10 +44,16 @@ fi # Set repository URL if not provided REPO_URL="${REPO_URL:-$DEFAULT_REPO_URL}" +# Determine BASE_DIR based on mount point +if mountpoint -q /data/media; then + BASE_DIR="/data/media/0/github" +else + BASE_DIR="/data/github" +fi + # Constants RUNNER_USER="github-runner" USER_GROUPS="comma,gpu,gpio,sudo" -BASE_DIR="/data/github" RUNNER_DIR="${BASE_DIR}/runner" BUILDS_DIR="${BASE_DIR}/builds" LOGS_DIR="${BASE_DIR}/logs" diff --git a/release/ci/uninstall_github_runner.sh b/release/ci/uninstall_github_runner.sh index a7b0ca3c0e..5f3acfbafd 100755 --- a/release/ci/uninstall_github_runner.sh +++ b/release/ci/uninstall_github_runner.sh @@ -1,6 +1,12 @@ #!/usr/bin/env bash +# Determine BASE_DIR based on mount point +if mountpoint -q /data/media; then + GITHUB_BASE_DIR="/data/media/0/github" +else + GITHUB_BASE_DIR="/data/github" +fi + # Define directories and user -GITHUB_BASE_DIR="/data/github" BIN_DIR="$GITHUB_BASE_DIR/bin" BUILDS_DIR="$GITHUB_BASE_DIR/builds" OPENPILOT_DIR="$GITHUB_BASE_DIR/openpilot" diff --git a/release/release_files.py b/release/release_files.py index 702255f965..dcebb19725 100755 --- a/release/release_files.py +++ b/release/release_files.py @@ -51,7 +51,6 @@ blacklist = [ # Sunnypilot blacklist sunnypilot_blacklist = [ "system/loggerd/sunnylink_uploader.py", # Temporarily, until we are ready to roll it out widely - "system/manager/gitlab_runner.sh", ".idea/", ".run/", ".*__pycache__/.*", diff --git a/selfdrive/ui/qt/offroad/developer_panel.cc b/selfdrive/ui/qt/offroad/developer_panel.cc index 5e127eb21f..bbd69f72ce 100644 --- a/selfdrive/ui/qt/offroad/developer_panel.cc +++ b/selfdrive/ui/qt/offroad/developer_panel.cc @@ -24,6 +24,9 @@ DeveloperPanel::DeveloperPanel(SettingsWindow *parent) : ListWidget(parent) { }); addItem(longManeuverToggle); + auto enableGithubRunner = new ParamControl("EnableGithubRunner", tr("Enable GitHub runner service"), tr("Enables or disables the github runner service."), ""); + addItem(enableGithubRunner); + // Joystick and longitudinal maneuvers should be hidden on release branches is_release = params.getBool("IsReleaseBranch"); diff --git a/selfdrive/ui/translations/main_ar.ts b/selfdrive/ui/translations/main_ar.ts index 2315268a72..0b18af4f25 100644 --- a/selfdrive/ui/translations/main_ar.ts +++ b/selfdrive/ui/translations/main_ar.ts @@ -123,6 +123,14 @@ Longitudinal Maneuver Mode وضع المناورة الطولية + + Enable GitHub runner service + + + + Enables or disables the github runner service. + + DevicePanel diff --git a/selfdrive/ui/translations/main_de.ts b/selfdrive/ui/translations/main_de.ts index 2abed6e305..1f40c2e40a 100644 --- a/selfdrive/ui/translations/main_de.ts +++ b/selfdrive/ui/translations/main_de.ts @@ -123,6 +123,14 @@ Longitudinal Maneuver Mode + + Enable GitHub runner service + + + + Enables or disables the github runner service. + + DevicePanel diff --git a/selfdrive/ui/translations/main_es.ts b/selfdrive/ui/translations/main_es.ts index 38ba964f11..6a27d49137 100644 --- a/selfdrive/ui/translations/main_es.ts +++ b/selfdrive/ui/translations/main_es.ts @@ -123,6 +123,14 @@ Longitudinal Maneuver Mode Modo de maniobra longitudinal + + Enable GitHub runner service + + + + Enables or disables the github runner service. + + DevicePanel diff --git a/selfdrive/ui/translations/main_fr.ts b/selfdrive/ui/translations/main_fr.ts index f4251bc41d..bcaff48bbe 100644 --- a/selfdrive/ui/translations/main_fr.ts +++ b/selfdrive/ui/translations/main_fr.ts @@ -123,6 +123,14 @@ Longitudinal Maneuver Mode + + Enable GitHub runner service + + + + Enables or disables the github runner service. + + DevicePanel diff --git a/selfdrive/ui/translations/main_ja.ts b/selfdrive/ui/translations/main_ja.ts index bc83d3f9ae..9e16d612f2 100644 --- a/selfdrive/ui/translations/main_ja.ts +++ b/selfdrive/ui/translations/main_ja.ts @@ -123,6 +123,14 @@ Longitudinal Maneuver Mode + + Enable GitHub runner service + + + + Enables or disables the github runner service. + + DevicePanel diff --git a/selfdrive/ui/translations/main_ko.ts b/selfdrive/ui/translations/main_ko.ts index 1d2a753e1f..4bbcae92a7 100644 --- a/selfdrive/ui/translations/main_ko.ts +++ b/selfdrive/ui/translations/main_ko.ts @@ -123,6 +123,14 @@ Longitudinal Maneuver Mode 롱컨 기동 모드 + + Enable GitHub runner service + + + + Enables or disables the github runner service. + + DevicePanel diff --git a/selfdrive/ui/translations/main_pt-BR.ts b/selfdrive/ui/translations/main_pt-BR.ts index a89c85510e..ec3d186287 100644 --- a/selfdrive/ui/translations/main_pt-BR.ts +++ b/selfdrive/ui/translations/main_pt-BR.ts @@ -123,6 +123,14 @@ Longitudinal Maneuver Mode Modo Longitudinal Maneuver + + Enable GitHub runner service + + + + Enables or disables the github runner service. + + DevicePanel diff --git a/selfdrive/ui/translations/main_th.ts b/selfdrive/ui/translations/main_th.ts index c2b2771830..82e135a856 100644 --- a/selfdrive/ui/translations/main_th.ts +++ b/selfdrive/ui/translations/main_th.ts @@ -123,6 +123,14 @@ Longitudinal Maneuver Mode + + Enable GitHub runner service + + + + Enables or disables the github runner service. + + DevicePanel diff --git a/selfdrive/ui/translations/main_tr.ts b/selfdrive/ui/translations/main_tr.ts index 9a53449276..58da05c4da 100644 --- a/selfdrive/ui/translations/main_tr.ts +++ b/selfdrive/ui/translations/main_tr.ts @@ -123,6 +123,14 @@ Longitudinal Maneuver Mode + + Enable GitHub runner service + + + + Enables or disables the github runner service. + + DevicePanel diff --git a/selfdrive/ui/translations/main_zh-CHS.ts b/selfdrive/ui/translations/main_zh-CHS.ts index f061322c45..ab679b7f09 100644 --- a/selfdrive/ui/translations/main_zh-CHS.ts +++ b/selfdrive/ui/translations/main_zh-CHS.ts @@ -123,6 +123,14 @@ Longitudinal Maneuver Mode + + Enable GitHub runner service + + + + Enables or disables the github runner service. + + DevicePanel diff --git a/selfdrive/ui/translations/main_zh-CHT.ts b/selfdrive/ui/translations/main_zh-CHT.ts index 04e76a8d95..f88b12abad 100644 --- a/selfdrive/ui/translations/main_zh-CHT.ts +++ b/selfdrive/ui/translations/main_zh-CHT.ts @@ -123,6 +123,14 @@ Longitudinal Maneuver Mode + + Enable GitHub runner service + + + + Enables or disables the github runner service. + + DevicePanel diff --git a/system/manager/github_runner.sh b/system/manager/github_runner.sh new file mode 100755 index 0000000000..f2170cfc70 --- /dev/null +++ b/system/manager/github_runner.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash + +# Define the service name +SERVICE_NAME="actions.runner.sunnypilot.$(uname -n)" + +# Function to control the service +control_service() { + local action=$1 # Store the function argument in a local variable + sudo systemctl $action ${SERVICE_NAME} +} + +service_exists_and_is_loaded() { + sudo systemctl status ${SERVICE_NAME} &>/dev/null + if [[ $? -ne 4 ]]; then + return 0 # Service is known to systemd (i.e., loaded) + else + return 1 # Service is unknown to systemd (i.e., not loaded) + fi +} + +# Check for required argument +if [[ -z $1 ]] || { [[ $1 != "start" ]] && [[ $1 != "stop" ]]; }; then + echo "Usage: $0 {start|stop}" + exit 1 +fi + +# Store the script argument in a descriptive variable +ACTION=$1 + +# Trap EXIT signal (Ctrl+C) and stop the service +trap 'control_service stop ; exit' SIGINT SIGKILL EXIT + +# Enter the main loop +while true; do + # Check if the service is actually present on the system + if service_exists_and_is_loaded; then + control_service $ACTION # Call the function with the specified action + fi + sleep 1 # Pause before the next iteration +done \ No newline at end of file diff --git a/system/manager/process_config.py b/system/manager/process_config.py index a25be615f4..09b897363c 100644 --- a/system/manager/process_config.py +++ b/system/manager/process_config.py @@ -54,6 +54,9 @@ def only_onroad(started: bool, params: Params, CP: car.CarParams) -> bool: def only_offroad(started: bool, params: Params, CP: car.CarParams) -> bool: return not started +def use_github_runner(started, params, CP: car.CarParams) -> bool: + return not PC and params.get_bool("EnableGithubRunner") and not params.get_bool("NetworkMetered") + def or_(*fns): return lambda *args: operator.or_(*(fn(*args) for fn in fns)) @@ -111,4 +114,7 @@ procs = [ PythonProcess("joystick", "tools.joystick.joystick_control", and_(joystick, iscar)), ] +if os.path.exists("./github_runner.sh"): + procs += [NativeProcess("github_runner_start", "system/manager", ["./github_runner.sh", "start"], and_(only_offroad, use_github_runner), sigkill=False)] + managed_processes = {p.name: p for p in procs}