Compare commits

..

1 Commits

Author SHA1 Message Date
github-actions[bot] 10f451cf35 sunnypilot v2026.03.01-4287
version: sunnypilot v2026.001.000 (feature-branch)
date: 2026-03-01T23:04:06
master commit: ea4e165020
2026-03-01 23:04:06 +00:00
3772 changed files with 1340294 additions and 81890 deletions
-19
View File
@@ -1,19 +0,0 @@
---
Checks: '
bugprone-*,
-bugprone-integer-division,
-bugprone-narrowing-conversions,
performance-*,
clang-analyzer-*,
misc-*,
-misc-unused-parameters,
modernize-*,
-modernize-avoid-c-arrays,
-modernize-deprecated-headers,
-modernize-use-auto,
-modernize-use-using,
-modernize-use-nullptr,
-modernize-use-trailing-return-type,
'
CheckOptions:
...
-21
View File
@@ -13,27 +13,6 @@
*.o-*
*.os
*.os-*
*.so
*.a
venv/
.venv/
notebooks
phone
massivemap
neos
installer
chffr/app2
chffr/backend/env
selfdrive/nav
selfdrive/baseui
selfdrive/test/simulator2
**/cache_data
xx/plus
xx/community
xx/projects
!xx/projects/eon_testing_master
!xx/projects/map3d
xx/ops
xx/junk
-21
View File
@@ -1,21 +0,0 @@
* text=auto
# to move existing files into LFS:
# git add --renormalize .
*.onnx filter=lfs diff=lfs merge=lfs -text
*.svg filter=lfs diff=lfs merge=lfs -text
*.png filter=lfs diff=lfs merge=lfs -text
*.gif filter=lfs diff=lfs merge=lfs -text
*.ttf filter=lfs diff=lfs merge=lfs -text
*.wav filter=lfs diff=lfs merge=lfs -text
selfdrive/car/tests/test_models_segs.txt filter=lfs diff=lfs merge=lfs -text
system/hardware/tici/updater filter=lfs diff=lfs merge=lfs -text
third_party/**/*.a filter=lfs diff=lfs merge=lfs -text
third_party/**/*.so filter=lfs diff=lfs merge=lfs -text
third_party/**/*.so.* filter=lfs diff=lfs merge=lfs -text
third_party/**/*.dylib filter=lfs diff=lfs merge=lfs -text
third_party/acados/*/t_renderer filter=lfs diff=lfs merge=lfs -text
third_party/qt5/larch64/bin/lrelease filter=lfs diff=lfs merge=lfs -text
third_party/qt5/larch64/bin/lupdate filter=lfs diff=lfs merge=lfs -text
third_party/catch2/include/catch2/catch.hpp filter=lfs diff=lfs merge=lfs -text
-11
View File
@@ -1,11 +0,0 @@
* @sunnypilot/dev-internal
/.github/ @devtekve @sunnyhaibin
/release/ci/ @devtekve @sunnyhaibin
/tinygrad_repo @devtekve @Discountchubbs
/tinygrad/ @devtekve @Discountchubbs
/selfdrive/controls/lib/longitudinal_planner.py @devtekve @Discountchubbs
/selfdrive/controls/lib/longitudinal_mpc_lib/long_mpc.py @devtekve @Discountchubbs
/selfdrive/modeld/ @devtekve @Discountchubbs
/sunnypilot/model* @devtekve @Discountchubbs
/sunnypilot/sunnylink/ @devtekve
/system/athena/ @devtekve
-47
View File
@@ -1,47 +0,0 @@
name: Bug report
description: For issues with running openpilot on your comma device
labels: ["bug"]
body:
- type: markdown
attributes:
value: >
Before creating a **bug report**, please check the following:
* If the issue likely only affects your car model or make, go back and open a **car bug report** instead.
* If the issue is related to the driving or driver monitoring models, you should open a [discussion](https://github.com/commaai/openpilot/discussions/categories/model-feedback) instead.
* Ensure you're running the latest openpilot release.
* Ensure you're using officially supported hardware. Issues running on PCs have a different issue template.
* Ensure there isn't an existing issue for your bug. If there is, leave a comment on the existing issue.
* Ensure you're running stock openpilot. We cannot look into bug reports from forks.
If you're unsure whether you've hit a bug, check out the #installation-help channel in the [community Discord server](https://discord.comma.ai).
- type: textarea
attributes:
label: Describe the bug
description: Also include a description of how to reproduce the bug
validations:
required: true
- type: input
id: route
attributes:
label: Provide a route where the issue occurs
description: Ensure the route is fully uploaded at https://useradmin.comma.ai. We cannot look into issues without routes, or at least a Dongle ID.
placeholder: 77611a1fac303767|2020-05-11--16-37-07
validations:
required: true
- type: input
id: version
attributes:
label: openpilot version
description: If you're not on release, provide the commit hash
placeholder: 0.8.10
validations:
required: true
- type: textarea
attributes:
label: Additional info
-14
View File
@@ -1,14 +0,0 @@
blank_issues_enabled: false
contact_links:
- name: Car bug report
url: https://github.com/commaai/opendbc/issues/new
about: For issues with a particular car make or model
- name: Join the Discord
url: https://discord.comma.ai
about: The community Discord is for both openpilot development and experience discussion
- name: Report driving behavior feedback
url: https://discord.com/channels/469524606043160576/1254834193066623017
about: Feedback for the driving and driver monitoring models goes in the #driving-feedback in Discord
- name: Community Wiki
url: https://github.com/commaai/openpilot/wiki
about: Check out our community wiki
-8
View File
@@ -1,8 +0,0 @@
---
name: Enhancement
about: For openpilot enhancement suggestions
title: ''
labels: 'enhancement'
assignees: ''
---
-42
View File
@@ -1,42 +0,0 @@
name: PC bug report
description: For issues with running openpilot on PC
labels: ["PC"]
body:
- type: markdown
attributes:
value: >
Before creating a **bug report**, please check the following:
* Ensure you're running the latest openpilot release.
* Ensure there isn't an existing issue for your bug. If there is, leave a comment on the existing issue.
* Ensure you're running stock openpilot. We cannot look into bug reports from forks.
If you're unsure whether you've hit a bug, check out the #installation-help channel in the [community Discord server](https://discord.comma.ai).
- type: textarea
attributes:
label: Describe the bug
description: Also include a description of how to reproduce the bug
validations:
required: true
- type: input
id: os-version
attributes:
label: OS Version
placeholder: Ubuntu 24.04
validations:
required: true
- type: input
id: version
attributes:
label: openpilot version or commit
placeholder: bd36f2ec8d3559909678eff2690c10a520938367
validations:
required: false
- type: textarea
attributes:
label: Additional info
-31
View File
@@ -1,31 +0,0 @@
ci:
- changed-files:
- any-glob-to-all-files: "{.github/**,**/test_*,**/test/**,Jenkinsfile}"
chore:
- changed-files:
- any-glob-to-all-files: "{.github/**}"
car:
- changed-files:
- any-glob-to-all-files: '{selfdrive/car/**,opendbc_repo}'
simulation:
- changed-files:
- any-glob-to-all-files: 'tools/sim/**'
ui:
- changed-files:
- any-glob-to-all-files: '{selfdrive/ui/**,system/ui/**}'
tools:
- changed-files:
- any-glob-to-all-files: 'tools/**'
multilanguage:
- changed-files:
- any-glob-to-all-files: 'selfdrive/ui/translations/**'
autonomy:
- changed-files:
- any-glob-to-all-files: "{selfdrive/modeld/models/**,selfdrive/test/process_replay/model_replay_ref_commit,sunnypilot/modeld*/models/**}"
-68
View File
@@ -1,68 +0,0 @@
<!-- Please copy and paste the relevant template -->
<!--- ***** Template: Fingerprint *****
**Car**
Which car (make, model, year) this fingerprint is for
**Route**
A route with the fingerprint
-->
<!--- ***** Template: Car Bugfix *****
**Description**
A description of the bug and the fix. Also link the issue if it exists.
**Verification**
Explain how you tested this bug fix.
**Route**
Route: [a route with the bug fix]
-->
<!--- ***** Template: Bugfix *****
**Description**
A description of the bug and the fix. Also link the issue if it exists.
**Verification**
Explain how you tested this bug fix.
-->
<!--- ***** Template: Car Port *****
**Checklist**
- [ ] added entry to CAR in selfdrive/car/*/values.py and ran `selfdrive/car/docs.py` to generate new docs
- [ ] test route added to [routes.py](https://github.com/commaai/openpilot/blob/master/selfdrive/car/tests/routes.py)
- [ ] route with openpilot:
- [ ] route with stock system:
- [ ] car harness used (if comma doesn't sell it, put N/A):
-->
<!--- ***** Template: Refactor *****
**Description**
A description of the refactor, including the goals it accomplishes.
**Verification**
Explain how you tested the refactor for regressions.
-->
-43
View File
@@ -1,43 +0,0 @@
exclude-labels:
- 'no-changelog'
categories:
- title: '🚀 Features'
labels:
- 'feature'
- 'enhancement'
- title: '🐛 Bug Fixes'
collapse-after: 5
labels:
- 'fix'
- 'bugfix'
- 'bug'
- title: '🧰 Maintenance'
collapse-after: 5
label: 'chore'
change-template: '- $TITLE @$AUTHOR (#$NUMBER)'
change-title-escapes: '\<*_&'
replacers:
- search: '/[Ss][Uu][Nn][Nn][Yy][Pp][Ii][Ll][Oo][Tt]/g'
replace: 'sunnypilot'
- search: '/\b[Ss][Pp]\b/g'
replace: 'SP'
version-resolver:
major:
labels:
- 'major'
minor:
labels:
- 'minor'
patch:
labels:
- 'patch'
default: patch
name-template: 'v$RESOLVED_VERSION 🚀'
tag-template: 'v$RESOLVED_VERSION'
version-template: "0.$MAJOR.$MINOR.$PATCH" # The day OP becomes v1, we need to bump this
tag-prefix: "v0." # The day OP becomes v1, we need to bump this
prerelease-identifier: "staging"
template: |
## Changes
$CHANGES
-58
View File
@@ -1,58 +0,0 @@
name: 'automatically cache based on current runner'
inputs:
path:
description: 'path to cache'
required: true
key:
description: 'key'
required: true
restore-keys:
description: 'restore-keys'
required: true
save:
description: 'whether to save the cache'
default: 'true'
required: false
outputs:
cache-hit:
description: 'cache hit occurred'
value: ${{ (contains(runner.name, 'nsc') && steps.ns-cache.outputs.cache-hit) ||
(!contains(runner.name, 'nsc') && inputs.save != 'false' && steps.gha-cache.outputs.cache-hit) ||
(!contains(runner.name, 'nsc') && inputs.save == 'false' && steps.gha-cache-ro.outputs.cache-hit) }}
runs:
using: "composite"
steps:
- name: setup namespace cache
id: ns-cache
if: ${{ contains(runner.name, 'nsc') }}
uses: namespacelabs/nscloud-cache-action@v1
with:
path: ${{ inputs.path }}
- name: setup github cache
id: gha-cache
if: ${{ !contains(runner.name, 'nsc') && inputs.save != 'false' }}
uses: 'actions/cache@v4'
with:
path: ${{ inputs.path }}
key: ${{ inputs.key }}
restore-keys: ${{ inputs.restore-keys }}
- name: setup github cache
id: gha-cache-ro
if: ${{ !contains(runner.name, 'nsc') && inputs.save == 'false' }}
uses: 'actions/cache/restore@v4'
with:
path: ${{ inputs.path }}
key: ${{ inputs.key }}
restore-keys: ${{ inputs.restore-keys }}
# make the directory manually in case we didn't get a hit, so it doesn't fail on future steps
- id: scons-cache-setup
shell: bash
run: |
mkdir -p ${{ inputs.path }}
sudo chmod -R 777 ${{ inputs.path }}
sudo chown -R $USER ${{ inputs.path }}
-91
View File
@@ -1,91 +0,0 @@
name: "PR review"
on:
pull_request_target:
types: [ opened, reopened, synchronize, edited ]
jobs:
labeler:
name: review
permissions:
contents: read
pull-requests: write
issues: write
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: false
# Label PRs
- uses: actions/labeler@v5.0.0
with:
dot: true
configuration-path: .github/labeler.yaml
# Check PR target branch
- name: check branch
uses: Vankka/pr-target-branch-action@def32ec9d93514138d6ac0132ee62e120a72aed5
if: github.repository == 'sunnypilot/sunnypilot'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
target: /^(?!master$).*/
exclude: /sunnypilot:.*/
change-to: ${{ github.base_ref }}
already-exists-action: close_this
already-exists-comment: "Your PR should be made against the `master` branch"
update-pr-labels:
name: Update fork's PR Labels
runs-on: ubuntu-latest
if: (github.event.pull_request.head.repo.fork && (contains(github.event_name, 'pull_request') && github.event.action == 'synchronize'))
env:
PR_LABEL: 'dev'
TRUST_FORK_PR_LABEL: 'trust-fork-pr'
steps:
- name: Check if PR has dev label
id: check-labels
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const prNumber = context.payload.pull_request.number;
const { data: labels } = await github.rest.issues.listLabelsOnIssue({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNumber
});
const hasDevC3Label = labels.some(label => label.name === process.env.PR_LABEL);
const hasTrustLabel = labels.some(label => label.name === process.env.TRUST_FORK_PR_LABEL);
console.log(`PR #${prNumber} has ${process.env.PR_LABEL} label: ${hasDevC3Label}`);
console.log(`PR #${prNumber} has ${process.env.TRUST_FORK_PR_LABEL} label: ${hasTrustLabel}`);
core.setOutput('has-dev', hasDevC3Label ? 'true' : 'false');
core.setOutput('has-trust', hasTrustLabel ? 'true' : 'false');
- name: Remove trust-fork-pr label if present
if: steps.check-labels.outputs.has-dev == 'true' && steps.check-labels.outputs.has-trust == 'true'
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const prNumber = context.payload.pull_request.number;
await github.rest.issues.removeLabel({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNumber,
name: process.env.TRUST_FORK_PR_LABEL
});
console.log(`Removed '${process.env.TRUST_FORK_PR_LABEL}' label from PR #${prNumber} as it received new commits`);
// Add a comment to the PR
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNumber,
body: `The \`${process.env.TRUST_FORK_PR_LABEL}\` label has been automatically removed because new commits were pushed to this PR. This PR will need to be re-reviewed before the label can be applied again.`
});
-37
View File
@@ -1,37 +0,0 @@
name: badges
on:
schedule:
- cron: '0 * * * *'
workflow_dispatch:
env:
BASE_IMAGE: sunnypilot-base
DOCKER_REGISTRY: ghcr.io/sunnypilot
RUN: docker run --shm-size 2G -v $PWD:/tmp/openpilot -w /tmp/openpilot -e PYTHONPATH=/tmp/openpilot -e NUM_JOBS -e JOB_ID -e GITHUB_ACTION -e GITHUB_REF -e GITHUB_HEAD_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_RUN_ID -v $GITHUB_WORKSPACE/.ci_cache/scons_cache:/tmp/scons_cache -v $GITHUB_WORKSPACE/.ci_cache/comma_download_cache:/tmp/comma_download_cache -v $GITHUB_WORKSPACE/.ci_cache/openpilot_cache:/tmp/openpilot_cache $DOCKER_REGISTRY/$BASE_IMAGE:latest /bin/bash -c
jobs:
badges:
name: create badges
runs-on: ubuntu-latest
if: github.repository == 'sunnypilot/sunnypilot'
permissions:
contents: write
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: ./.github/workflows/setup-with-retry
- name: Push badges
run: |
${{ env.RUN }} "scons -j$(nproc) && python3 selfdrive/ui/translations/create_badges.py"
rm .gitattributes
git checkout --orphan badges
git rm -rf --cached .
git config user.email "badge-researcher@sunnypilot.ai"
git config user.name "Badge Researcher"
git add translation_badge.svg
git commit -m "Add/Update badges"
git push -f origin HEAD
@@ -1,303 +0,0 @@
name: Build and push all tinygrad models
on:
workflow_dispatch:
inputs:
set_min_version:
description: 'Minimum selector version required for the models (see helpers.py or readme.md)'
required: true
type: string
jobs:
setup:
runs-on: ubuntu-latest
outputs:
json_version: ${{ steps.get-json.outputs.json_version }}
recompiled_dir: ${{ steps.create-recompiled-dir.outputs.recompiled_dir }}
json_file: ${{ steps.get-json.outputs.json_file }}
model_matrix: ${{ steps.set-matrix.outputs.model_matrix }}
tinygrad_ref: ${{ steps.get-tinygrad-ref.outputs.tinygrad_ref }}
steps:
- name: Checkout sunnypilot repo
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot
path: sunnypilot
submodules: recursive
- name: Get tinygrad_repo ref
id: get-tinygrad-ref
run: |
cd sunnypilot
export PYTHONPATH=$(pwd)
ref=$(python3 sunnypilot/models/tinygrad_ref.py)
echo "tinygrad_ref=$ref" >> $GITHUB_OUTPUT
echo "tinygrad_ref is $ref"
- name: Checkout docs repo (sunnypilot-docs, gh-pages)
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot-docs
ref: gh-pages
path: docs
ssh-key: ${{ secrets.CI_SUNNYPILOT_DOCS_PRIVATE_KEY }}
- name: Get next JSON version to use (from GitHub docs repo)
id: get-json
run: |
cd docs/docs
latest=$(ls driving_models_v*.json | sed -E 's/.*_v([0-9]+)\.json/\1/' | sort -n | tail -1)
next=$((latest+1))
json_file="driving_models_v${next}.json"
cp "driving_models_v${latest}.json" "$json_file"
echo "json_file=docs/docs/$json_file" >> $GITHUB_OUTPUT
echo "json_version=$((next+0))" >> $GITHUB_OUTPUT
echo "SRC_JSON_FILE=docs/docs/driving_models_v${latest}.json" >> $GITHUB_ENV
- name: Extract tinygrad models
id: set-matrix
working-directory: docs/docs
run: |
jq -c '[.bundles[] | select(.runner=="tinygrad") | {ref, display_name: (.display_name | gsub(" \\([^)]*\\)"; "")), is_20hz}]' "$(basename "${SRC_JSON_FILE}")" > matrix.json
echo "model_matrix=$(cat matrix.json)" >> $GITHUB_OUTPUT
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.GITLAB_SSH_PRIVATE_KEY }}
- run: |
mkdir -p ~/.ssh
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Clone GitLab docs repo and create new recompiled dir
id: create-recompiled-dir
env:
GIT_SSH_COMMAND: 'ssh -o UserKnownHostsFile=~/.ssh/known_hosts'
run: |
git clone --depth 1 --filter=tree:0 --sparse git@gitlab.com:sunnypilot/public/docs.sunnypilot.ai2.git gitlab_docs
cd gitlab_docs
git checkout main
git sparse-checkout set --no-cone models/
cd models
latest_dir=$(ls -d recompiled* 2>/dev/null | sed -E 's/recompiled([0-9]+)/\1/' | sort -n | tail -1)
if [[ -z "$latest_dir" ]]; then
next_dir=1
else
next_dir=$((latest_dir+1))
fi
recompiled_dir="${next_dir}"
mkdir -p "recompiled${recompiled_dir}"
touch "recompiled${recompiled_dir}/.gitkeep"
cd ../..
echo "recompiled_dir=$recompiled_dir" >> $GITHUB_OUTPUT
- name: Push empty recompiled dir to GitLab
run: |
cd gitlab_docs
git add models/recompiled${{ steps.create-recompiled-dir.outputs.recompiled_dir }}
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git commit -m "Add recompiled${{ steps.create-recompiled-dir.outputs.recompiled_dir }} for build-all" || echo "No changes to commit"
git push origin main
- name: Push new JSON to GitHub docs repo
run: |
cd docs
git pull origin gh-pages
git add docs/"$(basename ${{ steps.get-json.outputs.json_file }})"
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git commit -m "Add new ${{ steps.get-json.outputs.json_file }} for build-all" || echo "No changes to commit"
git push origin gh-pages
get_and_build:
needs: [setup]
strategy:
matrix:
model: ${{ fromJson(needs.setup.outputs.model_matrix) }}
fail-fast: false
uses: ./.github/workflows/build-single-tinygrad-model.yaml
with:
upstream_branch: ${{ matrix.model.ref }}
custom_name: ${{ matrix.model.display_name }}
recompiled_dir: ${{ needs.setup.outputs.recompiled_dir }}
json_version: ${{ needs.setup.outputs.json_version }}
secrets: inherit
retry_failed_models:
needs: [setup, get_and_build]
runs-on: ubuntu-latest
if: ${{ needs.setup.result != 'failure' && !cancelled() }}
outputs:
retry_matrix: ${{ steps.set-retry-matrix.outputs.retry_matrix }}
steps:
- uses: actions/download-artifact@v4
with:
pattern: model-*
path: output
- id: set-retry-matrix
run: |
echo '${{ needs.setup.outputs.model_matrix }}' > matrix.json
built=(); while IFS= read -r line; do built+=("$line"); done < <(
ls output | sed -E 's/^model-//' | sed -E 's/-[0-9]+$//' | sed -E 's/ \([^)]*\)//' | awk '{gsub(/^ +| +$/, ""); print}'
)
jq -c --argjson built "$(printf '%s\n' "${built[@]}" | jq -R . | jq -s .)" \
'map(select(.display_name as $n | ($built | index($n | gsub("^ +| +$"; "")) | not)))' matrix.json > retry_matrix.json
echo "retry_matrix=$(cat retry_matrix.json)" >> $GITHUB_OUTPUT
retry_get_and_build:
needs: [setup, get_and_build, retry_failed_models]
if: ${{ needs.get_and_build.result == 'failure' || (needs.retry_failed_models.outputs.retry_matrix != '[]' && needs.retry_failed_models.outputs.retry_matrix != '') }}
strategy:
matrix:
model: ${{ fromJson(needs.retry_failed_models.outputs.retry_matrix) }}
fail-fast: false
uses: ./.github/workflows/build-single-tinygrad-model.yaml
with:
upstream_branch: ${{ matrix.model.ref }}
custom_name: ${{ matrix.model.display_name }}
recompiled_dir: ${{ needs.setup.outputs.recompiled_dir }}
json_version: ${{ needs.setup.outputs.json_version }}
artifact_suffix: -retry
secrets: inherit
publish_models:
name: Publish models sequentially
needs: [setup, get_and_build, retry_failed_models, retry_get_and_build]
if: ${{ !cancelled() && (needs.get_and_build.result != 'failure' || needs.retry_get_and_build.result == 'success' || (needs.retry_failed_models.outputs.retry_matrix != '[]' && needs.retry_failed_models.outputs.retry_matrix != '')) }}
runs-on: ubuntu-latest
strategy:
max-parallel: 1
matrix:
model: ${{ fromJson(needs.setup.outputs.model_matrix) }}
env:
RECOMPILED_DIR: recompiled${{ needs.setup.outputs.recompiled_dir }}
JSON_FILE: ${{ needs.setup.outputs.json_file }}
ARTIFACT_NAME_INPUT: ${{ matrix.model.display_name }}
steps:
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.GITLAB_SSH_PRIVATE_KEY }}
- name: Add GitLab.com SSH key to known_hosts
run: |
mkdir -p ~/.ssh
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Clone GitLab docs repo
env:
GIT_SSH_COMMAND: 'ssh -o UserKnownHostsFile=~/.ssh/known_hosts'
run: |
echo "Cloning GitLab"
git clone --depth 1 --filter=tree:0 --sparse git@gitlab.com:sunnypilot/public/docs.sunnypilot.ai2.git gitlab_docs
cd gitlab_docs
echo "checkout models/${RECOMPILED_DIR}"
git sparse-checkout set --no-cone models/${RECOMPILED_DIR}
git checkout main
cd ..
- name: Checkout docs repo
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot-docs
ref: gh-pages
path: docs
ssh-key: ${{ secrets.CI_SUNNYPILOT_DOCS_PRIVATE_KEY }}
- name: Validate recompiled dir and JSON version
run: |
if [ ! -d "gitlab_docs/models/$RECOMPILED_DIR" ]; then
echo "Recompiled dir $RECOMPILED_DIR does not exist in GitLab repo"
exit 1
fi
if [ ! -f "$JSON_FILE" ]; then
echo "JSON file $JSON_FILE does not exist!"
exit 1
fi
- name: Download artifact name file
uses: actions/download-artifact@v4
with:
name: artifact-name-${{ env.ARTIFACT_NAME_INPUT }}
path: artifact_name
- name: Read artifact name
id: read-artifact-name
run: |
ARTIFACT_NAME=$(cat artifact_name/artifact_name.txt)
echo "artifact_name=$ARTIFACT_NAME" >> $GITHUB_OUTPUT
- name: Download model artifact
uses: actions/download-artifact@v4
with:
name: ${{ steps.read-artifact-name.outputs.artifact_name }}
path: output
- name: Remove onnx files bc not needed for recompiled dir since they already exist from single build
run: |
find output -type f -name '*.onnx' -delete
find output -type f -name 'big_*.pkl' -delete
find output -type f -name 'dmonitoring_model_tinygrad.pkl' -delete
- name: Copy model artifacts to gitlab
env:
ARTIFACT_NAME: ${{ steps.read-artifact-name.outputs.artifact_name }}
run: |
ARTIFACT_DIR="gitlab_docs/models/${RECOMPILED_DIR}/${ARTIFACT_NAME}"
mkdir -p "$ARTIFACT_DIR"
for path in output/*; do
if [ "$(basename "$path")" = "artifact_name.txt" ]; then
continue
fi
name="$(basename "$path")"
if [ -d "$path" ]; then
mkdir -p "$ARTIFACT_DIR/$name"
cp -r "$path"/* "$ARTIFACT_DIR/$name/"
echo "Copied dir $name -> $ARTIFACT_DIR/$name"
else
cp "$path" "$ARTIFACT_DIR/"
echo "Copied file $name -> $ARTIFACT_DIR/"
fi
done
- name: Push recompiled dir to GitLab
env:
GITLAB_SSH_PRIVATE_KEY: ${{ secrets.GITLAB_SSH_PRIVATE_KEY }}
run: |
cd gitlab_docs
git checkout main
git pull origin main
for d in models/"$RECOMPILED_DIR"/*/; do
git sparse-checkout add "$d"
done
git add models/"$RECOMPILED_DIR"
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git commit -m "Update $RECOMPILED_DIR with model from build-all-tinygrad-models" || echo "No changes to commit"
git push origin main
- run: |
cd docs
git pull origin gh-pages
- name: update json
run: |
ARGS=""
[ -n "${{ inputs.set_min_version }}" ] && ARGS="$ARGS --set-min-version \"${{ inputs.set_min_version }}\""
ARGS="$ARGS --sort-by-date"
ARGS="$ARGS --tinygrad-ref \"${{ needs.setup.outputs.tinygrad_ref }}\""
eval python3 docs/json_parser.py \
--json-path "$JSON_FILE" \
--recompiled-dir "gitlab_docs/models/$RECOMPILED_DIR" \
$ARGS
- name: Push updated json to GitHub
run: |
cd docs
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git checkout gh-pages
git add docs/"$(basename $JSON_FILE)"
git commit -m "Update $(basename $JSON_FILE) after recompiling model" || echo "No changes to commit"
git push origin gh-pages
@@ -1,228 +0,0 @@
name: Build Single Tinygrad Model and Push
on:
workflow_call:
inputs:
upstream_branch:
description: 'Upstream commit to build from'
required: true
type: string
custom_name:
description: 'Custom name for the model (no date, only name)'
required: false
type: string
recompiled_dir:
description: 'Existing recompiled directory number (e.g. 3 for recompiled3)'
required: true
type: string
json_version:
description: 'driving_models version number to update (e.g. 5 for driving_models_v5.json)'
required: true
type: string
artifact_suffix:
description: 'Suffix for artifact name'
required: false
type: string
default: ''
bypass_push:
description: 'Bypass pushing to GitLab for build-all'
required: false
default: true
type: boolean
workflow_dispatch:
inputs:
upstream_branch:
description: 'Upstream commit to build from'
required: true
type: string
custom_name:
description: 'Custom name for the model (no date, only name)'
required: false
type: string
recompiled_dir:
description: 'Existing recompiled directory number (e.g. 3 for recompiled3)'
required: true
type: string
json_version:
description: 'driving_models version number to update (e.g. 5 for driving_models_v5.json)'
required: true
type: string
model_folder:
description: 'Model folder'
type: choice
default: 'None'
options:
- None
- Simple Plan Models
- Space Lab Models
- TR Models
- DTR Models
- Custom Merge Models
- FOF series models
- Other
custom_model_folder:
description: 'Custom model folder name (if "Other" selected)'
required: false
type: string
generation:
description: 'Model generation'
required: false
type: string
version:
description: 'Minimum selector version'
required: false
type: string
env:
RECOMPILED_DIR: recompiled${{ inputs.recompiled_dir }}
JSON_FILE: docs/docs/driving_models_v${{ inputs.json_version }}.json
jobs:
build_model:
uses: ./.github/workflows/sunnypilot-build-model.yaml
with:
upstream_branch: ${{ inputs.upstream_branch }}
custom_name: ${{ inputs.custom_name || inputs.upstream_branch }}
is_20hz: true
artifact_suffix: ${{ inputs.artifact_suffix }}
secrets: inherit
publish_model:
if: ${{ !inputs.bypass_push && !cancelled() }}
concurrency:
group: gitlab-push-${{ inputs.recompiled_dir }}
cancel-in-progress: false
needs: build_model
runs-on: ubuntu-latest
steps:
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.GITLAB_SSH_PRIVATE_KEY }}
- name: Add GitLab.com SSH key to known_hosts
run: |
mkdir -p ~/.ssh
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Clone GitLab docs repo
env:
GIT_SSH_COMMAND: 'ssh -o UserKnownHostsFile=~/.ssh/known_hosts'
run: |
echo "Cloning GitLab"
git clone --depth 1 --filter=tree:0 --sparse git@gitlab.com:sunnypilot/public/docs.sunnypilot.ai2.git gitlab_docs
cd gitlab_docs
echo "checkout models/${RECOMPILED_DIR}"
git sparse-checkout set --no-cone models/${RECOMPILED_DIR}
git checkout main
cd ..
- name: Checkout docs repo
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot-docs
ref: gh-pages
path: docs
ssh-key: ${{ secrets.CI_SUNNYPILOT_DOCS_PRIVATE_KEY }}
- name: Validate recompiled dir and JSON version
run: |
if [ ! -d "gitlab_docs/models/$RECOMPILED_DIR" ]; then
echo "Recompiled dir $RECOMPILED_DIR does not exist in GitLab repo"
exit 1
fi
if [ ! -f "$JSON_FILE" ]; then
echo "JSON file $JSON_FILE does not exist!"
exit 1
fi
- name: Download artifact name file
uses: actions/download-artifact@v4
with:
name: artifact-name-${{ inputs.custom_name || inputs.upstream_branch }}
path: artifact_name
- name: Read artifact name
id: read-artifact-name
run: |
ARTIFACT_NAME=$(cat artifact_name/artifact_name.txt)
echo "artifact_name=$ARTIFACT_NAME" >> $GITHUB_OUTPUT
- name: Download and extract model artifact
uses: actions/download-artifact@v4
with:
name: ${{ steps.read-artifact-name.outputs.artifact_name }}
path: output
- name: Remove unwanted files
run: |
find output -type f -name 'dmonitoring_model_tinygrad.pkl' -delete
find output -type f -name 'dmonitoring_model.onnx' -delete
- name: Copy model artifact(s) to GitLab recompiled dir
env:
ARTIFACT_NAME: ${{ steps.read-artifact-name.outputs.artifact_name }}
run: |
ARTIFACT_DIR="gitlab_docs/models/${RECOMPILED_DIR}/${ARTIFACT_NAME}"
mkdir -p "$ARTIFACT_DIR"
for path in output/*; do
if [ "$(basename "$path")" = "artifact_name.txt" ]; then
continue
fi
name="$(basename "$path")"
if [ -d "$path" ]; then
mkdir -p "$ARTIFACT_DIR/$name"
cp -r "$path"/* "$ARTIFACT_DIR/$name/"
echo "Copied dir $name -> $ARTIFACT_DIR/$name"
else
cp "$path" "$ARTIFACT_DIR/"
echo "Copied file $name -> $ARTIFACT_DIR/"
fi
done
- name: Push recompiled dir to GitLab
env:
GITLAB_SSH_PRIVATE_KEY: ${{ secrets.GITLAB_SSH_PRIVATE_KEY }}
run: |
cd gitlab_docs
git checkout main
git pull origin main
for d in models/"$RECOMPILED_DIR"/*/; do
git sparse-checkout add "$d"
done
git add models/"$RECOMPILED_DIR"
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git commit -m "Create/Update $RECOMPILED_DIR with new/updated model from build-single-tinygrad-model" || echo "No changes to commit"
git push origin main
- run: |
cd docs
git pull origin gh-pages
- name: Run json_parser.py to update JSON
run: |
FOLDER="${{ inputs.model_folder }}"
if [ "$FOLDER" = "Other" ]; then
FOLDER="${{ inputs.custom_model_folder }}"
fi
ARGS=""
if [ "$FOLDER" != "None" ] && [ -n "$FOLDER" ]; then
ARGS="$ARGS --model-folder \"$FOLDER\""
fi
[ -n "${{ inputs.generation }}" ] && ARGS="$ARGS --generation \"${{ inputs.generation }}\""
[ -n "${{ inputs.version }}" ] && ARGS="$ARGS --version \"${{ inputs.version }}\""
eval python3 docs/json_parser.py \
--json-path "$JSON_FILE" \
--recompiled-dir "gitlab_docs/models/$RECOMPILED_DIR" \
--sort-by-date \
$ARGS
- name: Push updated JSON to GitHub docs repo
run: |
cd docs
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git checkout gh-pages
git add docs/"$(basename $JSON_FILE)"
git commit -m "Update $(basename $JSON_FILE) after recompiling model" || echo "No changes to commit"
git push origin gh-pages
-76
View File
@@ -1,76 +0,0 @@
name: cereal validation
on:
push:
branches:
- master
pull_request:
paths:
- 'cereal/**'
workflow_dispatch:
workflow_call:
inputs:
run_number:
default: '1'
required: true
type: string
concurrency:
group: cereal-validation-ci-run-${{ inputs.run_number }}-${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && github.run_id || github.head_ref || github.ref }}-${{ github.workflow }}-${{ github.event_name }}
cancel-in-progress: true
env:
PYTHONWARNINGS: error
BASE_IMAGE: openpilot-base
BUILD: selfdrive/test/docker_build.sh base
RUN: docker run --shm-size 2G -v $PWD:/tmp/openpilot -w /tmp/openpilot -e CI=1 -e PYTHONWARNINGS=error -e FILEREADER_CACHE=1 -e PYTHONPATH=/tmp/openpilot -e NUM_JOBS -e JOB_ID -e GITHUB_ACTION -e GITHUB_REF -e GITHUB_HEAD_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_RUN_ID -v $GITHUB_WORKSPACE/.ci_cache/scons_cache:/tmp/scons_cache -v $GITHUB_WORKSPACE/.ci_cache/comma_download_cache:/tmp/comma_download_cache -v $GITHUB_WORKSPACE/.ci_cache/openpilot_cache:/tmp/openpilot_cache $BASE_IMAGE /bin/bash -c
jobs:
generate_cereal_artifact:
name: Generate cereal validation artifacts
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: ./.github/workflows/setup-with-retry
- name: Build openpilot
run: ${{ env.RUN }} "scons -j$(nproc) cereal"
- name: Generate the log file
run: |
${{ env.RUN }} "cereal/messaging/tests/validate_sp_cereal_upstream.py -g -f schema_instances.bin" && \
ls -la
ls -la cereal/messaging/tests
- name: 'Prepare artifact'
run: |
mkdir -p "cereal/messaging/tests/cereal_validations"
cp cereal/messaging/tests/validate_sp_cereal_upstream.py "cereal/messaging/tests/cereal_validations/validate_sp_cereal_upstream.py"
cp schema_instances.bin "cereal/messaging/tests/cereal_validations/schema_instances.bin"
- name: 'Upload Artifact'
uses: actions/upload-artifact@v4
with:
name: cereal_validations
path: cereal/messaging/tests/cereal_validations
validate_cereal_with_upstream:
name: Validate cereal with Upstream
runs-on: ubuntu-24.04
needs: generate_cereal_artifact
steps:
- uses: actions/checkout@v4
with:
repository: 'commaai/openpilot'
submodules: true
ref: "refs/heads/master"
- uses: ./.github/workflows/setup-with-retry
- name: Build openpilot
run: ${{ env.RUN }} "scons -j$(nproc) cereal"
- name: Download build artifacts
uses: actions/download-artifact@v4
with:
name: cereal_validations
path: cereal/messaging/tests/cereal_validations
- name: 'Run the validation'
run: |
chmod +x cereal/messaging/tests/cereal_validations/validate_sp_cereal_upstream.py
${{ env.RUN }} "cereal/messaging/tests/cereal_validations/validate_sp_cereal_upstream.py -r -f cereal/messaging/tests/cereal_validations/schema_instances.bin"
-101
View File
@@ -1,101 +0,0 @@
name: weekly CI test report
on:
schedule:
- cron: '37 9 * * 1' # 9:37AM UTC -> 2:37AM PST every monday
workflow_dispatch:
inputs:
ci_runs:
description: 'The amount of runs to trigger in CI test report'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
env:
CI_RUNS: ${{ github.event.inputs.ci_runs || '50' }}
jobs:
setup:
if: github.repository == 'sunnypilot/sunnypilot'
runs-on: ubuntu-latest
outputs:
ci_runs: ${{ steps.ci_runs_setup.outputs.matrix }}
steps:
- id: ci_runs_setup
name: CI_RUNS=${{ env.CI_RUNS }}
run: |
matrix=$(python3 -c "import json; print(json.dumps({ 'run_number' : list(range(${{ env.CI_RUNS }})) }))")
echo "matrix=$matrix" >> $GITHUB_OUTPUT
ci_matrix_run:
needs: [ setup ]
strategy:
fail-fast: false
matrix: ${{fromJSON(needs.setup.outputs.ci_runs)}}
uses: sunnypilot/sunnypilot/.github/workflows/ci_weekly_run.yaml@master
with:
run_number: ${{ matrix.run_number }}
report:
needs: [ci_matrix_run]
runs-on: ubuntu-latest
if: always()
steps:
- name: Get job results
uses: actions/github-script@v7
id: get-job-results
with:
script: |
const jobs = await github
.paginate("GET /repos/{owner}/{repo}/actions/runs/{run_id}/attempts/{attempt}/jobs", {
owner: "commaai",
repo: "${{ github.event.repository.name }}",
run_id: "${{ github.run_id }}",
attempt: "${{ github.run_attempt }}",
})
var report = {}
jobs.slice(1, jobs.length-1).forEach(job => {
if (job.conclusion === "skipped") return;
const jobName = job.name.split(" / ")[2];
const runRegex = /\((.*?)\)/;
const run = job.name.match(runRegex)[1];
report[jobName] = report[jobName] || { successes: [], failures: [], canceled: [] };
switch (job.conclusion) {
case "success":
report[jobName].successes.push({ "run_number": run, "link": job.html_url}); break;
case "failure":
report[jobName].failures.push({ "run_number": run, "link": job.html_url }); break;
case "canceled":
report[jobName].canceled.push({ "run_number": run, "link": job.html_url }); break;
}
});
return JSON.stringify({"jobs": report});
- name: Add job results to summary
env:
JOB_RESULTS: ${{ fromJSON(steps.get-job-results.outputs.result) }}
run: |
cat <<EOF >> template.html
<table>
<thead>
<tr>
<th></th>
<th>Job</th>
<th>✅ Passing</th>
<th>❌ Failure Details</th>
</tr>
</thead>
<tbody>
{% for key in jobs.keys() %}<tr>
<td>{% for i in range(5) %}{% if i+1 <= (5 * jobs[key]["successes"]|length // ${{ env.CI_RUNS }}) %}🟩{% else %}🟥{% endif %}{% endfor%}</td>
<td>{{ key }}</td>
<td>{{ 100 * jobs[key]["successes"]|length // ${{ env.CI_RUNS }} }}%</td>
<td>{% if jobs[key]["failures"]|length > 0 %}<details>{% for failure in jobs[key]["failures"] %}<a href="{{ failure['link'] }}">Log for run #{{ failure['run_number'] }}</a><br>{% endfor %}</details>{% else %}{% endif %}</td>
</td>
</tr>{% endfor %}
</table>
EOF
pip install jinja2-cli
echo $JOB_RESULTS | jinja2 template.html > report.html
echo "# CI Test Report - ${{ env.CI_RUNS }} Runs" >> $GITHUB_STEP_SUMMARY
cat report.html >> $GITHUB_STEP_SUMMARY
-17
View File
@@ -1,17 +0,0 @@
name: weekly CI test run
on:
workflow_call:
inputs:
run_number:
required: true
type: string
concurrency:
group: ci-run-${{ inputs.run_number }}-${{ github.ref }}
cancel-in-progress: true
jobs:
selfdrive_tests:
uses: sunnypilot/sunnypilot/.github/workflows/selfdrive_tests.yaml@master
with:
run_number: ${{ inputs.run_number }}
@@ -1,21 +0,0 @@
name: 'compile openpilot'
runs:
using: "composite"
steps:
- shell: bash
name: Build openpilot with all flags
run: |
${{ env.RUN }} "scons -j$(nproc)"
${{ env.RUN }} "release/check-dirty.sh"
- shell: bash
name: Cleanup scons cache and rebuild
run: |
${{ env.RUN }} "rm -rf /tmp/scons_cache/* && \
scons -j$(nproc) --cache-populate"
- name: Save scons cache
uses: actions/cache/save@v4
if: github.ref == 'refs/heads/master'
with:
path: .ci_cache/scons_cache
key: scons-${{ runner.arch }}-${{ env.CACHE_COMMIT_DATE }}-${{ github.sha }}
-65
View File
@@ -1,65 +0,0 @@
name: docs
on:
push:
branches:
- master
pull_request:
workflow_call:
inputs:
run_number:
default: '1'
required: true
type: string
concurrency:
group: docs-tests-ci-run-${{ inputs.run_number }}-${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && github.run_id || github.head_ref || github.ref }}-${{ github.workflow }}-${{ github.event_name }}
cancel-in-progress: true
jobs:
docs:
name: build docs
runs-on: ubuntu-24.04
steps:
- uses: commaai/timeout@v1
- uses: actions/checkout@v4
with:
submodules: true
# Build
- name: Build docs
run: |
# TODO: can we install just the "docs" dependency group without the normal deps?
pip install mkdocs
mkdocs build
# Push to docs.comma.ai
- uses: actions/checkout@v4
if: github.ref == 'refs/heads/master' && github.repository == 'sunnypilot/sunnypilot'
with:
path: openpilot-docs
ssh-key: ${{ secrets.OPENPILOT_DOCS_KEY }}
repository: sunnypilot/sunnypilot-docs
- name: Push
if: github.ref == 'refs/heads/master' && github.repository == 'sunnypilot/sunnypilot'
run: |
set -x
source release/identity.sh
cd openpilot-docs
git checkout --orphan tmp
git rm -rf .
# copy over docs
cp -r ../docs_site/ docs/
# GitHub pages config
touch docs/.nojekyll
echo -n docs.comma.ai > docs/CNAME
git add -f .
git commit -m "build docs"
# docs live in different repo to not bloat openpilot's full clone size
git push -f origin tmp:gh-pages
-59
View File
@@ -1,59 +0,0 @@
name: jenkins scan
on:
issue_comment:
types: [created, edited]
jobs:
# TODO: gc old branches in a separate job in this workflow
scan-comments:
runs-on: ubuntu-latest
if: ${{ github.event.issue.pull_request }}
permissions:
contents: write
issues: write
steps:
- name: Check for trigger phrase
id: check_comment
uses: actions/github-script@v7
with:
script: |
const triggerPhrase = "trigger-jenkins";
const comment = context.payload.comment.body;
const commenter = context.payload.comment.user.login;
const { data: permissions } = await github.rest.repos.getCollaboratorPermissionLevel({
owner: context.repo.owner,
repo: context.repo.repo,
username: commenter
});
const hasWriteAccess = permissions.permission === 'write' || permissions.permission === 'admin';
return (hasWriteAccess && comment.includes(triggerPhrase));
result-encoding: json
- name: Checkout repository
if: steps.check_comment.outputs.result == 'true'
uses: actions/checkout@v4
with:
ref: refs/pull/${{ github.event.issue.number }}/head
- name: Push to tmp-jenkins branch
if: steps.check_comment.outputs.result == 'true'
run: |
git config --global user.name "github-actions[bot]"
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git checkout -b tmp-jenkins-${{ github.event.issue.number }}
GIT_LFS_SKIP_PUSH=1 git push -f origin tmp-jenkins-${{ github.event.issue.number }}
- name: Delete trigger comment
if: steps.check_comment.outputs.result == 'true' && always()
uses: actions/github-script@v7
with:
script: |
await github.rest.issues.deleteComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: context.payload.comment.id,
});
-72
View File
@@ -1,72 +0,0 @@
name: Sync comma's LFS
env:
LFS_URL: 'https://gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git/info/lfs'
LFS_PUSH_URL: 'ssh://git@gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git'
on:
schedule:
- cron: '0 0 * * *' # Runs at 00:00 UTC every day
push:
branches:
- 'master'
pull_request:
branches:
- 'master'
workflow_dispatch: # enables manual triggering
inputs:
upstream_branch:
default: 'master'
type: string
jobs:
sync:
runs-on: ubuntu-latest
# Skip if PR is in draft mode
if: (github.event_name != 'pull_request' || (github.event_name == 'pull_request' && github.event.pull_request.draft == false)) && !github.event.pull_request.head.repo.fork
steps:
- name: Checkout Repository
uses: actions/checkout@v4
with:
repository: 'commaai/openpilot'
ref: ${{ inputs.upstream_branch }}
- name: LFS Fetch
run: |
git lfs fetch
- name: Set up Git
run: |
git config --global user.name 'GitHub Action'
git config --global user.email 'action@github.com'
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
- name: Add GitLab public keys
run: |
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Ensure branch
run: |
if git symbolic-ref -q HEAD >/dev/null; then
echo "Already on a branch, proceeding with push"
else
echo "Detached HEAD state detected, creating temporary branch"
git checkout -b temp_branch
fi
- name: Update LFS Config
run: |
echo '[lfs]' > .lfsconfig
echo ' url = ${{ env.LFS_URL }}' >> .lfsconfig
echo ' pushurl = ${{ env.LFS_PUSH_URL }}' >> .lfsconfig
echo ' locksverify = false' >> .lfsconfig
- name: Push LFS
id: sync-and-commit
run: |
git lfs ls-files -l
git lfs push --all origin
-42
View File
@@ -1,42 +0,0 @@
name: "model review"
on:
pull_request:
types: [opened, reopened, synchronize]
paths:
- 'selfdrive/modeld/models/*.onnx'
workflow_dispatch:
jobs:
comment:
permissions:
contents: read
pull-requests: write
runs-on: ubuntu-latest
if: github.repository == 'commaai/openpilot'
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Checkout master
uses: actions/checkout@v4
with:
ref: master
path: base
- run: git lfs pull
- run: cd base && git lfs pull
- run: pip install onnx
- name: scripts/reporter.py
id: report
run: |
echo "content<<EOF" >> $GITHUB_OUTPUT
echo "## Model Review" >> $GITHUB_OUTPUT
MASTER_PATH=${{ github.workspace }}/base python scripts/reporter.py >> $GITHUB_OUTPUT
echo "EOF" >> $GITHUB_OUTPUT
- name: Post model report comment
uses: marocchino/sticky-pull-request-comment@baa7203ed60924babbe5dcd0ac8eae3b66ec5e16
with:
header: model-review
message: ${{ steps.report.outputs.content }}
@@ -1,105 +0,0 @@
name: 'Post to Discourse'
description: 'Posts a message to a Discourse topic (existing or new)'
inputs:
discourse-url:
description: 'Discourse instance URL (e.g., https://discourse.example.com)'
required: true
api-key:
description: 'Discourse API key'
required: true
api-username:
description: 'Discourse API username'
required: true
topic-id:
description: 'Discourse topic ID to post to (use this OR category-id + title)'
required: false
category-id:
description: 'Category ID for new topic (required if topic-id not provided)'
required: false
title:
description: 'Title for new topic (required if topic-id not provided)'
required: false
message:
description: 'Message content (markdown supported)'
required: true
outputs:
post-number:
description: 'The post number in the topic'
value: ${{ steps.post.outputs.post_number }}
post-url:
description: 'Direct URL to the post'
value: ${{ steps.post.outputs.post_url }}
topic-id:
description: 'The topic ID (useful when creating a new topic)'
value: ${{ steps.post.outputs.topic_id }}
runs:
using: "composite"
steps:
- name: Post to Discourse
id: post
shell: bash
run: |
# Validate inputs
if [ -z "${{ inputs.topic-id }}" ] && ([ -z "${{ inputs.category-id }}" ] || [ -z "${{ inputs.title }}" ]); then
echo "❌ Error: Must provide either topic-id OR both category-id and title"
exit 1
fi
if [ -n "${{ inputs.topic-id }}" ] && ([ -n "${{ inputs.category-id }}" ] || [ -n "${{ inputs.title }}" ]); then
echo "⚠️ Warning: Both topic-id and category-id/title provided. Will post to existing topic."
fi
# Determine if creating new topic or posting to existing
if [ -n "${{ inputs.topic-id }}" ]; then
echo "📝 Posting to existing topic ID: ${{ inputs.topic-id }}"
# Create JSON payload for posting to existing topic
PAYLOAD=$(jq -n \
--arg content '${{ inputs.message }}' \
--arg topic_id "${{ inputs.topic-id }}" \
'{topic_id: $topic_id, raw: $content}')
else
echo "✨ Creating new topic: ${{ inputs.title }}"
# Create JSON payload for new topic
PAYLOAD=$(jq -n \
--arg content '${{ inputs.message }}' \
--arg title "${{ inputs.title }}" \
--arg category "${{ inputs.category-id }}" \
'{title: $title, category: ($category | tonumber), raw: $content}')
fi
# Post to Discourse
RESPONSE=$(curl -s -w "\n%{http_code}" \
-X POST "${{ inputs.discourse-url }}/posts.json" \
-H "Content-Type: application/json" \
-H "Api-Key: ${{ inputs.api-key }}" \
-H "Api-Username: ${{ inputs.api-username }}" \
-d "$PAYLOAD")
HTTP_CODE=$(echo "$RESPONSE" | tail -n1)
BODY=$(echo "$RESPONSE" | sed '$d')
if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then
echo "✅ Successfully posted to Discourse!"
POST_NUMBER=$(echo "$BODY" | jq -r '.post_number // "unknown"')
TOPIC_ID=$(echo "$BODY" | jq -r '.topic_id // "${{ inputs.topic-id }}"')
POST_URL="${{ inputs.discourse-url }}/t/${TOPIC_ID}/${POST_NUMBER}"
echo "post_number=${POST_NUMBER}" >> $GITHUB_OUTPUT
echo "post_url=${POST_URL}" >> $GITHUB_OUTPUT
echo "topic_id=${TOPIC_ID}" >> $GITHUB_OUTPUT
echo "Topic ID: ${TOPIC_ID}"
echo "Post number: ${POST_NUMBER}"
echo "URL: ${POST_URL}"
else
echo "❌ Failed to post to Discourse"
echo "HTTP Code: ${HTTP_CODE}"
echo "Response: ${BODY}"
exit 1
fi
-39
View File
@@ -1,39 +0,0 @@
name: prebuilt
on:
schedule:
- cron: '0 * * * *'
workflow_dispatch:
env:
DOCKER_LOGIN: docker login ghcr.io -u ${{ github.actor }} -p ${{ secrets.GITHUB_TOKEN }}
BUILD: release/ci/docker_build_sp.sh prebuilt
jobs:
build_prebuilt:
name: build prebuilt
runs-on: ubuntu-latest
if: github.repository == 'sunnypilot/sunnypilot'
env:
PUSH_IMAGE: true
permissions:
checks: read
contents: read
packages: write
steps:
- name: Wait for green check mark
if: ${{ github.event_name != 'workflow_dispatch' }}
uses: lewagon/wait-on-check-action@ccfb013c15c8afb7bf2b7c028fb74dc5a068cccc
with:
ref: master
wait-interval: 30
running-workflow-name: 'build prebuilt'
repo-token: ${{ secrets.GITHUB_TOKEN }}
check-regexp: ^((?!.*(build master-ci).*).)*$
- uses: actions/checkout@v4
with:
submodules: true
- run: git lfs pull
- name: Build and Push docker image
run: |
$DOCKER_LOGIN
eval "$BUILD"
-28
View File
@@ -1,28 +0,0 @@
name: Release Drafter
on:
push:
branches:
- master
tags:
- 'v*'
pull_request_target:
types: [opened, reopened, synchronize]
workflow_dispatch:
permissions:
contents: read
jobs:
update_release_draft:
permissions:
contents: write
pull-requests: write
runs-on: ubuntu-latest
steps:
- uses: release-drafter/release-drafter@v6
with:
config-name: release-drafter.yml
prerelease: ${{ !startsWith(github.ref, 'refs/tags/v') }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-42
View File
@@ -1,42 +0,0 @@
name: release
on:
schedule:
- cron: '0 9 * * *'
workflow_dispatch:
jobs:
build___nightly:
name: build __nightly
env:
ImageOS: ubuntu24
container:
image: ghcr.io/sunnypilot/sunnypilot-base:latest
runs-on: ubuntu-latest
if: github.repository == 'sunnypilot/sunnypilot'
permissions:
checks: read
contents: write
steps:
- name: Install wait-on-check-action dependencies
run: |
sudo apt-get update
sudo apt-get install -y libyaml-dev
- name: Wait for green check mark
if: ${{ github.event_name == 'schedule' }}
uses: lewagon/wait-on-check-action@ccfb013c15c8afb7bf2b7c028fb74dc5a068cccc
with:
ref: master
wait-interval: 30
running-workflow-name: 'build __nightly'
repo-token: ${{ secrets.GITHUB_TOKEN }}
check-regexp: ^((?!.*(build prebuilt).*).)*$
- uses: actions/checkout@v4
with:
submodules: true
fetch-depth: 0
- name: Pull LFS
run: |
git config --global --add safe.directory '*'
git lfs pull
- name: Push __nightly
run: BRANCH=__nightly release/build_stripped.sh
-74
View File
@@ -1,74 +0,0 @@
name: repo maintenance
on:
schedule:
- cron: "0 14 * * 1" # every Monday at 2am UTC (6am PST)
workflow_dispatch:
env:
BASE_IMAGE: sunnypilot-base
BUILD: release/ci/docker_build_sp.sh base
RUN: docker run --shm-size 2G -v $PWD:/tmp/openpilot -w /tmp/openpilot -e CI=1 -e PYTHONWARNINGS=error -e FILEREADER_CACHE=1 -e PYTHONPATH=/tmp/openpilot -e NUM_JOBS -e JOB_ID -e GITHUB_ACTION -e GITHUB_REF -e GITHUB_HEAD_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_RUN_ID -v $GITHUB_WORKSPACE/.ci_cache/scons_cache:/tmp/scons_cache -v $GITHUB_WORKSPACE/.ci_cache/comma_download_cache:/tmp/comma_download_cache -v $GITHUB_WORKSPACE/.ci_cache/openpilot_cache:/tmp/openpilot_cache $BASE_IMAGE /bin/bash -c
jobs:
update_translations:
runs-on: ubuntu-latest
if: github.repository == 'sunnypilot/sunnypilot'
steps:
- uses: actions/checkout@v4
- uses: ./.github/workflows/setup-with-retry
- name: Update translations
run: |
${{ env.RUN }} "python3 selfdrive/ui/update_translations.py --vanish"
- name: Create Pull Request
uses: peter-evans/create-pull-request@9153d834b60caba6d51c9b9510b087acf9f33f83
with:
author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit-message: "Update translations"
title: "[bot] Update translations"
body: "Automatic PR from repo-maintenance -> update_translations"
branch: "update-translations"
base: "master"
delete-branch: true
labels: bot
package_updates:
name: package_updates
runs-on: ubuntu-latest
container:
image: ghcr.io/sunnypilot/sunnypilot-base:latest
if: github.repository == 'sunnypilot/sunnypilot'
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: uv lock
if: github.repository == 'commaai/openpilot'
run: |
python3 -m ensurepip --upgrade
pip3 install uv
uv lock --upgrade
- name: bump submodules
run: |
git config --global --add safe.directory '*'
git config submodule.tinygrad.update none
git submodule update --remote
git add .
- name: update car docs
run: |
export PYTHONPATH="$PWD"
scons -j$(nproc) --minimal opendbc_repo
python selfdrive/car/docs.py
git add docs/CARS.md
- name: Create Pull Request
uses: peter-evans/create-pull-request@9153d834b60caba6d51c9b9510b087acf9f33f83
with:
author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
token: ${{ github.repository == 'commaai/openpilot' && secrets.ACTIONS_CREATE_PR_PAT || secrets.GITHUB_TOKEN }}
commit-message: Update Python packages
title: '[bot] Update Python packages'
branch: auto-package-updates
base: master
delete-branch: true
body: 'Automatic PR from repo-maintenance -> package_updates'
labels: bot
-312
View File
@@ -1,312 +0,0 @@
name: selfdrive
on:
push:
branches:
- master
pull_request:
workflow_dispatch:
workflow_call:
inputs:
run_number:
default: '1'
required: true
type: string
concurrency:
group: selfdrive-tests-ci-run-${{ inputs.run_number }}-${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && github.run_id || github.head_ref || github.ref }}-${{ github.workflow }}-${{ github.event_name }}
cancel-in-progress: true
env:
PYTHONWARNINGS: error
BASE_IMAGE: sunnypilot-base
AZURE_TOKEN: ${{ secrets.AZURE_COMMADATACI_OPENPILOTCI_TOKEN }}
MAPBOX_TOKEN_CI: ${{ secrets.MAPBOX_TOKEN_CI }}
DOCKER_LOGIN: docker login ghcr.io -u ${{ github.actor }} -p ${{ secrets.GITHUB_TOKEN }}
BUILD: release/ci/docker_build_sp.sh base
RUN: docker run --shm-size 2G -v $PWD:/tmp/openpilot -w /tmp/openpilot -e CI=1 -e PYTHONWARNINGS=error -e FILEREADER_CACHE=1 -e PYTHONPATH=/tmp/openpilot -e NUM_JOBS -e JOB_ID -e GITHUB_ACTION -e GITHUB_REF -e GITHUB_HEAD_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_RUN_ID -e MAPBOX_TOKEN_CI=$MAPBOX_TOKEN_CI -v $GITHUB_WORKSPACE/.ci_cache/scons_cache:/tmp/scons_cache -v $GITHUB_WORKSPACE/.ci_cache/comma_download_cache:/tmp/comma_download_cache -v $GITHUB_WORKSPACE/.ci_cache/openpilot_cache:/tmp/openpilot_cache $BASE_IMAGE /bin/bash -c
PYTEST: pytest --continue-on-collection-errors --durations=0 -n logical
jobs:
build_release:
name: build release
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16", "namespace-experiments:docker.builds.local-cache=separate"]')
|| fromJSON('["ubuntu-24.04"]') }}
env:
STRIPPED_DIR: /tmp/releasepilot
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Getting LFS files
uses: nick-fields/retry@7152eba30c6575329ac0576536151aca5a72780e
with:
timeout_minutes: 2
max_attempts: 3
command: git lfs pull
- name: Build devel
timeout-minutes: 1
run: TARGET_DIR=$STRIPPED_DIR release/build_stripped.sh
- uses: ./.github/workflows/setup-with-retry
- name: Build openpilot and run checks
timeout-minutes: ${{ ((steps.restore-scons-cache.outputs.cache-hit == 'true') && 10 || 30) }} # allow more time when we missed the scons cache
run: |
cd $STRIPPED_DIR
${{ env.RUN }} "python3 system/manager/build.py"
- name: Run tests
timeout-minutes: 1
run: |
cd $STRIPPED_DIR
${{ env.RUN }} "release/check-dirty.sh"
- name: Check submodules
if: github.repository == 'sunnypilot/sunnypilot'
timeout-minutes: 3
run: |
if [ "${{ github.ref }}" != "refs/heads/master" ]; then
git fetch origin master:refs/remotes/origin/master
SUBMODULE_PATHS=$(git diff origin/master HEAD --name-only | grep -E '^[^/]+$' | while read path; do
if git ls-files --stage "$path" | grep -q "^160000"; then
echo "$path"
fi
done | tr '\n' ' ')
if [ -n "$SUBMODULE_PATHS" ]; then
echo "Changed submodule paths: $SUBMODULE_PATHS"
export SUBMODULE_PATHS="$SUBMODULE_PATHS"
export CHECK_PR_REFS=true
fi
fi
release/check-submodules.sh
build:
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16", "namespace-experiments:docker.builds.local-cache=separate"]')
|| fromJSON('["ubuntu-24.04"]') }}
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Setup docker push
if: github.ref == 'refs/heads/master' && github.event_name != 'pull_request' && github.repository == 'sunnypilot/sunnypilot'
run: |
echo "PUSH_IMAGE=true" >> "$GITHUB_ENV"
$DOCKER_LOGIN
- uses: ./.github/workflows/setup-with-retry
- uses: ./.github/workflows/compile-openpilot
timeout-minutes: 30
build_mac:
name: build macOS
runs-on: ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-macos-8x14' || 'macos-latest' }}
steps:
- uses: actions/checkout@v4
with:
submodules: true
- run: echo "CACHE_COMMIT_DATE=$(git log -1 --pretty='format:%cd' --date=format:'%Y-%m-%d-%H:%M')" >> $GITHUB_ENV
- name: Homebrew cache
uses: ./.github/workflows/auto-cache
if: false # disabling the cache for now because it is breaking macos builds...
with:
save: false # No need save here if we manually save it later conditionally
path: ~/Library/Caches/Homebrew
key: brew-macos-${{ env.CACHE_COMMIT_DATE }}-${{ github.sha }}
restore-keys: |
brew-macos-${{ env.CACHE_COMMIT_DATE }}
brew-macos
- name: Install dependencies
run: ./tools/mac_setup.sh
env:
PYTHONWARNINGS: default # package install has DeprecationWarnings
HOMEBREW_DISPLAY_INSTALL_TIMES: 1
- name: Save Homebrew cache
uses: actions/cache/save@v4
if: github.ref == 'refs/heads/master'
with:
path: ~/Library/Caches/Homebrew
key: brew-macos-${{ env.CACHE_COMMIT_DATE }}-${{ github.sha }}
- run: git lfs pull
- name: Getting scons cache
uses: ./.github/workflows/auto-cache
with:
save: false # No need save here if we manually save it later conditionally
path: /tmp/scons_cache
key: scons-${{ runner.arch }}-macos-${{ env.CACHE_COMMIT_DATE }}-${{ github.sha }}
restore-keys: |
scons-${{ runner.arch }}-macos-${{ env.CACHE_COMMIT_DATE }}
scons-${{ runner.arch }}-macos
- name: Building openpilot
run: . .venv/bin/activate && scons -j$(nproc)
- name: Save scons cache
uses: actions/cache/save@v4
if: github.ref == 'refs/heads/master'
with:
path: /tmp/scons_cache
key: scons-${{ runner.arch }}-macos-${{ env.CACHE_COMMIT_DATE }}-${{ github.sha }}
static_analysis:
name: static analysis
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16", "namespace-experiments:docker.builds.local-cache=separate"]')
|| fromJSON('["ubuntu-24.04"]') }}
env:
PYTHONWARNINGS: default
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: ./.github/workflows/setup-with-retry
- name: Static analysis
timeout-minutes: 1
run: ${{ env.RUN }} "scripts/lint/lint.sh"
unit_tests:
name: unit tests
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16", "namespace-experiments:docker.builds.local-cache=separate"]')
|| fromJSON('["ubuntu-24.04"]') }}
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: ./.github/workflows/setup-with-retry
id: setup-step
- name: Build openpilot
run: ${{ env.RUN }} "scons -j$(nproc)"
- name: Run unit tests
timeout-minutes: ${{ contains(runner.name, 'nsc') && ((steps.setup-step.outputs.duration < 18) && 1 || 2) || 999 }}
run: |
${{ env.RUN }} "source selfdrive/test/setup_xvfb.sh && \
# Pre-compile Python bytecode so each pytest worker doesn't need to
$PYTEST --collect-only -m 'not slow' -qq && \
MAX_EXAMPLES=1 $PYTEST -m 'not slow' && \
./selfdrive/ui/tests/create_test_translations.sh && \
QT_QPA_PLATFORM=offscreen ./selfdrive/ui/tests/test_translations && \
chmod -R 777 /tmp/comma_download_cache"
process_replay:
name: process replay
if: false # disable process_replay for forks
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16", "namespace-experiments:docker.builds.local-cache=separate"]')
|| fromJSON('["ubuntu-24.04"]') }}
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: ./.github/workflows/setup-with-retry
id: setup-step
- name: Cache test routes
id: dependency-cache
uses: actions/cache@v4
with:
path: .ci_cache/comma_download_cache
key: proc-replay-${{ hashFiles('selfdrive/test/process_replay/ref_commit', 'selfdrive/test/process_replay/test_processes.py') }}
- name: Build openpilot
run: |
${{ env.RUN }} "scons -j$(nproc)"
- name: Run replay
timeout-minutes: ${{ contains(runner.name, 'nsc') && (steps.dependency-cache.outputs.cache-hit == 'true') && ((steps.setup-step.outputs.duration < 18) && 1 || 2) || 20 }}
run: |
${{ env.RUN }} "selfdrive/test/process_replay/test_processes.py -j$(nproc) && \
chmod -R 777 /tmp/comma_download_cache"
- name: Print diff
id: print-diff
if: always()
run: cat selfdrive/test/process_replay/diff.txt
- uses: actions/upload-artifact@v4
if: always()
continue-on-error: true
with:
name: process_replay_diff.txt
path: selfdrive/test/process_replay/diff.txt
- name: Upload reference logs
if: false # TODO: move this to github instead of azure
run: |
${{ env.RUN }} "unset PYTHONWARNINGS && AZURE_TOKEN='$AZURE_TOKEN' python3 selfdrive/test/process_replay/test_processes.py -j$(nproc) --upload-only"
- name: Run regen
if: false
timeout-minutes: 4
run: |
${{ env.RUN }} "ONNXCPU=1 $PYTEST selfdrive/test/process_replay/test_regen.py && \
chmod -R 777 /tmp/comma_download_cache"
simulator_driving:
name: simulator driving
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16", "namespace-experiments:docker.builds.local-cache=separate"]')
|| fromJSON('["ubuntu-24.04"]') }}
if: (github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: ./.github/workflows/setup-with-retry
id: setup-step
- name: Build openpilot
run: |
${{ env.RUN }} "scons -j$(nproc)"
- name: Driving test
timeout-minutes: ${{ (steps.setup-step.outputs.duration < 18) && 1 || 2 }}
run: |
${{ env.RUN }} "source selfdrive/test/setup_xvfb.sh && \
source selfdrive/test/setup_vsound.sh && \
CI=1 pytest -s tools/sim/tests/test_metadrive_bridge.py"
create_ui_report:
# This job name needs to be the same as UI_JOB_NAME in ui_preview.yaml
name: Create UI Report
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16", "namespace-experiments:docker.builds.local-cache=separate"]')
|| fromJSON('["ubuntu-24.04"]') }}
if: false # FIXME: FrameReader is broken on CI runners
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: ./.github/workflows/setup-with-retry
- name: caching frames
id: frames-cache
uses: actions/cache@v4
with:
path: .ci_cache/comma_download_cache
key: ui_screenshots_test_${{ hashFiles('selfdrive/ui/tests/test_ui/run.py') }}
- name: Build openpilot
run: ${{ env.RUN }} "scons -j$(nproc)"
- name: Create Test Report
timeout-minutes: ${{ ((steps.frames-cache.outputs.cache-hit == 'true') && 2 || 4) }}
run: >
${{ env.RUN }} "PYTHONWARNINGS=ignore &&
source selfdrive/test/setup_xvfb.sh &&
CACHE_ROOT=/tmp/comma_download_cache python3 selfdrive/ui/tests/test_ui/run.py &&
chmod -R 777 /tmp/comma_download_cache"
- name: Upload Test Report
uses: actions/upload-artifact@v4
with:
name: report-${{ inputs.run_number || '1' }}-${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && 'master' || github.event.number }}
path: selfdrive/ui/tests/test_ui/report_1/screenshots
@@ -1,52 +0,0 @@
name: 'openpilot env setup, with retry on failure'
inputs:
docker_hub_pat:
description: 'Auth token for Docker Hub, required for BuildJet jobs'
required: false
default: ''
sleep_time:
description: 'Time to sleep between retries'
required: false
default: 30
outputs:
duration:
description: 'Duration of the setup process in seconds'
value: ${{ steps.get_duration.outputs.duration }}
runs:
using: "composite"
steps:
- id: start_time
shell: bash
run: echo "START_TIME=$(date +%s)" >> $GITHUB_ENV
- id: setup1
uses: ./.github/workflows/setup
continue-on-error: true
with:
is_retried: true
- if: steps.setup1.outcome == 'failure'
shell: bash
run: sleep ${{ inputs.sleep_time }}
- id: setup2
if: steps.setup1.outcome == 'failure'
uses: ./.github/workflows/setup
continue-on-error: true
with:
is_retried: true
- if: steps.setup2.outcome == 'failure'
shell: bash
run: sleep ${{ inputs.sleep_time }}
- id: setup3
if: steps.setup2.outcome == 'failure'
uses: ./.github/workflows/setup
with:
is_retried: true
- id: get_duration
shell: bash
run: |
END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))
echo "Total duration: $DURATION seconds"
echo "duration=$DURATION" >> $GITHUB_OUTPUT
-56
View File
@@ -1,56 +0,0 @@
name: 'openpilot env setup'
inputs:
is_retried:
description: 'A mock param that asserts that we use the setup-with-retry instead of this action directly'
required: false
default: 'false'
runs:
using: "composite"
steps:
# assert that this action is retried using the setup-with-retry
- shell: bash
if: ${{ inputs.is_retried == 'false' }}
run: |
echo "You should not run this action directly. Use setup-with-retry instead"
exit 1
- shell: bash
name: No retries!
run: |
if [ "${{ github.run_attempt }}" -gt ${{ github.event.pull_request.head.repo.fork && github.event.pull_request.author_association == 'NONE' && 2 || 1}} ]; then
echo -e "\033[0;31m##################################################"
echo -e "\033[0;31m Retries not allowed! Fix the flaky test! "
echo -e "\033[0;31m##################################################\033[0m"
exit 1
fi
# do this after checkout to ensure our custom LFS config is used to pull from GitLab
- shell: bash
run: git lfs pull
# build cache
- id: date
shell: bash
run: echo "CACHE_COMMIT_DATE=$(git log -1 --pretty='format:%cd' --date=format:'%Y-%m-%d-%H:%M')" >> $GITHUB_ENV
- shell: bash
run: echo "$CACHE_COMMIT_DATE"
- id: scons-cache
uses: ./.github/workflows/auto-cache
with:
path: .ci_cache/scons_cache
key: scons-${{ runner.arch }}-${{ env.CACHE_COMMIT_DATE }}-${{ github.sha }}
restore-keys: |
scons-${{ runner.arch }}-${{ env.CACHE_COMMIT_DATE }}
scons-${{ runner.arch }}
# as suggested here: https://github.com/moby/moby/issues/32816#issuecomment-910030001
- id: normalize-file-permissions
shell: bash
name: Normalize file permissions to ensure a consistent docker build cache
run: |
find . -type f -executable -not -perm 755 -exec chmod 755 {} \;
find . -type f -not -executable -not -perm 644 -exec chmod 644 {} \;
# build our docker image
- shell: bash
run: eval ${{ env.BUILD }}
-52
View File
@@ -1,52 +0,0 @@
name: stale
on:
schedule:
- cron: '30 1 * * *'
workflow_dispatch:
env:
DAYS_BEFORE_PR_CLOSE: 2
DAYS_BEFORE_PR_STALE: 9
DAYS_BEFORE_PR_STALE_DRAFT: 30
jobs:
stale:
runs-on: ubuntu-latest
steps:
- uses: actions/stale@v9
with:
exempt-all-milestones: true
# pull request config
stale-pr-message: 'This PR has had no activity for ${{ env.DAYS_BEFORE_PR_STALE }} days. It will be automatically closed in ${{ env.DAYS_BEFORE_PR_CLOSE }} days if there is no activity.'
close-pr-message: 'This PR has been automatically closed due to inactivity. Feel free to re-open once activity resumes.'
stale-pr-label: stale
delete-branch: ${{ github.event.pull_request.head.repo.full_name == 'sunnypilot/sunnypilot' }} # only delete branches on the main repo
exempt-pr-labels: "ignore stale,needs testing" # if wip or it needs testing from the community, don't mark as stale
days-before-pr-stale: ${{ env.DAYS_BEFORE_PR_STALE }}
days-before-pr-close: ${{ env.DAYS_BEFORE_PR_CLOSE }}
exempt-draft-pr: false
# issue config
days-before-issue-stale: -1 # ignore issues for now
# same as above, but give draft PRs more time
stale_drafts:
runs-on: ubuntu-latest
steps:
- uses: actions/stale@v9
with:
exempt-all-milestones: true
# pull request config
stale-pr-message: 'This PR has had no activity for ${{ env.DAYS_BEFORE_PR_STALE_DRAFT }} days. It will be automatically closed in ${{ env.DAYS_BEFORE_PR_CLOSE }} days if there is no activity.'
close-pr-message: 'This PR has been automatically closed due to inactivity. Feel free to re-open once activity resumes.'
stale-pr-label: stale
delete-branch: ${{ github.event.pull_request.head.repo.full_name == 'commaai/openpilot' }} # only delete branches on the main repo
exempt-pr-labels: "ignore stale,needs testing" # if wip or it needs testing from the community, don't mark as stale
days-before-pr-stale: ${{ env.DAYS_BEFORE_PR_STALE_DRAFT }}
days-before-pr-close: ${{ env.DAYS_BEFORE_PR_CLOSE }}
exempt-draft-pr: true
# issue config
days-before-issue-stale: -1 # ignore issues for now
@@ -1,219 +0,0 @@
name: Build Model from Upstream
env:
BUILD_DIR: "/data/openpilot"
OUTPUT_DIR: ${{ github.workspace }}/output
SCONS_CACHE_DIR: ${{ github.workspace }}/release/ci/scons_cache
UPSTREAM_REPO: "commaai/openpilot"
TINYGRAD_PATH: ${{ github.workspace }}/tinygrad_repo
MODELS_DIR: ${{ github.workspace }}/selfdrive/modeld/models
on:
workflow_call:
inputs:
upstream_branch:
description: 'Upstream branch to build from'
required: true
default: 'master'
type: string
custom_name:
description: 'Custom name for the model (no date, only name)'
required: false
type: string
is_20hz:
description: 'Is this a 20Hz model'
required: false
type: boolean
default: true
artifact_suffix:
description: 'Suffix for artifact name'
required: false
type: string
default: ''
workflow_dispatch:
inputs:
upstream_branch:
description: 'Upstream branch to build from'
required: true
default: 'master'
type: string
custom_name:
description: 'Custom name for the model (no date, only name)'
required: false
type: string
is_20hz:
description: 'Is this a 20Hz model'
required: false
type: boolean
default: true
run-name: Build model [${{ inputs.custom_name || inputs.upstream_branch }}] from ref [${{ inputs.upstream_branch }}]
jobs:
get_model:
runs-on: ubuntu-latest
env:
REF: ${{ inputs.upstream_branch }}
outputs:
model_date: ${{ steps.commit-date.outputs.model_date }}
steps:
# Note: To allow dynamic models from both openpilot and sunnypilot (merges/mashups), we try commaai as default,
# and fallback to sunnypilot if the ref checkout fails.
- name: Checkout commaai/openpilot
id: checkout_upstream
continue-on-error: true
uses: actions/checkout@v4
with:
repository: commaai/openpilot
ref: ${{ inputs.upstream_branch }}
submodules: recursive
path: openpilot
- name: Fallback to sunnypilot/sunnypilot
if: steps.checkout_upstream.outcome == 'failure'
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot
ref: ${{ inputs.upstream_branch }}
submodules: recursive
path: openpilot
- name: Get commit date
id: commit-date
run: |
cd ${{ github.workspace }}/openpilot
commit_date=$(git log -1 --format=%cd --date=format:'%B %d, %Y')
echo "model_date=${commit_date}" >> $GITHUB_OUTPUT
cat $GITHUB_OUTPUT
- run: |
cd ${{ github.workspace }}/openpilot
git lfs pull
- name: 'Upload Artifact'
uses: actions/upload-artifact@v4
with:
name: models-${{ env.REF }}${{ inputs.artifact_suffix }}
path: ${{ github.workspace }}/openpilot/selfdrive/modeld/models/*.onnx
build_model:
runs-on: [self-hosted, tici]
needs: get_model
env:
MODEL_NAME: ${{ inputs.custom_name || inputs.upstream_branch }} (${{ needs.get_model.outputs.model_date }})
REF: ${{ inputs.upstream_branch }}
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- run: git lfs pull
- name: Cache SCons
uses: actions/cache@v4
with:
path: ${{env.SCONS_CACHE_DIR}}
key: scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-model-${{ github.sha }}
# Note: GitHub Actions enforces cache isolation between different build sources (PR builds, workflow dispatches, etc.)
# for security. Only caches from the default branch are shared across all builds. This is by design and cannot be overridden.
restore-keys: |
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-model
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_NEW_BRANCH }}-model
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_BRANCH }}-model
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_NEW_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}
- name: Set environment variables
id: set-env
run: |
# Set up common environment
source /etc/profile;
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
printenv >> $GITHUB_ENV
if [[ "${{ runner.debug }}" == "1" ]]; then
cat $GITHUB_OUTPUT
fi
- name: Setup build environment
run: |
mkdir -p "${BUILD_DIR}/"
sudo find $BUILD_DIR/ -mindepth 1 -delete
echo "Starting build stage..."
echo "BUILD_DIR: ${BUILD_DIR}"
echo "CI_DIR: ${CI_DIR}"
echo "VERSION: ${{ steps.set-env.outputs.version }}"
echo "UV_PROJECT_ENVIRONMENT: ${UV_PROJECT_ENVIRONMENT}"
echo "VIRTUAL_ENV: ${VIRTUAL_ENV}"
echo "-------"
if [[ "${{ runner.debug }}" == "1" ]]; then
printenv
fi
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
rm -rf ${{ env.MODELS_DIR }}/*.onnx
- name: Download model artifacts
uses: actions/download-artifact@v4
with:
name: models-${{ env.REF }}${{ inputs.artifact_suffix }}
path: ${{ github.workspace }}/selfdrive/modeld/models
- name: Build Model
run: |
source /etc/profile
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
export PYTHONPATH="${PYTHONPATH}:${{ env.TINYGRAD_PATH }}"
# Loop through all .onnx files
find "${{ env.MODELS_DIR }}" -maxdepth 1 -name '*.onnx' | while IFS= read -r onnx_file; do
base_name=$(basename "$onnx_file" .onnx)
output_file="${{ env.MODELS_DIR }}/${base_name}_tinygrad.pkl"
echo "Compiling: $onnx_file -> $output_file"
QCOM=1 python3 "${{ env.TINYGRAD_PATH }}/examples/openpilot/compile3.py" "$onnx_file" "$output_file"
QCOM=1 python3 "${{ env.MODELS_DIR }}/../get_model_metadata.py" "$onnx_file" || true
done
- name: Prepare Output
run: |
sudo rm -rf ${{ env.OUTPUT_DIR }}
mkdir -p ${{ env.OUTPUT_DIR }}
# Copy the model files
rsync -avm \
--include='*.dlc' \
--include='*.thneed' \
--include='*.pkl' \
--include='*.onnx' \
--exclude='*' \
--delete-excluded \
--chown=comma:comma \
${{ env.MODELS_DIR }}/ ${{ env.OUTPUT_DIR }}/
python3 "${{ github.workspace }}/release/ci/model_generator.py" \
--model-dir "${{ env.MODELS_DIR }}" \
--output-dir "${{ env.OUTPUT_DIR }}" \
--custom-name "${{ env.MODEL_NAME }}" \
--upstream-branch "${{ inputs.upstream_branch }}" \
${{ inputs.is_20hz && '--is-20hz' || '' }}
- name: Write artifact name to file
run: echo "model-${{ env.MODEL_NAME }}${{ inputs.artifact_suffix }}-${{ github.run_number }}" > ${{ env.OUTPUT_DIR }}/artifact_name.txt
- name: Upload Build Artifacts
id: upload-artifact
uses: actions/upload-artifact@v4
with:
name: model-${{ env.MODEL_NAME }}${{ inputs.artifact_suffix }}-${{ github.run_number }}
path: ${{ env.OUTPUT_DIR }}
- name: Upload artifact name file
uses: actions/upload-artifact@v4
with:
name: artifact-name-${{ inputs.custom_name || inputs.upstream_branch }}
path: ${{ env.OUTPUT_DIR }}/artifact_name.txt
- name: Re-enable powersave
if: always()
run: |
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
@@ -1,372 +0,0 @@
name: sunnypilot prebuilt action
env:
BUILD_DIR: "/data/openpilot"
OUTPUT_DIR: ${{ github.workspace }}/output
CI_DIR: ${{ github.workspace }}/release/ci
SCONS_CACHE_DIR: ${{ github.workspace }}/release/ci/scons_cache
PUBLIC_REPO_URL: "https://github.com/sunnypilot/sunnypilot"
# Branch configurations
STAGING_SOURCE_BRANCH: 'master'
# Runtime configuration
SOURCE_BRANCH: "${{ github.head_ref || github.ref_name }}"
on:
push:
branches: [ master, master-dev ]
tags: [ 'release/*' ]
pull_request_target:
types: [ labeled ]
workflow_dispatch:
inputs:
wait_for_tests:
description: 'Wait for selfdrive_tests to finish'
required: false
type: boolean
default: false
jobs:
prepare_strategy:
runs-on: ubuntu-24.04
if: (!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
outputs:
environment: ${{ steps.strategy.outputs.environment }}
new_branch: ${{ steps.strategy.outputs.new_branch }}
extra_version_identifier: ${{ steps.strategy.outputs.extra_version_identifier }}
version: ${{ steps.strategy.outputs.version }}
cancel_publish_in_progress: ${{ steps.strategy.outputs.cancel_publish_in_progress }}
publish_concurrency_group: ${{ steps.strategy.outputs.publish_concurrency_group }}
is_stable_branch: ${{ steps.strategy.outputs.is_stable_branch }}
build: ${{ steps.strategy.outputs.build }}
steps:
- uses: actions/checkout@v4
- name: Extract deploy strategy
id: strategy
run: |
echo '::group::Strategy Extraction'
BRANCH="${{ github.head_ref || github.ref_name }}"
echo "Current branch: $BRANCH"
STRATEGY_JSON='${{ vars.DEPLOY_STRATEGY }}'
CONFIG=$(echo "$STRATEGY_JSON" | jq -r --arg branch "$BRANCH" '
.configs[] | select(.branch == $branch)
')
BUILD="$(date '+%Y.%m.%d')-${{ github.run_number }}"
if [[ -z "$CONFIG" || "$CONFIG" == "null" ]]; then
echo "No exact strategy match found. Falling back to feature/fork logic."
IS_FORK="${{ github.event.pull_request.head.repo.fork && 'true' || 'false' }}"
FORK_SUFFIX=$( [[ "$IS_FORK" == "true" ]] && echo "-fork" || echo "" )
NEW_BRANCH="${BRANCH}${FORK_SUFFIX}-prebuilt"
echo "new_branch=$NEW_BRANCH" >> $GITHUB_OUTPUT
echo "version=$BUILD" >> $GITHUB_OUTPUT
echo "cancel_publish_in_progress=true" >> $GITHUB_OUTPUT
echo "publish_concurrency_group=publish-${BRANCH}" >> $GITHUB_OUTPUT
echo "environment=feature-branch" >> $GITHUB_OUTPUT
echo "extra_version_identifier=feature-branch" >> $GITHUB_OUTPUT
else
echo "Matched config: $CONFIG"
environment=$(echo "$CONFIG" | jq -r '.environment')
echo "environment=$environment" >> $GITHUB_OUTPUT
echo "new_branch=$(echo "$CONFIG" | jq -r '.target_branch')" >> $GITHUB_OUTPUT
cancel="$(echo "$CONFIG" | jq -r '.cancel_publish_in_progress')";
echo "cancel_publish_in_progress=$( [ "$cancel" = "null" ] && echo "true" || echo $cancel)" >> $GITHUB_OUTPUT
echo "publish_concurrency_group=publish-${BRANCH}$( [ "$cancel" = "null" ] || [ "$cancel" = "true" ] || echo "${{ github.sha }}" )" >> $GITHUB_OUTPUT
is_stable_branch="$(echo "$CONFIG" | jq -r '.stable_branch // false')";
echo "is_stable_branch=$is_stable_branch" >> $GITHUB_OUTPUT
stable_version=$(cat sunnypilot/common/version.h | grep SUNNYPILOT_VERSION | sed -e 's/[^0-9|.]//g');
echo "version=$([ "$is_stable_branch" = "true" ] && echo "$stable_version" || echo "$BUILD")" >> $GITHUB_OUTPUT
echo "extra_version_identifier=${environment}" >> $GITHUB_OUTPUT
fi
echo "build=$BUILD" >> $GITHUB_OUTPUT
cat $GITHUB_OUTPUT
validate_tests:
runs-on: ubuntu-24.04
needs: [ prepare_strategy ]
if: ${{
((github.event_name == 'workflow_dispatch' && inputs.wait_for_tests) ||
(github.event_name == 'push' && needs.prepare_strategy.outputs.is_stable_branch == 'true') ||
contains(github.event_name, 'pull_request') && (github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
}}
steps:
- uses: actions/checkout@v4
- name: Wait for Tests
uses: ./.github/workflows/wait-for-action # Path to where you place the action
with:
workflow: selfdrive_tests.yaml # The workflow file to monitor
github-token: ${{ secrets.GITHUB_TOKEN }}
should-wait-for-start: ${{ github.event_name == 'push' && 'true' || 'false' }}
build:
needs: [ validate_tests, prepare_strategy ]
concurrency:
group: build-${{ github.head_ref || github.ref_name }}
cancel-in-progress: false
runs-on: [self-hosted, tici]
outputs:
new_branch: ${{ needs.prepare_strategy.outputs.new_branch }}
version: ${{ needs.prepare_strategy.outputs.version }}
extra_version_identifier: ${{ needs.prepare_strategy.outputs.extra_version_identifier }}
commit_sha: ${{ github.sha }}
if: ${{
(always() && !cancelled() && !failure()) &&
needs.prepare_strategy.result == 'success' &&
(needs.validate_tests.result == 'success' || needs.validate_tests.result == 'skipped') &&
(!contains(github.event_name, 'pull_request') ||
(github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
}}
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
ref: ${{ env.SOURCE_BRANCH }}
repository: ${{ github.event.pull_request.head.repo.fork && github.event.pull_request.head.repo.full_name || github.repository }}
- run: git lfs pull
- name: Cache SCons
uses: actions/cache@v4
with:
path: ${{env.SCONS_CACHE_DIR}}
key: scons-${{ runner.os }}-${{ runner.arch }}-${{ env.SOURCE_BRANCH }}-${{ github.sha }}
# Note: GitHub Actions enforces cache isolation between different build sources (PR builds, workflow dispatches, etc.)
# for security. Only caches from the default branch are shared across all builds. This is by design and cannot be overridden.
restore-keys: |
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.SOURCE_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.STAGING_SOURCE_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}
- name: Set environment variables
id: set-env
run: |
echo "new_branch=${{ needs.prepare_strategy.outputs.new_branch }}" >> $GITHUB_OUTPUT
echo "version=${{ needs.prepare_strategy.outputs.version }}" >> $GITHUB_OUTPUT
echo "extra_version_identifier=${{ needs.prepare_strategy.outputs.extra_version_identifier }}" >> $GITHUB_OUTPUT
echo "commit_sha=${{ github.sha }}" >> $GITHUB_OUTPUT
# Set up common environment
source /etc/profile;
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
printenv >> $GITHUB_ENV
if [[ "${{ runner.debug }}" == "1" ]]; then
cat $GITHUB_OUTPUT
fi
- name: Setup build environment
run: |
mkdir -p "${BUILD_DIR}/"
sudo find $BUILD_DIR/ -mindepth 1 -delete
echo "Starting build stage..."
echo "BUILD_DIR: ${BUILD_DIR}"
echo "CI_DIR: ${CI_DIR}"
echo "VERSION: ${{ steps.set-env.outputs.version }}"
echo "UV_PROJECT_ENVIRONMENT: ${UV_PROJECT_ENVIRONMENT}"
echo "VIRTUAL_ENV: ${VIRTUAL_ENV}"
echo "-------"
if [[ "${{ runner.debug }}" == "1" ]]; then
printenv
fi
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
- name: Build Main Project
run: |
export PYTHONPATH="$BUILD_DIR"
./release/release_files.py | sort | uniq | rsync -rRl${RUNNER_DEBUG:+v} --files-from=- . $BUILD_DIR/
cd $BUILD_DIR
sed -i '/from .board.jungle import PandaJungle, PandaJungleDFU/s/^/#/' panda/__init__.py
scons -j$(nproc) cache_dir=${{env.SCONS_CACHE_DIR}} --minimal
touch ${BUILD_DIR}/prebuilt
if [[ "${{ runner.debug }}" == "1" ]]; then
ls -la ${BUILD_DIR}
fi
- name: Prepare Output
run: |
sudo rm -rf ${OUTPUT_DIR}
mkdir -p ${OUTPUT_DIR}
rsync -am${RUNNER_DEBUG:+v} \
--include='**/panda/board/' \
--include='**/panda/board/obj' \
--include='**/panda/board/obj/panda.bin.signed' \
--include='**/panda/board/obj/panda_h7.bin.signed' \
--include='**/panda/board/obj/bootstub.panda.bin' \
--include='**/panda/board/obj/bootstub.panda_h7.bin' \
--exclude='.sconsign.dblite' \
--exclude='*.a' \
--exclude='*.o' \
--exclude='*.os' \
--exclude='*.pyc' \
--exclude='moc_*' \
--exclude='*.cc' \
--exclude='Jenkinsfile' \
--exclude='supercombo.onnx' \
--exclude='**/panda/board/*' \
--exclude='**/panda/board/obj/**' \
--exclude='**/panda/certs/' \
--exclude='**/panda/crypto/' \
--exclude='**/release/' \
--exclude='**/.github/' \
--exclude='**/selfdrive/ui/replay/' \
--exclude='**/__pycache__/' \
--exclude='**/selfdrive/ui/*.h' \
--exclude='**/selfdrive/ui/**/*.h' \
--exclude='**/selfdrive/ui/qt/offroad/sunnypilot/' \
--exclude='${{env.SCONS_CACHE_DIR}}' \
--exclude='**/.git/' \
--exclude='**/SConstruct' \
--exclude='**/SConscript' \
--exclude='**/.venv/' \
--delete-excluded \
--chown=comma:comma \
${BUILD_DIR}/ ${OUTPUT_DIR}/
- name: 'Tar.gz files'
run: |
tar czf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }} .
ls -la prebuilt.tar.gz
- name: 'Upload Artifact'
uses: actions/upload-artifact@v4
with:
name: prebuilt
path: prebuilt.tar.gz
- name: Re-enable powersave
if: always()
run: |
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
publish:
concurrency:
# We do a bit of a hack here to avoid canceling the publishing job if a new commit comes in while we're publishing by adding the sha to the group name.
# This means that if multiple commits come in while we're publishing, they will be queued up and publish one after the other.
# Otherwise, if a job is waiting to be published due to environment wait time, it would be canceled by a new commit and restart the wait time.
group: ${{ needs.prepare_strategy.outputs.publish_concurrency_group }}
cancel-in-progress: ${{ needs.prepare_strategy.outputs.cancel_publish_in_progress == 'true' }}
if: ${{ (always() && !cancelled() && !failure()) && needs.build.result == 'success' && needs.prepare_strategy.result == 'success' && (!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt')) }}
needs: [ build, prepare_strategy ]
runs-on: ubuntu-24.04
environment: ${{ needs.prepare_strategy.outputs.environment }}
steps:
- uses: actions/checkout@v4
- name: Download build artifacts
uses: actions/download-artifact@v4
with:
name: prebuilt
- name: Untar prebuilt
run: |
mkdir -p ${{ env.OUTPUT_DIR }}
tar xzf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }}
- name: Configure Git
run: |
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"
- name: Publish to Public Repository
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
echo '${{ toJSON(needs.build.outputs) }}'
ls -la ${{ env.OUTPUT_DIR }}
${{ env.CI_DIR }}/publish.sh \
"${{ github.workspace }}" \
"${{ env.OUTPUT_DIR }}" \
"${{ needs.build.outputs.new_branch }}" \
"${{ needs.build.outputs.version }}" \
"https://x-access-token:${{github.token}}@github.com/sunnypilot/sunnypilot.git" \
"${{ needs.build.outputs.extra_version_identifier }}"
echo ""
echo "---- ️ To update the list of branches that auto deploy prebuilts -----"
echo ""
echo "1. Go to: ${{ github.server_url }}/${{ github.repository }}/settings/variables/actions/AUTO_DEPLOY_PREBUILT_BRANCHES"
echo "2. Current value: ${{ vars.AUTO_DEPLOY_PREBUILT_BRANCHES }}"
echo "3. Update as needed (JSON array with no spaces)"
- name: Tag ${{ needs.prepare_strategy.outputs.environment }}
if: ${{ needs.prepare_strategy.outputs.is_stable_branch == 'true' && (github.event_name != 'push' || !startsWith(github.ref, 'refs/tags/')) }}
run: |
TAG="${{ needs.prepare_strategy.outputs.environment }}/${{ needs.prepare_strategy.outputs.version }}/${{ needs.prepare_strategy.outputs.build }}"
git tag -f -a ${TAG} -m "${{ needs.prepare_strategy.outputs.environment }} @ ${{ needs.prepare_strategy.outputs.version }} of build ${{ needs.build.outputs.build }}."
git push -f origin ${TAG}
notify:
needs:
- prepare_strategy
- build
- publish
runs-on: ubuntu-24.04
if: ${{ (always() && !cancelled() && !failure())
&& needs.publish.result == 'success'
&& (!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
&& (fromJSON(vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES_V2)[github.head_ref || github.ref_name] != null) }}
steps:
- uses: actions/checkout@v4
- name: Prepare notification message
id: message
run: |
TEMPLATE='${{ vars.DISCOURSE_GENERAL_UPDATE_NOTICE }}'
export VERSION="${{ needs.prepare_strategy.outputs.version }}"
export branch_name="${{ env.SOURCE_BRANCH }}"
export new_branch="${{ needs.prepare_strategy.outputs.new_branch }}"
export commit_sha="${{ github.sha }}"
export commit_short_sha="${{ github.sha }}"
export commit_short_sha="${commit_short_sha:0:7}"
export extra_version_identifier="${{ needs.prepare_strategy.outputs.extra_version_identifier || github.run_number }}"
export PUBLIC_REPO_URL="${{ env.PUBLIC_REPO_URL }}"
MESSAGE=$(cat << 'EOF' | envsubst
${{ vars.DISCOURSE_GENERAL_UPDATE_NOTICE }}
EOF
)
{
echo 'content<<EOFMARKER'
echo "$MESSAGE"
echo 'EOFMARKER'
} >> $GITHUB_OUTPUT
shell: bash
- name: Post to Discourse
uses: ./.github/workflows/post-to-discourse
with:
discourse-url: ${{ vars.DISCOURSE_URL }}
api-key: ${{ secrets.DISCOURSE_API_KEY }}
api-username: "system"
topic-id: ${{ fromJSON(vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES_V2)[github.head_ref || github.ref_name].topic_id }}
message: ${{ steps.message.outputs.content }}
manage-pr-labels:
name: Remove prebuilt label
runs-on: ubuntu-latest
if: (always() && contains(github.event_name, 'pull_request') && (github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
env:
LABEL: prebuilt
steps:
- name: Remove trust-fork-pr label if present
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const prNumber = context.payload.pull_request.number;
await github.rest.issues.removeLabel({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNumber,
name: process.env.LABEL
});
console.log(`Removed '${process.env.LABEL}' label from PR #${prNumber}`);
@@ -1,222 +0,0 @@
name: Build dev
env:
DEFAULT_SOURCE_BRANCH: "master"
DEFAULT_TARGET_BRANCH: "master-dev"
LFS_URL: 'https://gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git/info/lfs'
LFS_PUSH_URL: 'ssh://git@gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git'
on:
push:
branches:
- master
pull_request_target:
types: [ labeled ]
branches:
- 'master'
workflow_dispatch:
inputs:
source_branch:
description: 'Source branch to reset from'
required: true
default: 'master'
type: string
target_branch:
description: 'Target branch to reset and squash into'
required: true
default: 'master-dev'
type: string
cancel_in_progress:
description: 'Cancel any in-progress runs of this workflow'
required: false
default: true
type: boolean
concurrency:
group: ${{ github.workflow }}
cancel-in-progress: ${{ inputs.cancel_in_progress || github.event_name == 'push' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}
jobs:
reset-and-squash:
runs-on: ubuntu-latest
if: (
(github.event_name == 'workflow_dispatch')
|| (github.event_name == 'push' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch))
|| (contains(github.event_name, 'pull_request') && ((github.event.action == 'labeled' && (github.event.label.name == vars.PREBUILT_PR_LABEL || github.event.label.name == 'trust-fork-pr') && contains(github.event.pull_request.labels.*.name, vars.PREBUILT_PR_LABEL))))
)
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Fetch all history for all branches
token: ${{ secrets.GITHUB_TOKEN }}
- name: Wait for Tests
uses: ./.github/workflows/wait-for-action # Path to where you place the action
if: (
(github.event_name == 'push' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch))
|| (contains(github.event_name, 'pull_request') && ((github.event.action == 'labeled' && (github.event.label.name == vars.PREBUILT_PR_LABEL || github.event.label.name == 'trust-fork-pr') && contains(github.event.pull_request.labels.*.name, vars.PREBUILT_PR_LABEL))))
)
with:
workflow: selfdrive_tests.yaml # The workflow file to monitor
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Configure Git
run: |
git config --global user.name 'github-actions[bot]'
git config --global user.email 'github-actions[bot]@users.noreply.github.com'
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
- name: Add GitLab public keys
run: |
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install PyGithub
- name: Check branches exist
run: |
# Check if source branch exists
if ! git ls-remote --heads origin ${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }} | grep -q "${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}"; then
echo "Source branch ${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }} does not exist!"
exit 1
fi
# Make sure we have the latest source branch
git fetch origin ${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}
# Check if target branch exists
if ! git ls-remote --heads origin ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} | grep -q "${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}"; then
echo "Target branch ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} does not exist, creating it from ${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}"
git checkout -b ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} origin/${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}
git push origin ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}
else
# Fetch target branch if it exists
git fetch origin ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}
fi
- name: Reset target branch
run: |
echo "Resetting ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} to match ${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}"
# Delete if exists and recreate pointing to source
git branch -D ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} || true
git branch ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} origin/${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}
- name: Get PRs to squash
id: get-prs
run: |
# Use GitHub API to get PRs with specific label, ordered by creation date
PR_LIST=$(gh api graphql -f query='
query($search_query:String!) {
search(query: $search_query, type:ISSUE, first:40) {
nodes {
... on PullRequest {
number
headRefName
title
createdAt
labels(last:10) {
nodes {
name
}
}
headRepository {
name
nameWithOwner
url
isFork
}
commits(last: 1) {
nodes {
commit {
statusCheckRollup {
state
}
}
}
}
}
}
}
}' -F search_query="repo:${{ github.repository }} is:pr is:open label:${{ vars.PREBUILT_PR_LABEL }},${{ vars.PREBUILT_PR_LABEL }}-c3 draft:false sort:created-asc")
PR_LIST=${PR_LIST//\'/}
echo "PR_LIST=${PR_LIST}" >> $GITHUB_OUTPUT
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Process PRs
run: |
cp ${{ github.workspace }}/release/ci/squash_and_merge.py /tmp/squash_and_merge.py && \
chmod +x /tmp/squash_and_merge.py && \
python3 ${{ github.workspace }}/release/ci/squash_and_merge_prs.py \
--pr-data '${{ steps.get-prs.outputs.PR_LIST }}' \
--target-branch ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} \
--squash-script-path '/tmp/squash_and_merge.py'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Update LFS Config
run: |
echo '[lfs]' > .lfsconfig
echo ' url = ${{ env.LFS_URL }}' >> .lfsconfig
echo ' pushurl = ${{ env.LFS_PUSH_URL }}' >> .lfsconfig
echo ' locksverify = false' >> .lfsconfig
- name: Push changes if there are diffs
id: push-changes # Add an id so we can reference this step
run: |
TARGET_BRANCH="${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}"
# Fetch the latest from remote
git fetch origin $TARGET_BRANCH
# Check for diffs between local and remote
if git diff $TARGET_BRANCH origin/$TARGET_BRANCH --quiet; then
echo "No changes to push - local and remote branches are identical"
echo "has_changes=false" >> $GITHUB_OUTPUT
exit 0
fi
# If we get here, there are diffs, so push
if ! git push origin $TARGET_BRANCH --force; then
echo "Failed to push changes to $TARGET_BRANCH"
exit 1
fi
echo "Branch $TARGET_BRANCH has been reset and updated with squashed PRs"
echo "has_changes=true" >> $GITHUB_OUTPUT
- name: Trigger and wait for selfdrive tests
if: steps.push-changes.outputs.has_changes == 'true'
run: |
echo "Triggering selfdrive tests..."
gh workflow run selfdrive_tests.yaml --ref "${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}"
echo "Sleeping for 120s to give plenty of time for the action to start and then we wait"
sleep 120
echo "Getting latest run ID..."
RUN_ID=$(gh run list --workflow=selfdrive_tests.yaml --branch="${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}" --limit=1 --json databaseId --jq '.[0].databaseId')
echo "Watching run ID: $RUN_ID"
gh run watch "$RUN_ID"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Trigger prebuilt workflow
if: success() && steps.push-changes.outputs.has_changes == 'true'
run: |
gh workflow run sunnypilot-build-prebuilt.yaml --ref "${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-78
View File
@@ -1,78 +0,0 @@
name: Debug Discourse Posting
on:
push:
jobs:
test-discourse-post:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Post test message to Discourse
uses: ./.github/workflows/post-to-discourse
with:
discourse-url: ${{ vars.DISCOURSE_URL }}
api-key: ${{ secrets.DISCOURSE_API_KEY }}
api-username: ${{ secrets.DISCOURSE_API_USERNAME }}
topic-id: ${{ vars.DISCOURSE_UPDATES_TOPIC_ID }}
message: |
## 🧪 Test Post from GitHub Actions
**This is a test post to verify Discourse integration**
- **Workflow**: ${{ github.workflow }}
- **Run Number**: #${{ github.run_number }}
- **Branch**: `${{ github.ref_name }}`
- **Commit**: ${{ github.sha }}
- **Actor**: @${{ github.actor }}
- **Timestamp**: ${{ github.event.head_commit.timestamp }}
---
### Fake Build Info (for testing)
- **Version**: 0.9.8-test
- **Build**: #42
- **Branch**: release-test
[View workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})
*This is an automated test message. Drive safe! 🚗💨*
- name: Create topic on Discourse
uses: ./.github/workflows/post-to-discourse
with:
discourse-url: ${{ vars.DISCOURSE_URL }}
api-key: ${{ secrets.DISCOURSE_API_KEY }}
api-username: ${{ secrets.DISCOURSE_API_USERNAME }}
#topic-id: ${{ vars.DISCOURSE_UPDATES_TOPIC_ID }}
category-id: 4
title: "This is a test of a new topic instead of a reply"
message: |
## 🧪 Test Post from GitHub Actions
**This is a test post to verify Discourse integration**
- **Workflow**: ${{ github.workflow }}
- **Run Number**: #${{ github.run_number }}
- **Branch**: `${{ github.ref_name }}`
- **Commit**: ${{ github.sha }}
- **Actor**: @${{ github.actor }}
- **Timestamp**: ${{ github.event.head_commit.timestamp }}
---
### Fake Build Info (for testing)
- **Version**: 0.9.8-test
- **Build**: #42
- **Branch**: release-test
[View workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})
*This is an automated test message. Drive safe! 🚗💨*
- name: Display results
if: always()
run: |
echo "::notice::Discourse post test completed"
echo "Check your Discourse topic to verify the post appeared correctly"
-173
View File
@@ -1,173 +0,0 @@
name: "ui preview"
on:
push:
branches:
- master
pull_request_target:
types: [assigned, opened, synchronize, reopened, edited]
branches:
- 'master'
paths:
- 'selfdrive/ui/**'
workflow_dispatch:
env:
UI_JOB_NAME: "Create UI Report"
REPORT_NAME: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && 'master' || github.event.number }}
SHA: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && github.sha || github.event.pull_request.head.sha }}
BRANCH_NAME: "openpilot/pr-${{ github.event.number }}"
jobs:
preview:
if: github.repository == 'sunnypilot/sunnypilot'
name: preview
runs-on: ubuntu-latest
timeout-minutes: 20
permissions:
contents: read
pull-requests: write
actions: read
steps:
- name: Waiting for ui generation to start
run: sleep 30
- name: Waiting for ui generation to end
uses: lewagon/wait-on-check-action@v1.3.4
with:
ref: ${{ env.SHA }}
check-name: ${{ env.UI_JOB_NAME }}
repo-token: ${{ secrets.GITHUB_TOKEN }}
allowed-conclusions: success
wait-interval: 20
- name: Getting workflow run ID
id: get_run_id
run: |
echo "run_id=$(curl https://api.github.com/repos/${{ github.repository }}/commits/${{ env.SHA }}/check-runs | jq -r '.check_runs[] | select(.name == "${{ env.UI_JOB_NAME }}") | .html_url | capture("(?<number>[0-9]+)") | .number')" >> $GITHUB_OUTPUT
- name: Getting proposed ui
id: download-artifact
uses: dawidd6/action-download-artifact@v6
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
run_id: ${{ steps.get_run_id.outputs.run_id }}
search_artifacts: true
name: report-1-${{ env.REPORT_NAME }}
path: ${{ github.workspace }}/pr_ui
- name: Getting master ui
uses: actions/checkout@v4
with:
repository: sunnypilot/ci-artifacts
ssh-key: ${{ secrets.CI_ARTIFACTS_DEPLOY_KEY }}
path: ${{ github.workspace }}/master_ui
ref: openpilot_master_ui
- name: Saving new master ui
if: github.ref == 'refs/heads/master' && github.event_name == 'push'
working-directory: ${{ github.workspace }}/master_ui
run: |
git checkout --orphan=new_master_ui
git rm -rf *
git branch -D openpilot_master_ui
git branch -m openpilot_master_ui
git config user.name "GitHub Actions Bot"
git config user.email "<>"
mv ${{ github.workspace }}/pr_ui/*.png .
git add .
git commit -m "screenshots for commit ${{ env.SHA }}"
git push origin openpilot_master_ui --force
- name: Finding diff
if: github.event_name == 'pull_request_target'
id: find_diff
run: >-
sudo apt-get update && sudo apt-get install -y imagemagick
scenes=$(find ${{ github.workspace }}/pr_ui/*.png -type f -printf "%f\n" | cut -d '.' -f 1 | grep -v 'pair_device')
A=($scenes)
DIFF=""
TABLE="<details><summary>All Screenshots</summary>"
TABLE="${TABLE}<table>"
for ((i=0; i<${#A[*]}; i=i+1));
do
# Check if the master file exists
if [ ! -f "${{ github.workspace }}/master_ui/${A[$i]}.png" ]; then
# This is a new file in PR UI that doesn't exist in master
DIFF="${DIFF}<details open>"
DIFF="${DIFF}<summary>${A[$i]} : \$\${\\color{cyan}\\text{NEW}}\$\$</summary>"
DIFF="${DIFF}<table>"
DIFF="${DIFF}<tr>"
DIFF="${DIFF} <td> <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}.png\"> </td>"
DIFF="${DIFF}</tr>"
DIFF="${DIFF}</table>"
DIFF="${DIFF}</details>"
elif ! compare -fuzz 2% -highlight-color DeepSkyBlue1 -lowlight-color Black -compose Src ${{ github.workspace }}/master_ui/${A[$i]}.png ${{ github.workspace }}/pr_ui/${A[$i]}.png ${{ github.workspace }}/pr_ui/${A[$i]}_diff.png; then
convert ${{ github.workspace }}/pr_ui/${A[$i]}_diff.png -transparent black mask.png
composite mask.png ${{ github.workspace }}/master_ui/${A[$i]}.png composite_diff.png
convert -delay 100 ${{ github.workspace }}/master_ui/${A[$i]}.png composite_diff.png -loop 0 ${{ github.workspace }}/pr_ui/${A[$i]}_diff.gif
mv ${{ github.workspace }}/master_ui/${A[$i]}.png ${{ github.workspace }}/pr_ui/${A[$i]}_master_ref.png
DIFF="${DIFF}<details open>"
DIFF="${DIFF}<summary>${A[$i]} : \$\${\\color{red}\\text{DIFFERENT}}\$\$</summary>"
DIFF="${DIFF}<table>"
DIFF="${DIFF}<tr>"
DIFF="${DIFF} <td> master <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}_master_ref.png\"> </td>"
DIFF="${DIFF} <td> proposed <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}.png\"> </td>"
DIFF="${DIFF}</tr>"
DIFF="${DIFF}<tr>"
DIFF="${DIFF} <td> diff <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}_diff.png\"> </td>"
DIFF="${DIFF} <td> composite diff <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}_diff.gif\"> </td>"
DIFF="${DIFF}</tr>"
DIFF="${DIFF}</table>"
DIFF="${DIFF}</details>"
else
rm -f ${{ github.workspace }}/pr_ui/${A[$i]}_diff.png
fi
INDEX=$(($i % 2))
if [[ $INDEX -eq 0 ]]; then
TABLE="${TABLE}<tr>"
fi
TABLE="${TABLE} <td> <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}.png\"> </td>"
if [[ $INDEX -eq 1 || $(($i + 1)) -eq ${#A[*]} ]]; then
TABLE="${TABLE}</tr>"
fi
done
TABLE="${TABLE}</table></details>"
echo "DIFF=$DIFF$TABLE" >> "$GITHUB_OUTPUT"
- name: Saving proposed ui
if: github.event_name == 'pull_request_target'
working-directory: ${{ github.workspace }}/master_ui
run: |
git config user.name "GitHub Actions Bot"
git config user.email "<>"
git checkout --orphan=${{ env.BRANCH_NAME }}
git rm -rf *
mv ${{ github.workspace }}/pr_ui/* .
git add .
git commit -m "screenshots for PR #${{ github.event.number }}"
git push origin ${{ env.BRANCH_NAME }} --force
- name: Comment Screenshots on PR
if: github.event_name == 'pull_request_target'
uses: thollander/actions-comment-pull-request@v2
with:
message: |
<!-- _(run_id_screenshots **${{ github.run_id }}**)_ -->
## UI Preview
${{ steps.find_diff.outputs.DIFF }}
comment_tag: run_id_screenshots
pr_number: ${{ github.event.number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -1,52 +0,0 @@
name: 'Wait for Tests'
description: 'Action to wait for workflow tests to start and complete'
inputs:
workflow:
description: 'The workflow file name to monitor'
required: true
default: 'selfdrive_tests.yaml'
branch:
description: 'The branch to monitor (defaults to current branch)'
required: false
default: ''
github-token:
description: 'GitHub token for API access'
required: true
wait-time:
description: 'Initial sleep time in seconds before monitoring starts'
required: false
default: '30'
should-wait-for-start:
description: 'Whether to wait for tests to start'
required: false
default: false
runs:
using: 'composite'
steps:
- name: Wait for tests to start
if: inputs.should-wait-for-start == 'true'
shell: bash
run: |
echo "Sleeping for ${{ inputs.wait-time }} seconds to give some time for the action to start and then we'll wait"
sleep ${{ inputs.wait-time }}
- name: Wait for tests to finish
shell: bash
run: |
BRANCH="${{ inputs.branch || github.head_ref || github.ref_name }}"
echo "Looking for workflow runs of ${{ inputs.workflow }} on branch $BRANCH"
RUN_ID=$(gh run list --workflow=${{ inputs.workflow }} --branch="$BRANCH" --limit=1 --json databaseId --jq '.[0].databaseId')
echo "Watching run ID: $RUN_ID"
gh run watch "$RUN_ID"
CONCLUSION=$(gh run view "$RUN_ID" --json conclusion --jq '.conclusion')
echo "Run concluded with: $CONCLUSION"
if [[ "$CONCLUSION" != "success" ]]; then
echo "❌ Workflow run failed with conclusion: $CONCLUSION"
exit 1
fi
env:
GITHUB_TOKEN: ${{ inputs.github-token }}
+8 -9
View File
@@ -10,7 +10,6 @@ venv/
.overlay_init
.overlay_consistent
.sconsign.dblite
model2.png
a.out
.hypothesis
.cache/
@@ -37,29 +36,23 @@ a.out
*.class
*.pyxbldc
*.vcd
*.qm
*.mo
*_pyx.cpp
*.stats
config.json
clcache
compile_commands.json
compare_runtime*.html
persist
selfdrive/pandad/pandad
cereal/services.h
cereal/gen
cereal/messaging/bridge
selfdrive/mapd/default_speeds_by_region.json
selfdrive/ui/translations/tmp
selfdrive/test/longitudinal_maneuvers/out
selfdrive/car/tests/cars_dump
system/camerad/camerad
system/camerad/test/ae_gray_test
notebooks
hyperthneed
provisioning
.coverage*
coverage.xml
htmlcov
@@ -76,6 +69,7 @@ sunnypilot/modeld*/thneed/compile
sunnypilot/modeld*/models/*.thneed
sunnypilot/modeld*/models/*.pkl
# openpilot log files
*.bz2
*.zst
@@ -105,6 +99,11 @@ Pipfile
.history
.ionide
.claude/
.context/
PLAN.md
TASK.md
### JetBrains ###
!.idea/customTargets.xml
!.idea/tools/*
-21
View File
@@ -1,21 +0,0 @@
[submodule "panda"]
path = panda
url = https://github.com/sunnyhaibin/panda.git
[submodule "opendbc"]
path = opendbc_repo
url = https://github.com/sunnypilot/opendbc.git
[submodule "msgq"]
path = msgq_repo
url = https://github.com/sunnypilot/msgq.git
[submodule "rednose_repo"]
path = rednose_repo
url = https://github.com/commaai/rednose.git
[submodule "teleoprtc_repo"]
path = teleoprtc_repo
url = https://github.com/commaai/teleoprtc
[submodule "tinygrad"]
path = tinygrad_repo
url = https://github.com/tinygrad/tinygrad.git
[submodule "sunnypilot/neural_network_data"]
path = sunnypilot/neural_network_data
url = https://github.com/sunnypilot/neural-network-data.git
-25
View File
@@ -1,25 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="CLionExternalBuildManager">
<target id="a62f99e8-5ec4-434c-8122-49efed5af108" name="uv Scons Build Debug" defaultType="TOOL">
<configuration id="b93ec964-16e5-4962-a12e-3ed360ce8f02" name="uv Scons Build Debug">
<build type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Build Debug" />
</build>
<clean type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Clean" />
</clean>
</configuration>
</target>
<target id="edd8ad9d-183b-467c-a355-0d9a0ecab026" name="uv Scons Build Release" defaultType="TOOL">
<configuration id="09523339-5ce3-4223-ab9e-904f38ad7752" name="uv Scons Build Release">
<build type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Build Release" />
</build>
<clean type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Clean" />
</clean>
</configuration>
</target>
</component>
</project>
-23
View File
@@ -1,23 +0,0 @@
<toolSet name="External Tools">
<tool name="uv Scons Build Debug" showInMainMenu="false" showInEditor="false" showInProject="false" showInSearchPopup="false" disabled="false" useConsole="true" showConsoleOnStdOut="false" showConsoleOnStdErr="false" synchronizeAfterRun="true">
<exec>
<option name="COMMAND" value="bash" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -u -j$(nproc) --ccflags=\&quot;-fno-inline\&quot;&quot;" />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
<tool name="uv Scons Clean" showInMainMenu="false" showInEditor="false" showInProject="false" showInSearchPopup="false" disabled="false" useConsole="true" showConsoleOnStdOut="false" showConsoleOnStdErr="false" synchronizeAfterRun="true">
<exec>
<option name="COMMAND" value="bash" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -c&quot; " />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
<tool name="uv Scons Build Release" showInMainMenu="false" showInEditor="false" showInProject="false" showInSearchPopup="false" disabled="false" useConsole="true" showConsoleOnStdOut="false" showConsoleOnStdErr="false" synchronizeAfterRun="true">
<exec>
<option name="COMMAND" value="bash" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -u -j$(nproc)&quot; " />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
</toolSet>
-4
View File
@@ -1,4 +0,0 @@
[lfs]
url = https://gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git/info/lfs
pushurl = ssh://git@gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git
locksverify = false
-4
View File
@@ -1,4 +0,0 @@
[lfs]
url = https://gitlab.com/commaai/openpilot-lfs.git/info/lfs
pushurl = ssh://git@gitlab.com/commaai/openpilot-lfs.git
locksverify = false
-10
View File
@@ -1,10 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Build Debug" type="CLionExternalRunConfiguration" factoryName="Application" REDIRECT_INPUT="false" ELEVATE="false" USE_EXTERNAL_CONSOLE="false" EMULATE_TERMINAL="false" WORKING_DIR="file://$ProjectFileDir$/selfdrive/ui" PASS_PARENT_ENVS_2="true" PROJECT_NAME="sunnypilot" TARGET_NAME="uv Scons Build Debug" CONFIG_NAME="uv Scons Build Debug" RUN_PATH="ui">
<envs>
<env name="QT_DBL_CLICK_DIST" value="150" />
</envs>
<method v="2">
<option name="CLION.EXTERNAL.BUILD" enabled="true" />
</method>
</configuration>
</component>
-10
View File
@@ -1,10 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Build Release" type="CLionExternalRunConfiguration" factoryName="Application" REDIRECT_INPUT="false" ELEVATE="false" USE_EXTERNAL_CONSOLE="false" EMULATE_TERMINAL="false" WORKING_DIR="file://$ProjectFileDir$/selfdrive/ui" PASS_PARENT_ENVS_2="true" PROJECT_NAME="sunnypilot" TARGET_NAME="uv Scons Build Release" CONFIG_NAME="uv Scons Build Release" RUN_PATH="ui">
<envs>
<env name="QT_DBL_CLICK_DIST" value="150" />
</envs>
<method v="2">
<option name="CLION.EXTERNAL.BUILD" enabled="true" />
</method>
</configuration>
</component>
+100 -1
View File
@@ -1,5 +1,104 @@
sunnypilot Version 2025.003.000 (20xx-xx-xx)
sunnypilot Version 2026.001.000 (2026-03-xx)
========================
* What's Changed (sunnypilot/sunnypilot)
* Complete rewrite of the user interface from Qt C++ to Raylib Python
* comma four support
* ui: sunnypilot toggle style by @nayan8teen
* ui: fix scroll panel mouse wheel behavior by @nayan8teen
* ui: sunnypilot panels by @nayan8teen
* sunnylink: centralize key pair handling in sunnylink registration by @devtekve
* ui: reimplement sunnypilot branding with Raylib by @sunnyhaibin
* ui: Platform Selector by @Discountchubbs
* ui: vehicle brand settings by @Discountchubbs
* ui: sunnylink client-side implementation by @nayan8teen
* ui: `NetworkUISP` by @Discountchubbs
* ui: add sunnypilot font by @nayan8teen
* ui: sunnypilot sponsor tier color mapping by @sunnyhaibin
* ui: sunnylink panel by @nayan8teen
* ui: Models panel by @Discountchubbs
* ui: software panel by @Discountchubbs
* modeld_v2: support planplus outputs by @Discountchubbs
* ui: OSM panel by @Discountchubbs
* ui: Developer panel extension by @Discountchubbs
* sunnylink: Vehicle Selector support by @sunnyhaibin
* [TIZI/TICI] ui: Developer Metrics by @rav4kumar
* [comma 4] ui: sunnylink panel by @nayan8teen
* ui: lateral-only and longitudinal-only UI statuses support by @royjr
* sunnylink: elliptic curve keys support and improve key path handling by @nayan8teen
* sunnylink: block remote modification of SSH key parameters by @zikeji
* [TIZI/TICI] ui: rainbow path by @rav4kumar
* [TIZI/TICI] ui: chevron metrics by @rav4kumar
* ui: include MADS enabled state to `engaged` check by @sunnyhaibin
* Toyota: Enforce Factory Longitudinal Control by @sunnyhaibin
* ui: fix malformed dongle ID display on the PC if dongleID is not set by @dzid26
* SL: Re enable and validate ingestion of swaglogs by @devtekve
* modeld_v2: planplus model tuning by @Discountchubbs
* ui: fix Always Offroad button visibility by @nayan8teen
* Reimplement sunnypilot Terms of Service & sunnylink Consent Screens by @sunnyhaibin
* [TIZI/TICI] ui: update dmoji position and Developer UI adjustments by @rav4kumar
* modeld: configurable camera offset by @Discountchubbs
* [TIZI/TICI] ui: sunnylink status on sidebar by @Copilot
* ui: Global Brightness Override by @nayan8teen
* ui: Customizable Interactive Timeout by @sunnyhaibin
* sunnylink: add units to param metadata by @nayan8teen
* ui: Customizable Onroad Brightness by @sunnyhaibin
* [TIZI/TICI] ui: Steering panel by @nayan8teen
* [TIZI/TICI] ui: Rocket Fuel by @rav4kumar
* [TIZI/TICI] ui: MICI style turn signals by @rav4kumar
* [TIZI/TICI] ui: MICI style blindspot indicators by @sunnyhaibin
* [MICI] ui: display blindspot indicators when available by @rav4kumar
* [TIZI/TICI] ui: Road Name by @rav4kumar
* [TIZI/TICI] ui: Blue "Exit Always Offroad" button by @dzid26
* [TIZI/TICI] ui: Speed Limit by @rav4kumar
* Reapply "latcontrol_torque: lower kp and lower friction threshold (commaai/openpilot#36619)" by @sunnyhaibin
* [TIZI/TICI] ui: steering arc by @royjr
* [TIZI/TICI] ui: Smart Cruise Control elements by @sunnyhaibin
* [TIZI/TICI] ui: Green Light and Lead Departure elements by @sunnyhaibin
* [TIZI/TICI] ui: standstill timer by @sunnyhaibin
* [MICI] ui: driving models selector by @Discountchubbs
* [TIZI/TICI] ui: Hide vEgo and True vEgo by @sunnyhaibin
* [TIZI/TICI] ui: Visuals panel by @nayan8teen
* Device: Retain QuickBoot state after op switch by @nayan8teen
* [TIZI/TICI] ui: Trips panel by @sunnyhaibin
* [TIZI/TICI] ui: dynamic ICBM status by @sunnyhaibin
* [TIZI/TICI] ui: Cruise panel by @sunnyhaibin
* ui: better wake mode support by @nayan8teen
* Pause Lateral Control with Blinker: Post-Blinker Delay by @CHaucke89
* SCC-V: Use p97 for predicted lateral accel by @yasu-oh
* Controls: Support for Torque Lateral Control v0 Tune by @sunnyhaibin
* What's Changed (sunnypilot/opendbc)
* Honda: DBC for Accord 9th Generation by @mvl-boston
* FCA: update tire stiffness values for `RAM_HD` by @dparring
* Honda: Nidec hybrid baseline brake support by @mvl-boston
* Subaru Global Gen2: bump steering limits and update tuning by @sunnyhaibin
* Toyota: Enforce Stock Longitudinal Control by @rav4kumar
* Nissan: use MADS enabled status for LKAS HUD logic by @downquark7
* Reapply "Lateral: lower friction threshold (#2915)" (#378) by @sunnyhaibin
* HKG: add KIA_FORTE_2019_NON_SCC fingerprint by @royjr
* Nissan: Parse cruise control buttons by @downquark7
* Rivian: Add stalk down ACC behavior to match stock Rivian by @lukasloetkolben
* Tesla: remove `TESLA_MODEL_X` from `dashcamOnly` by @ssysm
* Hyundai Longitudinal: refactor tuning by @Discountchubbs
* Tesla: add fingerprint for Model 3 Performance HW4 by @sunnyhaibin
* Toyota: do not disable radar when smartDSU or CAN Filter detected by @sunnyhaibin
* Honda: add missing `GasInterceptor` messages to Taiwan Odyssey DBC by @mvl-boston
* GM: remove `CHEVROLET_EQUINOX_NON_ACC_3RD_GEN` from `dashcamOnly` by @sunnyhaibin
* GM: remove `CHEVROLET_BOLT_NON_ACC_2ND_GEN` from `dashcamOnly` by @sunnyhaibin
* New Contributors (sunnypilot/sunnypilot)
* @TheSecurityDev made their first contribution in "ui: fix sidebar scroll in UI screenshots"
* @zikeji made their first contribution in "sunnylink: block remote modification of SSH key parameters"
* @Candy0707 made their first contribution in "[TIZI/TICI] ui: Fix misaligned turn signals and blindspot indicators with sidebar"
* @CHaucke89 made their first contribution in "Pause Lateral Control with Blinker: Post-Blinker Delay"
* @yasu-oh made their first contribution in "SCC-V: Use p97 for predicted lateral accel"
* New Contributors (sunnypilot/opendbc)
* @AmyJeanes made their first contribution in "Tesla: Fix stock LKAS being blocked when MADS is enabled"
* @mvl-boston made their first contribution in "Honda: Update Clarity brake to renamed DBC message name"
* @dzid26 made their first contribution in "Tesla: Parse speed limit from CAN"
* @firestar5683 made their first contribution in "GM: Non-ACC platforms with steering only support"
* @downquark7 made their first contribution in "Nissan: use MADS enabled status for LKAS HUD logic"
* @royjr made their first contribution in "HKG: add KIA_FORTE_2019_NON_SCC fingerprint"
* @ssysm made their first contribution in "Tesla: remove `TESLA_MODEL_X` from `dashcamOnly`"
* Full Changelog: https://github.com/sunnypilot/sunnypilot/compare/v2025.002.000...v2026.001.000
sunnypilot Version 2025.002.000 (2025-11-06)
========================
+3 -1
View File
@@ -9,4 +9,6 @@ WORKDIR ${OPENPILOT_PATH}
COPY . ${OPENPILOT_PATH}/
RUN scons --cache-readonly -j$(nproc)
ENV UV_BIN="/home/batman/.local/bin/"
ENV PATH="$UV_BIN:$PATH"
RUN UV_PROJECT_ENVIRONMENT=$VIRTUAL_ENV uv run scons --cache-readonly -j$(nproc)
+3 -1
View File
@@ -9,4 +9,6 @@ WORKDIR ${OPENPILOT_PATH}
COPY . ${OPENPILOT_PATH}/
RUN scons --cache-readonly -j$(nproc)
ENV UV_BIN="/home/batman/.local/bin/"
ENV PATH="$UV_BIN:$PATH"
RUN UV_PROJECT_ENVIRONMENT=$VIRTUAL_ENV uv run scons --cache-readonly -j$(nproc)
Vendored
-269
View File
@@ -1,269 +0,0 @@
def retryWithDelay(int maxRetries, int delay, Closure body) {
for (int i = 0; i < maxRetries; i++) {
try {
return body()
} catch (Exception e) {
sleep(delay)
}
}
throw Exception("Failed after ${maxRetries} retries")
}
def device(String ip, String step_label, String cmd) {
withCredentials([file(credentialsId: 'id_rsa', variable: 'key_file')]) {
def ssh_cmd = """
ssh -o ConnectTimeout=5 -o ServerAliveInterval=5 -o ServerAliveCountMax=2 -o BatchMode=yes -o StrictHostKeyChecking=no -i ${key_file} 'comma@${ip}' exec /usr/bin/bash <<'END'
set -e
export TERM=xterm-256color
shopt -s huponexit # kill all child processes when the shell exits
export CI=1
export PYTHONWARNINGS=error
export LOGPRINT=debug
export TEST_DIR=${env.TEST_DIR}
export SOURCE_DIR=${env.SOURCE_DIR}
export GIT_BRANCH=${env.GIT_BRANCH}
export GIT_COMMIT=${env.GIT_COMMIT}
export CI_ARTIFACTS_TOKEN=${env.CI_ARTIFACTS_TOKEN}
export GITHUB_COMMENTS_TOKEN=${env.GITHUB_COMMENTS_TOKEN}
export AZURE_TOKEN='${env.AZURE_TOKEN}'
# only use 1 thread for tici tests since most require HIL
export PYTEST_ADDOPTS="-n0 -s"
export GIT_SSH_COMMAND="ssh -i /data/gitkey"
source ~/.bash_profile
if [ -f /TICI ]; then
source /etc/profile
rm -rf /tmp/tmp*
rm -rf ~/.commacache
rm -rf /dev/shm/*
rm -rf /dev/tmp/tmp*
if ! systemctl is-active --quiet systemd-resolved; then
echo "restarting resolved"
sudo systemctl start systemd-resolved
sleep 3
fi
# restart aux USB
if [ -e /sys/bus/usb/drivers/hub/3-0:1.0 ]; then
echo "restarting aux usb"
echo "3-0:1.0" | sudo tee /sys/bus/usb/drivers/hub/unbind
sleep 0.5
echo "3-0:1.0" | sudo tee /sys/bus/usb/drivers/hub/bind
fi
fi
if [ -f /data/openpilot/launch_env.sh ]; then
source /data/openpilot/launch_env.sh
fi
ln -snf ${env.TEST_DIR} /data/pythonpath
cd ${env.TEST_DIR} || true
time ${cmd}
END"""
sh script: ssh_cmd, label: step_label
}
}
def deviceStage(String stageName, String deviceType, List extra_env, def steps) {
stage(stageName) {
if (currentBuild.result != null) {
return
}
if (isReplay()) {
error("REPLAYING TESTS IS NOT ALLOWED. FIX THEM INSTEAD.")
}
def extra = extra_env.collect { "export ${it}" }.join('\n');
def branch = env.BRANCH_NAME ?: 'master';
def gitDiff = sh returnStdout: true, script: 'curl -s -H "Authorization: Bearer ${GITHUB_COMMENTS_TOKEN}" https://api.github.com/repos/commaai/openpilot/compare/master...${GIT_BRANCH} | jq .files[].filename || echo "/"', label: 'Getting changes'
lock(resource: "", label: deviceType, inversePrecedence: true, variable: 'device_ip', quantity: 1, resourceSelectStrategy: 'random') {
docker.image('ghcr.io/commaai/alpine-ssh').inside('--user=root') {
timeout(time: 35, unit: 'MINUTES') {
retry (3) {
def date = sh(script: 'date', returnStdout: true).trim();
device(device_ip, "set time", "date -s '" + date + "'")
device(device_ip, "git checkout", extra + "\n" + readFile("selfdrive/test/setup_device_ci.sh"))
}
steps.each { item ->
def name = item[0]
def cmd = item[1]
def args = item[2]
def diffPaths = args.diffPaths ?: []
def cmdTimeout = args.timeout ?: 9999
if (branch != "master" && !branch.contains("__jenkins_loop_") && diffPaths && !hasPathChanged(gitDiff, diffPaths)) {
println "Skipping ${name}: no changes in ${diffPaths}."
return
} else {
timeout(time: cmdTimeout, unit: 'SECONDS') {
device(device_ip, name, cmd)
}
}
}
}
}
}
}
}
def hasPathChanged(String gitDiff, List<String> paths) {
for (path in paths) {
if (gitDiff.contains(path)) {
return true
}
}
return false
}
def isReplay() {
def replayClass = "org.jenkinsci.plugins.workflow.cps.replay.ReplayCause"
return currentBuild.rawBuild.getCauses().any{ cause -> cause.toString().contains(replayClass) }
}
def setupCredentials() {
withCredentials([
string(credentialsId: 'azure_token', variable: 'AZURE_TOKEN'),
]) {
env.AZURE_TOKEN = "${AZURE_TOKEN}"
}
withCredentials([
string(credentialsId: 'ci_artifacts_pat', variable: 'CI_ARTIFACTS_TOKEN'),
]) {
env.CI_ARTIFACTS_TOKEN = "${CI_ARTIFACTS_TOKEN}"
}
withCredentials([
string(credentialsId: 'post_comments_github_pat', variable: 'GITHUB_COMMENTS_TOKEN'),
]) {
env.GITHUB_COMMENTS_TOKEN = "${GITHUB_COMMENTS_TOKEN}"
}
}
def step(String name, String cmd, Map args = [:]) {
return [name, cmd, args]
}
node {
env.CI = "1"
env.PYTHONWARNINGS = "error"
env.TEST_DIR = "/data/openpilot"
env.SOURCE_DIR = "/data/openpilot_source/"
setupCredentials()
env.GIT_BRANCH = checkout(scm).GIT_BRANCH
env.GIT_COMMIT = checkout(scm).GIT_COMMIT
def excludeBranches = ['__nightly', 'devel', 'devel-staging', 'release3', 'release3-staging',
'release-tici', 'testing-closet*', 'hotfix-*']
def excludeRegex = excludeBranches.join('|').replaceAll('\\*', '.*')
if (env.BRANCH_NAME != 'master' && !env.BRANCH_NAME.contains('__jenkins_loop_')) {
properties([
disableConcurrentBuilds(abortPrevious: true)
])
}
try {
if (env.BRANCH_NAME == 'devel-staging') {
deviceStage("build release3-staging", "tizi-needs-can", [], [
step("build release3-staging", "RELEASE_BRANCH=release3-staging $SOURCE_DIR/release/build_release.sh"),
])
}
if (env.BRANCH_NAME == '__nightly') {
parallel (
'nightly': {
deviceStage("build nightly", "tizi-needs-can", [], [
step("build nightly", "RELEASE_BRANCH=nightly $SOURCE_DIR/release/build_release.sh"),
])
},
'nightly-dev': {
deviceStage("build nightly-dev", "tizi-needs-can", [], [
step("build nightly-dev", "PANDA_DEBUG_BUILD=1 RELEASE_BRANCH=nightly-dev $SOURCE_DIR/release/build_release.sh"),
])
},
)
}
if (!env.BRANCH_NAME.matches(excludeRegex)) {
parallel (
'onroad tests': {
deviceStage("onroad", "tizi-needs-can", ["UNSAFE=1"], [
step("build openpilot", "cd system/manager && ./build.py"),
step("check dirty", "release/check-dirty.sh"),
step("onroad tests", "pytest selfdrive/test/test_onroad.py -s", [timeout: 60]),
])
},
'HW + Unit Tests': {
deviceStage("tizi-hardware", "tizi-common", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test pandad", "pytest selfdrive/pandad/tests/test_pandad.py", [diffPaths: ["panda", "selfdrive/pandad/"]]),
step("test power draw", "pytest -s system/hardware/tici/tests/test_power_draw.py"),
step("test encoder", "LD_LIBRARY_PATH=/usr/local/lib pytest system/loggerd/tests/test_encoder.py", [diffPaths: ["system/loggerd/"]]),
step("test manager", "pytest system/manager/test/test_manager.py"),
])
},
'loopback': {
deviceStage("loopback", "tizi-loopback", ["UNSAFE=1"], [
step("build openpilot", "cd system/manager && ./build.py"),
step("test pandad loopback", "pytest selfdrive/pandad/tests/test_pandad_loopback.py"),
])
},
'camerad OX03C10': {
deviceStage("OX03C10", "tizi-ox03c10", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test camerad", "pytest system/camerad/test/test_camerad.py", [timeout: 60]),
step("test exposure", "pytest system/camerad/test/test_exposure.py"),
])
},
'camerad OS04C10': {
deviceStage("OS04C10", "tici-os04c10", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test camerad", "pytest system/camerad/test/test_camerad.py", [timeout: 60]),
step("test exposure", "pytest system/camerad/test/test_exposure.py"),
])
},
'sensord': {
deviceStage("LSM + MMC", "tizi-lsmc", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test sensord", "pytest system/sensord/tests/test_sensord.py"),
])
},
'replay': {
deviceStage("model-replay", "tizi-replay", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py", [diffPaths: ["selfdrive/modeld/", "tinygrad_repo", "selfdrive/test/process_replay/model_replay.py"]]),
step("model replay", "selfdrive/test/process_replay/model_replay.py", [diffPaths: ["selfdrive/modeld/", "tinygrad_repo", "selfdrive/test/process_replay/model_replay.py"]]),
])
},
'tizi': {
deviceStage("tizi", "tizi", ["UNSAFE=1"], [
step("build openpilot", "cd system/manager && ./build.py"),
step("test pandad loopback", "SINGLE_PANDA=1 pytest selfdrive/pandad/tests/test_pandad_loopback.py"),
step("test pandad spi", "pytest selfdrive/pandad/tests/test_pandad_spi.py"),
step("test amp", "pytest system/hardware/tici/tests/test_amplifier.py"),
// TODO: enable once new AGNOS is available
// step("test esim", "pytest system/hardware/tici/tests/test_esim.py"),
step("test qcomgpsd", "pytest system/qcomgpsd/tests/test_qcomgpsd.py", [diffPaths: ["system/qcomgpsd/"]]),
])
},
)
}
} catch (Exception e) {
currentBuild.result = 'FAILED'
throw e
}
}
+2 -58
View File
@@ -11,66 +11,10 @@ Join the official sunnypilot community forum to stay up to date with all the lat
https://docs.sunnypilot.ai/ is your one stop shop for everything from features to installation to FAQ about the sunnypilot
## 🚘 Running on a dedicated device in a car
* A supported device to run this software
* a [comma three](https://comma.ai/shop/products/three) or a [C3X](https://comma.ai/shop/comma-3x)
* This software
* One of [the 325+ supported cars](https://github.com/sunnypilot/sunnypilot/blob/master/docs/CARS.md). We support Honda, Toyota, Hyundai, Nissan, Kia, Chrysler, Lexus, Acura, Audi, VW, Ford, and more. If your car is not supported but has adaptive cruise control and lane-keeping assist, it's likely able to run sunnypilot.
* A [car harness](https://comma.ai/shop/products/car-harness) to connect to your car
Detailed instructions for [how to mount the device in a car](https://comma.ai/setup).
First, check out this list of items you'll need to [get started](https://community.sunnypilot.ai/t/getting-started-using-sunnypilot-in-your-supported-car/251).
## Installation
Please refer to [Recommended Branches](#recommended-branches) to find your preferred/supported branch. This guide will assume you want to install the latest `staging` branch.
### If you want to use our newest branches (our rewrite)
> [!TIP]
>You can see the rewrite state on our [rewrite project board](https://github.com/orgs/sunnypilot/projects/2), and to install the new branches, you can use the following links
* sunnypilot not installed or you installed a version before 0.8.17?
1. [Factory reset/uninstall](https://github.com/commaai/openpilot/wiki/FAQ#how-can-i-reset-the-device) the previous software if you have another software/fork installed.
2. After factory reset/uninstall and upon reboot, select `Custom Software` when given the option.
3. Input the installation URL per [Recommended Branches](#recommended-branches). Example: ```https://staging.sunnypilot.ai```.
4. Complete the rest of the installation following the onscreen instructions.
* sunnypilot already installed and you installed a version after 0.8.17?
1. On the comma three/3X, go to `Settings` ▶️ `Software`.
2. At the `Download` option, press `CHECK`. This will fetch the list of latest branches from sunnypilot.
3. At the `Target Branch` option, press `SELECT` to open the Target Branch selector.
4. Scroll to select the desired branch per Recommended Branches (see below). Example: `staging`
### Recommended Branches
| Branch | Installation URL |
|:---------------:|:---------------------------------------------:|
| `release` | `https://release.sunnypilot.ai` |
| `staging` | `https://staging.sunnypilot.ai` |
| `dev` | `https://dev.sunnypilot.ai` |
| `custom-branch` | `https://install.sunnypilot.ai/{branch_name}` |
> [!TIP]
> You can use sunnypilot/targetbranch as an install URL. Example: 'sunnypilot/staging'.
> [!NOTE]
> Do you require further assistance with software installation? Join the [sunnypilot community forum](https://community.sunnypilot.ai/new-topic?category=general/qa) and create a topic in the General/Q&A Category channel.
<details>
<summary>Older legacy branches</summary>
### If you want to use our older legacy branches (*not recommended*)
> [**IMPORTANT**]
> It is recommended to [re-flash AGNOS](https://flash.comma.ai/) if you intend to downgrade from the new branches.
> You can still restore the latest sunnylink backup made on the old branches.
| Branch | Installation URL |
|:------------:|:--------------------------------:|
| `release-c3` | https://release-c3.sunnypilot.ai |
| `staging-c3` | https://staging-c3.sunnypilot.ai |
| `dev-c3` | https://dev-c3.sunnypilot.ai |
</details>
Next, refer to the sunnypilot community forum for [installation instructions](https://community.sunnypilot.ai/t/read-before-installing-sunnypilot/254), as well as a complete list of [Recommended Branch Installations](https://community.sunnypilot.ai/t/recommended-branch-installations/235).
## 🎆 Pull Requests
We welcome both pull requests and issues on GitHub. Bug fixes are encouraged.
+22 -1
View File
@@ -1,13 +1,34 @@
Version 0.10.4 (2026-02-17)
========================
* Lexus LS 2018 support thanks to Hacheoy!
Version 0.10.3 (2025-12-17)
========================
* New driving model #36249
* New temporal policy architecture
* New on-policy training physics noise model
* New driver monitoring model #36409
* Trained on a new dataset, including comma four data
* Improved inter-process communication memory efficiency
Version 0.10.2 (2025-11-19)
========================
* comma four support
Version 0.10.1 (2025-09-08)
========================
* New driving model
* New driving model #36276
* World Model: removed global localization inputs
* World Model: 2x the number of parameters
* World Model: trained on 4x the number of segments
* VAE Compression Model: new architecture and training objective
* Driving Vision Model: trained on 4x the number of segments
* New Driver Monitoring model #36198
* Acura TLX 2021 support thanks to MVL!
* Honda City 2023 support thanks to vanillagorillaa and drFritz!
* Honda N-Box 2018 support thanks to miettal!
* Honda Odyssey 2021-25 support thanks to csouers and MVL!
* Honda Passport 2026 support thanks to vanillagorillaa and MVL!
Version 0.10.0 (2025-08-05)
========================
-380
View File
@@ -1,380 +0,0 @@
import os
import subprocess
import sys
import sysconfig
import platform
import numpy as np
import SCons.Errors
SCons.Warnings.warningAsException(True)
# pending upstream fix - https://github.com/SCons/scons/issues/4461
#SetOption('warn', 'all')
TICI = os.path.isfile('/TICI')
AGNOS = TICI
Decider('MD5-timestamp')
SetOption('num_jobs', max(1, int(os.cpu_count()/2)))
AddOption('--kaitai',
action='store_true',
help='Regenerate kaitai struct parsers')
AddOption('--asan',
action='store_true',
help='turn on ASAN')
AddOption('--ubsan',
action='store_true',
help='turn on UBSan')
AddOption('--coverage',
action='store_true',
help='build with test coverage options')
AddOption('--clazy',
action='store_true',
help='build with clazy')
AddOption('--ccflags',
action='store',
type='string',
default='',
help='pass arbitrary flags over the command line')
AddOption('--external-sconscript',
action='store',
metavar='FILE',
dest='external_sconscript',
help='add an external SConscript to the build')
AddOption('--mutation',
action='store_true',
help='generate mutation-ready code')
AddOption('--minimal',
action='store_false',
dest='extras',
default=os.path.exists(File('#.lfsconfig').abspath), # minimal by default on release branch (where there's no LFS)
help='the minimum build to run openpilot. no tests, tools, etc.')
AddOption('--stock-ui',
action='store_true',
dest='stock_ui',
default=False,
help='Build stock openpilot UI instead of sunnypilot UI')
## Architecture name breakdown (arch)
## - larch64: linux tici aarch64
## - aarch64: linux pc aarch64
## - x86_64: linux pc x64
## - Darwin: mac x64 or arm64
real_arch = arch = subprocess.check_output(["uname", "-m"], encoding='utf8').rstrip()
if platform.system() == "Darwin":
arch = "Darwin"
brew_prefix = subprocess.check_output(['brew', '--prefix'], encoding='utf8').strip()
elif arch == "aarch64" and AGNOS:
arch = "larch64"
assert arch in ["larch64", "aarch64", "x86_64", "Darwin"]
lenv = {
"PATH": os.environ['PATH'],
"PYTHONPATH": Dir("#").abspath + ':' + Dir(f"#third_party/acados").abspath,
"ACADOS_SOURCE_DIR": Dir("#third_party/acados").abspath,
"ACADOS_PYTHON_INTERFACE_PATH": Dir("#third_party/acados/acados_template").abspath,
"TERA_PATH": Dir("#").abspath + f"/third_party/acados/{arch}/t_renderer"
}
rpath = []
if arch == "larch64":
cpppath = [
"#third_party/opencl/include",
]
libpath = [
"/usr/local/lib",
"/system/vendor/lib64",
f"#third_party/acados/{arch}/lib",
]
libpath += [
"#third_party/snpe/larch64",
"#third_party/libyuv/larch64/lib",
"/usr/lib/aarch64-linux-gnu"
]
cflags = ["-DQCOM2", "-mcpu=cortex-a57"]
cxxflags = ["-DQCOM2", "-mcpu=cortex-a57"]
rpath += ["/usr/local/lib"]
else:
cflags = []
cxxflags = []
cpppath = []
rpath += []
# MacOS
if arch == "Darwin":
libpath = [
f"#third_party/libyuv/{arch}/lib",
f"#third_party/acados/{arch}/lib",
f"{brew_prefix}/lib",
f"{brew_prefix}/opt/openssl@3.0/lib",
"/System/Library/Frameworks/OpenGL.framework/Libraries",
]
cflags += ["-DGL_SILENCE_DEPRECATION"]
cxxflags += ["-DGL_SILENCE_DEPRECATION"]
cpppath += [
f"{brew_prefix}/include",
f"{brew_prefix}/opt/openssl@3.0/include",
]
# Linux
else:
libpath = [
f"#third_party/acados/{arch}/lib",
f"#third_party/libyuv/{arch}/lib",
"/usr/lib",
"/usr/local/lib",
]
if arch == "x86_64":
libpath += [
f"#third_party/snpe/{arch}"
]
rpath += [
Dir(f"#third_party/snpe/{arch}").abspath,
]
if GetOption('asan'):
ccflags = ["-fsanitize=address", "-fno-omit-frame-pointer"]
ldflags = ["-fsanitize=address"]
elif GetOption('ubsan'):
ccflags = ["-fsanitize=undefined"]
ldflags = ["-fsanitize=undefined"]
else:
ccflags = []
ldflags = []
# no --as-needed on mac linker
if arch != "Darwin":
ldflags += ["-Wl,--as-needed", "-Wl,--no-undefined"]
if not GetOption('stock_ui'):
cflags += ["-DSUNNYPILOT"]
cxxflags += ["-DSUNNYPILOT"]
ccflags_option = GetOption('ccflags')
if ccflags_option:
ccflags += ccflags_option.split(' ')
env = Environment(
ENV=lenv,
CCFLAGS=[
"-g",
"-fPIC",
"-O2",
"-Wunused",
"-Werror",
"-Wshadow",
"-Wno-unknown-warning-option",
"-Wno-inconsistent-missing-override",
"-Wno-c99-designator",
"-Wno-reorder-init-list",
"-Wno-vla-cxx-extension",
] + cflags + ccflags,
CPPPATH=cpppath + [
"#",
"#third_party/acados/include",
"#third_party/acados/include/blasfeo/include",
"#third_party/acados/include/hpipm/include",
"#third_party/catch2/include",
"#third_party/libyuv/include",
"#third_party/json11",
"#third_party/linux/include",
"#third_party/snpe/include",
"#third_party",
"#msgq",
],
CC='clang',
CXX='clang++',
LINKFLAGS=ldflags,
RPATH=rpath,
CFLAGS=["-std=gnu11"] + cflags,
CXXFLAGS=["-std=c++1z"] + cxxflags,
LIBPATH=libpath + [
"#msgq_repo",
"#third_party",
"#selfdrive/pandad",
"#common",
"#rednose/helpers",
],
CYTHONCFILESUFFIX=".cpp",
COMPILATIONDB_USE_ABSPATH=True,
REDNOSE_ROOT="#",
tools=["default", "cython", "compilation_db", "rednose_filter"],
toolpath=["#site_scons/site_tools", "#rednose_repo/site_scons/site_tools"],
)
if arch == "Darwin":
# RPATH is not supported on macOS, instead use the linker flags
darwin_rpath_link_flags = [f"-Wl,-rpath,{path}" for path in env["RPATH"]]
env["LINKFLAGS"] += darwin_rpath_link_flags
env.CompilationDatabase('compile_commands.json')
# Setup cache dir
default_cache_dir = '/data/scons_cache' if AGNOS else '/tmp/scons_cache'
cache_dir = ARGUMENTS.get('cache_dir', default_cache_dir)
CacheDir(cache_dir)
Clean(["."], cache_dir)
node_interval = 5
node_count = 0
def progress_function(node):
global node_count
node_count += node_interval
sys.stderr.write("progress: %d\n" % node_count)
if os.environ.get('SCONS_PROGRESS'):
Progress(progress_function, interval=node_interval)
# Cython build environment
py_include = sysconfig.get_paths()['include']
envCython = env.Clone()
envCython["CPPPATH"] += [py_include, np.get_include()]
envCython["CCFLAGS"] += ["-Wno-#warnings", "-Wno-shadow", "-Wno-deprecated-declarations"]
envCython["CCFLAGS"].remove("-Werror")
envCython["LIBS"] = []
if arch == "Darwin":
envCython["LINKFLAGS"] = ["-bundle", "-undefined", "dynamic_lookup"] + darwin_rpath_link_flags
else:
envCython["LINKFLAGS"] = ["-pthread", "-shared"]
np_version = SCons.Script.Value(np.__version__)
Export('envCython', 'np_version')
# Qt build environment
qt_env = env.Clone()
qt_modules = ["Widgets", "Gui", "Core", "Network", "Concurrent", "DBus", "Xml"]
qt_libs = []
if arch == "Darwin":
qt_env['QTDIR'] = f"{brew_prefix}/opt/qt@5"
qt_dirs = [
os.path.join(qt_env['QTDIR'], "include"),
]
qt_dirs += [f"{qt_env['QTDIR']}/include/Qt{m}" for m in qt_modules]
qt_env["LINKFLAGS"] += ["-F" + os.path.join(qt_env['QTDIR'], "lib")]
qt_env["FRAMEWORKS"] += [f"Qt{m}" for m in qt_modules] + ["OpenGL"]
qt_env.AppendENVPath('PATH', os.path.join(qt_env['QTDIR'], "bin"))
else:
qt_install_prefix = subprocess.check_output(['qmake', '-query', 'QT_INSTALL_PREFIX'], encoding='utf8').strip()
qt_install_headers = subprocess.check_output(['qmake', '-query', 'QT_INSTALL_HEADERS'], encoding='utf8').strip()
qt_env['QTDIR'] = qt_install_prefix
qt_dirs = [
f"{qt_install_headers}",
]
qt_gui_path = os.path.join(qt_install_headers, "QtGui")
qt_gui_dirs = [d for d in os.listdir(qt_gui_path) if os.path.isdir(os.path.join(qt_gui_path, d))]
qt_dirs += [f"{qt_install_headers}/QtGui/{qt_gui_dirs[0]}/QtGui", ] if qt_gui_dirs else []
qt_dirs += [f"{qt_install_headers}/Qt{m}" for m in qt_modules]
qt_libs = [f"Qt5{m}" for m in qt_modules]
if arch == "larch64":
qt_libs += ["GLESv2", "wayland-client"]
qt_env.PrependENVPath('PATH', Dir("#third_party/qt5/larch64/bin/").abspath)
elif arch != "Darwin":
qt_libs += ["GL"]
qt_env['QT3DIR'] = qt_env['QTDIR']
qt_env.Tool('qt3')
qt_env['CPPPATH'] += qt_dirs + ["#third_party/qrcode"]
qt_flags = [
"-D_REENTRANT",
"-DQT_NO_DEBUG",
"-DQT_WIDGETS_LIB",
"-DQT_GUI_LIB",
"-DQT_CORE_LIB",
"-DQT_MESSAGELOGCONTEXT",
]
qt_env['CXXFLAGS'] += qt_flags
qt_env['LIBPATH'] += ['#selfdrive/ui', ]
qt_env['LIBS'] = qt_libs
if GetOption("clazy"):
checks = [
"level0",
"level1",
"no-range-loop",
"no-non-pod-global-static",
]
qt_env['CXX'] = 'clazy'
qt_env['ENV']['CLAZY_IGNORE_DIRS'] = qt_dirs[0]
qt_env['ENV']['CLAZY_CHECKS'] = ','.join(checks)
Export('env', 'qt_env', 'arch', 'real_arch')
# Build common module
SConscript(['common/SConscript'])
Import('_common', '_gpucommon')
common = [_common, 'json11', 'zmq']
gpucommon = [_gpucommon]
Export('common', 'gpucommon')
# Build messaging (cereal + msgq + socketmaster + their dependencies)
# Enable swaglog include in submodules
env_swaglog = env.Clone()
env_swaglog['CXXFLAGS'].append('-DSWAGLOG="\\"common/swaglog.h\\""')
SConscript(['msgq_repo/SConscript'], exports={'env': env_swaglog})
SConscript(['opendbc_repo/SConscript'], exports={'env': env_swaglog})
SConscript(['cereal/SConscript'])
Import('socketmaster', 'msgq')
messaging = [socketmaster, msgq, 'capnp', 'kj',]
Export('messaging')
# Build other submodules
SConscript(['panda/SConscript'])
# Build rednose library
SConscript(['rednose/SConscript'])
# Build system services
SConscript([
'system/ubloxd/SConscript',
'system/loggerd/SConscript',
])
if arch == "larch64":
SConscript(['system/camerad/SConscript'])
# Build openpilot
SConscript(['third_party/SConscript'])
SConscript(['selfdrive/SConscript'])
SConscript(['sunnypilot/SConscript'])
if Dir('#tools/cabana/').exists() and GetOption('extras'):
SConscript(['tools/replay/SConscript'])
if arch != "larch64":
SConscript(['tools/cabana/SConscript'])
external_sconscript = GetOption('external_sconscript')
if external_sconscript:
SConscript([external_sconscript])
-20
View File
@@ -1,20 +0,0 @@
Import('env', 'common', 'msgq')
cereal_dir = Dir('.')
gen_dir = Dir('gen')
# Build cereal
schema_files = ['log.capnp', 'car.capnp', 'legacy.capnp', 'custom.capnp']
env.Command([f'gen/cpp/{s}.c++' for s in schema_files] + [f'gen/cpp/{s}.h' for s in schema_files],
schema_files,
f"capnpc --src-prefix={cereal_dir.path} $SOURCES -o c++:{gen_dir.path}/cpp/")
cereal = env.Library('cereal', [f'gen/cpp/{s}.c++' for s in schema_files])
# Build messaging
services_h = env.Command(['services.h'], ['services.py'], 'python3 ' + cereal_dir.path + '/services.py > $TARGET')
env.Program('messaging/bridge', ['messaging/bridge.cc', 'messaging/msgq_to_zmq.cc'], LIBS=[msgq, common, 'pthread'])
socketmaster = env.Library('socketmaster', ['messaging/socketmaster.cc'])
Export('cereal', 'socketmaster')
+2 -14
View File
@@ -153,6 +153,7 @@ struct ModelManagerSP @0xaedffd8f31e7b55d {
navigation @1;
vision @2;
policy @3;
offPolicy @4;
}
}
@@ -454,20 +455,7 @@ struct ModelDataV2SP @0xa1680744031fdb2d {
}
}
struct Navigationd @0xcb9fd56c7057593a {
upcomingTurn @0 :Text;
currentSpeedLimit @1 :UInt64;
bannerInstructions @2 :Text;
distanceFromRoute @3 :Float64;
allManeuvers @4 :List(Maneuver);
valid @5 :Bool;
struct Maneuver {
distance @0 :Float64;
type @1 :Text;
modifier @2 :Text;
instruction @3 :Text;
}
struct CustomReserved10 @0xcb9fd56c7057593a {
}
struct CustomReserved11 @0xc2243c65e0340384 {
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+22 -10
View File
@@ -87,6 +87,7 @@ struct OnroadEvent @0xc4fa6047f024e718 {
laneChange @50;
lowMemory @51;
stockAeb @52;
stockLkas @98;
ldw @53;
carUnrecognized @54;
invalidLkasSetting @55;
@@ -918,6 +919,8 @@ struct ControlsState @0x97ff69c53601abf1 {
saturated @7 :Bool;
actualLateralAccel @9 :Float32;
desiredLateralAccel @10 :Float32;
desiredLateralJerk @11 :Float32;
version @12 :Int32;
}
struct LateralLQRState {
@@ -1475,6 +1478,11 @@ struct ProcLog {
cmdline @15 :List(Text);
exe @16 :Text;
# from /proc/<pid>/smaps_rollup (proportional/private memory)
memPss @17 :UInt64; # Pss — shared pages split by mapper count
memPssAnon @18 :UInt64; # Pss_Anon — private anonymous (heap, stack)
memPssShmem @19 :UInt64; # Pss_Shmem — proportional MSGQ/tmpfs share
}
struct CPUTimes {
@@ -2146,13 +2154,10 @@ struct Joystick {
struct DriverStateV2 {
frameId @0 :UInt32;
modelExecutionTime @1 :Float32;
dspExecutionTimeDEPRECATED @2 :Float32;
gpuExecutionTime @8 :Float32;
rawPredictions @3 :Data;
poorVisionProb @4 :Float32;
wheelOnRightProb @5 :Float32;
leftDriverData @6 :DriverData;
rightDriverData @7 :DriverData;
@@ -2167,10 +2172,14 @@ struct DriverStateV2 {
leftBlinkProb @7 :Float32;
rightBlinkProb @8 :Float32;
sunglassesProb @9 :Float32;
occludedProb @10 :Float32;
readyProb @11 :List(Float32);
notReadyProb @12 :List(Float32);
phoneProb @13 :Float32;
notReadyProbDEPRECATED @12 :List(Float32);
occludedProbDEPRECATED @10 :Float32;
readyProbDEPRECATED @11 :List(Float32);
}
dspExecutionTimeDEPRECATED @2 :Float32;
poorVisionProbDEPRECATED @4 :Float32;
}
struct DriverStateDEPRECATED @0xb83c6cc593ed0a00 {
@@ -2222,6 +2231,9 @@ struct DriverMonitoringState @0xb83cda094a1da284 {
hiStdCount @14 :UInt32;
isActiveMode @16 :Bool;
isRHD @4 :Bool;
uncertainCount @19 :UInt32;
phoneProbOffset @20 :Float32;
phoneProbValidCount @21 :UInt32;
isPreviewDEPRECATED @15 :Bool;
rhdCheckedDEPRECATED @5 :Bool;
@@ -2518,13 +2530,10 @@ struct Event {
controlsState @7 :ControlsState;
selfdriveState @130 :SelfdriveState;
gyroscope @99 :SensorEventData;
gyroscope2 @100 :SensorEventData;
accelerometer @98 :SensorEventData;
accelerometer2 @101 :SensorEventData;
magnetometer @95 :SensorEventData;
lightSensor @96 :SensorEventData;
temperatureSensor @97 :SensorEventData;
temperatureSensor2 @123 :SensorEventData;
pandaStates @81 :List(PandaState);
peripheralState @80 :PeripheralState;
radarState @13 :RadarState;
@@ -2632,7 +2641,7 @@ struct Event {
carStateSP @114 :Custom.CarStateSP;
liveMapDataSP @115 :Custom.LiveMapDataSP;
modelDataV2SP @116 :Custom.ModelDataV2SP;
navigationd @136 :Custom.Navigationd;
customReserved10 @136 :Custom.CustomReserved10;
customReserved11 @137 :Custom.CustomReserved11;
customReserved12 @138 :Custom.CustomReserved12;
customReserved13 @139 :Custom.CustomReserved13;
@@ -2687,5 +2696,8 @@ struct Event {
liveLocationKalman @72 :LiveLocationKalman;
liveTracksDEPRECATED @16 :List(LiveTracksDEPRECATED);
onroadEventsDEPRECATED @68: List(Car.OnroadEventDEPRECATED);
gyroscope2DEPRECATED @100 :SensorEventData;
accelerometer2DEPRECATED @101 :SensorEventData;
temperatureSensor2DEPRECATED @123 :SensorEventData;
}
}
+18 -6
View File
@@ -1,10 +1,8 @@
# must be built with scons
from msgq.ipc_pyx import Context, Poller, SubSocket, PubSocket, SocketEventHandle, toggle_fake_events, \
set_fake_prefix, get_fake_prefix, delete_fake_prefix, wait_for_one_event
from msgq.ipc_pyx import MultiplePublishersError, IpcError
from msgq import fake_event_handle, pub_sock, sub_sock, drain_sock_raw
from msgq import fake_event_handle, drain_sock_raw, MultiplePublishersError, IpcError, \
Context, Poller, SubSocket, PubSocket, SocketEventHandle, toggle_fake_events, \
set_fake_prefix, get_fake_prefix, delete_fake_prefix, wait_for_one_event
import msgq
import os
import capnp
import time
@@ -13,11 +11,25 @@ from typing import Optional, List, Union, Dict
from cereal import log
from cereal.services import SERVICE_LIST
from openpilot.common.util import MovingAverage
from openpilot.common.utils import MovingAverage
NO_TRAVERSAL_LIMIT = 2**64-1
def pub_sock(endpoint: str) -> PubSocket:
service = SERVICE_LIST.get(endpoint)
segment_size = service.queue_size if service else 0
return msgq.pub_sock(endpoint, segment_size)
def sub_sock(endpoint: str, poller: Optional[Poller] = None, addr: str = "127.0.0.1",
conflate: bool = False, timeout: Optional[int] = None) -> SubSocket:
service = SERVICE_LIST.get(endpoint)
segment_size = service.queue_size if service else 0
return msgq.sub_sock(endpoint, poller=poller, addr=addr, conflate=conflate,
timeout=timeout, segment_size=segment_size)
def reset_context():
msgq.context = Context()
BIN
View File
Binary file not shown.
+8 -7
View File
@@ -25,15 +25,16 @@ void msgq_to_zmq(const std::vector<std::string> &endpoints, const std::string &i
}
void zmq_to_msgq(const std::vector<std::string> &endpoints, const std::string &ip) {
auto poller = std::make_unique<ZMQPoller>();
auto pub_context = std::make_unique<MSGQContext>();
auto sub_context = std::make_unique<ZMQContext>();
std::map<SubSocket *, PubSocket *> sub2pub;
auto poller = std::make_unique<BridgeZmqPoller>();
auto pub_context = std::make_unique<Context>();
auto sub_context = std::make_unique<BridgeZmqContext>();
std::map<BridgeZmqSubSocket *, PubSocket *> sub2pub;
for (auto endpoint : endpoints) {
auto pub_sock = new MSGQPubSocket();
auto sub_sock = new ZMQSubSocket();
pub_sock->connect(pub_context.get(), endpoint);
auto pub_sock = new PubSocket();
auto sub_sock = new BridgeZmqSubSocket();
size_t queue_size = services.at(endpoint).queue_size;
pub_sock->connect(pub_context.get(), endpoint, true, queue_size);
sub_sock->connect(sub_context.get(), endpoint, ip, false);
poller->registerSocket(sub_sock);
+170
View File
@@ -0,0 +1,170 @@
#include "cereal/messaging/bridge_zmq.h"
#include <cassert>
#include <cstring>
#include <unistd.h>
static size_t fnv1a_hash(const std::string &str) {
const size_t fnv_prime = 0x100000001b3;
size_t hash_value = 0xcbf29ce484222325;
for (char c : str) {
hash_value ^= (unsigned char)c;
hash_value *= fnv_prime;
}
return hash_value;
}
// FIXME: This is a hack to get the port number from the socket name, might have collisions.
static int get_port(std::string endpoint) {
size_t hash_value = fnv1a_hash(endpoint);
int start_port = 8023;
int max_port = 65535;
return start_port + (hash_value % (max_port - start_port));
}
BridgeZmqContext::BridgeZmqContext() {
context = zmq_ctx_new();
}
BridgeZmqContext::~BridgeZmqContext() {
if (context != nullptr) {
zmq_ctx_term(context);
}
}
void BridgeZmqMessage::init(size_t sz) {
size = sz;
data = new char[size];
}
void BridgeZmqMessage::init(char *d, size_t sz) {
size = sz;
data = new char[size];
memcpy(data, d, size);
}
void BridgeZmqMessage::close() {
if (size > 0) {
delete[] data;
}
data = nullptr;
size = 0;
}
BridgeZmqMessage::~BridgeZmqMessage() {
close();
}
int BridgeZmqSubSocket::connect(BridgeZmqContext *context, std::string endpoint, std::string address, bool conflate, bool check_endpoint) {
sock = zmq_socket(context->getRawContext(), ZMQ_SUB);
if (sock == nullptr) {
return -1;
}
zmq_setsockopt(sock, ZMQ_SUBSCRIBE, "", 0);
if (conflate) {
int arg = 1;
zmq_setsockopt(sock, ZMQ_CONFLATE, &arg, sizeof(int));
}
int reconnect_ivl = 500;
zmq_setsockopt(sock, ZMQ_RECONNECT_IVL_MAX, &reconnect_ivl, sizeof(reconnect_ivl));
full_endpoint = "tcp://" + address + ":";
if (check_endpoint) {
full_endpoint += std::to_string(get_port(endpoint));
} else {
full_endpoint += endpoint;
}
return zmq_connect(sock, full_endpoint.c_str());
}
void BridgeZmqSubSocket::setTimeout(int timeout) {
zmq_setsockopt(sock, ZMQ_RCVTIMEO, &timeout, sizeof(int));
}
Message *BridgeZmqSubSocket::receive(bool non_blocking) {
zmq_msg_t msg;
assert(zmq_msg_init(&msg) == 0);
int flags = non_blocking ? ZMQ_DONTWAIT : 0;
int rc = zmq_msg_recv(&msg, sock, flags);
Message *ret = nullptr;
if (rc >= 0) {
ret = new BridgeZmqMessage;
ret->init((char *)zmq_msg_data(&msg), zmq_msg_size(&msg));
}
zmq_msg_close(&msg);
return ret;
}
BridgeZmqSubSocket::~BridgeZmqSubSocket() {
if (sock != nullptr) {
zmq_close(sock);
}
}
int BridgeZmqPubSocket::connect(BridgeZmqContext *context, std::string endpoint, bool check_endpoint) {
sock = zmq_socket(context->getRawContext(), ZMQ_PUB);
if (sock == nullptr) {
return -1;
}
full_endpoint = "tcp://*:";
if (check_endpoint) {
full_endpoint += std::to_string(get_port(endpoint));
} else {
full_endpoint += endpoint;
}
// ZMQ pub sockets cannot be shared between processes, so we need to ensure pid stays the same.
pid = getpid();
return zmq_bind(sock, full_endpoint.c_str());
}
int BridgeZmqPubSocket::sendMessage(Message *message) {
assert(pid == getpid());
return zmq_send(sock, message->getData(), message->getSize(), ZMQ_DONTWAIT);
}
int BridgeZmqPubSocket::send(char *data, size_t size) {
assert(pid == getpid());
return zmq_send(sock, data, size, ZMQ_DONTWAIT);
}
BridgeZmqPubSocket::~BridgeZmqPubSocket() {
if (sock != nullptr) {
zmq_close(sock);
}
}
void BridgeZmqPoller::registerSocket(BridgeZmqSubSocket *socket) {
assert(num_polls + 1 < (sizeof(polls) / sizeof(polls[0])));
polls[num_polls].socket = socket->getRawSocket();
polls[num_polls].events = ZMQ_POLLIN;
sockets.push_back(socket);
num_polls++;
}
std::vector<BridgeZmqSubSocket *> BridgeZmqPoller::poll(int timeout) {
std::vector<BridgeZmqSubSocket *> ret;
int rc = zmq_poll(polls, num_polls, timeout);
if (rc < 0) {
return ret;
}
for (size_t i = 0; i < num_polls; i++) {
if (polls[i].revents) {
ret.push_back(sockets[i]);
}
}
return ret;
}
+72
View File
@@ -0,0 +1,72 @@
#pragma once
#include <cstddef>
#include <string>
#include <vector>
#include <zmq.h>
#include "msgq/ipc.h"
class BridgeZmqContext {
public:
BridgeZmqContext();
void *getRawContext() { return context; }
~BridgeZmqContext();
private:
void *context = nullptr;
};
class BridgeZmqMessage : public Message {
public:
void init(size_t size);
void init(char *data, size_t size);
void close();
size_t getSize() { return size; }
char *getData() { return data; }
~BridgeZmqMessage();
private:
char *data = nullptr;
size_t size = 0;
};
class BridgeZmqSubSocket {
public:
int connect(BridgeZmqContext *context, std::string endpoint, std::string address, bool conflate = false, bool check_endpoint = true);
void setTimeout(int timeout);
Message *receive(bool non_blocking = false);
void *getRawSocket() { return sock; }
~BridgeZmqSubSocket();
private:
void *sock = nullptr;
std::string full_endpoint;
};
class BridgeZmqPubSocket {
public:
int connect(BridgeZmqContext *context, std::string endpoint, bool check_endpoint = true);
int sendMessage(Message *message);
int send(char *data, size_t size);
void *getRawSocket() { return sock; }
~BridgeZmqPubSocket();
private:
void *sock = nullptr;
std::string full_endpoint;
int pid = -1;
};
class BridgeZmqPoller {
public:
void registerSocket(BridgeZmqSubSocket *socket);
std::vector<BridgeZmqSubSocket *> poll(int timeout);
private:
static constexpr size_t MAX_BRIDGE_ZMQ_POLLERS = 128;
std::vector<BridgeZmqSubSocket *> sockets;
zmq_pollitem_t polls[MAX_BRIDGE_ZMQ_POLLERS] = {};
size_t num_polls = 0;
};
+9 -7
View File
@@ -2,6 +2,7 @@
#include <cassert>
#include "cereal/services.h"
#include "common/util.h"
extern ExitHandler do_exit;
@@ -21,14 +22,14 @@ static std::string recv_zmq_msg(void *sock) {
}
void MsgqToZmq::run(const std::vector<std::string> &endpoints, const std::string &ip) {
zmq_context = std::make_unique<ZMQContext>();
msgq_context = std::make_unique<MSGQContext>();
zmq_context = std::make_unique<BridgeZmqContext>();
msgq_context = std::make_unique<Context>();
// Create ZMQPubSockets for each endpoint
for (const auto &endpoint : endpoints) {
auto &socket_pair = socket_pairs.emplace_back();
socket_pair.endpoint = endpoint;
socket_pair.pub_sock = std::make_unique<ZMQPubSocket>();
socket_pair.pub_sock = std::make_unique<BridgeZmqPubSocket>();
int ret = socket_pair.pub_sock->connect(zmq_context.get(), endpoint);
if (ret != 0) {
printf("Failed to create ZMQ publisher for [%s]: %s\n", endpoint.c_str(), zmq_strerror(zmq_errno()));
@@ -48,7 +49,7 @@ void MsgqToZmq::run(const std::vector<std::string> &endpoints, const std::string
for (auto sub_sock : msgq_poller->poll(100)) {
// Process messages for each socket
ZMQPubSocket *pub_sock = sub2pub.at(sub_sock);
BridgeZmqPubSocket *pub_sock = sub2pub.at(sub_sock);
for (int i = 0; i < MAX_MESSAGES_PER_SOCKET; ++i) {
auto msg = std::unique_ptr<Message>(sub_sock->receive(true));
if (!msg) break;
@@ -71,7 +72,7 @@ void MsgqToZmq::zmqMonitorThread() {
// Set up ZMQ monitor for each pub socket
for (int i = 0; i < socket_pairs.size(); ++i) {
std::string addr = "inproc://op-bridge-monitor-" + std::to_string(i);
zmq_socket_monitor(socket_pairs[i].pub_sock->sock, addr.c_str(), ZMQ_EVENT_ACCEPTED | ZMQ_EVENT_DISCONNECTED);
zmq_socket_monitor(socket_pairs[i].pub_sock->getRawSocket(), addr.c_str(), ZMQ_EVENT_ACCEPTED | ZMQ_EVENT_DISCONNECTED);
void *monitor_socket = zmq_socket(zmq_context->getRawContext(), ZMQ_PAIR);
zmq_connect(monitor_socket, addr.c_str());
@@ -108,7 +109,8 @@ void MsgqToZmq::zmqMonitorThread() {
if (++pair.connected_clients == 1) {
// Create new MSGQ subscriber socket and map to ZMQ publisher
pair.sub_sock = std::make_unique<MSGQSubSocket>();
pair.sub_sock->connect(msgq_context.get(), pair.endpoint, "127.0.0.1");
size_t queue_size = services.at(pair.endpoint).queue_size;
pair.sub_sock->connect(msgq_context.get(), pair.endpoint, "127.0.0.1", false, true, queue_size);
sub2pub[pair.sub_sock.get()] = pair.pub_sock.get();
registerSockets();
}
@@ -128,7 +130,7 @@ void MsgqToZmq::zmqMonitorThread() {
// Clean up monitor sockets
for (int i = 0; i < pollitems.size(); ++i) {
zmq_socket_monitor(socket_pairs[i].pub_sock->sock, nullptr, 0);
zmq_socket_monitor(socket_pairs[i].pub_sock->getRawSocket(), nullptr, 0);
zmq_close(pollitems[i].socket);
}
cv.notify_one();
+5 -6
View File
@@ -7,9 +7,8 @@
#include <string>
#include <vector>
#define private public
#include "msgq/impl_msgq.h"
#include "msgq/impl_zmq.h"
#include "cereal/messaging/bridge_zmq.h"
class MsgqToZmq {
public:
@@ -22,16 +21,16 @@ protected:
struct SocketPair {
std::string endpoint;
std::unique_ptr<ZMQPubSocket> pub_sock;
std::unique_ptr<BridgeZmqPubSocket> pub_sock;
std::unique_ptr<MSGQSubSocket> sub_sock;
int connected_clients = 0;
};
std::unique_ptr<MSGQContext> msgq_context;
std::unique_ptr<ZMQContext> zmq_context;
std::unique_ptr<Context> msgq_context;
std::unique_ptr<BridgeZmqContext> zmq_context;
std::mutex mutex;
std::condition_variable cv;
std::unique_ptr<MSGQPoller> msgq_poller;
std::map<SubSocket *, ZMQPubSocket *> sub2pub;
std::map<SubSocket *, BridgeZmqPubSocket *> sub2pub;
std::vector<SocketPair> socket_pairs;
};
+3 -2
View File
@@ -50,7 +50,7 @@ SubMaster::SubMaster(const std::vector<const char *> &service_list, const std::v
assert(services.count(std::string(name)) > 0);
service serv = services.at(std::string(name));
SubSocket *socket = SubSocket::create(message_context.context(), name, address ? address : "127.0.0.1", true);
SubSocket *socket = SubSocket::create(message_context.context(), name, address ? address : "127.0.0.1", true, true, serv.queue_size);
assert(socket != 0);
bool is_polled = inList(poll, name) || poll.empty();
if (is_polled) poller_->registerSocket(socket);
@@ -187,7 +187,8 @@ SubMaster::~SubMaster() {
PubMaster::PubMaster(const std::vector<const char *> &service_list) {
for (auto name : service_list) {
assert(services.count(name) > 0);
PubSocket *socket = PubSocket::create(message_context.context(), name);
service serv = services.at(std::string(name));
PubSocket *socket = PubSocket::create(message_context.context(), name, true, serv.queue_size);
assert(socket);
sockets_[name] = socket;
}
+99
View File
@@ -0,0 +1,99 @@
/* THIS IS AN AUTOGENERATED FILE, PLEASE EDIT services.py */
#ifndef __SERVICES_H
#define __SERVICES_H
#include <map>
#include <string>
struct service { std::string name; bool should_log; float frequency; int decimation; size_t queue_size; };
static std::map<std::string, service> services = {
{ "gyroscope", {"gyroscope", true, 104.000000, 104, 256000}},
{ "accelerometer", {"accelerometer", true, 104.000000, 104, 256000}},
{ "magnetometer", {"magnetometer", true, 25.000000, -1, 256000}},
{ "lightSensor", {"lightSensor", true, 100.000000, 100, 256000}},
{ "temperatureSensor", {"temperatureSensor", true, 2.000000, 200, 256000}},
{ "gpsNMEA", {"gpsNMEA", true, 9.000000, -1, 256000}},
{ "deviceState", {"deviceState", true, 2.000000, 1, 256000}},
{ "touch", {"touch", true, 20.000000, 1, 256000}},
{ "can", {"can", true, 100.000000, 2053, 10485760}},
{ "controlsState", {"controlsState", true, 100.000000, 10, 2097152}},
{ "selfdriveState", {"selfdriveState", true, 100.000000, 10, 256000}},
{ "pandaStates", {"pandaStates", true, 10.000000, 1, 256000}},
{ "peripheralState", {"peripheralState", true, 2.000000, 1, 256000}},
{ "radarState", {"radarState", true, 20.000000, 5, 256000}},
{ "roadEncodeIdx", {"roadEncodeIdx", false, 20.000000, 1, 256000}},
{ "liveTracks", {"liveTracks", true, 20.000000, -1, 256000}},
{ "sendcan", {"sendcan", true, 100.000000, 139, 2097152}},
{ "logMessage", {"logMessage", true, 0.000000, -1, 256000}},
{ "errorLogMessage", {"errorLogMessage", true, 0.000000, 1, 256000}},
{ "liveCalibration", {"liveCalibration", true, 4.000000, 4, 256000}},
{ "liveTorqueParameters", {"liveTorqueParameters", true, 4.000000, 1, 256000}},
{ "liveDelay", {"liveDelay", true, 4.000000, 1, 256000}},
{ "androidLog", {"androidLog", true, 0.000000, -1, 256000}},
{ "carState", {"carState", true, 100.000000, 10, 256000}},
{ "carControl", {"carControl", true, 100.000000, 10, 256000}},
{ "carOutput", {"carOutput", true, 100.000000, 10, 256000}},
{ "longitudinalPlan", {"longitudinalPlan", true, 20.000000, 10, 256000}},
{ "driverAssistance", {"driverAssistance", true, 20.000000, 20, 256000}},
{ "procLog", {"procLog", true, 0.500000, 15, 10485760}},
{ "gpsLocationExternal", {"gpsLocationExternal", true, 10.000000, 10, 256000}},
{ "gpsLocation", {"gpsLocation", true, 1.000000, 1, 256000}},
{ "ubloxGnss", {"ubloxGnss", true, 10.000000, -1, 256000}},
{ "qcomGnss", {"qcomGnss", true, 2.000000, -1, 256000}},
{ "gnssMeasurements", {"gnssMeasurements", true, 10.000000, 10, 256000}},
{ "clocks", {"clocks", true, 0.100000, 1, 256000}},
{ "ubloxRaw", {"ubloxRaw", true, 20.000000, -1, 256000}},
{ "livePose", {"livePose", true, 20.000000, 4, 256000}},
{ "liveParameters", {"liveParameters", true, 20.000000, 5, 256000}},
{ "cameraOdometry", {"cameraOdometry", true, 20.000000, 10, 256000}},
{ "thumbnail", {"thumbnail", true, 0.016667, 1, 256000}},
{ "onroadEvents", {"onroadEvents", true, 1.000000, 1, 256000}},
{ "carParams", {"carParams", true, 0.020000, 1, 256000}},
{ "roadCameraState", {"roadCameraState", true, 20.000000, 20, 256000}},
{ "driverCameraState", {"driverCameraState", true, 20.000000, 20, 256000}},
{ "driverEncodeIdx", {"driverEncodeIdx", false, 20.000000, 1, 256000}},
{ "driverStateV2", {"driverStateV2", true, 20.000000, 10, 256000}},
{ "driverMonitoringState", {"driverMonitoringState", true, 20.000000, 10, 256000}},
{ "wideRoadEncodeIdx", {"wideRoadEncodeIdx", false, 20.000000, 1, 256000}},
{ "wideRoadCameraState", {"wideRoadCameraState", true, 20.000000, 20, 256000}},
{ "drivingModelData", {"drivingModelData", true, 20.000000, 10, 256000}},
{ "modelV2", {"modelV2", true, 20.000000, -1, 10485760}},
{ "managerState", {"managerState", true, 2.000000, 1, 256000}},
{ "uploaderState", {"uploaderState", true, 0.000000, 1, 256000}},
{ "navInstruction", {"navInstruction", true, 1.000000, 10, 256000}},
{ "navRoute", {"navRoute", true, 0.000000, -1, 256000}},
{ "navThumbnail", {"navThumbnail", true, 0.000000, -1, 256000}},
{ "qRoadEncodeIdx", {"qRoadEncodeIdx", false, 20.000000, -1, 256000}},
{ "userBookmark", {"userBookmark", true, 0.000000, 1, 256000}},
{ "soundPressure", {"soundPressure", true, 10.000000, 10, 256000}},
{ "rawAudioData", {"rawAudioData", false, 20.000000, -1, 256000}},
{ "bookmarkButton", {"bookmarkButton", true, 0.000000, 1, 256000}},
{ "audioFeedback", {"audioFeedback", true, 0.000000, 1, 256000}},
{ "roadEncodeData", {"roadEncodeData", false, 20.000000, -1, 10485760}},
{ "driverEncodeData", {"driverEncodeData", false, 20.000000, -1, 10485760}},
{ "wideRoadEncodeData", {"wideRoadEncodeData", false, 20.000000, -1, 10485760}},
{ "qRoadEncodeData", {"qRoadEncodeData", false, 20.000000, -1, 10485760}},
{ "modelManagerSP", {"modelManagerSP", false, 1.000000, 1, 10485760}},
{ "backupManagerSP", {"backupManagerSP", false, 1.000000, 1, 10485760}},
{ "selfdriveStateSP", {"selfdriveStateSP", true, 100.000000, 10, 256000}},
{ "longitudinalPlanSP", {"longitudinalPlanSP", true, 20.000000, 10, 256000}},
{ "onroadEventsSP", {"onroadEventsSP", true, 1.000000, 1, 256000}},
{ "carParamsSP", {"carParamsSP", true, 0.020000, 1, 256000}},
{ "carControlSP", {"carControlSP", true, 100.000000, 10, 256000}},
{ "carStateSP", {"carStateSP", true, 100.000000, 10, 256000}},
{ "liveMapDataSP", {"liveMapDataSP", true, 1.000000, 1, 256000}},
{ "modelDataV2SP", {"modelDataV2SP", true, 20.000000, -1, 10485760}},
{ "liveLocationKalman", {"liveLocationKalman", true, 20.000000, -1, 256000}},
{ "uiDebug", {"uiDebug", true, 0.000000, 1, 256000}},
{ "testJoystick", {"testJoystick", true, 0.000000, -1, 256000}},
{ "alertDebug", {"alertDebug", true, 20.000000, 5, 256000}},
{ "livestreamWideRoadEncodeIdx", {"livestreamWideRoadEncodeIdx", false, 20.000000, -1, 256000}},
{ "livestreamRoadEncodeIdx", {"livestreamRoadEncodeIdx", false, 20.000000, -1, 256000}},
{ "livestreamDriverEncodeIdx", {"livestreamDriverEncodeIdx", false, 20.000000, -1, 256000}},
{ "livestreamWideRoadEncodeData", {"livestreamWideRoadEncodeData", false, 20.000000, -1, 2097152}},
{ "livestreamRoadEncodeData", {"livestreamRoadEncodeData", false, 20.000000, -1, 2097152}},
{ "livestreamDriverEncodeData", {"livestreamDriverEncodeData", false, 20.000000, -1, 2097152}},
{ "customReservedRawData0", {"customReservedRawData0", true, 0.000000, -1, 256000}},
{ "customReservedRawData1", {"customReservedRawData1", true, 0.000000, -1, 256000}},
{ "customReservedRawData2", {"customReservedRawData2", true, 0.000000, -1, 256000}},
};
#endif
+29 -23
View File
@@ -1,37 +1,44 @@
#!/usr/bin/env python3
from enum import IntEnum
from typing import Optional
# TODO: this should be automatically determined using the capnp schema
class QueueSize(IntEnum):
BIG = 10 * 1024 * 1024 # 10MB - video frames, large AI outputs
MEDIUM = 2 * 1024 * 1024 # 2MB - high freq (CAN), livestream
SMALL = 250 * 1024 # 250KB - most services
class Service:
def __init__(self, should_log: bool, frequency: float, decimation: Optional[int] = None):
def __init__(self, should_log: bool, frequency: float, decimation: Optional[int] = None,
queue_size: QueueSize = QueueSize.SMALL):
self.should_log = should_log
self.frequency = frequency
self.decimation = decimation
self.queue_size = queue_size
_services: dict[str, tuple] = {
# service: (should_log, frequency, qlog decimation (optional))
# note: the "EncodeIdx" packets will still be in the log
"gyroscope": (True, 104., 104),
"gyroscope2": (True, 100., 100),
"accelerometer": (True, 104., 104),
"accelerometer2": (True, 100., 100),
"magnetometer": (True, 25.),
"lightSensor": (True, 100., 100),
"temperatureSensor": (True, 2., 200),
"temperatureSensor2": (True, 2., 200),
"gpsNMEA": (True, 9.),
"deviceState": (True, 2., 1),
"touch": (True, 20., 1),
"can": (True, 100., 2053), # decimation gives ~3 msgs in a full segment
"controlsState": (True, 100., 10),
"can": (True, 100., 2053, QueueSize.BIG), # decimation gives ~3 msgs in a full segment
"controlsState": (True, 100., 10, QueueSize.MEDIUM),
"selfdriveState": (True, 100., 10),
"pandaStates": (True, 10., 1),
"peripheralState": (True, 2., 1),
"radarState": (True, 20., 5),
"roadEncodeIdx": (False, 20., 1),
"liveTracks": (True, 20.),
"sendcan": (True, 100., 139),
"sendcan": (True, 100., 139, QueueSize.MEDIUM),
"logMessage": (True, 0.),
"errorLogMessage": (True, 0., 1),
"liveCalibration": (True, 4., 4),
@@ -43,7 +50,7 @@ _services: dict[str, tuple] = {
"carOutput": (True, 100., 10),
"longitudinalPlan": (True, 20., 10),
"driverAssistance": (True, 20., 20),
"procLog": (True, 0.5, 15),
"procLog": (True, 0.5, 15, QueueSize.BIG),
"gpsLocationExternal": (True, 10., 10),
"gpsLocation": (True, 1., 1),
"ubloxGnss": (True, 10.),
@@ -65,7 +72,7 @@ _services: dict[str, tuple] = {
"wideRoadEncodeIdx": (False, 20., 1),
"wideRoadCameraState": (True, 20., 20),
"drivingModelData": (True, 20., 10),
"modelV2": (True, 20.),
"modelV2": (True, 20., None, QueueSize.BIG),
"managerState": (True, 2., 1),
"uploaderState": (True, 0., 1),
"navInstruction": (True, 1., 10),
@@ -77,10 +84,14 @@ _services: dict[str, tuple] = {
"rawAudioData": (False, 20.),
"bookmarkButton": (True, 0., 1),
"audioFeedback": (True, 0., 1),
"roadEncodeData": (False, 20., None, QueueSize.BIG),
"driverEncodeData": (False, 20., None, QueueSize.BIG),
"wideRoadEncodeData": (False, 20., None, QueueSize.BIG),
"qRoadEncodeData": (False, 20., None, QueueSize.BIG),
# sunnypilot
"modelManagerSP": (False, 1., 1),
"backupManagerSP": (False, 1., 1),
"modelManagerSP": (False, 1., 1, QueueSize.BIG),
"backupManagerSP": (False, 1., 1, QueueSize.BIG),
"selfdriveStateSP": (True, 100., 10),
"longitudinalPlanSP": (True, 20., 10),
"onroadEventsSP": (True, 1., 1),
@@ -88,24 +99,19 @@ _services: dict[str, tuple] = {
"carControlSP": (True, 100., 10),
"carStateSP": (True, 100., 10),
"liveMapDataSP": (True, 1., 1),
"modelDataV2SP": (True, 20.),
"navigationd": (True, 3.),
"modelDataV2SP": (True, 20., None, QueueSize.BIG),
"liveLocationKalman": (True, 20.),
# debug
"uiDebug": (True, 0., 1),
"testJoystick": (True, 0.),
"alertDebug": (True, 20., 5),
"roadEncodeData": (False, 20.),
"driverEncodeData": (False, 20.),
"wideRoadEncodeData": (False, 20.),
"qRoadEncodeData": (False, 20.),
"livestreamWideRoadEncodeIdx": (False, 20.),
"livestreamRoadEncodeIdx": (False, 20.),
"livestreamDriverEncodeIdx": (False, 20.),
"livestreamWideRoadEncodeData": (False, 20.),
"livestreamRoadEncodeData": (False, 20.),
"livestreamDriverEncodeData": (False, 20.),
"livestreamWideRoadEncodeData": (False, 20., None, QueueSize.MEDIUM),
"livestreamRoadEncodeData": (False, 20., None, QueueSize.MEDIUM),
"livestreamDriverEncodeData": (False, 20., None, QueueSize.MEDIUM),
"customReservedRawData0": (True, 0.),
"customReservedRawData1": (True, 0.),
"customReservedRawData2": (True, 0.),
@@ -123,13 +129,13 @@ def build_header():
h += "#include <map>\n"
h += "#include <string>\n"
h += "struct service { std::string name; bool should_log; float frequency; int decimation; };\n"
h += "struct service { std::string name; bool should_log; float frequency; int decimation; size_t queue_size; };\n"
h += "static std::map<std::string, service> services = {\n"
for k, v in SERVICE_LIST.items():
should_log = "true" if v.should_log else "false"
decimation = -1 if v.decimation is None else v.decimation
h += ' { "%s", {"%s", %s, %f, %d}},\n' % \
(k, k, should_log, v.frequency, decimation)
h += ' { "%s", {"%s", %s, %f, %d, %d}},\n' % \
(k, k, should_log, v.frequency, decimation, v.queue_size)
h += "};\n"
h += "#endif\n"
-35
View File
@@ -1,35 +0,0 @@
Import('env', 'envCython', 'arch')
common_libs = [
'params.cc',
'swaglog.cc',
'util.cc',
'watchdog.cc',
'ratekeeper.cc'
]
_common = env.Library('common', common_libs, LIBS="json11")
files = [
'clutil.cc',
]
_gpucommon = env.Library('gpucommon', files)
Export('_common', '_gpucommon')
if GetOption('extras'):
env.Program('tests/test_common',
['tests/test_runner.cc', 'tests/test_params.cc', 'tests/test_util.cc', 'tests/test_swaglog.cc'],
LIBS=[_common, 'json11', 'zmq', 'pthread'])
# Cython bindings
params_python = envCython.Program('params_pyx.so', 'params_pyx.pyx', LIBS=envCython['LIBS'] + [_common, 'zmq', 'json11'])
SConscript([
'transformations/SConscript',
])
Import('transformations_python')
common_python = [params_python, transformations_python]
Export('common_python')
+8 -4
View File
@@ -14,9 +14,13 @@ class Api:
def post(self, *args, **kwargs):
return self.service.post(*args, **kwargs)
def get_token(self, expiry_hours=1):
return self.service.get_token(expiry_hours)
def get_token(self, payload_extra=None, expiry_hours=1):
return self.service.get_token(payload_extra, expiry_hours)
def api_get(endpoint, method='GET', timeout=None, access_token=None, **params):
return CommaConnectApi(None).api_get(endpoint, method, timeout, access_token, **params)
def api_get(endpoint, method='GET', timeout=None, access_token=None, session=None, **params):
return CommaConnectApi(None).api_get(endpoint, method, timeout, access_token, session, **params)
def get_key_pair() -> tuple[str, str, str] | tuple[None, None, None]:
return CommaConnectApi(None).get_key_pair()
+24 -8
View File
@@ -1,18 +1,22 @@
import jwt
import os
import requests
import unicodedata
from datetime import datetime, timedelta, UTC
from openpilot.system.hardware.hw import Paths
from openpilot.system.version import get_version
# name: jwt signature algorithm
KEYS = {"id_rsa": "RS256",
"id_ecdsa": "ES256"}
class BaseApi:
def __init__(self, dongle_id, api_host, user_agent="openpilot-"):
self.dongle_id = dongle_id
self.api_host = api_host
self.user_agent = user_agent
with open(f'{Paths.persist_root()}/comma/id_rsa') as f:
self.private_key = f.read()
self.jwt_algorithm, self.private_key, _ = self.get_key_pair()
def get(self, *args, **kwargs):
return self.request('GET', *args, **kwargs)
@@ -23,7 +27,7 @@ class BaseApi:
def request(self, method, endpoint, timeout=None, access_token=None, **params):
return self.api_get(endpoint, method=method, timeout=timeout, access_token=access_token, **params)
def _get_token(self, expiry_hours=1, **extra_payload):
def _get_token(self, payload_extra=None, expiry_hours=1, **extra_payload):
now = datetime.now(UTC).replace(tzinfo=None)
payload = {
'identity': self.dongle_id,
@@ -32,20 +36,22 @@ class BaseApi:
'exp': now + timedelta(hours=expiry_hours),
**extra_payload
}
token = jwt.encode(payload, self.private_key, algorithm='RS256')
if payload_extra is not None:
payload.update(payload_extra)
token = jwt.encode(payload, self.private_key, algorithm=self.jwt_algorithm)
if isinstance(token, bytes):
token = token.decode('utf8')
return token
def get_token(self, expiry_hours=1):
return self._get_token(expiry_hours)
def get_token(self, payload_extra=None, expiry_hours=1):
return self._get_token(payload_extra, expiry_hours)
def remove_non_ascii_chars(self, text):
normalized_text = unicodedata.normalize('NFD', text)
ascii_encoded_text = normalized_text.encode('ascii', 'ignore')
return ascii_encoded_text.decode()
def api_get(self, endpoint, method='GET', timeout=None, access_token=None, json=None, **params):
def api_get(self, endpoint, method='GET', timeout=None, access_token=None, session=None, json=None, **params):
headers = {}
if access_token is not None:
headers['Authorization'] = "JWT " + access_token
@@ -53,4 +59,14 @@ class BaseApi:
version = self.remove_non_ascii_chars(get_version())
headers['User-Agent'] = self.user_agent + version
return requests.request(method, f"{self.api_host}/{endpoint}", timeout=timeout, headers=headers, json=json, params=params)
# TODO: add session to Api
req = requests if session is None else session
return req.request(method, f"{self.api_host}/{endpoint}", timeout=timeout, headers=headers, json=json, params=params)
@staticmethod
def get_key_pair() -> tuple[str, str, str] | tuple[None, None, None]:
for key in KEYS:
if os.path.isfile(Paths.persist_root() + f'/comma/{key}') and os.path.isfile(Paths.persist_root() + f'/comma/{key}.pub'):
with open(Paths.persist_root() + f'/comma/{key}') as private, open(Paths.persist_root() + f'/comma/{key}.pub') as public:
return KEYS[key], private.read(), public.read()
return None, None, None
-9
View File
@@ -1,9 +0,0 @@
# remove all keys that end in DEPRECATED
def strip_deprecated_keys(d):
for k in list(d.keys()):
if isinstance(k, str):
if k.endswith('DEPRECATED'):
d.pop(k)
elif isinstance(d[k], dict):
strip_deprecated_keys(d[k])
return d
-58
View File
@@ -1,58 +0,0 @@
import io
import os
import tempfile
import contextlib
import zstandard as zstd
LOG_COMPRESSION_LEVEL = 10 # little benefit up to level 15. level ~17 is a small step change
class CallbackReader:
"""Wraps a file, but overrides the read method to also
call a callback function with the number of bytes read so far."""
def __init__(self, f, callback, *args):
self.f = f
self.callback = callback
self.cb_args = args
self.total_read = 0
def __getattr__(self, attr):
return getattr(self.f, attr)
def read(self, *args, **kwargs):
chunk = self.f.read(*args, **kwargs)
self.total_read += len(chunk)
self.callback(*self.cb_args, self.total_read)
return chunk
@contextlib.contextmanager
def atomic_write_in_dir(path: str, mode: str = 'w', buffering: int = -1, encoding: str = None, newline: str = None,
overwrite: bool = False):
"""Write to a file atomically using a temporary file in the same directory as the destination file."""
dir_name = os.path.dirname(path)
if not overwrite and os.path.exists(path):
raise FileExistsError(f"File '{path}' already exists. To overwrite it, set 'overwrite' to True.")
with tempfile.NamedTemporaryFile(mode=mode, buffering=buffering, encoding=encoding, newline=newline, dir=dir_name, delete=False) as tmp_file:
yield tmp_file
tmp_file_name = tmp_file.name
os.replace(tmp_file_name, path)
def get_upload_stream(filepath: str, should_compress: bool) -> tuple[io.BufferedIOBase, int]:
if not should_compress:
file_size = os.path.getsize(filepath)
file_stream = open(filepath, "rb")
return file_stream, file_size
# Compress the file on the fly
compressed_stream = io.BytesIO()
compressor = zstd.ZstdCompressor(level=LOG_COMPRESSION_LEVEL)
with open(filepath, "rb") as f:
compressor.copy_stream(f, compressed_stream)
compressed_size = compressed_stream.tell()
compressed_stream.seek(0)
return compressed_stream, compressed_size
+17
View File
@@ -15,3 +15,20 @@ class FirstOrderFilter:
self.initialized = True
self.x = x
return self.x
class BounceFilter(FirstOrderFilter):
def __init__(self, x0, rc, dt, initialized=True, bounce=2):
self.velocity = FirstOrderFilter(0.0, 0.15, dt)
self.bounce = bounce
super().__init__(x0, rc, dt, initialized)
def update(self, x):
super().update(x)
scale = self.dt / (1.0 / 60.0) # tuned at 60 fps
self.velocity.x += (x - self.x) * self.bounce * scale * self.dt
self.velocity.update(0.0)
if abs(self.velocity.x) < 1e-5:
self.velocity.x = 0.0
self.x += self.velocity.x
return self.x
+7 -7
View File
@@ -1,30 +1,30 @@
from functools import cache
import subprocess
from openpilot.common.run import run_cmd, run_cmd_default
from openpilot.common.utils import run_cmd, run_cmd_default
@cache
def get_commit(cwd: str = None, branch: str = "HEAD") -> str:
def get_commit(cwd: str | None = None, branch: str = "HEAD") -> str:
return run_cmd_default(["git", "rev-parse", branch], cwd=cwd)
@cache
def get_commit_date(cwd: str = None, commit: str = "HEAD") -> str:
def get_commit_date(cwd: str | None = None, commit: str = "HEAD") -> str:
return run_cmd_default(["git", "show", "--no-patch", "--format='%ct %ci'", commit], cwd=cwd)
@cache
def get_short_branch(cwd: str = None) -> str:
def get_short_branch(cwd: str | None = None) -> str:
return run_cmd_default(["git", "rev-parse", "--abbrev-ref", "HEAD"], cwd=cwd)
@cache
def get_branch(cwd: str = None) -> str:
def get_branch(cwd: str | None = None) -> str:
return run_cmd_default(["git", "rev-parse", "--abbrev-ref", "--symbolic-full-name", "@{u}"], cwd=cwd)
@cache
def get_origin(cwd: str = None) -> str:
def get_origin(cwd: str | None = None) -> str:
try:
local_branch = run_cmd(["git", "name-rev", "--name-only", "HEAD"], cwd=cwd)
tracking_remote = run_cmd(["git", "config", "branch." + local_branch + ".remote"], cwd=cwd)
@@ -34,7 +34,7 @@ def get_origin(cwd: str = None) -> str:
@cache
def get_normalized_origin(cwd: str = None) -> str:
def get_normalized_origin(cwd: str | None = None) -> str:
return get_origin(cwd) \
.replace("git@", "", 1) \
.replace(".git", "", 1) \
+81
View File
@@ -0,0 +1,81 @@
import os
import fcntl
import ctypes
# I2C constants from /usr/include/linux/i2c-dev.h
I2C_SLAVE = 0x0703
I2C_SLAVE_FORCE = 0x0706
I2C_SMBUS = 0x0720
# SMBus transfer types
I2C_SMBUS_READ = 1
I2C_SMBUS_WRITE = 0
I2C_SMBUS_BYTE_DATA = 2
I2C_SMBUS_I2C_BLOCK_DATA = 8
I2C_SMBUS_BLOCK_MAX = 32
class _I2cSmbusData(ctypes.Union):
_fields_ = [
("byte", ctypes.c_uint8),
("word", ctypes.c_uint16),
("block", ctypes.c_uint8 * (I2C_SMBUS_BLOCK_MAX + 2)),
]
class _I2cSmbusIoctlData(ctypes.Structure):
_fields_ = [
("read_write", ctypes.c_uint8),
("command", ctypes.c_uint8),
("size", ctypes.c_uint32),
("data", ctypes.POINTER(_I2cSmbusData)),
]
class SMBus:
def __init__(self, bus: int):
self._fd = os.open(f'/dev/i2c-{bus}', os.O_RDWR)
def __enter__(self) -> 'SMBus':
return self
def __exit__(self, *args) -> None:
self.close()
def close(self) -> None:
if hasattr(self, '_fd') and self._fd >= 0:
os.close(self._fd)
self._fd = -1
def _set_address(self, addr: int, force: bool = False) -> None:
ioctl_arg = I2C_SLAVE_FORCE if force else I2C_SLAVE
fcntl.ioctl(self._fd, ioctl_arg, addr)
def _smbus_access(self, read_write: int, command: int, size: int, data: _I2cSmbusData) -> None:
ioctl_data = _I2cSmbusIoctlData(read_write, command, size, ctypes.pointer(data))
fcntl.ioctl(self._fd, I2C_SMBUS, ioctl_data)
def read_byte_data(self, addr: int, register: int, force: bool = False) -> int:
self._set_address(addr, force)
data = _I2cSmbusData()
self._smbus_access(I2C_SMBUS_READ, register, I2C_SMBUS_BYTE_DATA, data)
return int(data.byte)
def write_byte_data(self, addr: int, register: int, value: int, force: bool = False) -> None:
self._set_address(addr, force)
data = _I2cSmbusData()
data.byte = value & 0xFF
self._smbus_access(I2C_SMBUS_WRITE, register, I2C_SMBUS_BYTE_DATA, data)
def read_i2c_block_data(self, addr: int, register: int, length: int, force: bool = False) -> list[int]:
self._set_address(addr, force)
if not (0 <= length <= I2C_SMBUS_BLOCK_MAX):
raise ValueError(f"length must be 0..{I2C_SMBUS_BLOCK_MAX}")
data = _I2cSmbusData()
data.block[0] = length
self._smbus_access(I2C_SMBUS_READ, register, I2C_SMBUS_I2C_BLOCK_DATA, data)
read_len = int(data.block[0]) or length
read_len = min(read_len, length)
return [int(b) for b in data.block[1 : read_len + 1]]
+1 -1
View File
@@ -1 +1 @@
#define DEFAULT_MODEL "Firehose (Default)"
#define DEFAULT_MODEL "CD210 (Default)"
+15 -19
View File
@@ -66,11 +66,12 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"IsTakingSnapshot", {CLEAR_ON_MANAGER_START, BOOL}},
{"IsTestedBranch", {CLEAR_ON_MANAGER_START, BOOL}},
{"JoystickDebugMode", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, BOOL}},
{"LanguageSetting", {PERSISTENT | BACKUP, STRING, "main_en"}},
{"LanguageSetting", {PERSISTENT | BACKUP, STRING, "en"}},
{"LastAthenaPingTime", {CLEAR_ON_MANAGER_START, INT}},
{"LastGPSPosition", {PERSISTENT, STRING}},
{"LastManagerExitReason", {CLEAR_ON_MANAGER_START, STRING}},
{"LastOffroadStatusPacket", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, JSON}},
{"LastAgnosPowerMonitorShutdown", {CLEAR_ON_MANAGER_START, STRING}},
{"LastPowerDropDetected", {CLEAR_ON_MANAGER_START, STRING}},
{"LastUpdateException", {CLEAR_ON_MANAGER_START, STRING}},
{"LastUpdateRouteCount", {PERSISTENT, INT, "0"}},
@@ -97,6 +98,7 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"Offroad_TemperatureTooHigh", {CLEAR_ON_MANAGER_START, JSON}},
{"Offroad_UnregisteredHardware", {CLEAR_ON_MANAGER_START, JSON}},
{"Offroad_UpdateFailed", {CLEAR_ON_MANAGER_START, JSON}},
{"Offroad_DriverMonitoringUncertain", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, JSON}},
{"OnroadCycleRequested", {CLEAR_ON_MANAGER_START, BOOL}},
{"OpenpilotEnabledToggle", {PERSISTENT | BACKUP, BOOL, "1"}},
{"PandaHeartbeatLost", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, BOOL}},
@@ -108,10 +110,12 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"RecordFront", {PERSISTENT | BACKUP, BOOL}},
{"RecordFrontLock", {PERSISTENT, BOOL}}, // for the internal fleet
{"SecOCKey", {PERSISTENT | DONT_LOG | BACKUP, STRING}},
{"ShowDebugInfo", {PERSISTENT, BOOL}},
{"RouteCount", {PERSISTENT, INT, "0"}},
{"SnoozeUpdate", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, BOOL}},
{"SshEnabled", {PERSISTENT | BACKUP, BOOL}},
{"TermsVersion", {PERSISTENT, STRING}},
{"TorqueBar", {PERSISTENT | BACKUP, BOOL, "0"}},
{"TrainingVersion", {PERSISTENT, STRING}},
{"UbloxAvailable", {PERSISTENT, BOOL}},
{"UpdateAvailable", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, BOOL}},
@@ -133,14 +137,17 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"ApiCache_DriveStats", {PERSISTENT, JSON}},
{"AutoLaneChangeBsmDelay", {PERSISTENT | BACKUP, BOOL, "0"}},
{"AutoLaneChangeTimer", {PERSISTENT | BACKUP, INT, "0"}},
{"BlinkerLateralReengageDelay", {PERSISTENT | BACKUP, INT, "0"}}, // seconds
{"BlinkerMinLateralControlSpeed", {PERSISTENT | BACKUP, INT, "20"}}, // MPH or km/h
{"BlinkerPauseLateralControl", {PERSISTENT | BACKUP, INT, "0"}},
{"Brightness", {PERSISTENT | BACKUP, INT, "0"}},
{"CarList", {PERSISTENT, JSON}},
{"CarParamsSP", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, BYTES}},
{"CarParamsSPCache", {CLEAR_ON_MANAGER_START, BYTES}},
{"CarParamsSPPersistent", {PERSISTENT, BYTES}},
{"CarPlatformBundle", {PERSISTENT | BACKUP, JSON}},
{"ChevronInfo", {PERSISTENT | BACKUP, INT, "4"}},
{"CompletedSunnylinkConsentVersion", {PERSISTENT, STRING, "0"}},
{"CustomAccIncrementsEnabled", {PERSISTENT | BACKUP, BOOL, "0"}},
{"CustomAccLongPressIncrement", {PERSISTENT | BACKUP, INT, "5"}},
{"CustomAccShortPressIncrement", {PERSISTENT | BACKUP, INT, "1"}},
@@ -150,6 +157,7 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"EnableGithubRunner", {PERSISTENT | BACKUP, BOOL}},
{"GreenLightAlert", {PERSISTENT | BACKUP, BOOL, "0"}},
{"GithubRunnerSufficientVoltage", {CLEAR_ON_MANAGER_START , BOOL}},
{"HasAcceptedTermsSP", {PERSISTENT, STRING, "0"}},
{"HideVEgoUI", {PERSISTENT | BACKUP, BOOL, "0"}},
{"IntelligentCruiseButtonManagement", {PERSISTENT | BACKUP , BOOL}},
{"InteractivityTimeout", {PERSISTENT | BACKUP, INT, "0"}},
@@ -162,24 +170,16 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"OffroadMode", {CLEAR_ON_MANAGER_START, BOOL}},
{"Offroad_TiciSupport", {CLEAR_ON_MANAGER_START, JSON}},
{"OnroadScreenOffBrightness", {PERSISTENT | BACKUP, INT, "0"}},
{"OnroadScreenOffControl", {PERSISTENT | BACKUP, BOOL}},
{"OnroadScreenOffTimer", {PERSISTENT | BACKUP, INT, "15"}},
{"OnroadUploads", {PERSISTENT | BACKUP, BOOL, "1"}},
{"QuickBootToggle", {PERSISTENT | BACKUP, BOOL, "0"}},
{"QuietMode", {PERSISTENT | BACKUP, BOOL, "0"}},
{"RainbowMode", {PERSISTENT | BACKUP, BOOL, "0"}},
{"RocketFuel", {PERSISTENT | BACKUP, BOOL, "0"}},
{"ShowAdvancedControls", {PERSISTENT | BACKUP, BOOL, "0"}},
{"ShowTurnSignals", {PERSISTENT | BACKUP, BOOL, "0"}},
{"StandstillTimer", {PERSISTENT | BACKUP, BOOL, "0"}},
{"TrueVEgoUI", {PERSISTENT | BACKUP, BOOL, "0"}},
{"VisualRadarTracks", {PERSISTENT | BACKUP, BOOL, "0"}},
{"VisualRadarTracksDelay", {PERSISTENT | BACKUP, FLOAT, "0.0"}},
{"VisualWideCam", {PERSISTENT | BACKUP, BOOL, "0"}},
{"VisualStyle", {PERSISTENT | BACKUP, INT, "0"}},
{"VisualStyleZoom", {PERSISTENT | BACKUP, BOOL, "0"}},
{"VisualStyleOverhead", {PERSISTENT | BACKUP, BOOL, "0"}},
{"VisualStyleOverheadZoom", {PERSISTENT | BACKUP, BOOL, "0"}},
{"VisualStyleOverheadThreshold", {PERSISTENT | BACKUP, INT, "20"}},
// MADS params
{"Mads", {PERSISTENT | BACKUP, BOOL, "1"}},
@@ -195,14 +195,6 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"ModelManager_LastSyncTime", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, INT, "0"}},
{"ModelManager_ModelsCache", {PERSISTENT | BACKUP, JSON}},
// Navigation params
{"AllowNavigation", {PERSISTENT | BACKUP, BOOL, "0"}},
{"MapboxToken", {PERSISTENT | BACKUP, STRING}},
{"MapboxSettings", {CLEAR_ON_MANAGER_START, JSON}},
{"MapboxRoute", {PERSISTENT, STRING}},
{"MapboxRecompute", {PERSISTENT | BACKUP, BOOL, "0"}},
{"NavDesiresAllowed", {PERSISTENT | BACKUP, BOOL, "0"}},
// Neural Network Lateral Control
{"NeuralNetworkLateralControl", {PERSISTENT | BACKUP, BOOL, "0"}},
@@ -225,16 +217,19 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"SubaruStopAndGo", {PERSISTENT | BACKUP, BOOL, "0"}},
{"SubaruStopAndGoManualParkingBrake", {PERSISTENT | BACKUP, BOOL, "0"}},
{"TeslaCoopSteering", {PERSISTENT | BACKUP, BOOL, "0"}},
{"ToyotaEnforceStockLongitudinal", {PERSISTENT | BACKUP, BOOL, "0"}},
{"DynamicExperimentalControl", {PERSISTENT | BACKUP, BOOL, "0"}},
{"BlindSpot", {PERSISTENT | BACKUP, BOOL, "0"}},
// sunnypilot model params
{"CameraOffset", {PERSISTENT | BACKUP, FLOAT, "0.0"}},
{"LagdToggle", {PERSISTENT | BACKUP, BOOL, "1"}},
{"LagdToggleDelay", {PERSISTENT | BACKUP, FLOAT, "0.2"}},
{"LagdValueCache", {PERSISTENT, FLOAT, "0.2"}},
{"LaneTurnDesire", {PERSISTENT | BACKUP, BOOL, "0"}},
{"LaneTurnValue", {PERSISTENT | BACKUP, FLOAT, "19.0"}},
{"PlanplusControl", {PERSISTENT | BACKUP, FLOAT, "1.0"}},
// mapd
{"MapAdvisorySpeedLimit", {CLEAR_ON_ONROAD_TRANSITION, FLOAT}},
@@ -255,7 +250,7 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"OsmStateTitle", {PERSISTENT, STRING}},
{"OsmWayTest", {PERSISTENT, STRING}},
{"RoadName", {CLEAR_ON_ONROAD_TRANSITION, STRING}},
{"RoadNameToggle", {PERSISTENT, STRING}},
{"RoadNameToggle", {PERSISTENT | BACKUP, BOOL, "0"}},
// Speed Limit
{"SpeedLimitMode", {PERSISTENT | BACKUP, INT, "1"}},
@@ -273,6 +268,7 @@ inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"EnforceTorqueControl", {PERSISTENT | BACKUP, BOOL}},
{"LiveTorqueParamsToggle", {PERSISTENT | BACKUP , BOOL}},
{"LiveTorqueParamsRelaxedToggle", {PERSISTENT | BACKUP , BOOL}},
{"TorqueControlTune", {PERSISTENT | BACKUP, FLOAT}},
{"TorqueParamsOverrideEnabled", {PERSISTENT | BACKUP, BOOL, "0"}},
{"TorqueParamsOverrideFriction", {PERSISTENT | BACKUP, FLOAT, "0.1"}},
{"TorqueParamsOverrideLatAccelFactor", {PERSISTENT | BACKUP, FLOAT, "2.5"}},
+17857
View File
File diff suppressed because it is too large Load Diff
BIN
View File
Binary file not shown.
+9 -16
View File
@@ -2,21 +2,14 @@ import numpy as np
from numbers import Number
class PIDController:
def __init__(self, k_p, k_i, k_f=0., k_d=0., pos_limit=1e308, neg_limit=-1e308, rate=100):
self._k_p = k_p
self._k_i = k_i
self._k_d = k_d
self.k_f = k_f # feedforward gain
if isinstance(self._k_p, Number):
self._k_p = [[0], [self._k_p]]
if isinstance(self._k_i, Number):
self._k_i = [[0], [self._k_i]]
if isinstance(self._k_d, Number):
self._k_d = [[0], [self._k_d]]
def __init__(self, k_p, k_i, k_d=0., pos_limit=1e308, neg_limit=-1e308, rate=100):
self._k_p: list[list[float]] = [[0], [k_p]] if isinstance(k_p, Number) else k_p
self._k_i: list[list[float]] = [[0], [k_i]] if isinstance(k_i, Number) else k_i
self._k_d: list[list[float]] = [[0], [k_d]] if isinstance(k_d, Number) else k_d
self.set_limits(pos_limit, neg_limit)
self.i_rate = 1.0 / rate
self.i_dt = 1.0 / rate
self.speed = 0.0
self.reset()
@@ -46,12 +39,12 @@ class PIDController:
def update(self, error, error_rate=0.0, speed=0.0, feedforward=0., freeze_integrator=False):
self.speed = speed
self.p = float(error) * self.k_p
self.f = feedforward * self.k_f
self.d = error_rate * self.k_d
self.p = self.k_p * float(error)
self.d = self.k_d * error_rate
self.f = feedforward
if not freeze_integrator:
i = self.i + error * self.k_i * self.i_rate
i = self.i + self.k_i * self.i_dt * error
# Don't allow windup if already clipping
test_control = self.p + i + self.d + self.f
+5 -1
View File
@@ -13,7 +13,11 @@ public:
if (prefix.empty()) {
prefix = util::random_string(15);
}
msgq_path = Path::shm_path() + "/" + prefix;
#ifdef __APPLE__
msgq_path = "/tmp/msgq_" + prefix;
#else
msgq_path = "/dev/shm/msgq_" + prefix;
#endif
bool ret = util::create_directories(msgq_path, 0777);
assert(ret);
setenv("OPENPILOT_PREFIX", prefix.c_str(), 1);
+4 -2
View File
@@ -1,4 +1,5 @@
import os
import platform
import shutil
import uuid
@@ -9,9 +10,10 @@ from openpilot.system.hardware.hw import Paths
from openpilot.system.hardware.hw import DEFAULT_DOWNLOAD_CACHE_ROOT
class OpenpilotPrefix:
def __init__(self, prefix: str = None, create_dirs_on_enter: bool = True, clean_dirs_on_exit: bool = True, shared_download_cache: bool = False):
def __init__(self, prefix: str | None = None, create_dirs_on_enter: bool = True, clean_dirs_on_exit: bool = True, shared_download_cache: bool = False):
self.prefix = prefix if prefix else str(uuid.uuid4().hex[0:15])
self.msgq_path = os.path.join(Paths.shm_path(), self.prefix)
shm_path = "/tmp" if platform.system() == "Darwin" else "/dev/shm"
self.msgq_path = os.path.join(shm_path, "msgq_" + self.prefix)
self.create_dirs_on_enter = create_dirs_on_enter
self.clean_dirs_on_exit = clean_dirs_on_exit
self.shared_download_cache = shared_download_cache
+1 -1
View File
@@ -6,7 +6,7 @@ import time
from setproctitle import getproctitle
from openpilot.common.util import MovingAverage
from openpilot.common.utils import MovingAverage
from openpilot.system.hardware import PC
-30
View File
@@ -1,30 +0,0 @@
import time
import functools
from openpilot.common.swaglog import cloudlog
def retry(attempts=3, delay=1.0, ignore_failure=False):
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
for _ in range(attempts):
try:
return func(*args, **kwargs)
except Exception:
cloudlog.exception(f"{func.__name__} failed, trying again")
time.sleep(delay)
if ignore_failure:
cloudlog.error(f"{func.__name__} failed after retry")
else:
raise Exception(f"{func.__name__} failed after retry")
return wrapper
return decorator
if __name__ == "__main__":
@retry(attempts=10)
def abc():
raise ValueError("abc failed :(")
abc()
-28
View File
@@ -1,28 +0,0 @@
import subprocess
from contextlib import contextmanager
from subprocess import Popen, PIPE, TimeoutExpired
def run_cmd(cmd: list[str], cwd=None, env=None) -> str:
return subprocess.check_output(cmd, encoding='utf8', cwd=cwd, env=env).strip()
def run_cmd_default(cmd: list[str], default: str = "", cwd=None, env=None) -> str:
try:
return run_cmd(cmd, cwd=cwd, env=env)
except subprocess.CalledProcessError:
return default
@contextmanager
def managed_proc(cmd: list[str], env: dict[str, str]):
proc = Popen(cmd, env=env, stdout=PIPE, stderr=PIPE)
try:
yield proc
finally:
if proc.poll() is None:
proc.terminate()
try:
proc.wait(timeout=5)
except TimeoutExpired:
proc.kill()
+3 -3
View File
@@ -1,7 +1,7 @@
import os
from uuid import uuid4
from openpilot.common.file_helpers import atomic_write_in_dir
from openpilot.common.utils import atomic_write
class TestFileHelpers:
@@ -15,5 +15,5 @@ class TestFileHelpers:
assert f.read() == "test"
os.remove(path)
def test_atomic_write_in_dir(self):
self.run_atomic_write_func(atomic_write_in_dir)
def test_atomic_write(self):
self.run_atomic_write_func(atomic_write)
-5
View File
@@ -1,5 +0,0 @@
Import('env', 'envCython')
transformations = env.Library('transformations', ['orientation.cc', 'coordinates.cc'])
transformations_python = envCython.Program('transformations.so', 'transformations.pyx')
Export('transformations', 'transformations_python')
@@ -102,3 +102,36 @@ class TestNED:
np.testing.assert_allclose(converter.ned2ecef(ned_offsets_batch),
ecef_positions_offset_batch,
rtol=1e-9, atol=1e-7)
def test_errors(self):
# Test wrong shape/type for geodetic2ecef
# numpy_wrap raises IndexError for scalar input
with np.testing.assert_raises(IndexError):
coord.geodetic2ecef(1.0)
with np.testing.assert_raises_regex(ValueError, "Geodetic must be size 3"):
coord.geodetic2ecef([0, 0])
with np.testing.assert_raises_regex(ValueError, "Geodetic must be size 3"):
coord.geodetic2ecef([0, 0, 0, 0])
with np.testing.assert_raises(TypeError):
coord.geodetic2ecef(['a', 'b', 'c'])
# Test LocalCoord constructor errors
with np.testing.assert_raises(ValueError):
coord.LocalCoord.from_geodetic([0, 0])
with np.testing.assert_raises(ValueError):
coord.LocalCoord.from_geodetic(1)
with np.testing.assert_raises(TypeError):
coord.LocalCoord.from_geodetic(['a', 'b', 'c'])
# Test wrong shape/type for ecef2geodetic
with np.testing.assert_raises(ValueError):
coord.ecef2geodetic([1, 2])
with np.testing.assert_raises(ValueError):
coord.ecef2geodetic([1, 2, 3, 4])
with np.testing.assert_raises(IndexError):
coord.ecef2geodetic(1.0)
@@ -1,4 +1,5 @@
import numpy as np
import pytest
from openpilot.common.transformations.orientation import euler2quat, quat2euler, euler2rot, rot2euler, \
rot2quat, quat2rot, \
@@ -59,3 +60,32 @@ class TestOrientation:
np.testing.assert_allclose(ned_eulers[i], ned_euler_from_ecef(ecef_positions[i], eulers[i]), rtol=1e-7)
#np.testing.assert_allclose(eulers[i], ecef_euler_from_ned(ecef_positions[i], ned_eulers[i]), rtol=1e-7)
# np.testing.assert_allclose(ned_eulers, ned_euler_from_ecef(ecef_positions, eulers), rtol=1e-7)
def test_inputs(self):
with pytest.raises(ValueError):
euler2quat([1, 2])
with pytest.raises(ValueError):
quat2rot([1, 2, 3])
with pytest.raises(IndexError):
rot2quat(np.zeros((2, 2)))
def test_euler_rot_consistency(self):
rpy = [0.1, 0.2, 0.3]
R = euler2rot(rpy)
# R -> q -> R
q = rot2quat(R)
R_new = quat2rot(q)
np.testing.assert_allclose(R, R_new, atol=1e-15)
# q -> R -> Euler (quat2euler) -> R
rpy_new = quat2euler(q)
R_new2 = euler2rot(rpy_new)
np.testing.assert_allclose(R, R_new2, atol=1e-15)
# R -> Euler (rot2euler) -> R
rpy_from_rot = rot2euler(R)
R_new3 = euler2rot(rpy_from_rot)
np.testing.assert_allclose(R, R_new3, atol=1e-15)
@@ -1,72 +0,0 @@
# cython: language_level=3
from libcpp cimport bool
cdef extern from "orientation.cc":
pass
cdef extern from "orientation.hpp":
cdef cppclass Quaternion "Eigen::Quaterniond":
Quaternion()
Quaternion(double, double, double, double)
double w()
double x()
double y()
double z()
cdef cppclass Vector3 "Eigen::Vector3d":
Vector3()
Vector3(double, double, double)
double operator()(int)
cdef cppclass Matrix3 "Eigen::Matrix3d":
Matrix3()
Matrix3(double*)
double operator()(int, int)
Quaternion euler2quat(const Vector3 &)
Vector3 quat2euler(const Quaternion &)
Matrix3 quat2rot(const Quaternion &)
Quaternion rot2quat(const Matrix3 &)
Vector3 rot2euler(const Matrix3 &)
Matrix3 euler2rot(const Vector3 &)
Matrix3 rot_matrix(double, double, double)
Vector3 ecef_euler_from_ned(const ECEF &, const Vector3 &)
Vector3 ned_euler_from_ecef(const ECEF &, const Vector3 &)
cdef extern from "coordinates.cc":
cdef struct ECEF:
double x
double y
double z
cdef struct NED:
double n
double e
double d
cdef struct Geodetic:
double lat
double lon
double alt
bool radians
ECEF geodetic2ecef(const Geodetic &)
Geodetic ecef2geodetic(const ECEF &)
cdef cppclass LocalCoord_c "LocalCoord":
Matrix3 ned2ecef_matrix
Matrix3 ecef2ned_matrix
LocalCoord_c(const Geodetic &, const ECEF &)
LocalCoord_c(const Geodetic &)
LocalCoord_c(const ECEF &)
NED ecef2ned(const ECEF &)
ECEF ned2ecef(const NED &)
NED geodetic2ned(const Geodetic &)
Geodetic ned2geodetic(const NED &)
cdef extern from "coordinates.hpp":
pass
+342
View File
@@ -0,0 +1,342 @@
import numpy as np
# Constants
a = 6378137.0
b = 6356752.3142
esq = 6.69437999014e-3
e1sq = 6.73949674228e-3
def geodetic2ecef_single(g):
"""
Convert geodetic coordinates (latitude, longitude, altitude) to ECEF.
"""
try:
if len(g) != 3:
raise ValueError("Geodetic must be size 3")
except TypeError:
raise ValueError("Geodetic must be a sequence of length 3") from None
lat, lon, alt = g
lat = np.radians(lat)
lon = np.radians(lon)
xi = np.sqrt(1.0 - esq * np.sin(lat)**2)
x = (a / xi + alt) * np.cos(lat) * np.cos(lon)
y = (a / xi + alt) * np.cos(lat) * np.sin(lon)
z = (a / xi * (1.0 - esq) + alt) * np.sin(lat)
return np.array([x, y, z])
def ecef2geodetic_single(e):
"""
Convert ECEF to geodetic coordinates using Ferrari's solution.
"""
x, y, z = e
r = np.sqrt(x**2 + y**2)
Esq = a**2 - b**2
F = 54 * b**2 * z**2
G = r**2 + (1 - esq) * z**2 - esq * Esq
C = (esq**2 * F * r**2) / (G**3)
S = np.cbrt(1 + C + np.sqrt(C**2 + 2 * C))
P = F / (3 * (S + 1 / S + 1)**2 * G**2)
Q = np.sqrt(1 + 2 * esq**2 * P)
r_0 = -(P * esq * r) / (1 + Q) + np.sqrt(0.5 * a**2 * (1 + 1.0 / Q) - P * (1 - esq) * z**2 / (Q * (1 + Q)) - 0.5 * P * r**2)
U = np.sqrt((r - esq * r_0)**2 + z**2)
V = np.sqrt((r - esq * r_0)**2 + (1 - esq) * z**2)
Z_0 = b**2 * z / (a * V)
h = U * (1 - b**2 / (a * V))
lat = np.arctan((z + e1sq * Z_0) / r)
lon = np.arctan2(y, x)
return np.array([np.degrees(lat), np.degrees(lon), h])
def euler2quat_single(euler):
"""
Convert Euler angles (roll, pitch, yaw) to a quaternion.
Rotation order: Z-Y-X (yaw, pitch, roll).
"""
phi, theta, psi = euler
c_phi, s_phi = np.cos(phi / 2), np.sin(phi / 2)
c_theta, s_theta = np.cos(theta / 2), np.sin(theta / 2)
c_psi, s_psi = np.cos(psi / 2), np.sin(psi / 2)
w = c_phi * c_theta * c_psi + s_phi * s_theta * s_psi
x = s_phi * c_theta * c_psi - c_phi * s_theta * s_psi
y = c_phi * s_theta * c_psi + s_phi * c_theta * s_psi
z = c_phi * c_theta * s_psi - s_phi * s_theta * c_psi
if w < 0:
return np.array([-w, -x, -y, -z])
return np.array([w, x, y, z])
def quat2euler_single(q):
"""
Convert a quaternion to Euler angles (roll, pitch, yaw).
"""
w, x, y, z = q
gamma = np.arctan2(2 * (w * x + y * z), 1 - 2 * (x**2 + y**2))
sin_arg = 2 * (w * y - z * x)
sin_arg = np.clip(sin_arg, -1.0, 1.0)
theta = np.arcsin(sin_arg)
psi = np.arctan2(2 * (w * z + x * y), 1 - 2 * (y**2 + z**2))
return np.array([gamma, theta, psi])
def quat2rot_single(q):
"""
Convert a quaternion to a 3x3 rotation matrix.
"""
w, x, y, z = q
xx, yy, zz = x * x, y * y, z * z
xy, xz, yz = x * y, x * z, y * z
wx, wy, wz = w * x, w * y, w * z
mat = np.array([
[1 - 2 * (yy + zz), 2 * (xy - wz), 2 * (xz + wy)],
[2 * (xy + wz), 1 - 2 * (xx + zz), 2 * (yz - wx)],
[2 * (xz - wy), 2 * (yz + wx), 1 - 2 * (xx + yy)]
])
return mat
def rot2quat_single(rot):
"""
Convert a 3x3 rotation matrix to a quaternion.
"""
trace = np.trace(rot)
if trace > 0:
s = 0.5 / np.sqrt(trace + 1.0)
w = 0.25 / s
x = (rot[2, 1] - rot[1, 2]) * s
y = (rot[0, 2] - rot[2, 0]) * s
z = (rot[1, 0] - rot[0, 1]) * s
else:
if rot[0, 0] > rot[1, 1] and rot[0, 0] > rot[2, 2]:
s = 2.0 * np.sqrt(1.0 + rot[0, 0] - rot[1, 1] - rot[2, 2])
w = (rot[2, 1] - rot[1, 2]) / s
x = 0.25 * s
y = (rot[0, 1] + rot[1, 0]) / s
z = (rot[0, 2] + rot[2, 0]) / s
elif rot[1, 1] > rot[2, 2]:
s = 2.0 * np.sqrt(1.0 + rot[1, 1] - rot[0, 0] - rot[2, 2])
w = (rot[0, 2] - rot[2, 0]) / s
x = (rot[0, 1] + rot[1, 0]) / s
y = 0.25 * s
z = (rot[1, 2] + rot[2, 1]) / s
else:
s = 2.0 * np.sqrt(1.0 + rot[2, 2] - rot[0, 0] - rot[1, 1])
w = (rot[1, 0] - rot[0, 1]) / s
x = (rot[0, 2] + rot[2, 0]) / s
y = (rot[1, 2] + rot[2, 1]) / s
z = 0.25 * s
if w < 0:
return np.array([-w, -x, -y, -z])
return np.array([w, x, y, z])
def euler2rot_single(euler):
"""
Convert Euler angles (roll, pitch, yaw) to a 3x3 rotation matrix.
Rotation order: Z-Y-X (yaw, pitch, roll).
"""
phi, theta, psi = euler
cx, sx = np.cos(phi), np.sin(phi)
cy, sy = np.cos(theta), np.sin(theta)
cz, sz = np.cos(psi), np.sin(psi)
Rx = np.array([[1, 0, 0], [0, cx, -sx], [0, sx, cx]])
Ry = np.array([[cy, 0, sy], [0, 1, 0], [-sy, 0, cy]])
Rz = np.array([[cz, -sz, 0], [sz, cz, 0], [0, 0, 1]])
return Rz @ Ry @ Rx
def rot2euler_single(rot):
"""
Convert a 3x3 rotation matrix to Euler angles (roll, pitch, yaw).
"""
return quat2euler_single(rot2quat_single(rot))
def rot_matrix(roll, pitch, yaw):
"""
Create a 3x3 rotation matrix from roll, pitch, and yaw angles.
"""
return euler2rot_single([roll, pitch, yaw])
def axis_angle_to_rot(axis, angle):
"""
Convert an axis-angle representation to a 3x3 rotation matrix.
"""
c = np.cos(angle / 2)
s = np.sin(angle / 2)
q = np.array([c, s*axis[0], s*axis[1], s*axis[2]])
return quat2rot_single(q)
class LocalCoord:
"""
A class to handle conversions between ECEF and local NED coordinates.
"""
def __init__(self, geodetic=None, ecef=None):
"""
Initialize LocalCoord with either geodetic or ECEF coordinates.
"""
if geodetic is not None:
self.init_ecef = geodetic2ecef_single(geodetic)
lat, lon, _ = geodetic
elif ecef is not None:
self.init_ecef = np.array(ecef)
lat, lon, _ = ecef2geodetic_single(ecef)
else:
raise ValueError("Must provide geodetic or ecef")
lat = np.radians(lat)
lon = np.radians(lon)
self.ned2ecef_matrix = np.array([
[-np.sin(lat) * np.cos(lon), -np.sin(lon), -np.cos(lat) * np.cos(lon)],
[-np.sin(lat) * np.sin(lon), np.cos(lon), -np.cos(lat) * np.sin(lon)],
[np.cos(lat), 0, -np.sin(lat)]
])
self.ecef2ned_matrix = self.ned2ecef_matrix.T
@classmethod
def from_geodetic(cls, geodetic):
"""
Create a LocalCoord instance from geodetic coordinates.
"""
return cls(geodetic=geodetic)
@classmethod
def from_ecef(cls, ecef):
"""
Create a LocalCoord instance from ECEF coordinates.
"""
return cls(ecef=ecef)
def ecef2ned_single(self, ecef):
"""
Convert a single ECEF point to NED coordinates relative to the origin.
"""
return self.ecef2ned_matrix @ (ecef - self.init_ecef)
def ned2ecef_single(self, ned):
"""
Convert a single NED point to ECEF coordinates.
"""
return self.ned2ecef_matrix @ ned + self.init_ecef
def geodetic2ned_single(self, geodetic):
"""
Convert a single geodetic point to NED coordinates.
"""
ecef = geodetic2ecef_single(geodetic)
return self.ecef2ned_single(ecef)
def ned2geodetic_single(self, ned):
"""
Convert a single NED point to geodetic coordinates.
"""
ecef = self.ned2ecef_single(ned)
return ecef2geodetic_single(ecef)
@property
def ned_from_ecef_matrix(self):
"""
Returns the rotation matrix from ECEF to NED coordinates.
"""
return self.ecef2ned_matrix
@property
def ecef_from_ned_matrix(self):
"""
Returns the rotation matrix from NED to ECEF coordinates.
"""
return self.ned2ecef_matrix
def ecef_euler_from_ned_single(ecef_init, ned_pose):
"""
Convert NED Euler angles (roll, pitch, yaw) at a given ECEF origin
to equivalent ECEF Euler angles.
"""
converter = LocalCoord(ecef=ecef_init)
zero = np.array(ecef_init)
x0 = converter.ned2ecef_single([1, 0, 0]) - zero
y0 = converter.ned2ecef_single([0, 1, 0]) - zero
z0 = converter.ned2ecef_single([0, 0, 1]) - zero
phi, theta, psi = ned_pose
x1 = axis_angle_to_rot(z0, psi) @ x0
y1 = axis_angle_to_rot(z0, psi) @ y0
z1 = axis_angle_to_rot(z0, psi) @ z0
x2 = axis_angle_to_rot(y1, theta) @ x1
y2 = axis_angle_to_rot(y1, theta) @ y1
z2 = axis_angle_to_rot(y1, theta) @ z1
x3 = axis_angle_to_rot(x2, phi) @ x2
y3 = axis_angle_to_rot(x2, phi) @ y2
x0 = np.array([1.0, 0, 0])
y0 = np.array([0, 1.0, 0])
z0 = np.array([0, 0, 1.0])
psi_out = np.arctan2(np.dot(x3, y0), np.dot(x3, x0))
theta_out = np.arctan2(-np.dot(x3, z0), np.sqrt(np.dot(x3, x0)**2 + np.dot(x3, y0)**2))
y2 = axis_angle_to_rot(z0, psi_out) @ y0
z2 = axis_angle_to_rot(y2, theta_out) @ z0
phi_out = np.arctan2(np.dot(y3, z2), np.dot(y3, y2))
return np.array([phi_out, theta_out, psi_out])
def ned_euler_from_ecef_single(ecef_init, ecef_pose):
"""
Convert ECEF Euler angles (roll, pitch, yaw) at a given ECEF origin
to equivalent NED Euler angles.
"""
converter = LocalCoord(ecef=ecef_init)
x0 = np.array([1.0, 0, 0])
y0 = np.array([0, 1.0, 0])
z0 = np.array([0, 0, 1.0])
phi, theta, psi = ecef_pose
x1 = axis_angle_to_rot(z0, psi) @ x0
y1 = axis_angle_to_rot(z0, psi) @ y0
z1 = axis_angle_to_rot(z0, psi) @ z0
x2 = axis_angle_to_rot(y1, theta) @ x1
y2 = axis_angle_to_rot(y1, theta) @ y1
z2 = axis_angle_to_rot(y1, theta) @ z1
x3 = axis_angle_to_rot(x2, phi) @ x2
y3 = axis_angle_to_rot(x2, phi) @ y2
zero = np.array(ecef_init)
x0 = converter.ned2ecef_single([1, 0, 0]) - zero
y0 = converter.ned2ecef_single([0, 1, 0]) - zero
z0 = converter.ned2ecef_single([0, 0, 1]) - zero
psi_out = np.arctan2(np.dot(x3, y0), np.dot(x3, x0))
theta_out = np.arctan2(-np.dot(x3, z0), np.sqrt(np.dot(x3, x0)**2 + np.dot(x3, y0)**2))
y2 = axis_angle_to_rot(z0, psi_out) @ y0
z2 = axis_angle_to_rot(y2, theta_out) @ z0
phi_out = np.arctan2(np.dot(y3, z2), np.dot(y3, y2))
return np.array([phi_out, theta_out, psi_out])
-173
View File
@@ -1,173 +0,0 @@
# distutils: language = c++
# cython: language_level = 3
from openpilot.common.transformations.transformations cimport Matrix3, Vector3, Quaternion
from openpilot.common.transformations.transformations cimport ECEF, NED, Geodetic
from openpilot.common.transformations.transformations cimport euler2quat as euler2quat_c
from openpilot.common.transformations.transformations cimport quat2euler as quat2euler_c
from openpilot.common.transformations.transformations cimport quat2rot as quat2rot_c
from openpilot.common.transformations.transformations cimport rot2quat as rot2quat_c
from openpilot.common.transformations.transformations cimport euler2rot as euler2rot_c
from openpilot.common.transformations.transformations cimport rot2euler as rot2euler_c
from openpilot.common.transformations.transformations cimport rot_matrix as rot_matrix_c
from openpilot.common.transformations.transformations cimport ecef_euler_from_ned as ecef_euler_from_ned_c
from openpilot.common.transformations.transformations cimport ned_euler_from_ecef as ned_euler_from_ecef_c
from openpilot.common.transformations.transformations cimport geodetic2ecef as geodetic2ecef_c
from openpilot.common.transformations.transformations cimport ecef2geodetic as ecef2geodetic_c
from openpilot.common.transformations.transformations cimport LocalCoord_c
import numpy as np
cimport numpy as np
cdef np.ndarray[double, ndim=2] matrix2numpy(Matrix3 m):
return np.array([
[m(0, 0), m(0, 1), m(0, 2)],
[m(1, 0), m(1, 1), m(1, 2)],
[m(2, 0), m(2, 1), m(2, 2)],
])
cdef Matrix3 numpy2matrix(np.ndarray[double, ndim=2, mode="fortran"] m):
assert m.shape[0] == 3
assert m.shape[1] == 3
return Matrix3(<double*>m.data)
cdef ECEF list2ecef(ecef):
cdef ECEF e
e.x = ecef[0]
e.y = ecef[1]
e.z = ecef[2]
return e
cdef NED list2ned(ned):
cdef NED n
n.n = ned[0]
n.e = ned[1]
n.d = ned[2]
return n
cdef Geodetic list2geodetic(geodetic):
cdef Geodetic g
g.lat = geodetic[0]
g.lon = geodetic[1]
g.alt = geodetic[2]
return g
def euler2quat_single(euler):
cdef Vector3 e = Vector3(euler[0], euler[1], euler[2])
cdef Quaternion q = euler2quat_c(e)
return [q.w(), q.x(), q.y(), q.z()]
def quat2euler_single(quat):
cdef Quaternion q = Quaternion(quat[0], quat[1], quat[2], quat[3])
cdef Vector3 e = quat2euler_c(q)
return [e(0), e(1), e(2)]
def quat2rot_single(quat):
cdef Quaternion q = Quaternion(quat[0], quat[1], quat[2], quat[3])
cdef Matrix3 r = quat2rot_c(q)
return matrix2numpy(r)
def rot2quat_single(rot):
cdef Matrix3 r = numpy2matrix(np.asfortranarray(rot, dtype=np.double))
cdef Quaternion q = rot2quat_c(r)
return [q.w(), q.x(), q.y(), q.z()]
def euler2rot_single(euler):
cdef Vector3 e = Vector3(euler[0], euler[1], euler[2])
cdef Matrix3 r = euler2rot_c(e)
return matrix2numpy(r)
def rot2euler_single(rot):
cdef Matrix3 r = numpy2matrix(np.asfortranarray(rot, dtype=np.double))
cdef Vector3 e = rot2euler_c(r)
return [e(0), e(1), e(2)]
def rot_matrix(roll, pitch, yaw):
return matrix2numpy(rot_matrix_c(roll, pitch, yaw))
def ecef_euler_from_ned_single(ecef_init, ned_pose):
cdef ECEF init = list2ecef(ecef_init)
cdef Vector3 pose = Vector3(ned_pose[0], ned_pose[1], ned_pose[2])
cdef Vector3 e = ecef_euler_from_ned_c(init, pose)
return [e(0), e(1), e(2)]
def ned_euler_from_ecef_single(ecef_init, ecef_pose):
cdef ECEF init = list2ecef(ecef_init)
cdef Vector3 pose = Vector3(ecef_pose[0], ecef_pose[1], ecef_pose[2])
cdef Vector3 e = ned_euler_from_ecef_c(init, pose)
return [e(0), e(1), e(2)]
def geodetic2ecef_single(geodetic):
cdef Geodetic g = list2geodetic(geodetic)
cdef ECEF e = geodetic2ecef_c(g)
return [e.x, e.y, e.z]
def ecef2geodetic_single(ecef):
cdef ECEF e = list2ecef(ecef)
cdef Geodetic g = ecef2geodetic_c(e)
return [g.lat, g.lon, g.alt]
cdef class LocalCoord:
cdef LocalCoord_c * lc
def __init__(self, geodetic=None, ecef=None):
assert (geodetic is not None) or (ecef is not None)
if geodetic is not None:
self.lc = new LocalCoord_c(list2geodetic(geodetic))
elif ecef is not None:
self.lc = new LocalCoord_c(list2ecef(ecef))
@property
def ned2ecef_matrix(self):
return matrix2numpy(self.lc.ned2ecef_matrix)
@property
def ecef2ned_matrix(self):
return matrix2numpy(self.lc.ecef2ned_matrix)
@property
def ned_from_ecef_matrix(self):
return self.ecef2ned_matrix
@property
def ecef_from_ned_matrix(self):
return self.ned2ecef_matrix
@classmethod
def from_geodetic(cls, geodetic):
return cls(geodetic=geodetic)
@classmethod
def from_ecef(cls, ecef):
return cls(ecef=ecef)
def ecef2ned_single(self, ecef):
assert self.lc
cdef ECEF e = list2ecef(ecef)
cdef NED n = self.lc.ecef2ned(e)
return [n.n, n.e, n.d]
def ned2ecef_single(self, ned):
assert self.lc
cdef NED n = list2ned(ned)
cdef ECEF e = self.lc.ned2ecef(n)
return [e.x, e.y, e.z]
def geodetic2ned_single(self, geodetic):
assert self.lc
cdef Geodetic g = list2geodetic(geodetic)
cdef NED n = self.lc.geodetic2ned(g)
return [n.n, n.e, n.d]
def ned2geodetic_single(self, ned):
assert self.lc
cdef NED n = list2ned(ned)
cdef Geodetic g = self.lc.ned2geodetic(n)
return [g.lat, g.lon, g.alt]
def __dealloc__(self):
del self.lc

Some files were not shown because too many files have changed in this diff Show More