Compare commits

..

1 Commits

Author SHA1 Message Date
github-actions[bot] 1f7233cb98 sunnypilot v0.10.1
version: sunnypilot v0.10.1 (staging-tici)
date: 2025-10-13T01:35:37
master commit: 737a6c4236
2025-10-13 01:35:37 +00:00
3820 changed files with 1581295 additions and 116772 deletions
+19
View File
@@ -0,0 +1,19 @@
---
Checks: '
bugprone-*,
-bugprone-integer-division,
-bugprone-narrowing-conversions,
performance-*,
clang-analyzer-*,
misc-*,
-misc-unused-parameters,
modernize-*,
-modernize-avoid-c-arrays,
-modernize-deprecated-headers,
-modernize-use-auto,
-modernize-use-using,
-modernize-use-nullptr,
-modernize-use-trailing-return-type,
'
CheckOptions:
...
-1
View File
@@ -3,4 +3,3 @@ REGIST
PullRequest
cancelled
FOF
NoO
+39
View File
@@ -0,0 +1,39 @@
**/.git
.DS_Store
*.dylib
*.DSYM
*.d
*.pyc
*.pyo
.*.swp
.*.swo
.*.un~
*.tmp
*.o
*.o-*
*.os
*.os-*
*.so
*.a
venv/
.venv/
notebooks
phone
massivemap
neos
installer
chffr/app2
chffr/backend/env
selfdrive/nav
selfdrive/baseui
selfdrive/test/simulator2
**/cache_data
xx/plus
xx/community
xx/projects
!xx/projects/eon_testing_master
!xx/projects/map3d
xx/ops
xx/junk
-14
View File
@@ -1,14 +0,0 @@
* text=auto
# to move existing files into LFS:
# git add --renormalize .
*.onnx filter=lfs diff=lfs merge=lfs -text
*.svg filter=lfs diff=lfs merge=lfs -text
*.png filter=lfs diff=lfs merge=lfs -text
*.gif filter=lfs diff=lfs merge=lfs -text
*.ttf filter=lfs diff=lfs merge=lfs -text
*.otf filter=lfs diff=lfs merge=lfs -text
*.wav filter=lfs diff=lfs merge=lfs -text
selfdrive/car/tests/test_models_segs.txt filter=lfs diff=lfs merge=lfs -text
common/hardware/tici/updater filter=lfs diff=lfs merge=lfs -text
-11
View File
@@ -1,11 +0,0 @@
* @sunnypilot/dev-internal
/.github/ @devtekve @sunnyhaibin
/release/ci/ @devtekve @sunnyhaibin
/tinygrad_repo @devtekve @Discountchubbs
/tinygrad/ @devtekve @Discountchubbs
/selfdrive/controls/lib/longitudinal_planner.py @devtekve @Discountchubbs
/selfdrive/controls/lib/longitudinal_mpc_lib/long_mpc.py @devtekve @Discountchubbs
/selfdrive/modeld/ @devtekve @Discountchubbs
/sunnypilot/model* @devtekve @Discountchubbs
/sunnypilot/sunnylink/ @devtekve
/system/athena/ @devtekve
-47
View File
@@ -1,47 +0,0 @@
name: Bug report
description: For issues with running openpilot on your comma device
labels: ["bug"]
body:
- type: markdown
attributes:
value: >
Before creating a **bug report**, please check the following:
* If the issue likely only affects your car model or make, go back and open a **car bug report** instead.
* If the issue is related to the driving or driver monitoring models, you should open a [discussion](https://github.com/commaai/openpilot/discussions/categories/model-feedback) instead.
* Ensure you're running the latest openpilot release.
* Ensure you're using officially supported hardware. Issues running on PCs have a different issue template.
* Ensure there isn't an existing issue for your bug. If there is, leave a comment on the existing issue.
* Ensure you're running stock openpilot. We cannot look into bug reports from forks.
If you're unsure whether you've hit a bug, check out the #installation-help channel in the [community Discord server](https://discord.comma.ai).
- type: textarea
attributes:
label: Describe the bug
description: Also include a description of how to reproduce the bug
validations:
required: true
- type: input
id: route
attributes:
label: Provide a route where the issue occurs
description: Ensure the route is fully uploaded at https://useradmin.comma.ai. We cannot look into issues without routes, or at least a Dongle ID.
placeholder: 77611a1fac303767|2020-05-11--16-37-07
validations:
required: true
- type: input
id: version
attributes:
label: openpilot version
description: If you're not on release, provide the commit hash
placeholder: 0.8.10
validations:
required: true
- type: textarea
attributes:
label: Additional info
-14
View File
@@ -1,14 +0,0 @@
blank_issues_enabled: false
contact_links:
- name: Car bug report
url: https://github.com/commaai/opendbc/issues/new
about: For issues with a particular car make or model
- name: Join the Discord
url: https://discord.comma.ai
about: The community Discord is for both openpilot development and experience discussion
- name: Report driving behavior feedback
url: https://discord.com/channels/469524606043160576/1254834193066623017
about: Feedback for the driving and driver monitoring models goes in the #driving-feedback in Discord
- name: Community Wiki
url: https://github.com/commaai/openpilot/wiki
about: Check out our community wiki
-42
View File
@@ -1,42 +0,0 @@
name: PC bug report
description: For issues with running openpilot on PC
labels: ["PC"]
body:
- type: markdown
attributes:
value: >
Before creating a **bug report**, please check the following:
* Ensure you're running the latest openpilot release.
* Ensure there isn't an existing issue for your bug. If there is, leave a comment on the existing issue.
* Ensure you're running stock openpilot. We cannot look into bug reports from forks.
If you're unsure whether you've hit a bug, check out the #installation-help channel in the [community Discord server](https://discord.comma.ai).
- type: textarea
attributes:
label: Describe the bug
description: Also include a description of how to reproduce the bug
validations:
required: true
- type: input
id: os-version
attributes:
label: OS Version
placeholder: Ubuntu 24.04
validations:
required: true
- type: input
id: version
attributes:
label: openpilot version or commit
placeholder: bd36f2ec8d3559909678eff2690c10a520938367
validations:
required: false
- type: textarea
attributes:
label: Additional info
-31
View File
@@ -1,31 +0,0 @@
ci:
- changed-files:
- any-glob-to-all-files: "{.github/**,**/test_*,**/test/**,Jenkinsfile}"
chore:
- changed-files:
- any-glob-to-all-files: "{.github/**}"
car:
- changed-files:
- any-glob-to-all-files: '{selfdrive/car/**,opendbc_repo}'
simulation:
- changed-files:
- any-glob-to-all-files: 'tools/sim/**'
ui:
- changed-files:
- any-glob-to-all-files: '{selfdrive/assets/**,selfdrive/ui/**,system/ui/**}'
tools:
- changed-files:
- any-glob-to-all-files: 'tools/**'
multilanguage:
- changed-files:
- any-glob-to-all-files: 'selfdrive/ui/translations/**'
autonomy:
- changed-files:
- any-glob-to-all-files: "{selfdrive/modeld/models/**,selfdrive/test/process_replay/model_replay_ref_commit,sunnypilot/modeld*/models/**}"
-68
View File
@@ -1,68 +0,0 @@
<!-- Please copy and paste the relevant template -->
<!--- ***** Template: Fingerprint *****
**Car**
Which car (make, model, year) this fingerprint is for
**Route**
A route with the fingerprint
-->
<!--- ***** Template: Car Bugfix *****
**Description**
A description of the bug and the fix. Also link the issue if it exists.
**Verification**
Explain how you tested this bug fix.
**Route**
Route: [a route with the bug fix]
-->
<!--- ***** Template: Bugfix *****
**Description**
A description of the bug and the fix. Also link the issue if it exists.
**Verification**
Explain how you tested this bug fix.
-->
<!--- ***** Template: Car Port *****
**Checklist**
- [ ] added entry to CAR in selfdrive/car/*/values.py and ran `selfdrive/car/docs.py` to generate new docs
- [ ] test route added to [routes.py](https://github.com/commaai/openpilot/blob/master/selfdrive/car/tests/routes.py)
- [ ] route with openpilot:
- [ ] route with stock system:
- [ ] car harness used (if comma doesn't sell it, put N/A):
-->
<!--- ***** Template: Refactor *****
**Description**
A description of the refactor, including the goals it accomplishes.
**Verification**
Explain how you tested the refactor for regressions.
-->
-43
View File
@@ -1,43 +0,0 @@
exclude-labels:
- 'no-changelog'
categories:
- title: '🚀 Features'
labels:
- 'feature'
- 'enhancement'
- title: '🐛 Bug Fixes'
collapse-after: 5
labels:
- 'fix'
- 'bugfix'
- 'bug'
- title: '🧰 Maintenance'
collapse-after: 5
label: 'chore'
change-template: '- $TITLE @$AUTHOR (#$NUMBER)'
change-title-escapes: '\<*_&'
replacers:
- search: '/[Ss][Uu][Nn][Nn][Yy][Pp][Ii][Ll][Oo][Tt]/g'
replace: 'sunnypilot'
- search: '/\b[Ss][Pp]\b/g'
replace: 'SP'
version-resolver:
major:
labels:
- 'major'
minor:
labels:
- 'minor'
patch:
labels:
- 'patch'
default: patch
name-template: 'v$RESOLVED_VERSION 🚀'
tag-template: 'v$RESOLVED_VERSION'
version-template: "0.$MAJOR.$MINOR.$PATCH" # The day OP becomes v1, we need to bump this
tag-prefix: "v0." # The day OP becomes v1, we need to bump this
prerelease-identifier: "staging"
template: |
## Changes
$CHANGES
-91
View File
@@ -1,91 +0,0 @@
name: "PR review"
on:
pull_request_target:
types: [ opened, reopened, synchronize, edited ]
jobs:
labeler:
name: review
permissions:
contents: read
pull-requests: write
issues: write
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
submodules: false
# Label PRs
- uses: actions/labeler@v6
with:
dot: true
configuration-path: .github/labeler.yaml
# Check PR target branch
- name: check branch
uses: Vankka/pr-target-branch-action@def32ec9d93514138d6ac0132ee62e120a72aed5
if: github.repository == 'sunnypilot/sunnypilot'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
target: /^(?!master$).*/
exclude: /sunnypilot:.*/
change-to: ${{ github.base_ref }}
already-exists-action: close_this
already-exists-comment: "Your PR should be made against the `master` branch"
update-pr-labels:
name: Update fork's PR Labels
runs-on: ubuntu-latest
if: (github.event.pull_request.head.repo.fork && (contains(github.event_name, 'pull_request') && github.event.action == 'synchronize'))
env:
PR_LABEL: 'dev'
TRUST_FORK_PR_LABEL: 'trust-fork-pr'
steps:
- name: Check if PR has dev label
id: check-labels
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const prNumber = context.payload.pull_request.number;
const { data: labels } = await github.rest.issues.listLabelsOnIssue({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNumber
});
const hasDevC3Label = labels.some(label => label.name === process.env.PR_LABEL);
const hasTrustLabel = labels.some(label => label.name === process.env.TRUST_FORK_PR_LABEL);
console.log(`PR #${prNumber} has ${process.env.PR_LABEL} label: ${hasDevC3Label}`);
console.log(`PR #${prNumber} has ${process.env.TRUST_FORK_PR_LABEL} label: ${hasTrustLabel}`);
core.setOutput('has-dev', hasDevC3Label ? 'true' : 'false');
core.setOutput('has-trust', hasTrustLabel ? 'true' : 'false');
- name: Remove trust-fork-pr label if present
if: steps.check-labels.outputs.has-dev == 'true' && steps.check-labels.outputs.has-trust == 'true'
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const prNumber = context.payload.pull_request.number;
await github.rest.issues.removeLabel({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNumber,
name: process.env.TRUST_FORK_PR_LABEL
});
console.log(`Removed '${process.env.TRUST_FORK_PR_LABEL}' label from PR #${prNumber} as it received new commits`);
// Add a comment to the PR
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNumber,
body: `The \`${process.env.TRUST_FORK_PR_LABEL}\` label has been automatically removed because new commits were pushed to this PR. This PR will need to be re-reviewed before the label can be applied again.`
});
-35
View File
@@ -1,35 +0,0 @@
name: badges
on:
schedule:
- cron: '0 * * * *'
workflow_dispatch:
env:
PYTHONPATH: ${{ github.workspace }}
jobs:
badges:
name: create badges
runs-on: ubuntu-latest
if: github.repository == 'sunnypilot/sunnypilot'
permissions:
contents: write
steps:
- uses: actions/checkout@v6
with:
submodules: true
- run: ./tools/op.sh setup
- name: Push badges
run: |
python3 selfdrive/ui/translations/create_badges.py
rm .gitattributes
git checkout --orphan badges
git rm -rf --cached .
git config user.email "badge-researcher@sunnypilot.ai"
git config user.name "Badge Researcher"
git add translation_badge.svg
git commit -m "Add/Update badges"
git push -f origin HEAD
@@ -1,306 +0,0 @@
name: Build and push all tinygrad models
on:
workflow_dispatch:
inputs:
set_min_version:
description: 'Minimum selector version required for the models (see helpers.py or readme.md)'
required: true
type: string
jobs:
setup:
runs-on: ubuntu-latest
outputs:
json_version: ${{ steps.get-json.outputs.json_version }}
recompiled_dir: ${{ steps.create-recompiled-dir.outputs.recompiled_dir }}
json_file: ${{ steps.get-json.outputs.json_file }}
model_matrix: ${{ steps.set-matrix.outputs.model_matrix }}
tinygrad_ref: ${{ steps.get-tinygrad-ref.outputs.tinygrad_ref }}
steps:
- name: Checkout sunnypilot repo
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot
path: sunnypilot
submodules: recursive
- name: Get tinygrad_repo ref
id: get-tinygrad-ref
run: |
cd sunnypilot
export PYTHONPATH=$(pwd)
ref=$(python3 sunnypilot/models/tinygrad_ref.py)
echo "tinygrad_ref=$ref" >> $GITHUB_OUTPUT
echo "tinygrad_ref is $ref"
- name: Checkout docs repo (sunnypilot-models, gh-pages)
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot-models
ref: gh-pages
path: docs
ssh-key: ${{ secrets.CI_SUNNYPILOT_DOCS_PRIVATE_KEY }}
- name: Get next JSON version to use (from GitHub docs repo)
id: get-json
run: |
cd docs/docs
latest=$(ls driving_models_v*.json | sed -E 's/.*_v([0-9]+)\.json/\1/' | sort -n | tail -1)
next=$((latest+1))
json_file="driving_models_v${next}.json"
cp "driving_models_v${latest}.json" "$json_file"
echo "json_file=docs/docs/$json_file" >> $GITHUB_OUTPUT
echo "json_version=$((next+0))" >> $GITHUB_OUTPUT
echo "SRC_JSON_FILE=docs/docs/driving_models_v${latest}.json" >> $GITHUB_ENV
- name: Extract tinygrad models
id: set-matrix
working-directory: docs/docs
run: |
jq -c '[.bundles[] | select(.runner=="tinygrad") | {ref, display_name: (.display_name | gsub(" \\([^)]*\\)"; "")), is_20hz}]' "$(basename "${SRC_JSON_FILE}")" > matrix.json
echo "model_matrix=$(cat matrix.json)" >> $GITHUB_OUTPUT
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.GITLAB_SSH_PRIVATE_KEY }}
- run: |
mkdir -p ~/.ssh
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Clone GitLab docs repo and create new recompiled dir
id: create-recompiled-dir
env:
GIT_SSH_COMMAND: 'ssh -o UserKnownHostsFile=~/.ssh/known_hosts'
run: |
git clone --depth 1 --filter=tree:0 --sparse git@gitlab.com:sunnypilot/public/${{ vars.MODELS_GITLAB }} gitlab_docs
cd gitlab_docs
git checkout main
git sparse-checkout set --no-cone models/
cd models
latest_dir=$(ls -d recompiled* 2>/dev/null | sed -E 's/recompiled([0-9]+)/\1/' | sort -n | tail -1)
if [[ -z "$latest_dir" ]]; then
next_dir=1
else
next_dir=$((latest_dir+1))
fi
recompiled_dir="${next_dir}"
mkdir -p "recompiled${recompiled_dir}"
touch "recompiled${recompiled_dir}/.gitkeep"
cd ../..
echo "recompiled_dir=$recompiled_dir" >> $GITHUB_OUTPUT
- name: Push empty recompiled dir to GitLab
run: |
cd gitlab_docs
git add models/recompiled${{ steps.create-recompiled-dir.outputs.recompiled_dir }}
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git commit -m "Add recompiled${{ steps.create-recompiled-dir.outputs.recompiled_dir }} for build-all" || echo "No changes to commit"
git push origin main
- name: Push new JSON to GitHub docs repo
run: |
cd docs
git pull origin gh-pages
git add docs/"$(basename ${{ steps.get-json.outputs.json_file }})"
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git commit -m "Add new ${{ steps.get-json.outputs.json_file }} for build-all" || echo "No changes to commit"
git push origin gh-pages
get_and_build:
needs: [setup]
strategy:
matrix:
model: ${{ fromJson(needs.setup.outputs.model_matrix) }}
fail-fast: false
uses: ./.github/workflows/build-single-tinygrad-model.yaml
with:
upstream_branch: ${{ matrix.model.ref }}
custom_name: ${{ matrix.model.display_name }}
is_20hz: ${{ matrix.model.is_20hz }}
recompiled_dir: ${{ needs.setup.outputs.recompiled_dir }}
json_version: ${{ needs.setup.outputs.json_version }}
secrets: inherit
retry_failed_models:
needs: [setup, get_and_build]
runs-on: ubuntu-latest
if: ${{ needs.setup.result != 'failure' && !cancelled() }}
outputs:
retry_matrix: ${{ steps.set-retry-matrix.outputs.retry_matrix }}
steps:
- uses: actions/download-artifact@v4
with:
pattern: model-*
path: output
- id: set-retry-matrix
run: |
echo '${{ needs.setup.outputs.model_matrix }}' > matrix.json
built=(); while IFS= read -r line; do built+=("$line"); done < <(
find output -maxdepth 1 -name 'model-*' -printf "%f\n" | sed -E 's/^model-//' | sed -E 's/-[0-9]+$//' | sed -E 's/ \([^)]*\)//' | awk '{gsub(/^ +| +$/, ""); print}'
)
jq -c --argjson built "$(printf '%s\n' "${built[@]}" | jq -R . | jq -s .)" \
'map(select(.display_name as $n | ($built | index($n | gsub("^ +| +$"; "")) | not)))' matrix.json > retry_matrix.json
echo "retry_matrix=$(cat retry_matrix.json)" >> $GITHUB_OUTPUT
retry_get_and_build:
needs: [setup, get_and_build, retry_failed_models]
if: ${{ needs.get_and_build.result == 'failure' || (needs.retry_failed_models.outputs.retry_matrix != '[]' && needs.retry_failed_models.outputs.retry_matrix != '') }}
strategy:
matrix:
model: ${{ fromJson(needs.retry_failed_models.outputs.retry_matrix) }}
fail-fast: false
uses: ./.github/workflows/build-single-tinygrad-model.yaml
with:
upstream_branch: ${{ matrix.model.ref }}
custom_name: ${{ matrix.model.display_name }}
is_20hz: ${{ matrix.model.is_20hz }}
recompiled_dir: ${{ needs.setup.outputs.recompiled_dir }}
json_version: ${{ needs.setup.outputs.json_version }}
artifact_suffix: -retry
secrets: inherit
publish_models:
name: Publish models sequentially
needs: [setup, get_and_build, retry_failed_models, retry_get_and_build]
if: ${{ !cancelled() && (needs.get_and_build.result != 'failure' || needs.retry_get_and_build.result == 'success' || (needs.retry_failed_models.outputs.retry_matrix != '[]' && needs.retry_failed_models.outputs.retry_matrix != '')) }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
max-parallel: 1
matrix:
model: ${{ fromJson(needs.setup.outputs.model_matrix) }}
env:
RECOMPILED_DIR: recompiled${{ needs.setup.outputs.recompiled_dir }}
JSON_FILE: ${{ needs.setup.outputs.json_file }}
ARTIFACT_NAME_INPUT: ${{ matrix.model.display_name }}
steps:
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.GITLAB_SSH_PRIVATE_KEY }}
- name: Add GitLab.com SSH key to known_hosts
run: |
mkdir -p ~/.ssh
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Clone GitLab docs repo
env:
GIT_SSH_COMMAND: 'ssh -o UserKnownHostsFile=~/.ssh/known_hosts'
run: |
echo "Cloning GitLab"
git clone --depth 1 --filter=tree:0 --sparse git@gitlab.com:sunnypilot/public/${{ vars.MODELS_GITLAB }} gitlab_docs
cd gitlab_docs
echo "checkout models/${RECOMPILED_DIR}"
git sparse-checkout set --no-cone models/${RECOMPILED_DIR}
git checkout main
cd ..
- name: Checkout docs repo
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot-models
ref: gh-pages
path: docs
ssh-key: ${{ secrets.CI_SUNNYPILOT_DOCS_PRIVATE_KEY }}
- name: Validate recompiled dir and JSON version
run: |
if [ ! -d "gitlab_docs/models/$RECOMPILED_DIR" ]; then
echo "Recompiled dir $RECOMPILED_DIR does not exist in GitLab repo"
exit 1
fi
if [ ! -f "$JSON_FILE" ]; then
echo "JSON file $JSON_FILE does not exist!"
exit 1
fi
- name: Download artifact name file
uses: actions/download-artifact@v4
with:
name: artifact-name-${{ env.ARTIFACT_NAME_INPUT }}
path: artifact_name
- name: Read artifact name
id: read-artifact-name
run: |
ARTIFACT_NAME=$(cat artifact_name/artifact_name.txt)
echo "artifact_name=$ARTIFACT_NAME" >> $GITHUB_OUTPUT
- name: Download model artifact
uses: actions/download-artifact@v4
with:
name: ${{ steps.read-artifact-name.outputs.artifact_name }}
path: output
- name: Remove onnx files bc not needed for recompiled dir since they already exist from single build
run: |
find output -type f -name '*.onnx' -delete
find output -type f -name 'big_*.pkl' -delete
find output -type f -name 'dmonitoring_model_tinygrad.pkl' -delete
- name: Copy model artifacts to gitlab
env:
ARTIFACT_NAME: ${{ steps.read-artifact-name.outputs.artifact_name }}
run: |
ARTIFACT_DIR="gitlab_docs/models/${RECOMPILED_DIR}/${ARTIFACT_NAME}"
mkdir -p "$ARTIFACT_DIR"
for path in output/*; do
if [ "$(basename "$path")" = "artifact_name.txt" ]; then
continue
fi
name="$(basename "$path")"
if [ -d "$path" ]; then
mkdir -p "$ARTIFACT_DIR/$name"
cp -r "$path"/* "$ARTIFACT_DIR/$name/"
echo "Copied dir $name -> $ARTIFACT_DIR/$name"
else
cp "$path" "$ARTIFACT_DIR/"
echo "Copied file $name -> $ARTIFACT_DIR/"
fi
done
- name: Push recompiled dir to GitLab
env:
GITLAB_SSH_PRIVATE_KEY: ${{ secrets.GITLAB_SSH_PRIVATE_KEY }}
run: |
cd gitlab_docs
git checkout main
git pull origin main
for d in models/"$RECOMPILED_DIR"/*/; do
git sparse-checkout add "$d"
done
git add models/"$RECOMPILED_DIR"
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git commit -m "Update $RECOMPILED_DIR with model from build-all-tinygrad-models" || echo "No changes to commit"
git push origin main
- run: |
cd docs
git pull origin gh-pages
- name: update json
run: |
ARGS=""
[ -n "${{ inputs.set_min_version }}" ] && ARGS="$ARGS --set-min-version \"${{ inputs.set_min_version }}\""
ARGS="$ARGS --sort-by-date"
ARGS="$ARGS --tinygrad-ref \"${{ needs.setup.outputs.tinygrad_ref }}\""
eval python3 docs/json_parser.py \
--json-path "$JSON_FILE" \
--recompiled-dir "gitlab_docs/models/$RECOMPILED_DIR" \
$ARGS
- name: Push updated json to GitHub
run: |
cd docs
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git checkout gh-pages
git add docs/"$(basename $JSON_FILE)"
git commit -m "Update $(basename $JSON_FILE) after recompiling model" || echo "No changes to commit"
git push origin gh-pages
@@ -1,238 +0,0 @@
name: Build Single Tinygrad Model and Push
on:
workflow_call:
inputs:
upstream_branch:
description: 'Upstream commit to build from'
required: true
type: string
custom_name:
description: 'Custom name for the model (no date, only name)'
required: false
type: string
recompiled_dir:
description: 'Existing recompiled directory number (e.g. 3 for recompiled3)'
required: true
type: string
json_version:
description: 'driving_models version number to update (e.g. 5 for driving_models_v5.json)'
required: true
type: string
artifact_suffix:
description: 'Suffix for artifact name'
required: false
type: string
default: ''
is_20hz:
description: 'Is this a 20Hz model'
required: false
type: boolean
default: true
bypass_push:
description: 'Bypass pushing to GitLab for build-all'
required: false
default: true
type: boolean
workflow_dispatch:
inputs:
upstream_branch:
description: 'Upstream commit to build from'
required: true
type: string
custom_name:
description: 'Custom name for the model (no date, only name)'
required: false
type: string
is_20hz:
description: 'Is this a 20Hz model'
required: false
type: boolean
default: true
recompiled_dir:
description: 'Existing recompiled directory number (e.g. 3 for recompiled3)'
required: true
type: string
json_version:
description: 'driving_models version number to update (e.g. 5 for driving_models_v5.json)'
required: true
type: string
model_folder:
description: 'Model folder'
type: choice
default: 'None'
options:
- None
- Simple Plan Models
- Space Lab Models
- TR Models
- DTR Models
- Custom Merge Models
- FOF series models
- Other
custom_model_folder:
description: 'Custom model folder name (if "Other" selected)'
required: false
type: string
generation:
description: 'Model generation'
required: false
type: string
version:
description: 'Minimum selector version'
required: false
type: string
env:
RECOMPILED_DIR: recompiled${{ inputs.recompiled_dir }}
JSON_FILE: docs/docs/driving_models_v${{ inputs.json_version }}.json
jobs:
build_model:
uses: ./.github/workflows/sunnypilot-build-model.yaml
with:
upstream_branch: ${{ inputs.upstream_branch }}
custom_name: ${{ inputs.custom_name || inputs.upstream_branch }}
is_20hz: ${{ inputs.is_20hz }}
artifact_suffix: ${{ inputs.artifact_suffix }}
secrets: inherit
publish_model:
if: ${{ !inputs.bypass_push && !cancelled() }}
concurrency:
group: gitlab-push-${{ inputs.recompiled_dir }}
cancel-in-progress: false
needs: build_model
runs-on: ubuntu-latest
steps:
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.GITLAB_SSH_PRIVATE_KEY }}
- name: Add GitLab.com SSH key to known_hosts
run: |
mkdir -p ~/.ssh
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Clone GitLab docs repo
env:
GIT_SSH_COMMAND: 'ssh -o UserKnownHostsFile=~/.ssh/known_hosts'
run: |
echo "Cloning GitLab"
git clone --depth 1 --filter=tree:0 --sparse git@gitlab.com:sunnypilot/public/${{ vars.MODELS_GITLAB }} gitlab_docs
cd gitlab_docs
echo "checkout models/${RECOMPILED_DIR}"
git sparse-checkout set --no-cone models/${RECOMPILED_DIR}
git checkout main
cd ..
- name: Checkout docs repo
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot-models
ref: gh-pages
path: docs
ssh-key: ${{ secrets.CI_SUNNYPILOT_DOCS_PRIVATE_KEY }}
- name: Validate recompiled dir and JSON version
run: |
if [ ! -d "gitlab_docs/models/$RECOMPILED_DIR" ]; then
echo "Recompiled dir $RECOMPILED_DIR does not exist in GitLab repo"
exit 1
fi
if [ ! -f "$JSON_FILE" ]; then
echo "JSON file $JSON_FILE does not exist!"
exit 1
fi
- name: Download artifact name file
uses: actions/download-artifact@v4
with:
name: artifact-name-${{ inputs.custom_name || inputs.upstream_branch }}
path: artifact_name
- name: Read artifact name
id: read-artifact-name
run: |
ARTIFACT_NAME=$(cat artifact_name/artifact_name.txt)
echo "artifact_name=$ARTIFACT_NAME" >> $GITHUB_OUTPUT
- name: Download and extract model artifact
uses: actions/download-artifact@v4
with:
name: ${{ steps.read-artifact-name.outputs.artifact_name }}
path: output
- name: Remove unwanted files
run: |
find output -type f -name 'dmonitoring_model_tinygrad.pkl' -delete
find output -type f -name 'dmonitoring_model.onnx' -delete
- name: Copy model artifact(s) to GitLab recompiled dir
env:
ARTIFACT_NAME: ${{ steps.read-artifact-name.outputs.artifact_name }}
run: |
ARTIFACT_DIR="gitlab_docs/models/${RECOMPILED_DIR}/${ARTIFACT_NAME}"
mkdir -p "$ARTIFACT_DIR"
for path in output/*; do
if [ "$(basename "$path")" = "artifact_name.txt" ]; then
continue
fi
name="$(basename "$path")"
if [ -d "$path" ]; then
mkdir -p "$ARTIFACT_DIR/$name"
cp -r "$path"/* "$ARTIFACT_DIR/$name/"
echo "Copied dir $name -> $ARTIFACT_DIR/$name"
else
cp "$path" "$ARTIFACT_DIR/"
echo "Copied file $name -> $ARTIFACT_DIR/"
fi
done
- name: Push recompiled dir to GitLab
env:
GITLAB_SSH_PRIVATE_KEY: ${{ secrets.GITLAB_SSH_PRIVATE_KEY }}
run: |
cd gitlab_docs
git checkout main
git pull origin main
for d in models/"$RECOMPILED_DIR"/*/; do
git sparse-checkout add "$d"
done
git add models/"$RECOMPILED_DIR"
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git commit -m "Create/Update $RECOMPILED_DIR with new/updated model from build-single-tinygrad-model" || echo "No changes to commit"
git push origin main
- run: |
cd docs
git pull origin gh-pages
- name: Run json_parser.py to update JSON
run: |
FOLDER="${{ inputs.model_folder }}"
if [ "$FOLDER" = "Other" ]; then
FOLDER="${{ inputs.custom_model_folder }}"
fi
ARGS=""
if [ "$FOLDER" != "None" ] && [ -n "$FOLDER" ]; then
ARGS="$ARGS --model-folder \"$FOLDER\""
fi
[ -n "${{ inputs.generation }}" ] && ARGS="$ARGS --generation \"${{ inputs.generation }}\""
[ -n "${{ inputs.version }}" ] && ARGS="$ARGS --version \"${{ inputs.version }}\""
eval python3 docs/json_parser.py \
--json-path "$JSON_FILE" \
--recompiled-dir "gitlab_docs/models/$RECOMPILED_DIR" \
--sort-by-date \
$ARGS
- name: Push updated JSON to GitHub docs repo
run: |
cd docs
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git checkout gh-pages
git add docs/"$(basename $JSON_FILE)"
git commit -m "Update $(basename $JSON_FILE) after recompiling model" || echo "No changes to commit"
git push origin gh-pages
-84
View File
@@ -1,84 +0,0 @@
name: cereal validation
on:
push:
branches:
- master
pull_request:
paths:
- 'cereal/**'
workflow_dispatch:
workflow_call:
inputs:
run_number:
default: '1'
required: true
type: string
concurrency:
group: cereal-validation-ci-run-${{ inputs.run_number }}-${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && github.run_id || github.head_ref || github.ref }}-${{ github.workflow }}-${{ github.event_name }}
cancel-in-progress: true
env:
CI: 1
jobs:
validate_cereal_with_upstream:
name: Validate cereal with Upstream
runs-on: ubuntu-24.04
steps:
- name: Checkout sunnypilot cereal
uses: actions/checkout@v6
with:
sparse-checkout: cereal
- name: Init sunnypilot opendbc submodule
run: git submodule update --init --depth 1 opendbc_repo
- name: Checkout upstream openpilot
uses: actions/checkout@v6
with:
repository: 'commaai/openpilot'
path: upstream_openpilot
ref: "refs/heads/master"
- name: Init upstream opendbc submodule
working-directory: upstream_openpilot
run: git submodule update --init --depth 1 opendbc_repo
- name: Locate upstream capnp paths
id: locate-capnp
run: |
CEREAL_DIR=$(find upstream_openpilot -maxdepth 4 -name log.capnp -path '*/cereal/log.capnp' -printf '%h\n' -quit)
if [ -z "$CEREAL_DIR" ]; then
echo "::error::Could not locate cereal/log.capnp in upstream openpilot"
exit 1
fi
echo "cereal_dir=$CEREAL_DIR" >> "$GITHUB_OUTPUT"
echo "Found upstream cereal at: $CEREAL_DIR"
IMPORT_ARGS=""
CAR_CAPNP=$(find upstream_openpilot -maxdepth 5 -name car.capnp -path '*/opendbc/car/car.capnp' -printf '%h\n' -quit)
if [ -n "$CAR_CAPNP" ]; then
IMPORT_ARGS="-I $CAR_CAPNP"
echo "Found car.capnp at: $CAR_CAPNP"
fi
echo "import_args=$IMPORT_ARGS" >> "$GITHUB_OUTPUT"
- name: Install uv
run: pip install uv
- name: Generate sunnypilot schema
run: |
PYCAPNP_VER=$(python3 -c "import re; m=re.search(r'name = \"pycapnp\"\nversion = \"([^\"]+)\"', open('uv.lock').read()); print(m.group(1))")
uv run --isolated --with "pycapnp==${PYCAPNP_VER}" \
python3 cereal/messaging/tests/validate_sp_cereal_upstream.py \
-g -f /tmp/sp_schema.json --cereal-dir cereal
- name: Validate against upstream
run: |
PYCAPNP_VER=$(python3 -c "import re; m=re.search(r'name = \"pycapnp\"\nversion = \"([^\"]+)\"', open('uv.lock').read()); print(m.group(1))")
uv run --isolated --with "pycapnp==${PYCAPNP_VER}" \
python3 cereal/messaging/tests/validate_sp_cereal_upstream.py \
-r -f /tmp/sp_schema.json --cereal-dir ${{ steps.locate-capnp.outputs.cereal_dir }} \
${{ steps.locate-capnp.outputs.import_args }}
-45
View File
@@ -1,45 +0,0 @@
name: diff report
on:
pull_request_target:
types: [opened, synchronize, reopened]
jobs:
comment:
name: comment
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
pull-requests: write
actions: read
steps:
- name: Wait for process replay
id: wait
continue-on-error: true
uses: lewagon/wait-on-check-action@v1.3.4
with:
ref: ${{ github.event.pull_request.head.sha }}
check-name: process replay
repo-token: ${{ secrets.GITHUB_TOKEN }}
allowed-conclusions: success,failure
wait-interval: 20
- name: Download diff
if: steps.wait.outcome == 'success'
uses: dawidd6/action-download-artifact@v6
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
workflow: tests.yaml
workflow_conclusion: ''
pr: ${{ github.event.number }}
name: diff_report_${{ github.event.number }}
path: .
allow_forks: true
- name: Comment on PR
if: steps.wait.outcome == 'success'
uses: thollander/actions-comment-pull-request@v2
with:
filePath: diff_report.txt
comment_tag: diff_report
pr_number: ${{ github.event.number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-65
View File
@@ -1,65 +0,0 @@
name: docs
on:
push:
branches:
- master
pull_request:
workflow_call:
inputs:
run_number:
default: '1'
required: true
type: string
concurrency:
group: docs-tests-ci-run-${{ inputs.run_number }}-${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && github.run_id || github.head_ref || github.ref }}-${{ github.workflow }}-${{ github.event_name }}
cancel-in-progress: true
jobs:
docs:
name: build docs
runs-on: ubuntu-24.04
steps:
- uses: commaai/timeout@v1
- uses: actions/checkout@v6
with:
submodules: true
# Build
- name: Build docs
run: |
git lfs pull
pip install zensical
python scripts/docs.py build
# Push to docs.comma.ai
- uses: actions/checkout@v6
if: github.ref == 'refs/heads/master' && github.repository == 'sunnypilot/sunnypilot'
with:
path: openpilot-docs
ssh-key: ${{ secrets.OPENPILOT_DOCS_KEY }}
repository: sunnypilot/sunnypilot-docs
- name: Push
if: github.ref == 'refs/heads/master' && github.repository == 'sunnypilot/sunnypilot'
run: |
set -x
source release/identity.sh
cd openpilot-docs
git checkout --orphan tmp
git rm -rf .
# copy over docs
cp -r ../docs_site/ docs/
# GitHub pages config
touch docs/.nojekyll
echo -n docs.comma.ai > docs/CNAME
git add -f .
git commit -m "build docs"
# docs live in different repo to not bloat openpilot's full clone size
git push -f origin tmp:gh-pages
-96
View File
@@ -1,96 +0,0 @@
name: jenkins scan
on:
issue_comment:
types: [created, edited]
jobs:
cleanup-branches:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Delete stale Jenkins branches
uses: actions/github-script@v8
with:
script: |
const cutoff = Date.now() - 24 * 60 * 60 * 1000;
const prefixes = ['tmp-jenkins', '__jenkins'];
for await (const response of github.paginate.iterator(github.rest.repos.listBranches, {
owner: context.repo.owner,
repo: context.repo.repo,
per_page: 100,
})) {
for (const branch of response.data) {
if (!prefixes.some(p => branch.name.startsWith(p))) continue;
const { data: commit } = await github.rest.repos.getCommit({
owner: context.repo.owner,
repo: context.repo.repo,
ref: branch.commit.sha,
});
const commitDate = new Date(commit.commit.committer.date).getTime();
if (commitDate < cutoff) {
console.log(`Deleting branch: ${branch.name} (last commit: ${commit.commit.committer.date})`);
await github.rest.git.deleteRef({
owner: context.repo.owner,
repo: context.repo.repo,
ref: `heads/${branch.name}`,
});
}
}
}
scan-comments:
runs-on: ubuntu-latest
if: ${{ github.event.issue.pull_request }}
permissions:
contents: write
issues: write
steps:
- name: Check for trigger phrase
id: check_comment
uses: actions/github-script@v8
with:
script: |
const triggerPhrase = "trigger-jenkins";
const comment = context.payload.comment.body;
const commenter = context.payload.comment.user.login;
const { data: permissions } = await github.rest.repos.getCollaboratorPermissionLevel({
owner: context.repo.owner,
repo: context.repo.repo,
username: commenter
});
const hasWriteAccess = permissions.permission === 'write' || permissions.permission === 'admin';
return (hasWriteAccess && comment.includes(triggerPhrase));
result-encoding: json
- name: Checkout repository
if: steps.check_comment.outputs.result == 'true'
uses: actions/checkout@v6
with:
ref: refs/pull/${{ github.event.issue.number }}/head
- name: Push to tmp-jenkins branch
if: steps.check_comment.outputs.result == 'true'
run: |
git config --global user.name "github-actions[bot]"
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git checkout -b tmp-jenkins-${{ github.event.issue.number }}
GIT_LFS_SKIP_PUSH=1 git push -f origin tmp-jenkins-${{ github.event.issue.number }}
- name: Delete trigger comment
if: steps.check_comment.outputs.result == 'true' && always()
uses: actions/github-script@v8
with:
script: |
await github.rest.issues.deleteComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: context.payload.comment.id,
});
-72
View File
@@ -1,72 +0,0 @@
name: Sync comma's LFS
env:
LFS_URL: 'https://gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git/info/lfs'
LFS_PUSH_URL: 'ssh://git@gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git'
on:
schedule:
- cron: '0 0 * * *' # Runs at 00:00 UTC every day
push:
branches:
- 'master'
pull_request:
branches:
- 'master'
workflow_dispatch: # enables manual triggering
inputs:
upstream_branch:
default: 'master'
type: string
jobs:
sync:
runs-on: ubuntu-latest
# Skip if PR is in draft mode
if: (github.event_name != 'pull_request' || (github.event_name == 'pull_request' && github.event.pull_request.draft == false)) && !github.event.pull_request.head.repo.fork
steps:
- name: Checkout Repository
uses: actions/checkout@v4
with:
repository: 'commaai/openpilot'
ref: ${{ inputs.upstream_branch }}
- name: LFS Fetch
run: |
git lfs fetch
- name: Set up Git
run: |
git config --global user.name 'GitHub Action'
git config --global user.email 'action@github.com'
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
- name: Add GitLab public keys
run: |
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Ensure branch
run: |
if git symbolic-ref -q HEAD >/dev/null; then
echo "Already on a branch, proceeding with push"
else
echo "Detached HEAD state detected, creating temporary branch"
git checkout -b temp_branch
fi
- name: Update LFS Config
run: |
echo '[lfs]' > .lfsconfig
echo ' url = ${{ env.LFS_URL }}' >> .lfsconfig
echo ' pushurl = ${{ env.LFS_PUSH_URL }}' >> .lfsconfig
echo ' locksverify = false' >> .lfsconfig
- name: Push LFS
id: sync-and-commit
run: |
git lfs ls-files -l
git lfs push --all origin
-42
View File
@@ -1,42 +0,0 @@
name: "model review"
on:
pull_request:
types: [opened, reopened, synchronize]
paths:
- 'selfdrive/modeld/models/*.onnx'
workflow_dispatch:
jobs:
comment:
permissions:
contents: read
pull-requests: write
runs-on: ubuntu-latest
if: github.repository == 'commaai/openpilot'
steps:
- name: Checkout
uses: actions/checkout@v6
with:
submodules: true
- name: Checkout master
uses: actions/checkout@v6
with:
ref: master
path: base
- run: git lfs pull
- run: cd base && git lfs pull
- name: scripts/reporter.py
id: report
run: |
echo "content<<EOF" >> $GITHUB_OUTPUT
echo "## Model Review" >> $GITHUB_OUTPUT
PYTHONPATH=${{ github.workspace }} MASTER_PATH=${{ github.workspace }}/base python scripts/reporter.py >> $GITHUB_OUTPUT
echo "EOF" >> $GITHUB_OUTPUT
- name: Post model report comment
uses: marocchino/sticky-pull-request-comment@baa7203ed60924babbe5dcd0ac8eae3b66ec5e16
with:
header: model-review
message: ${{ steps.report.outputs.content }}
@@ -1,105 +0,0 @@
name: 'Post to Discourse'
description: 'Posts a message to a Discourse topic (existing or new)'
inputs:
discourse-url:
description: 'Discourse instance URL (e.g., https://discourse.example.com)'
required: true
api-key:
description: 'Discourse API key'
required: true
api-username:
description: 'Discourse API username'
required: true
topic-id:
description: 'Discourse topic ID to post to (use this OR category-id + title)'
required: false
category-id:
description: 'Category ID for new topic (required if topic-id not provided)'
required: false
title:
description: 'Title for new topic (required if topic-id not provided)'
required: false
message:
description: 'Message content (markdown supported)'
required: true
outputs:
post-number:
description: 'The post number in the topic'
value: ${{ steps.post.outputs.post_number }}
post-url:
description: 'Direct URL to the post'
value: ${{ steps.post.outputs.post_url }}
topic-id:
description: 'The topic ID (useful when creating a new topic)'
value: ${{ steps.post.outputs.topic_id }}
runs:
using: "composite"
steps:
- name: Post to Discourse
id: post
shell: bash
run: |
# Validate inputs
if [ -z "${{ inputs.topic-id }}" ] && ([ -z "${{ inputs.category-id }}" ] || [ -z "${{ inputs.title }}" ]); then
echo "❌ Error: Must provide either topic-id OR both category-id and title"
exit 1
fi
if [ -n "${{ inputs.topic-id }}" ] && ([ -n "${{ inputs.category-id }}" ] || [ -n "${{ inputs.title }}" ]); then
echo "⚠️ Warning: Both topic-id and category-id/title provided. Will post to existing topic."
fi
# Determine if creating new topic or posting to existing
if [ -n "${{ inputs.topic-id }}" ]; then
echo "📝 Posting to existing topic ID: ${{ inputs.topic-id }}"
# Create JSON payload for posting to existing topic
PAYLOAD=$(jq -n \
--arg content '${{ inputs.message }}' \
--arg topic_id "${{ inputs.topic-id }}" \
'{topic_id: $topic_id, raw: $content}')
else
echo "✨ Creating new topic: ${{ inputs.title }}"
# Create JSON payload for new topic
PAYLOAD=$(jq -n \
--arg content '${{ inputs.message }}' \
--arg title "${{ inputs.title }}" \
--arg category "${{ inputs.category-id }}" \
'{title: $title, category: ($category | tonumber), raw: $content}')
fi
# Post to Discourse
RESPONSE=$(curl -s -w "\n%{http_code}" \
-X POST "${{ inputs.discourse-url }}/posts.json" \
-H "Content-Type: application/json" \
-H "Api-Key: ${{ inputs.api-key }}" \
-H "Api-Username: ${{ inputs.api-username }}" \
-d "$PAYLOAD")
HTTP_CODE=$(echo "$RESPONSE" | tail -n1)
BODY=$(echo "$RESPONSE" | sed '$d')
if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then
echo "✅ Successfully posted to Discourse!"
POST_NUMBER=$(echo "$BODY" | jq -r '.post_number // "unknown"')
TOPIC_ID=$(echo "$BODY" | jq -r '.topic_id // "${{ inputs.topic-id }}"')
POST_URL="${{ inputs.discourse-url }}/t/${TOPIC_ID}/${POST_NUMBER}"
echo "post_number=${POST_NUMBER}" >> $GITHUB_OUTPUT
echo "post_url=${POST_URL}" >> $GITHUB_OUTPUT
echo "topic_id=${TOPIC_ID}" >> $GITHUB_OUTPUT
echo "Topic ID: ${TOPIC_ID}"
echo "Post number: ${POST_NUMBER}"
echo "URL: ${POST_URL}"
else
echo "❌ Failed to post to Discourse"
echo "HTTP Code: ${HTTP_CODE}"
echo "Response: ${BODY}"
exit 1
fi
-39
View File
@@ -1,39 +0,0 @@
name: prebuilt
on:
schedule:
- cron: '0 * * * *'
workflow_dispatch:
env:
DOCKER_LOGIN: docker login ghcr.io -u ${{ github.actor }} -p ${{ secrets.GITHUB_TOKEN }}
BUILD: release/ci/docker_build_sp.sh
jobs:
build_prebuilt:
name: build prebuilt
runs-on: ubuntu-latest
if: github.repository == 'sunnypilot/sunnypilot'
env:
PUSH_IMAGE: true
permissions:
checks: read
contents: read
packages: write
steps:
- name: Wait for green check mark
if: ${{ github.event_name != 'workflow_dispatch' }}
uses: lewagon/wait-on-check-action@ccfb013c15c8afb7bf2b7c028fb74dc5a068cccc
with:
ref: master
wait-interval: 30
running-workflow-name: 'build prebuilt'
repo-token: ${{ secrets.GITHUB_TOKEN }}
check-regexp: ^((?!.*(build master-ci|create badges).*).)*$
- uses: actions/checkout@v6
with:
submodules: true
- run: git lfs pull
- name: Build and Push docker image
run: |
$DOCKER_LOGIN
eval "$BUILD"
-28
View File
@@ -1,28 +0,0 @@
name: Release Drafter
on:
push:
branches:
- master
tags:
- 'v*'
pull_request_target:
types: [opened, reopened, synchronize]
workflow_dispatch:
permissions:
contents: read
jobs:
update_release_draft:
permissions:
contents: write
pull-requests: write
runs-on: ubuntu-latest
steps:
- uses: release-drafter/release-drafter@v6
with:
config-name: release-drafter.yml
prerelease: ${{ !startsWith(github.ref, 'refs/tags/v') }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-31
View File
@@ -1,31 +0,0 @@
name: release
on:
schedule:
- cron: '0 9 * * *'
workflow_dispatch:
jobs:
build___nightly:
name: build __nightly
runs-on: ubuntu-latest
if: github.repository == 'sunnypilot/sunnypilot'
permissions:
checks: read
contents: write
steps:
- name: Wait for green check mark
if: ${{ github.event_name == 'schedule' }}
uses: lewagon/wait-on-check-action@ccfb013c15c8afb7bf2b7c028fb74dc5a068cccc
with:
ref: master
wait-interval: 30
running-workflow-name: 'build __nightly'
repo-token: ${{ secrets.GITHUB_TOKEN }}
check-regexp: ^((?!.*(build prebuilt|create badges).*).)*$
- uses: actions/checkout@v4
with:
submodules: true
fetch-depth: 0
- run: ./tools/op.sh setup
- name: Push __nightly
run: BRANCH=__nightly release/build_stripped.sh
-77
View File
@@ -1,77 +0,0 @@
name: repo maintenance
on:
schedule:
- cron: "0 14 * * 1" # every Monday at 2am UTC (6am PST)
workflow_dispatch:
env:
PYTHONPATH: ${{ github.workspace }}
jobs:
package_updates:
name: package_updates
runs-on: ubuntu-latest
if: github.repository == 'sunnypilot/sunnypilot'
steps:
- uses: actions/checkout@v6
with:
submodules: true
- run: ./tools/op.sh setup
- name: uv lock
run: uv lock --upgrade
- name: uv pip tree
id: pip_tree
run: |
echo 'PIP_TREE<<EOF' >> $GITHUB_OUTPUT
uv pip tree >> $GITHUB_OUTPUT
echo 'EOF' >> $GITHUB_OUTPUT
- name: venv size
id: venv_size
run: |
echo 'VENV_SIZE<<EOF' >> $GITHUB_OUTPUT
echo "Total: $(du -sh .venv | cut -f1)" >> $GITHUB_OUTPUT
echo "" >> $GITHUB_OUTPUT
echo "Top 10 by size:" >> $GITHUB_OUTPUT
du -sh .venv/lib/python*/site-packages/* 2>/dev/null \
| grep -v '\.dist-info' \
| grep -v '__pycache__' \
| sort -rh \
| head -10 \
| while IFS=$'\t' read size path; do echo "$size ${path##*/}"; done >> $GITHUB_OUTPUT
echo 'EOF' >> $GITHUB_OUTPUT
- name: bump submodules
run: |
git config submodule.msgq.update none
git config submodule.rednose_repo.update none
git config submodule.teleoprtc_repo.update none
git config submodule.tinygrad.update none
git submodule update --remote
git add .
- name: update car docs
run: |
python selfdrive/car/docs.py
git add docs/CARS.md
- name: Create Pull Request
uses: peter-evans/create-pull-request@c0f553fe549906ede9cf27b5156039d195d2ece0
with:
author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
token: ${{ github.repository == 'commaai/openpilot' && secrets.ACTIONS_CREATE_PR_PAT || secrets.GITHUB_TOKEN }}
commit-message: Update Python packages
title: '[bot] Update Python packages'
branch: auto-package-updates
base: master
delete-branch: true
body: |
Automatic PR from repo-maintenance -> package_updates
```
$ du -sh .venv && du -sh .venv/lib/python*/site-packages/* | sort -rh | head -10
${{ steps.venv_size.outputs.VENV_SIZE }}
```
```
$ uv pip tree
${{ steps.pip_tree.outputs.PIP_TREE }}
```
labels: bot
-52
View File
@@ -1,52 +0,0 @@
name: stale
on:
schedule:
- cron: '30 1 * * *'
workflow_dispatch:
env:
DAYS_BEFORE_PR_CLOSE: 7
DAYS_BEFORE_PR_STALE: 24
DAYS_BEFORE_PR_STALE_DRAFT: 30
jobs:
stale:
runs-on: ubuntu-latest
steps:
- uses: actions/stale@v10
with:
exempt-all-milestones: true
# pull request config
stale-pr-message: 'This PR has had no activity for ${{ env.DAYS_BEFORE_PR_STALE }} days. It will be automatically closed in ${{ env.DAYS_BEFORE_PR_CLOSE }} days if there is no activity.'
close-pr-message: 'This PR has been automatically closed due to inactivity. Feel free to re-open once activity resumes.'
stale-pr-label: stale
delete-branch: ${{ github.event.pull_request.head.repo.full_name == 'sunnypilot/sunnypilot' }} # only delete branches on the main repo
exempt-pr-labels: "ignore stale,needs testing" # if wip or it needs testing from the community, don't mark as stale
days-before-pr-stale: ${{ env.DAYS_BEFORE_PR_STALE }}
days-before-pr-close: ${{ env.DAYS_BEFORE_PR_CLOSE }}
exempt-draft-pr: false
# issue config
days-before-issue-stale: -1 # ignore issues for now
# same as above, but give draft PRs more time
stale_drafts:
runs-on: ubuntu-latest
steps:
- uses: actions/stale@v10
with:
exempt-all-milestones: true
# pull request config
stale-pr-message: 'This PR has had no activity for ${{ env.DAYS_BEFORE_PR_STALE_DRAFT }} days. It will be automatically closed in ${{ env.DAYS_BEFORE_PR_CLOSE }} days if there is no activity.'
close-pr-message: 'This PR has been automatically closed due to inactivity. Feel free to re-open once activity resumes.'
stale-pr-label: stale
delete-branch: ${{ github.event.pull_request.head.repo.full_name == 'commaai/openpilot' }} # only delete branches on the main repo
exempt-pr-labels: "ignore stale,needs testing" # if wip or it needs testing from the community, don't mark as stale
days-before-pr-stale: ${{ env.DAYS_BEFORE_PR_STALE_DRAFT }}
days-before-pr-close: ${{ env.DAYS_BEFORE_PR_CLOSE }}
exempt-draft-pr: true
# issue config
days-before-issue-stale: -1 # ignore issues for now
@@ -1,267 +0,0 @@
name: Build Model from Upstream
env:
BUILD_DIR: "/data/openpilot"
OUTPUT_DIR: ${{ github.workspace }}/output
SCONS_CACHE_DIR: ${{ github.workspace }}/release/ci/scons_cache
UPSTREAM_REPO: "commaai/openpilot"
TINYGRAD_PATH: ${{ github.workspace }}/tinygrad_repo
MODELS_DIR: ${{ github.workspace }}/selfdrive/modeld/models
on:
workflow_call:
inputs:
upstream_branch:
description: 'Upstream branch to build from'
required: true
default: 'master'
type: string
custom_name:
description: 'Custom name for the model (no date, only name)'
required: false
type: string
is_20hz:
description: 'Is this a 20Hz model'
required: false
type: boolean
default: true
artifact_suffix:
description: 'Suffix for artifact name'
required: false
type: string
default: ''
workflow_dispatch:
inputs:
upstream_branch:
description: 'Upstream branch to build from'
required: true
default: 'master'
type: string
custom_name:
description: 'Custom name for the model (no date, only name)'
required: false
type: string
is_20hz:
description: 'Is this a 20Hz model'
required: false
type: boolean
default: true
run-name: Build model [${{ inputs.custom_name || inputs.upstream_branch }}] from ref [${{ inputs.upstream_branch }}]
jobs:
get_model:
runs-on: ubuntu-latest
env:
REF: ${{ inputs.upstream_branch }}
outputs:
model_date: ${{ steps.commit-date.outputs.model_date }}
steps:
# Note: To allow dynamic models from both openpilot and sunnypilot (merges/mashups), we try commaai as default,
# and fallback to sunnypilot if the ref checkout fails.
- name: Checkout commaai/openpilot
id: checkout_upstream
continue-on-error: true
uses: actions/checkout@v4
with:
repository: commaai/openpilot
ref: ${{ inputs.upstream_branch }}
submodules: recursive
path: openpilot
- name: Fallback to sunnypilot/sunnypilot
if: steps.checkout_upstream.outcome == 'failure'
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot
ref: ${{ inputs.upstream_branch }}
submodules: recursive
path: openpilot
- name: Get commit date
id: commit-date
run: |
cd ${{ github.workspace }}/openpilot
commit_date=$(git log -1 --format=%cd --date=format:'%B %d, %Y')
echo "model_date=${commit_date}" >> $GITHUB_OUTPUT
cat $GITHUB_OUTPUT
- run: |
cd ${{ github.workspace }}/openpilot
git lfs pull
- name: 'Upload Artifact'
uses: actions/upload-artifact@v4
with:
name: models-${{ env.REF }}${{ inputs.artifact_suffix }}
path: ${{ github.workspace }}/openpilot/selfdrive/modeld/models/*.onnx
build_model:
runs-on: [self-hosted, tici]
needs: get_model
env:
MODEL_NAME: ${{ inputs.custom_name || inputs.upstream_branch }} (${{ needs.get_model.outputs.model_date }})
REF: ${{ inputs.upstream_branch }}
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- run: git lfs pull
- name: Cache SCons
uses: actions/cache@v4
with:
path: ${{env.SCONS_CACHE_DIR}}
key: scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-model-${{ github.sha }}
# Note: GitHub Actions enforces cache isolation between different build sources (PR builds, workflow dispatches, etc.)
# for security. Only caches from the default branch are shared across all builds. This is by design and cannot be overridden.
restore-keys: |
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-model
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_NEW_BRANCH }}-model
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_BRANCH }}-model
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_NEW_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}
- name: Set environment variables
id: set-env
run: |
# Set up common environment
source /etc/profile;
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
printenv >> $GITHUB_ENV
if [[ "${{ runner.debug }}" == "1" ]]; then
cat $GITHUB_OUTPUT
fi
- name: Setup build environment
run: |
mkdir -p "${BUILD_DIR}/"
sudo find $BUILD_DIR/ -mindepth 1 -delete
echo "Starting build stage..."
echo "BUILD_DIR: ${BUILD_DIR}"
echo "CI_DIR: ${CI_DIR}"
echo "VERSION: ${{ steps.set-env.outputs.version }}"
echo "UV_PROJECT_ENVIRONMENT: ${UV_PROJECT_ENVIRONMENT}"
echo "VIRTUAL_ENV: ${VIRTUAL_ENV}"
echo "-------"
if [[ "${{ runner.debug }}" == "1" ]]; then
printenv
fi
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
rm -rf ${{ env.MODELS_DIR }}/*.onnx
- name: Download model artifacts
uses: actions/download-artifact@v4
with:
name: models-${{ env.REF }}${{ inputs.artifact_suffix }}
path: ${{ github.workspace }}/selfdrive/modeld/models
- run: |
rm -f ${{ github.workspace }}/selfdrive/modeld/models/{dmonitoring_model,big_driving_policy,big_driving_vision}.onnx
- name: Build Model
run: |
source /etc/profile
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
export PYTHONPATH="${PYTHONPATH}:${{ env.TINYGRAD_PATH }}:${{ github.workspace }}"
COMPILE_MODELD="${{ github.workspace }}/sunnypilot/modeld_v2/compile_modeld.py"
MODEL_SIZE=$(python3 -c "from openpilot.common.transformations.model import MEDMODEL_INPUT_SIZE as s; print(f'{s[0]}x{s[1]}')")
CAMERA_RES=$(python3 -c "from openpilot.common.transformations.camera import _ar_ox_fisheye as a, _os_fisheye as o; print(f'{a.width}x{a.height} {o.width}x{o.height}')")
TG_FLAGS="DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1"
# Generate metadata for all ONNX files
find "${{ env.MODELS_DIR }}" -maxdepth 1 -name '*.onnx' | while IFS= read -r onnx_file; do
echo "Generating metadata: $onnx_file"
env ${TG_FLAGS} python3 "${{ env.MODELS_DIR }}/../get_model_metadata.py" "$onnx_file" || true
done
# Detect model type and build compile args
VISION_ONNX="${{ env.MODELS_DIR }}/driving_vision.onnx"
POLICY_ONNX="${{ env.MODELS_DIR }}/driving_policy.onnx"
OFF_POLICY_ONNX="${{ env.MODELS_DIR }}/driving_off_policy.onnx"
ON_POLICY_ONNX="${{ env.MODELS_DIR }}/driving_on_policy.onnx"
SUPERCOMBO_ONNX="${{ env.MODELS_DIR }}/supercombo.onnx"
MODEL_TYPE="" ONNX_ARGS="" OUTPUT_NAME=""
if [ -f "$VISION_ONNX" ]; then
ONNX_ARGS="--vision-onnx $VISION_ONNX"
if [ -f "$ON_POLICY_ONNX" ] && [ -f "$OFF_POLICY_ONNX" ]; then
MODEL_TYPE=vision_multi_policy
ONNX_ARGS="$ONNX_ARGS --off-policy-onnx $OFF_POLICY_ONNX --on-policy-onnx $ON_POLICY_ONNX"
elif [ -f "$OFF_POLICY_ONNX" ] && [ -f "$POLICY_ONNX" ]; then
MODEL_TYPE=vision_multi_policy
ONNX_ARGS="$ONNX_ARGS --policy-onnx $POLICY_ONNX --off-policy-onnx $OFF_POLICY_ONNX"
elif [ -f "$POLICY_ONNX" ]; then
MODEL_TYPE=vision_policy
ONNX_ARGS="$ONNX_ARGS --policy-onnx $POLICY_ONNX"
fi
elif [ -f "$SUPERCOMBO_ONNX" ]; then
MODEL_TYPE=supercombo
ONNX_ARGS="--supercombo-onnx $SUPERCOMBO_ONNX"
fi
if [ -n "$MODEL_TYPE" ]; then
echo "Detected: $MODEL_TYPE -> driving_tinygrad.pkl"
env ${TG_FLAGS} python3 "$COMPILE_MODELD" \
--model-type $MODEL_TYPE \
--model-size $MODEL_SIZE \
--camera-resolutions $CAMERA_RES \
$ONNX_ARGS \
--output "${{ env.MODELS_DIR }}/driving_tinygrad.pkl"
fi
- name: Validate Model Outputs
run: |
source /etc/profile
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
python3 "${{ github.workspace }}/release/ci/model_generator.py" \
--validate-only \
--model-dir "${{ env.MODELS_DIR }}"
- name: Prepare Output
run: |
sudo rm -rf ${{ env.OUTPUT_DIR }}
mkdir -p ${{ env.OUTPUT_DIR }}
# Copy the model files
rsync -avm \
--include='*.dlc' \
--include='*.pkl' \
--include='*.chunk*' \
--include='*.chunkmanifest' \
--include='*.onnx' \
--exclude='*' \
--delete-excluded \
--chown=comma:comma \
${{ env.MODELS_DIR }}/ ${{ env.OUTPUT_DIR }}/
python3 "${{ github.workspace }}/release/ci/model_generator.py" \
--model-dir "${{ env.MODELS_DIR }}" \
--output-dir "${{ env.OUTPUT_DIR }}" \
--custom-name "${{ env.MODEL_NAME }}" \
--upstream-branch "${{ inputs.upstream_branch }}" \
${{ inputs.is_20hz && '--is-20hz' || '' }}
- name: Write artifact name to file
run: echo "model-${{ env.MODEL_NAME }}${{ inputs.artifact_suffix }}-${{ github.run_number }}" > ${{ env.OUTPUT_DIR }}/artifact_name.txt
- name: Upload Build Artifacts
id: upload-artifact
uses: actions/upload-artifact@v4
with:
name: model-${{ env.MODEL_NAME }}${{ inputs.artifact_suffix }}-${{ github.run_number }}
path: ${{ env.OUTPUT_DIR }}
- name: Upload artifact name file
uses: actions/upload-artifact@v4
with:
name: artifact-name-${{ inputs.custom_name || inputs.upstream_branch }}
path: ${{ env.OUTPUT_DIR }}/artifact_name.txt
- name: Re-enable powersave
if: always()
run: |
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
@@ -1,369 +0,0 @@
name: sunnypilot prebuilt action
env:
BUILD_DIR: "/data/openpilot"
OUTPUT_DIR: ${{ github.workspace }}/output
CI_DIR: ${{ github.workspace }}/release/ci
SCONS_CACHE_DIR: ${{ github.workspace }}/release/ci/scons_cache
PUBLIC_REPO_URL: "https://github.com/sunnypilot/sunnypilot"
# Branch configurations
STAGING_SOURCE_BRANCH: 'master'
# Runtime configuration
SOURCE_BRANCH: "${{ github.head_ref || github.ref_name }}"
on:
push:
branches: [ master, master-dev ]
tags: [ 'release/*' ]
pull_request_target:
types: [ labeled ]
workflow_dispatch:
inputs:
wait_for_tests:
description: 'Wait for tests to finish'
required: false
type: boolean
default: false
jobs:
prepare_strategy:
runs-on: ubuntu-24.04
if: (!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
outputs:
environment: ${{ steps.strategy.outputs.environment }}
new_branch: ${{ steps.strategy.outputs.new_branch }}
extra_version_identifier: ${{ steps.strategy.outputs.extra_version_identifier }}
version: ${{ steps.strategy.outputs.version }}
cancel_publish_in_progress: ${{ steps.strategy.outputs.cancel_publish_in_progress }}
publish_concurrency_group: ${{ steps.strategy.outputs.publish_concurrency_group }}
is_stable_branch: ${{ steps.strategy.outputs.is_stable_branch }}
build: ${{ steps.strategy.outputs.build }}
steps:
- uses: actions/checkout@v4
- name: Extract deploy strategy
id: strategy
run: |
echo '::group::Strategy Extraction'
BRANCH="${{ github.head_ref || github.ref_name }}"
echo "Current branch: $BRANCH"
STRATEGY_JSON='${{ vars.DEPLOY_STRATEGY }}'
CONFIG=$(echo "$STRATEGY_JSON" | jq -r --arg branch "$BRANCH" '
.configs[] | select(.branch == $branch)
')
BUILD="$(date '+%Y.%m.%d')-${{ github.run_number }}"
if [[ -z "$CONFIG" || "$CONFIG" == "null" ]]; then
echo "No exact strategy match found. Falling back to feature/fork logic."
IS_FORK="${{ github.event.pull_request.head.repo.fork && 'true' || 'false' }}"
FORK_SUFFIX=$( [[ "$IS_FORK" == "true" ]] && echo "-fork" || echo "" )
NEW_BRANCH="${BRANCH}${FORK_SUFFIX}-prebuilt"
echo "new_branch=$NEW_BRANCH" >> $GITHUB_OUTPUT
echo "version=$BUILD" >> $GITHUB_OUTPUT
echo "cancel_publish_in_progress=true" >> $GITHUB_OUTPUT
echo "publish_concurrency_group=publish-${BRANCH}" >> $GITHUB_OUTPUT
echo "environment=feature-branch" >> $GITHUB_OUTPUT
echo "extra_version_identifier=feature-branch" >> $GITHUB_OUTPUT
else
echo "Matched config: $CONFIG"
environment=$(echo "$CONFIG" | jq -r '.environment')
echo "environment=$environment" >> $GITHUB_OUTPUT
echo "new_branch=$(echo "$CONFIG" | jq -r '.target_branch')" >> $GITHUB_OUTPUT
cancel="$(echo "$CONFIG" | jq -r '.cancel_publish_in_progress')";
echo "cancel_publish_in_progress=$( [ "$cancel" = "null" ] && echo "true" || echo $cancel)" >> $GITHUB_OUTPUT
echo "publish_concurrency_group=publish-${BRANCH}$( [ "$cancel" = "null" ] || [ "$cancel" = "true" ] || echo "${{ github.sha }}" )" >> $GITHUB_OUTPUT
is_stable_branch="$(echo "$CONFIG" | jq -r '.stable_branch // false')";
echo "is_stable_branch=$is_stable_branch" >> $GITHUB_OUTPUT
stable_version=$(cat sunnypilot/common/version.h | grep SUNNYPILOT_VERSION | sed -e 's/[^0-9|.]//g');
echo "version=$([ "$is_stable_branch" = "true" ] && echo "$stable_version" || echo "$BUILD")" >> $GITHUB_OUTPUT
echo "extra_version_identifier=${environment}" >> $GITHUB_OUTPUT
fi
echo "build=$BUILD" >> $GITHUB_OUTPUT
cat $GITHUB_OUTPUT
validate_tests:
runs-on: ubuntu-24.04
needs: [ prepare_strategy ]
if: ${{
((github.event_name == 'workflow_dispatch' && inputs.wait_for_tests) ||
(github.event_name == 'push' && needs.prepare_strategy.outputs.is_stable_branch == 'true') ||
contains(github.event_name, 'pull_request') && (github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
}}
steps:
- uses: actions/checkout@v4
- name: Wait for Tests
uses: ./.github/workflows/wait-for-action # Path to where you place the action
with:
workflow: tests.yaml # The workflow file to monitor
github-token: ${{ secrets.GITHUB_TOKEN }}
should-wait-for-start: ${{ github.event_name == 'push' && 'true' || 'false' }}
build:
needs: [ validate_tests, prepare_strategy ]
concurrency:
group: build-${{ github.head_ref || github.ref_name }}
cancel-in-progress: false
runs-on: [self-hosted, tici]
outputs:
new_branch: ${{ needs.prepare_strategy.outputs.new_branch }}
version: ${{ needs.prepare_strategy.outputs.version }}
extra_version_identifier: ${{ needs.prepare_strategy.outputs.extra_version_identifier }}
commit_sha: ${{ github.sha }}
if: ${{
(always() && !cancelled() && !failure()) &&
needs.prepare_strategy.result == 'success' &&
(needs.validate_tests.result == 'success' || needs.validate_tests.result == 'skipped') &&
(!contains(github.event_name, 'pull_request') ||
(github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
}}
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
ref: ${{ env.SOURCE_BRANCH }}
repository: ${{ github.event.pull_request.head.repo.fork && github.event.pull_request.head.repo.full_name || github.repository }}
- run: git lfs pull
- name: Cache SCons
uses: actions/cache@v4
with:
path: ${{env.SCONS_CACHE_DIR}}
key: scons-${{ runner.os }}-${{ runner.arch }}-${{ env.SOURCE_BRANCH }}-${{ github.sha }}
# Note: GitHub Actions enforces cache isolation between different build sources (PR builds, workflow dispatches, etc.)
# for security. Only caches from the default branch are shared across all builds. This is by design and cannot be overridden.
restore-keys: |
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.SOURCE_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.STAGING_SOURCE_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}
- name: Set environment variables
id: set-env
run: |
echo "new_branch=${{ needs.prepare_strategy.outputs.new_branch }}" >> $GITHUB_OUTPUT
echo "version=${{ needs.prepare_strategy.outputs.version }}" >> $GITHUB_OUTPUT
echo "extra_version_identifier=${{ needs.prepare_strategy.outputs.extra_version_identifier }}" >> $GITHUB_OUTPUT
echo "commit_sha=${{ github.sha }}" >> $GITHUB_OUTPUT
# Set up common environment
source /etc/profile;
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
printenv >> $GITHUB_ENV
if [[ "${{ runner.debug }}" == "1" ]]; then
cat $GITHUB_OUTPUT
fi
- name: Setup build environment
run: |
mkdir -p "${BUILD_DIR}/"
sudo find $BUILD_DIR/ -mindepth 1 -delete
echo "Starting build stage..."
echo "BUILD_DIR: ${BUILD_DIR}"
echo "CI_DIR: ${CI_DIR}"
echo "VERSION: ${{ steps.set-env.outputs.version }}"
echo "UV_PROJECT_ENVIRONMENT: ${UV_PROJECT_ENVIRONMENT}"
echo "VIRTUAL_ENV: ${VIRTUAL_ENV}"
echo "-------"
if [[ "${{ runner.debug }}" == "1" ]]; then
printenv
fi
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
- name: Build Main Project
run: |
export PYTHONPATH="$BUILD_DIR"
./release/release_files.py | sort | uniq | rsync -rRl${RUNNER_DEBUG:+v} --files-from=- . $BUILD_DIR/
cd $BUILD_DIR
sed -i '/from .board.jungle import PandaJungle, PandaJungleDFU/s/^/#/' panda/__init__.py
echo "Building sunnypilot's modeld_v2..."
scons -j$(nproc) cache_dir=${{env.SCONS_CACHE_DIR}} --minimal sunnypilot/modeld_v2
echo "Building sunnypilot's locationd..."
scons -j2 cache_dir=${{env.SCONS_CACHE_DIR}} --minimal sunnypilot/selfdrive/locationd
echo "Building openpilot's locationd..."
scons -j1 cache_dir=${{env.SCONS_CACHE_DIR}} --minimal selfdrive/locationd
echo "Building rest of sunnypilot"
scons -j$(nproc) cache_dir=${{env.SCONS_CACHE_DIR}} --minimal
touch ${BUILD_DIR}/prebuilt
if [[ "${{ runner.debug }}" == "1" ]]; then
ls -la ${BUILD_DIR}
fi
- name: Prepare Output
run: |
sudo rm -rf ${OUTPUT_DIR}
mkdir -p ${OUTPUT_DIR}
rsync -am${RUNNER_DEBUG:+v} \
--exclude='.sconsign.dblite' \
--exclude='*.a' \
--exclude='*.o' \
--exclude='*.os' \
--exclude='*.pyc' \
--exclude='moc_*' \
--exclude='__pycache__' \
--exclude='Jenkinsfile' \
--exclude='**/release/' \
--exclude='**/.github/' \
--exclude='**/selfdrive/ui/replay/' \
--exclude='**/__pycache__/' \
--exclude='${{env.SCONS_CACHE_DIR}}' \
--exclude='**/.git/' \
--exclude='**/SConstruct' \
--exclude='**/SConscript' \
--exclude='**/.venv/' \
--exclude='selfdrive/modeld/models/*.onnx*' \
--exclude='sunnypilot/modeld*/models/*.onnx*' \
--exclude='third_party/*x86*' \
--exclude='third_party/*Darwin*' \
--delete-excluded \
--chown=comma:comma \
${BUILD_DIR}/ ${OUTPUT_DIR}/
- name: 'Tar.gz files'
run: |
tar czf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }} .
ls -la prebuilt.tar.gz
- name: 'Upload Artifact'
uses: actions/upload-artifact@v4
with:
name: prebuilt
path: prebuilt.tar.gz
- name: Re-enable powersave
if: always()
run: |
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
publish:
concurrency:
# We do a bit of a hack here to avoid canceling the publishing job if a new commit comes in while we're publishing by adding the sha to the group name.
# This means that if multiple commits come in while we're publishing, they will be queued up and publish one after the other.
# Otherwise, if a job is waiting to be published due to environment wait time, it would be canceled by a new commit and restart the wait time.
group: ${{ needs.prepare_strategy.outputs.publish_concurrency_group }}
cancel-in-progress: ${{ needs.prepare_strategy.outputs.cancel_publish_in_progress == 'true' }}
if: ${{ (always() && !cancelled() && !failure()) && needs.build.result == 'success' && needs.prepare_strategy.result == 'success' && (!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt')) }}
needs: [ build, prepare_strategy ]
runs-on: ubuntu-24.04
environment: ${{ needs.prepare_strategy.outputs.environment }}
steps:
- uses: actions/checkout@v4
- name: Download build artifacts
uses: actions/download-artifact@v4
with:
name: prebuilt
- name: Untar prebuilt
run: |
mkdir -p ${{ env.OUTPUT_DIR }}
tar xzf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }}
- name: Configure Git
run: |
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"
- name: Publish to Public Repository
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
echo '${{ toJSON(needs.build.outputs) }}'
ls -la ${{ env.OUTPUT_DIR }}
${{ env.CI_DIR }}/publish.sh \
"${{ github.workspace }}" \
"${{ env.OUTPUT_DIR }}" \
"${{ needs.build.outputs.new_branch }}" \
"${{ needs.build.outputs.version }}" \
"https://x-access-token:${{github.token}}@github.com/sunnypilot/sunnypilot.git" \
"${{ needs.build.outputs.extra_version_identifier }}"
echo ""
echo "---- ️ To update the list of branches that auto deploy prebuilts -----"
echo ""
echo "1. Go to: ${{ github.server_url }}/${{ github.repository }}/settings/variables/actions/AUTO_DEPLOY_PREBUILT_BRANCHES"
echo "2. Current value: ${{ vars.AUTO_DEPLOY_PREBUILT_BRANCHES }}"
echo "3. Update as needed (JSON array with no spaces)"
- name: Tag ${{ needs.prepare_strategy.outputs.environment }}
if: ${{ needs.prepare_strategy.outputs.is_stable_branch == 'true' && (github.event_name != 'push' || !startsWith(github.ref, 'refs/tags/')) }}
run: |
TAG="${{ needs.prepare_strategy.outputs.environment }}/${{ needs.prepare_strategy.outputs.version }}/${{ needs.prepare_strategy.outputs.build }}"
git tag -f -a ${TAG} -m "${{ needs.prepare_strategy.outputs.environment }} @ ${{ needs.prepare_strategy.outputs.version }} of build ${{ needs.build.outputs.build }}."
git push -f origin ${TAG}
notify:
needs:
- prepare_strategy
- build
- publish
runs-on: ubuntu-24.04
if: ${{ (always() && !cancelled() && !failure())
&& needs.publish.result == 'success'
&& (!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
&& (fromJSON(vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES_V2)[github.head_ref || github.ref_name] != null) }}
steps:
- uses: actions/checkout@v4
- name: Prepare notification message
id: message
run: |
TEMPLATE='${{ vars.DISCOURSE_GENERAL_UPDATE_NOTICE }}'
export VERSION="${{ needs.prepare_strategy.outputs.version }}"
export branch_name="${{ env.SOURCE_BRANCH }}"
export new_branch="${{ needs.prepare_strategy.outputs.new_branch }}"
export commit_sha="${{ github.sha }}"
export commit_short_sha="${{ github.sha }}"
export commit_short_sha="${commit_short_sha:0:7}"
export extra_version_identifier="${{ needs.prepare_strategy.outputs.extra_version_identifier || github.run_number }}"
export PUBLIC_REPO_URL="${{ env.PUBLIC_REPO_URL }}"
MESSAGE=$(cat << 'EOF' | envsubst
${{ vars.DISCOURSE_GENERAL_UPDATE_NOTICE }}
EOF
)
{
echo 'content<<EOFMARKER'
echo "$MESSAGE"
echo 'EOFMARKER'
} >> $GITHUB_OUTPUT
shell: bash
- name: Post to Discourse
uses: ./.github/workflows/post-to-discourse
with:
discourse-url: ${{ vars.DISCOURSE_URL }}
api-key: ${{ secrets.DISCOURSE_API_KEY }}
api-username: "system"
topic-id: ${{ fromJSON(vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES_V2)[github.head_ref || github.ref_name].topic_id }}
message: ${{ steps.message.outputs.content }}
manage-pr-labels:
name: Remove prebuilt label
runs-on: ubuntu-latest
if: (always() && contains(github.event_name, 'pull_request') && (github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
env:
LABEL: prebuilt
steps:
- name: Remove trust-fork-pr label if present
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const prNumber = context.payload.pull_request.number;
await github.rest.issues.removeLabel({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNumber,
name: process.env.LABEL
});
console.log(`Removed '${process.env.LABEL}' label from PR #${prNumber}`);
@@ -1,243 +0,0 @@
name: Build dev
env:
DEFAULT_SOURCE_BRANCH: "master"
DEFAULT_TARGET_BRANCH: "master-dev"
LFS_URL: 'https://gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git/info/lfs'
LFS_PUSH_URL: 'ssh://git@gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git'
on:
push:
branches:
- master
pull_request_target:
types: [ labeled ]
branches:
- 'master'
workflow_dispatch:
inputs:
source_branch:
description: 'Source branch to reset from'
required: true
default: 'master'
type: string
target_branch:
description: 'Target branch to reset and squash into'
required: true
default: 'master-dev'
type: string
cancel_in_progress:
description: 'Cancel any in-progress runs of this workflow'
required: false
default: true
type: boolean
concurrency:
group: ${{ github.workflow }}
cancel-in-progress: ${{ inputs.cancel_in_progress || github.event_name == 'push' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}
jobs:
reset-and-squash:
runs-on: ubuntu-latest
if: (
(github.event_name == 'workflow_dispatch')
|| (github.event_name == 'push' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch))
|| (contains(github.event_name, 'pull_request') && ((github.event.action == 'labeled' && (github.event.label.name == vars.PREBUILT_PR_LABEL || github.event.label.name == 'trust-fork-pr') && contains(github.event.pull_request.labels.*.name, vars.PREBUILT_PR_LABEL))))
)
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Fetch all history for all branches
token: ${{ secrets.GITHUB_TOKEN }}
persist-credentials: false
- name: Wait for Tests
uses: ./.github/workflows/wait-for-action # Path to where you place the action
if: (
(github.event_name == 'push' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch))
|| (contains(github.event_name, 'pull_request') && ((github.event.action == 'labeled' && (github.event.label.name == vars.PREBUILT_PR_LABEL || github.event.label.name == 'trust-fork-pr') && contains(github.event.pull_request.labels.*.name, vars.PREBUILT_PR_LABEL))))
)
with:
workflow: tests.yaml # The workflow file to monitor
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Configure Git
run: |
git config --global user.name 'github-actions[bot]'
git config --global user.email 'github-actions[bot]@users.noreply.github.com'
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
- name: Add GitLab public keys
run: |
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install PyGithub
- name: Check branches exist
run: |
# Check if source branch exists
if ! git ls-remote --heads origin ${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }} | grep -q "${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}"; then
echo "Source branch ${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }} does not exist!"
exit 1
fi
# Make sure we have the latest source branch
git fetch origin ${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}
# Check if target branch exists
if ! git ls-remote --heads origin ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} | grep -q "${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}"; then
echo "Target branch ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} does not exist, creating it from ${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}"
git checkout -b ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} origin/${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}
git push origin ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}
else
# Fetch target branch if it exists
git fetch origin ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}
fi
- name: Reset target branch
run: |
echo "Resetting ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} to match ${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}"
# Delete if exists and recreate pointing to source
git branch -D ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} || true
git branch ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} origin/${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}
- name: Get PRs to squash
id: get-prs
run: |
# Use GitHub API to get PRs with specific label, ordered by creation date
PR_LIST=$(gh api graphql -f query='
query($search_query:String!) {
search(query: $search_query, type:ISSUE, first:40) {
nodes {
... on PullRequest {
number
headRefName
title
createdAt
labels(last:10) {
nodes {
name
}
}
headRepository {
name
nameWithOwner
url
isFork
}
commits(last: 1) {
nodes {
commit {
statusCheckRollup {
state
}
}
}
}
}
}
}
}' -F search_query="repo:${{ github.repository }} is:pr is:open label:${{ vars.PREBUILT_PR_LABEL }},${{ vars.PREBUILT_PR_LABEL }}-c3 draft:false sort:created-asc")
PR_LIST=${PR_LIST//\'/}
echo "PR_LIST=${PR_LIST}" >> $GITHUB_OUTPUT
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Process PRs
run: |
cp ${{ github.workspace }}/release/ci/squash_and_merge.py /tmp/squash_and_merge.py && \
chmod +x /tmp/squash_and_merge.py && \
python3 ${{ github.workspace }}/release/ci/squash_and_merge_prs.py \
--pr-data '${{ steps.get-prs.outputs.PR_LIST }}' \
--target-branch ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} \
--squash-script-path '/tmp/squash_and_merge.py'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Update LFS Config
run: |
echo '[lfs]' > .lfsconfig
echo ' url = ${{ env.LFS_URL }}' >> .lfsconfig
echo ' pushurl = ${{ env.LFS_PUSH_URL }}' >> .lfsconfig
echo ' locksverify = false' >> .lfsconfig
- name: Restore workflows from source
run: |
TARGET_BRANCH="${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}"
SOURCE_BRANCH="${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}"
# Ensure we are on the target branch
git checkout $TARGET_BRANCH
echo "Restoring .github/workflows from $SOURCE_BRANCH"
git checkout origin/$SOURCE_BRANCH -- .github/workflows
if ! git diff --cached --quiet; then
echo "Workflows differ. Committing restoration."
git commit -m "chore: restore .github/workflows from $SOURCE_BRANCH"
else
echo "Workflows match $SOURCE_BRANCH."
fi
- uses: actions/create-github-app-token@v2
id: ci-token
with:
app-id: ${{ secrets.CI_GITHUB_ACTIONS_TOKEN_APP_ID }}
private-key: ${{ secrets.CI_GITHUB_ACTIONS_TOKEN_APP_PRIVATE_KEY }}
- name: Push changes if there are diffs
id: push-changes
run: |
TARGET_BRANCH="${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}"
# Use the App Token to set the remote URL with authentication
git remote set-url origin "https://x-access-token:${{ steps.ci-token.outputs.token }}@github.com/${{ github.repository }}.git"
# Fetch the latest from remote
git fetch origin $TARGET_BRANCH
# Check for diffs between local and remote
if git diff $TARGET_BRANCH origin/$TARGET_BRANCH --quiet; then
echo "No changes to push - local and remote branches are identical"
echo "has_changes=false" >> $GITHUB_OUTPUT
exit 0
fi
# Push with the authenticated origin
if ! git push origin $TARGET_BRANCH --force; then
echo "Failed to push changes to $TARGET_BRANCH"
exit 1
fi
echo "Branch $TARGET_BRANCH has been reset and updated with squashed PRs"
echo "has_changes=true" >> $GITHUB_OUTPUT
- name: Trigger and wait for selfdrive tests
if: steps.push-changes.outputs.has_changes == 'true'
run: |
echo "Triggering selfdrive tests..."
gh workflow run tests.yaml --ref "${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}"
echo "Sleeping for 120s to give plenty of time for the action to start and then we wait"
sleep 120
echo "Getting latest run ID..."
RUN_ID=$(gh run list --workflow=tests.yaml --branch="${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}" --limit=1 --json databaseId --jq '.[0].databaseId')
echo "Watching run ID: $RUN_ID"
gh run watch "$RUN_ID"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-78
View File
@@ -1,78 +0,0 @@
name: Debug Discourse Posting
on:
push:
jobs:
test-discourse-post:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Post test message to Discourse
uses: ./.github/workflows/post-to-discourse
with:
discourse-url: ${{ vars.DISCOURSE_URL }}
api-key: ${{ secrets.DISCOURSE_API_KEY }}
api-username: ${{ secrets.DISCOURSE_API_USERNAME }}
topic-id: ${{ vars.DISCOURSE_UPDATES_TOPIC_ID }}
message: |
## 🧪 Test Post from GitHub Actions
**This is a test post to verify Discourse integration**
- **Workflow**: ${{ github.workflow }}
- **Run Number**: #${{ github.run_number }}
- **Branch**: `${{ github.ref_name }}`
- **Commit**: ${{ github.sha }}
- **Actor**: @${{ github.actor }}
- **Timestamp**: ${{ github.event.head_commit.timestamp }}
---
### Fake Build Info (for testing)
- **Version**: 0.9.8-test
- **Build**: #42
- **Branch**: release-test
[View workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})
*This is an automated test message. Drive safe! 🚗💨*
- name: Create topic on Discourse
uses: ./.github/workflows/post-to-discourse
with:
discourse-url: ${{ vars.DISCOURSE_URL }}
api-key: ${{ secrets.DISCOURSE_API_KEY }}
api-username: ${{ secrets.DISCOURSE_API_USERNAME }}
#topic-id: ${{ vars.DISCOURSE_UPDATES_TOPIC_ID }}
category-id: 4
title: "This is a test of a new topic instead of a reply"
message: |
## 🧪 Test Post from GitHub Actions
**This is a test post to verify Discourse integration**
- **Workflow**: ${{ github.workflow }}
- **Run Number**: #${{ github.run_number }}
- **Branch**: `${{ github.ref_name }}`
- **Commit**: ${{ github.sha }}
- **Actor**: @${{ github.actor }}
- **Timestamp**: ${{ github.event.head_commit.timestamp }}
---
### Fake Build Info (for testing)
- **Version**: 0.9.8-test
- **Build**: #42
- **Branch**: release-test
[View workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})
*This is an automated test message. Drive safe! 🚗💨*
- name: Display results
if: always()
run: |
echo "::notice::Discourse post test completed"
echo "Check your Discourse topic to verify the post appeared correctly"
-254
View File
@@ -1,254 +0,0 @@
name: tests
on:
push:
branches:
- master
pull_request:
workflow_dispatch:
workflow_call:
inputs:
run_number:
default: '1'
required: true
type: string
concurrency:
group: tests-ci-run-${{ inputs.run_number }}-${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && github.run_id || github.head_ref || github.ref }}-${{ github.workflow }}-${{ github.event_name }}
cancel-in-progress: true
env:
CI: 1
PYTHONPATH: ${{ github.workspace }}
PYTEST: pytest --continue-on-collection-errors --durations=0 -n logical
jobs:
build_release:
name: build release
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16"]')
|| fromJSON('["ubuntu-24.04"]') }}
env:
STRIPPED_DIR: /tmp/releasepilot
PYTHONPATH: /tmp/releasepilot
steps:
- uses: actions/checkout@v6
with:
submodules: true
- name: Getting LFS files
uses: nick-fields/retry@7152eba30c6575329ac0576536151aca5a72780e
with:
timeout_minutes: 2
max_attempts: 3
command: git lfs pull
- name: Build devel
timeout-minutes: 1
run: TARGET_DIR=$STRIPPED_DIR release/build_stripped.sh
- run: ./tools/op.sh setup
- name: Build openpilot and run checks
timeout-minutes: 30
working-directory: ${{ env.STRIPPED_DIR }}
run: python3 system/manager/build.py
- name: Run tests
timeout-minutes: 1
working-directory: ${{ env.STRIPPED_DIR }}
run: release/check-dirty.sh
- name: Check submodules
if: github.repository == 'sunnypilot/sunnypilot'
timeout-minutes: 3
run: |
if [ "${{ github.ref }}" != "refs/heads/master" ]; then
git fetch origin master:refs/remotes/origin/master
SUBMODULE_PATHS=$(git diff origin/master HEAD --name-only | grep -E '^[^/]+$' | while read path; do
if git ls-files --stage "$path" | grep -q "^160000"; then
echo "$path"
fi
done | tr '\n' ' ')
if [ -n "$SUBMODULE_PATHS" ]; then
echo "Changed submodule paths: $SUBMODULE_PATHS"
export SUBMODULE_PATHS="$SUBMODULE_PATHS"
export CHECK_PR_REFS=true
fi
fi
release/check-submodules.sh
build_mac:
name: build macOS
runs-on: ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-macos-8x14' || 'macos-latest' }}
steps:
- uses: actions/checkout@v6
with:
submodules: true
- name: Remove Homebrew from environment
run: |
FILTERED=$(echo "$PATH" | tr ':' '\n' | grep -v '/opt/homebrew' | tr '\n' ':')
echo "PATH=${FILTERED}/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin" >> $GITHUB_ENV
- run: ./tools/op.sh setup
- name: Building openpilot
run: scons
static_analysis:
name: static analysis
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16"]')
|| fromJSON('["ubuntu-24.04"]') }}
steps:
- uses: actions/checkout@v6
with:
submodules: true
- run: ./tools/op.sh setup
- name: Static analysis
timeout-minutes: 1
run: scripts/lint/lint.sh
unit_tests:
name: unit tests
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16"]')
|| fromJSON('["ubuntu-24.04"]') }}
steps:
- uses: actions/checkout@v6
with:
submodules: true
- run: ./tools/op.sh setup
- name: Build openpilot
run: scons
- name: Run unit tests
timeout-minutes: ${{ contains(runner.name, 'nsc') && 2 || 999 }}
run: |
source selfdrive/test/setup_xvfb.sh
# Pre-compile Python bytecode so each pytest worker doesn't need to
$PYTEST --collect-only -m 'not slow' -qq
MAX_EXAMPLES=1 $PYTEST -m 'not slow'
process_replay:
name: process replay
if: false # disable process_replay for forks
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16"]')
|| fromJSON('["ubuntu-24.04"]') }}
steps:
- uses: actions/checkout@v6
with:
submodules: true
- run: ./tools/op.sh setup
- name: Build openpilot
run: scons
- name: Run replay
timeout-minutes: ${{ contains(runner.name, 'nsc') && 2 || 20 }}
continue-on-error: ${{ github.ref == 'refs/heads/master' }}
run: selfdrive/test/process_replay/test_processes.py -j$(nproc)
- name: Print diff
id: print-diff
if: always()
run: cat selfdrive/test/process_replay/diff.txt
- name: Print diff report
if: always()
run: cat selfdrive/test/process_replay/diff_report.txt
- uses: actions/upload-artifact@v6
if: always()
continue-on-error: true
with:
name: process_replay_diff.txt
path: selfdrive/test/process_replay/diff.txt
- name: Upload diff report
uses: actions/upload-artifact@v6
if: always() && github.event_name == 'pull_request'
continue-on-error: true
with:
name: diff_report_${{ github.event.number }}
path: selfdrive/test/process_replay/diff_report.txt
- name: Checkout ci-artifacts
if: github.repository == 'commaai/openpilot' && github.ref == 'refs/heads/master'
uses: actions/checkout@v4
with:
repository: commaai/ci-artifacts
ssh-key: ${{ secrets.CI_ARTIFACTS_DEPLOY_KEY }}
path: ${{ github.workspace }}/ci-artifacts
- name: Prepare refs
if: github.repository == 'commaai/openpilot' && github.ref == 'refs/heads/master'
working-directory: ${{ github.workspace }}/ci-artifacts
run: |
git config user.name "GitHub Actions Bot"
git config user.email "<>"
git fetch origin process-replay || true
git checkout process-replay 2>/dev/null || git checkout --orphan process-replay
cp ${{ github.workspace }}/selfdrive/test/process_replay/fakedata/*.zst .
echo "${{ github.sha }}" > ref_commit
git add .
git commit -m "process-replay refs for ${{ github.repository }}@${{ github.sha }}" || echo "No changes to commit"
- name: Push refs
if: github.repository == 'commaai/openpilot' && github.ref == 'refs/heads/master'
uses: nick-fields/retry@7152eba30c6575329ac0576536151aca5a72780e
with:
timeout_minutes: 2
max_attempts: 3
command: cd ${{ github.workspace }}/ci-artifacts && git push origin process-replay --force
- name: Run regen
if: false
timeout-minutes: 4
env:
ONNXCPU: 1
run: $PYTEST selfdrive/test/process_replay/test_regen.py
simulator_driving:
name: simulator driving
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16"]')
|| fromJSON('["ubuntu-24.04"]') }}
if: false # FIXME: Started to timeout recently
steps:
- uses: actions/checkout@v6
with:
submodules: true
- run: ./tools/op.sh setup
- name: Build openpilot
run: scons
- name: Driving test
timeout-minutes: 2
run: |
source selfdrive/test/setup_xvfb.sh
pytest -s tools/sim/tests/test_metadrive_bridge.py
create_ui_report:
name: Create UI Report
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16"]')
|| fromJSON('["ubuntu-24.04"]') }}
steps:
- uses: actions/checkout@v6
with:
submodules: true
- run: ./tools/op.sh setup
- name: Build openpilot
run: scons
- name: Create UI Report
run: |
source selfdrive/test/setup_xvfb.sh
python3 selfdrive/ui/tests/diff/replay.py
python3 selfdrive/ui/tests/diff/replay.py --big
- name: Upload UI Report
uses: actions/upload-artifact@v6
with:
name: ui-report-${{ inputs.run_number || '1' }}-${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && 'master' || github.event.number }}
path: selfdrive/ui/tests/diff/report
-175
View File
@@ -1,175 +0,0 @@
name: "ui preview"
on:
push:
branches:
- master
pull_request_target:
types: [assigned, opened, synchronize, reopened, edited]
branches:
- 'master'
paths:
- 'selfdrive/assets/**'
- 'selfdrive/ui/**'
- 'system/ui/**'
workflow_dispatch:
env:
UI_JOB_NAME: "Create UI Report"
REPORT_NAME: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && 'master' || github.event.number }}
SHA: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && github.sha || github.event.pull_request.head.sha }}
BRANCH_NAME: "openpilot/pr-${{ github.event.number }}-ui-preview"
REPORT_FILES_BRANCH_NAME: "mici-raylib-ui-reports"
# variant:video_prefix:master_branch
VARIANTS: "mici:mici_ui_replay:openpilot_master_ui_mici_raylib big:tizi_ui_replay:openpilot_master_ui_big_raylib"
jobs:
preview:
if: github.repository == 'sunnypilot/sunnypilot'
name: preview
runs-on: ubuntu-latest
timeout-minutes: 20
permissions:
contents: read
pull-requests: write
actions: read
steps:
- uses: actions/checkout@v6
with:
submodules: true
- name: Waiting for ui generation to end
uses: lewagon/wait-on-check-action@v1.3.4
with:
ref: ${{ env.SHA }}
check-name: ${{ env.UI_JOB_NAME }}
repo-token: ${{ secrets.GITHUB_TOKEN }}
allowed-conclusions: success
wait-interval: 20
- name: Getting workflow run ID
id: get_run_id
run: |
echo "run_id=$(curl https://api.github.com/repos/${{ github.repository }}/commits/${{ env.SHA }}/check-runs | jq -r '.check_runs[] | select(.name == "${{ env.UI_JOB_NAME }}") | .html_url | capture("(?<number>[0-9]+)") | .number')" >> $GITHUB_OUTPUT
- name: Getting proposed ui
uses: dawidd6/action-download-artifact@v6
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
run_id: ${{ steps.get_run_id.outputs.run_id }}
search_artifacts: true
name: ui-report-1-${{ env.REPORT_NAME }}
path: ${{ github.workspace }}/pr_ui
- name: Getting mici master ui
uses: actions/checkout@v6
with:
repository: sunnypilot/ci-artifacts
ssh-key: ${{ secrets.CI_ARTIFACTS_DEPLOY_KEY }}
path: ${{ github.workspace }}/master_mici
ref: openpilot_master_ui_mici_raylib
- name: Getting big master ui
uses: actions/checkout@v6
with:
repository: sunnypilot/ci-artifacts
ssh-key: ${{ secrets.CI_ARTIFACTS_DEPLOY_KEY }}
path: ${{ github.workspace }}/master_big
ref: openpilot_master_ui_big_raylib
- name: Saving new master ui
if: github.ref == 'refs/heads/master' && github.event_name == 'push'
run: |
for variant in $VARIANTS; do
IFS=':' read -r name video branch <<< "$variant"
master_dir="${{ github.workspace }}/master_${name}"
cd "$master_dir"
git checkout --orphan=new_branch
git rm -rf *
git branch -D "$branch"
git branch -m "$branch"
git config user.name "GitHub Actions Bot"
git config user.email "<>"
cp "${{ github.workspace }}/pr_ui/${video}.mp4" .
git add .
git commit -m "${name} video for commit ${{ env.SHA }}"
git push origin "$branch" --force
done
- name: Setup FFmpeg
uses: AnimMouse/setup-ffmpeg@ae28d57dabbb148eff63170b6bf7f2b60062cbae
- name: Finding diffs
if: github.event_name == 'pull_request_target'
id: find_diff
run: |
export PYTHONPATH=${{ github.workspace }}
baseurl="https://github.com/sunnypilot/ci-artifacts/raw/refs/heads/${{ env.BRANCH_NAME }}"
COMMENT=""
for variant in $VARIANTS; do
IFS=':' read -r name video _ <<< "$variant"
diff_name="${name}_diff"
mv "${{ github.workspace }}/pr_ui/${video}.mp4" "${{ github.workspace }}/pr_ui/${video}_proposed.mp4"
cp "${{ github.workspace }}/master_${name}/${video}.mp4" "${{ github.workspace }}/pr_ui/${video}_master.mp4"
diff_exit_code=0
python3 ${{ github.workspace }}/selfdrive/ui/tests/diff/diff.py \
"${{ github.workspace }}/pr_ui/${video}_master.mp4" \
"${{ github.workspace }}/pr_ui/${video}_proposed.mp4" \
"${diff_name}.html" --basedir "$baseurl" --no-open || diff_exit_code=$?
cp "${{ github.workspace }}/selfdrive/ui/tests/diff/report/${diff_name}.html" "${{ github.workspace }}/pr_ui/"
cp "${{ github.workspace }}/selfdrive/ui/tests/diff/report/${diff_name}.mp4" "${{ github.workspace }}/pr_ui/"
REPORT_URL="https://sunnypilot.github.io/ci-artifacts/${diff_name}_pr_${{ github.event.number }}.html"
if [ $diff_exit_code -eq 0 ]; then
COMMENT+="**${name}**: Videos are identical! [View Diff Report]($REPORT_URL)"$'\n'
else
COMMENT+="**${name}**: ⚠️ <strong>Videos differ!</strong> [View Diff Report]($REPORT_URL)"$'\n'
fi
done
{
echo "COMMENT<<EOF"
echo "$COMMENT"
echo "EOF"
} >> "$GITHUB_OUTPUT"
- name: Saving proposed ui
if: github.event_name == 'pull_request_target'
working-directory: ${{ github.workspace }}/master_mici
run: |
git config user.name "GitHub Actions Bot"
git config user.email "<>"
git checkout --orphan=${{ env.BRANCH_NAME }}
git rm -rf *
mv ${{ github.workspace }}/pr_ui/* .
git add .
git commit -m "ui videos for PR #${{ github.event.number }}"
git push origin ${{ env.BRANCH_NAME }} --force
# Append diff reports to report files branch
git fetch origin ${{ env.REPORT_FILES_BRANCH_NAME }}
git checkout ${{ env.REPORT_FILES_BRANCH_NAME }}
for variant in $VARIANTS; do
IFS=':' read -r name _ _ <<< "$variant"
diff_name="${name}_diff"
cp "${{ github.workspace }}/selfdrive/ui/tests/diff/report/${diff_name}.html" "${diff_name}_pr_${{ github.event.number }}.html"
git add "${diff_name}_pr_${{ github.event.number }}.html"
done
git commit -m "ui diff reports for PR #${{ github.event.number }}" || echo "No changes to commit"
git push origin ${{ env.REPORT_FILES_BRANCH_NAME }}
- name: Comment on PR
if: github.event_name == 'pull_request_target'
uses: thollander/actions-comment-pull-request@v2
with:
message: |
<!-- _(run_id_ui_preview **${{ github.run_id }}**)_ -->
## UI Preview
${{ steps.find_diff.outputs.COMMENT }}
comment_tag: run_id_ui_preview
pr_number: ${{ github.event.number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -1,52 +0,0 @@
name: 'Wait for Tests'
description: 'Action to wait for workflow tests to start and complete'
inputs:
workflow:
description: 'The workflow file name to monitor'
required: true
default: 'tests.yaml'
branch:
description: 'The branch to monitor (defaults to current branch)'
required: false
default: ''
github-token:
description: 'GitHub token for API access'
required: true
wait-time:
description: 'Initial sleep time in seconds before monitoring starts'
required: false
default: '30'
should-wait-for-start:
description: 'Whether to wait for tests to start'
required: false
default: false
runs:
using: 'composite'
steps:
- name: Wait for tests to start
if: inputs.should-wait-for-start == 'true'
shell: bash
run: |
echo "Sleeping for ${{ inputs.wait-time }} seconds to give some time for the action to start and then we'll wait"
sleep ${{ inputs.wait-time }}
- name: Wait for tests to finish
shell: bash
run: |
BRANCH="${{ inputs.branch || github.head_ref || github.ref_name }}"
echo "Looking for workflow runs of ${{ inputs.workflow }} on branch $BRANCH"
RUN_ID=$(gh run list --workflow=${{ inputs.workflow }} --branch="$BRANCH" --limit=1 --json databaseId --jq '.[0].databaseId')
echo "Watching run ID: $RUN_ID"
gh run watch "$RUN_ID"
CONCLUSION=$(gh run view "$RUN_ID" --json conclusion --jq '.conclusion')
echo "Run concluded with: $CONCLUSION"
if [[ "$CONCLUSION" != "success" ]]; then
echo "❌ Workflow run failed with conclusion: $CONCLUSION"
exit 1
fi
env:
GITHUB_TOKEN: ${{ inputs.github-token }}
+37 -22
View File
@@ -10,16 +10,17 @@ venv/
.overlay_init
.overlay_consistent
.sconsign.dblite
model2.png
a.out
.hypothesis
.cache/
bin/
/docs_site/
*.mp4
*.dylib
*.DSYM
*.d
*.pem
*.pyc
*.pyo
.*.swp
@@ -36,60 +37,74 @@ bin/
*.class
*.pyxbldc
*.vcd
*.mo
*.qm
*_pyx.cpp
*.stats
*.pkl
*.pkl*
config.json
clcache
compile_commands.json
compare_runtime*.html
selfdrive/modeld/models/tg_input_devices.json
# build artifacts
docs_site/
persist
selfdrive/pandad/pandad
cereal/services.h
cereal/gen
cereal/messaging/bridge
selfdrive/mapd/default_speeds_by_region.json
system/proclogd/proclogd
selfdrive/ui/translations/tmp
selfdrive/test/longitudinal_maneuvers/out
selfdrive/car/tests/cars_dump
system/camerad/camerad
system/camerad/test/ae_gray_test
notebooks
hyperthneed
provisioning
.coverage*
coverage.xml
htmlcov
pandaextra
.mypy_cache/
flycheck_*
cppcheck_report.txt
comma*.sh
selfdrive/modeld/models/*.pkl
sunnypilot/modeld*/thneed/compile
sunnypilot/modeld*/models/*.thneed
sunnypilot/modeld*/models/*.pkl
# openpilot log files
*.bz2
*.zst
*.rlog
build/
!**/.gitkeep
poetry.toml
Pipfile
### VisualStudioCode ###
*.vsix
.history
.ionide
.vscode/*
.history/
!.vscode/settings.json
!.vscode/tasks.json
!.vscode/launch.json
!.vscode/extensions.json
!.vscode/*.code-snippets
# agents
.claude/
.context/
PLAN.md
TASK.md
CLAUDE.md
SKILL.md
# Local History for Visual Studio Code
.history/
# Built Visual Studio Code Extensions
*.vsix
### VisualStudioCode Patch ###
# Ignore all local history of files
.history
.ionide
### JetBrains ###
!.idea/customTargets.xml
-21
View File
@@ -1,21 +0,0 @@
[submodule "panda"]
path = panda
url = https://github.com/sunnyhaibin/panda.git
[submodule "opendbc"]
path = opendbc_repo
url = https://github.com/sunnypilot/opendbc.git
[submodule "msgq"]
path = msgq_repo
url = https://github.com/sunnypilot/msgq.git
[submodule "rednose_repo"]
path = rednose_repo
url = https://github.com/commaai/rednose.git
[submodule "teleoprtc_repo"]
path = teleoprtc_repo
url = https://github.com/commaai/teleoprtc
[submodule "tinygrad"]
path = tinygrad_repo
url = https://github.com/sunnypilot/tinygrad.git
[submodule "sunnypilot/neural_network_data"]
path = sunnypilot/neural_network_data
url = https://github.com/sunnypilot/neural-network-data.git
-25
View File
@@ -1,25 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="CLionExternalBuildManager">
<target id="a62f99e8-5ec4-434c-8122-49efed5af108" name="uv Scons Build Debug" defaultType="TOOL">
<configuration id="b93ec964-16e5-4962-a12e-3ed360ce8f02" name="uv Scons Build Debug">
<build type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Build Debug" />
</build>
<clean type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Clean" />
</clean>
</configuration>
</target>
<target id="edd8ad9d-183b-467c-a355-0d9a0ecab026" name="uv Scons Build Release" defaultType="TOOL">
<configuration id="09523339-5ce3-4223-ab9e-904f38ad7752" name="uv Scons Build Release">
<build type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Build Release" />
</build>
<clean type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Clean" />
</clean>
</configuration>
</target>
</component>
</project>
-23
View File
@@ -1,23 +0,0 @@
<toolSet name="External Tools">
<tool name="uv Scons Build Debug" showInMainMenu="false" showInEditor="false" showInProject="false" showInSearchPopup="false" disabled="false" useConsole="true" showConsoleOnStdOut="false" showConsoleOnStdErr="false" synchronizeAfterRun="true">
<exec>
<option name="COMMAND" value="bash" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -u -j$(nproc) --ccflags=\&quot;-fno-inline\&quot;&quot;" />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
<tool name="uv Scons Clean" showInMainMenu="false" showInEditor="false" showInProject="false" showInSearchPopup="false" disabled="false" useConsole="true" showConsoleOnStdOut="false" showConsoleOnStdErr="false" synchronizeAfterRun="true">
<exec>
<option name="COMMAND" value="bash" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -c&quot; " />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
<tool name="uv Scons Build Release" showInMainMenu="false" showInEditor="false" showInProject="false" showInSearchPopup="false" disabled="false" useConsole="true" showConsoleOnStdOut="false" showConsoleOnStdErr="false" synchronizeAfterRun="true">
<exec>
<option name="COMMAND" value="bash" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -u -j$(nproc)&quot; " />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
</toolSet>
-4
View File
@@ -1,4 +0,0 @@
[lfs]
url = https://gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git/info/lfs
pushurl = ssh://git@gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git
locksverify = false
-4
View File
@@ -1,4 +0,0 @@
[lfs]
url = https://gitlab.com/commaai/openpilot-lfs.git/info/lfs
pushurl = ssh://git@gitlab.com/commaai/openpilot-lfs.git
locksverify = false
-1
View File
@@ -1 +0,0 @@
3.12.13
-10
View File
@@ -1,10 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Build Debug" type="CLionExternalRunConfiguration" factoryName="Application" REDIRECT_INPUT="false" ELEVATE="false" USE_EXTERNAL_CONSOLE="false" EMULATE_TERMINAL="false" WORKING_DIR="file://$ProjectFileDir$/selfdrive/ui" PASS_PARENT_ENVS_2="true" PROJECT_NAME="sunnypilot" TARGET_NAME="uv Scons Build Debug" CONFIG_NAME="uv Scons Build Debug" RUN_PATH="ui">
<envs>
<env name="QT_DBL_CLICK_DIST" value="150" />
</envs>
<method v="2">
<option name="CLION.EXTERNAL.BUILD" enabled="true" />
</method>
</configuration>
</component>
-10
View File
@@ -1,10 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Build Release" type="CLionExternalRunConfiguration" factoryName="Application" REDIRECT_INPUT="false" ELEVATE="false" USE_EXTERNAL_CONSOLE="false" EMULATE_TERMINAL="false" WORKING_DIR="file://$ProjectFileDir$/selfdrive/ui" PASS_PARENT_ENVS_2="true" PROJECT_NAME="sunnypilot" TARGET_NAME="uv Scons Build Release" CONFIG_NAME="uv Scons Build Release" RUN_PATH="ui">
<envs>
<env name="QT_DBL_CLICK_DIST" value="150" />
</envs>
<method v="2">
<option name="CLION.EXTERNAL.BUILD" enabled="true" />
</method>
</configuration>
</component>
-26
View File
@@ -1,26 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Build_BIG_UI" type="PythonConfigurationType" factoryName="Python">
<module name="sunnypilot" />
<option name="ENV_FILES" value="" />
<option name="INTERPRETER_OPTIONS" value="" />
<option name="PARENT_ENVS" value="true" />
<envs>
<env name="BIG" value="1" />
</envs>
<option name="SDK_HOME" value="" />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$/" />
<option name="IS_MODULE_SDK" value="true" />
<option name="ADD_CONTENT_ROOTS" value="true" />
<option name="ADD_SOURCE_ROOTS" value="true" />
<option name="SCRIPT_NAME" value="$ProjectFileDir$/selfdrive/ui/ui.py" />
<option name="PARAMETERS" value="" />
<option name="SHOW_COMMAND_LINE" value="false" />
<option name="EMULATE_TERMINAL" value="false" />
<option name="MODULE_MODE" value="false" />
<option name="REDIRECT_INPUT" value="false" />
<option name="INPUT_FILE" value="" />
<method v="2">
<option name="ToolBeforeRunTask" enabled="true" actionId="Tool_External Tools_uv Scons Build Debug" />
</method>
</configuration>
</component>
-23
View File
@@ -1,23 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Build_SMALL_UI" type="PythonConfigurationType" factoryName="Python">
<module name="sunnypilot" />
<option name="ENV_FILES" value="" />
<option name="INTERPRETER_OPTIONS" value="" />
<option name="PARENT_ENVS" value="true" />
<option name="SDK_HOME" value="" />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$/" />
<option name="IS_MODULE_SDK" value="true" />
<option name="ADD_CONTENT_ROOTS" value="true" />
<option name="ADD_SOURCE_ROOTS" value="true" />
<option name="SCRIPT_NAME" value="$ProjectFileDir$/selfdrive/ui/ui.py" />
<option name="PARAMETERS" value="" />
<option name="SHOW_COMMAND_LINE" value="false" />
<option name="EMULATE_TERMINAL" value="false" />
<option name="MODULE_MODE" value="false" />
<option name="REDIRECT_INPUT" value="false" />
<option name="INPUT_FILE" value="" />
<method v="2">
<option name="ToolBeforeRunTask" enabled="true" actionId="Tool_External Tools_uv Scons Build Debug" />
</method>
</configuration>
</component>
+1 -43
View File
@@ -23,11 +23,6 @@
"id": "args",
"description": "Arguments to pass to the process",
"type": "promptString"
},
{
"id": "replayArg",
"type": "promptString",
"description": "Enter route or segment to replay."
}
],
"configurations": [
@@ -45,44 +40,7 @@
"type": "cppdbg",
"request": "launch",
"program": "${workspaceFolder}/${input:cpp_process}",
"cwd": "${workspaceFolder}"
},
{
"name": "Attach LLDB to Replay drive",
"type": "lldb",
"request": "attach",
"pid": "${command:pickMyProcess}",
"sourceMap": {
".": "${workspaceFolder}/opendbc/safety"
},
"initCommands": [
"script import time; time.sleep(3)"
]
},
{
"name": "Replay drive",
"type": "debugpy",
"request": "launch",
"program": "${workspaceFolder}/opendbc/safety/tests/safety_replay/replay_drive.py",
"args": [
"${input:replayArg}"
],
"console": "integratedTerminal",
"justMyCode": false,
"env": {
"PYTHONPATH": "${workspaceFolder}"
},
"subProcess": true,
"stopOnEntry": false
}
],
"compounds": [
{
"name": "Replay drive + Safety LLDB",
"configurations": [
"Replay drive",
"Attach LLDB to Replay drive"
]
"cwd": "${workspaceFolder}",
}
]
}
+1
View File
@@ -21,6 +21,7 @@
"common/**",
"selfdrive/**",
"system/**",
"third_party/**",
"tools/**",
]
}
-1320
View File
File diff suppressed because it is too large Load Diff
+12
View File
@@ -0,0 +1,12 @@
FROM ghcr.io/commaai/openpilot-base:latest
ENV PYTHONUNBUFFERED=1
ENV OPENPILOT_PATH=/home/batman/openpilot
RUN mkdir -p ${OPENPILOT_PATH}
WORKDIR ${OPENPILOT_PATH}
COPY . ${OPENPILOT_PATH}/
RUN scons --cache-readonly -j$(nproc)
+82
View File
@@ -0,0 +1,82 @@
FROM ubuntu:24.04
ENV PYTHONUNBUFFERED=1
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -y --no-install-recommends sudo tzdata locales ssh pulseaudio xvfb x11-xserver-utils gnome-screenshot python3-tk python3-dev && \
rm -rf /var/lib/apt/lists/*
RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && locale-gen
ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US:en
ENV LC_ALL=en_US.UTF-8
COPY tools/install_ubuntu_dependencies.sh /tmp/tools/
RUN /tmp/tools/install_ubuntu_dependencies.sh && \
rm -rf /var/lib/apt/lists/* /tmp/* && \
cd /usr/lib/gcc/arm-none-eabi/* && \
rm -rf arm/ thumb/nofp thumb/v6* thumb/v8* thumb/v7+fp thumb/v7-r+fp.sp
# Add OpenCL
RUN apt-get update && apt-get install -y --no-install-recommends \
apt-utils \
alien \
unzip \
tar \
curl \
xz-utils \
dbus \
gcc-arm-none-eabi \
tmux \
vim \
libx11-6 \
wget \
&& rm -rf /var/lib/apt/lists/*
RUN mkdir -p /tmp/opencl-driver-intel && \
cd /tmp/opencl-driver-intel && \
wget https://github.com/intel/llvm/releases/download/2024-WW14/oclcpuexp-2024.17.3.0.09_rel.tar.gz && \
wget https://github.com/oneapi-src/oneTBB/releases/download/v2021.12.0/oneapi-tbb-2021.12.0-lin.tgz && \
mkdir -p /opt/intel/oclcpuexp_2024.17.3.0.09_rel && \
cd /opt/intel/oclcpuexp_2024.17.3.0.09_rel && \
tar -zxvf /tmp/opencl-driver-intel/oclcpuexp-2024.17.3.0.09_rel.tar.gz && \
mkdir -p /etc/OpenCL/vendors && \
echo /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64/libintelocl.so > /etc/OpenCL/vendors/intel_expcpu.icd && \
cd /opt/intel && \
tar -zxvf /tmp/opencl-driver-intel/oneapi-tbb-2021.12.0-lin.tgz && \
ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbb.so /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbbmalloc.so /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbb.so.12 /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbbmalloc.so.2 /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
mkdir -p /etc/ld.so.conf.d && \
echo /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 > /etc/ld.so.conf.d/libintelopenclexp.conf && \
ldconfig -f /etc/ld.so.conf.d/libintelopenclexp.conf && \
cd / && \
rm -rf /tmp/opencl-driver-intel
ENV NVIDIA_VISIBLE_DEVICES=all
ENV NVIDIA_DRIVER_CAPABILITIES=graphics,utility,compute
ENV QTWEBENGINE_DISABLE_SANDBOX=1
RUN dbus-uuidgen > /etc/machine-id
RUN apt-get update && apt-get install -y fonts-noto-cjk fonts-noto-color-emoji
ARG USER=batman
ARG USER_UID=1001
RUN useradd -m -s /bin/bash -u $USER_UID $USER
RUN usermod -aG sudo $USER
RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
USER $USER
COPY --chown=$USER pyproject.toml uv.lock /home/$USER
COPY --chown=$USER tools/install_python_dependencies.sh /home/$USER/tools/
ENV VIRTUAL_ENV=/home/$USER/.venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
RUN cd /home/$USER && \
tools/install_python_dependencies.sh && \
rm -rf tools/ pyproject.toml uv.lock .cache
USER root
RUN sudo git config --global --add safe.directory /tmp/openpilot
+12
View File
@@ -0,0 +1,12 @@
FROM ghcr.io/sunnypilot/sunnypilot-tici-base:latest
ENV PYTHONUNBUFFERED=1
ENV OPENPILOT_PATH=/home/batman/openpilot
RUN mkdir -p ${OPENPILOT_PATH}
WORKDIR ${OPENPILOT_PATH}
COPY . ${OPENPILOT_PATH}/
RUN scons --cache-readonly -j$(nproc)
+83
View File
@@ -0,0 +1,83 @@
FROM ubuntu:24.04
ENV PYTHONUNBUFFERED=1
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -y --no-install-recommends sudo tzdata locales ssh pulseaudio xvfb x11-xserver-utils gnome-screenshot python3-tk python3-dev && \
rm -rf /var/lib/apt/lists/*
RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && locale-gen
ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US:en
ENV LC_ALL=en_US.UTF-8
COPY tools/install_ubuntu_dependencies.sh /tmp/tools/
RUN /tmp/tools/install_ubuntu_dependencies.sh && \
rm -rf /var/lib/apt/lists/* /tmp/* && \
cd /usr/lib/gcc/arm-none-eabi/* && \
rm -rf arm/ thumb/nofp thumb/v6* thumb/v8* thumb/v7+fp thumb/v7-r+fp.sp
# Add OpenCL
RUN apt-get update && apt-get install -y --no-install-recommends \
apt-utils \
alien \
unzip \
tar \
curl \
xz-utils \
dbus \
gcc-arm-none-eabi \
tmux \
vim \
libx11-6 \
wget \
rsync \
&& rm -rf /var/lib/apt/lists/*
RUN mkdir -p /tmp/opencl-driver-intel && \
cd /tmp/opencl-driver-intel && \
wget https://github.com/intel/llvm/releases/download/2024-WW14/oclcpuexp-2024.17.3.0.09_rel.tar.gz && \
wget https://github.com/oneapi-src/oneTBB/releases/download/v2021.12.0/oneapi-tbb-2021.12.0-lin.tgz && \
mkdir -p /opt/intel/oclcpuexp_2024.17.3.0.09_rel && \
cd /opt/intel/oclcpuexp_2024.17.3.0.09_rel && \
tar -zxvf /tmp/opencl-driver-intel/oclcpuexp-2024.17.3.0.09_rel.tar.gz && \
mkdir -p /etc/OpenCL/vendors && \
echo /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64/libintelocl.so > /etc/OpenCL/vendors/intel_expcpu.icd && \
cd /opt/intel && \
tar -zxvf /tmp/opencl-driver-intel/oneapi-tbb-2021.12.0-lin.tgz && \
ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbb.so /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbbmalloc.so /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbb.so.12 /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbbmalloc.so.2 /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
mkdir -p /etc/ld.so.conf.d && \
echo /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 > /etc/ld.so.conf.d/libintelopenclexp.conf && \
ldconfig -f /etc/ld.so.conf.d/libintelopenclexp.conf && \
cd / && \
rm -rf /tmp/opencl-driver-intel
ENV NVIDIA_VISIBLE_DEVICES=all
ENV NVIDIA_DRIVER_CAPABILITIES=graphics,utility,compute
ENV QTWEBENGINE_DISABLE_SANDBOX=1
RUN dbus-uuidgen > /etc/machine-id
RUN apt-get update && apt-get install -y fonts-noto-cjk fonts-noto-color-emoji
ARG USER=batman
ARG USER_UID=1001
RUN useradd -m -s /bin/bash -u $USER_UID $USER
RUN usermod -aG sudo $USER
RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
USER $USER
COPY --chown=$USER pyproject.toml uv.lock /home/$USER
COPY --chown=$USER tools/install_python_dependencies.sh /home/$USER/tools/
ENV VIRTUAL_ENV=/home/$USER/.venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
RUN cd /home/$USER && \
tools/install_python_dependencies.sh && \
rm -rf tools/ pyproject.toml uv.lock .cache
USER root
RUN sudo git config --global --add safe.directory /tmp/openpilot
Vendored
-259
View File
@@ -1,259 +0,0 @@
def retryWithDelay(int maxRetries, int delay, Closure body) {
for (int i = 0; i < maxRetries; i++) {
try {
return body()
} catch (Exception e) {
sleep(delay)
}
}
throw Exception("Failed after ${maxRetries} retries")
}
def device(String ip, String step_label, String cmd) {
withCredentials([file(credentialsId: 'id_rsa', variable: 'key_file')]) {
def ssh_cmd = """
ssh -o ControlMaster=auto -o ControlPath=/tmp/ssh_control_%C -o ControlPersist=yes -o ConnectTimeout=5 -o ServerAliveInterval=5 -o ServerAliveCountMax=2 -o BatchMode=yes -o StrictHostKeyChecking=no -i ${key_file} 'comma@${ip}' exec /usr/bin/bash <<'END'
set -e
export TERM=xterm-256color
shopt -s huponexit # kill all child processes when the shell exits
export CI=1
export PYTHONWARNINGS=error
#export LOGPRINT=debug # this has gotten too spammy...
export TEST_DIR=${env.TEST_DIR}
export SOURCE_DIR=${env.SOURCE_DIR}
export GIT_BRANCH=${env.GIT_BRANCH}
export GIT_COMMIT=${env.GIT_COMMIT}
export CI_ARTIFACTS_TOKEN=${env.CI_ARTIFACTS_TOKEN}
export GITHUB_COMMENTS_TOKEN=${env.GITHUB_COMMENTS_TOKEN}
export AZURE_TOKEN='${env.AZURE_TOKEN}'
# only use 1 thread for tici tests since most require HIL
export PYTEST_ADDOPTS="-n0 -s"
export GIT_SSH_COMMAND="ssh -i /data/gitkey"
source ~/.bash_profile
if [ -f /TICI ]; then
source /etc/profile
rm -rf /tmp/tmp*
rm -rf ~/.commacache
rm -rf /dev/shm/*
rm -rf /dev/tmp/tmp*
if ! systemctl is-active --quiet systemd-resolved; then
echo "restarting resolved"
sudo systemctl start systemd-resolved
sleep 3
fi
# restart aux USB
if [ -e /sys/bus/usb/drivers/hub/3-0:1.0 ]; then
echo "restarting aux usb"
echo "3-0:1.0" | sudo tee /sys/bus/usb/drivers/hub/unbind
sleep 0.5
echo "3-0:1.0" | sudo tee /sys/bus/usb/drivers/hub/bind
fi
fi
if [ -f /data/openpilot/launch_env.sh ]; then
source /data/openpilot/launch_env.sh
fi
ln -snf ${env.TEST_DIR} /data/pythonpath
cd ${env.TEST_DIR} || true
time ${cmd}
END"""
sh script: ssh_cmd, label: step_label
}
}
def deviceStage(String stageName, String deviceType, List extra_env, def steps) {
stage(stageName) {
if (currentBuild.result != null) {
return
}
if (isReplay()) {
error("REPLAYING TESTS IS NOT ALLOWED. FIX THEM INSTEAD.")
}
def extra = extra_env.collect { "export ${it}" }.join('\n');
def branch = env.BRANCH_NAME ?: 'master';
def gitDiff = sh returnStdout: true, script: 'curl -s -H "Authorization: Bearer ${GITHUB_COMMENTS_TOKEN}" https://api.github.com/repos/commaai/openpilot/compare/master...${GIT_BRANCH} | jq .files[].filename || echo "/"', label: 'Getting changes'
lock(resource: "", label: deviceType, inversePrecedence: true, variable: 'device_ip', quantity: 1, resourceSelectStrategy: 'random') {
docker.image('ghcr.io/commaai/alpine-ssh').inside('--user=root') {
timeout(time: 35, unit: 'MINUTES') {
retry (3) {
def date = sh(script: 'date', returnStdout: true).trim();
device(device_ip, "set time", "date -s '" + date + "'")
device(device_ip, "git checkout", extra + "\n" + readFile("selfdrive/test/setup_device_ci.sh"))
}
steps.each { item ->
def name = item[0]
def cmd = item[1]
def args = item[2]
def diffPaths = args.diffPaths ?: []
def cmdTimeout = args.timeout ?: 9999
if (branch != "master" && !branch.contains("__jenkins_loop_") && diffPaths && !hasPathChanged(gitDiff, diffPaths)) {
println "Skipping ${name}: no changes in ${diffPaths}."
return
} else {
timeout(time: cmdTimeout, unit: 'SECONDS') {
device(device_ip, name, cmd)
}
}
}
}
}
}
}
}
def hasPathChanged(String gitDiff, List<String> paths) {
for (path in paths) {
if (gitDiff.contains(path)) {
return true
}
}
return false
}
def isReplay() {
def replayClass = "org.jenkinsci.plugins.workflow.cps.replay.ReplayCause"
return currentBuild.rawBuild.getCauses().any{ cause -> cause.toString().contains(replayClass) }
}
def setupCredentials() {
withCredentials([
string(credentialsId: 'azure_token', variable: 'AZURE_TOKEN'),
]) {
env.AZURE_TOKEN = "${AZURE_TOKEN}"
}
withCredentials([
string(credentialsId: 'ci_artifacts_pat', variable: 'CI_ARTIFACTS_TOKEN'),
]) {
env.CI_ARTIFACTS_TOKEN = "${CI_ARTIFACTS_TOKEN}"
}
withCredentials([
string(credentialsId: 'post_comments_github_pat', variable: 'GITHUB_COMMENTS_TOKEN'),
]) {
env.GITHUB_COMMENTS_TOKEN = "${GITHUB_COMMENTS_TOKEN}"
}
}
def step(String name, String cmd, Map args = [:]) {
return [name, cmd, args]
}
node {
env.CI = "1"
env.PYTHONWARNINGS = "error"
env.TEST_DIR = "/data/openpilot"
env.SOURCE_DIR = "/data/openpilot_source/"
setupCredentials()
env.GIT_BRANCH = checkout(scm).GIT_BRANCH
env.GIT_COMMIT = checkout(scm).GIT_COMMIT
def excludeBranches = ['__nightly', 'devel', 'devel-staging',
'release-tizi', 'release-tizi-staging', 'release-mici', 'release-mici-staging', 'testing-closet*', 'hotfix-*']
def excludeRegex = excludeBranches.join('|').replaceAll('\\*', '.*')
if (env.BRANCH_NAME != 'master' && !env.BRANCH_NAME.contains('__jenkins_loop_')) {
properties([
disableConcurrentBuilds(abortPrevious: true)
])
}
try {
if (env.BRANCH_NAME == 'devel-staging') {
deviceStage("build release-tizi-staging", "tizi-needs-can", [], [
step("build release-tizi-staging", "RELEASE_BRANCH=release-tizi-staging,release-mici-staging $SOURCE_DIR/release/build_release.sh"),
])
}
if (env.BRANCH_NAME == '__nightly') {
parallel (
'nightly': {
deviceStage("build nightly", "tizi-needs-can", [], [
step("build nightly", "RELEASE_BRANCH=nightly $SOURCE_DIR/release/build_release.sh"),
])
},
'nightly-dev': {
deviceStage("build nightly-dev", "tizi-needs-can", [], [
step("build nightly-dev", "PANDA_DEBUG_BUILD=1 RELEASE_BRANCH=nightly-dev $SOURCE_DIR/release/build_release.sh"),
])
},
)
}
if (!env.BRANCH_NAME.matches(excludeRegex)) {
parallel (
'onroad tests': {
deviceStage("onroad", "tizi-needs-can", ["UNSAFE=1"], [
step("build openpilot", "cd system/manager && ./build.py"),
step("check dirty", "release/check-dirty.sh"),
step("onroad tests", "pytest selfdrive/test/test_onroad.py -s", [timeout: 60]),
])
},
'HW + Unit Tests': {
deviceStage("tizi-hardware", "tizi-common", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test power draw", "pytest -s selfdrive/test//test_power_draw.py"),
step("test encoder", "LD_LIBRARY_PATH=/usr/local/lib pytest system/loggerd/tests/test_encoder.py", [diffPaths: ["system/loggerd/"]]),
step("test manager", "pytest system/manager/test/test_manager.py"),
])
},
'camerad OX03C10': {
deviceStage("OX03C10", "tizi-ox03c10", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test pandad", "pytest selfdrive/pandad/tests/test_pandad.py"),
step("test camerad", "pytest system/camerad/test/test_camerad.py", [timeout: 90]),
])
},
'camerad OS04C10': {
deviceStage("OS04C10", "tici-os04c10", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test pandad", "pytest selfdrive/pandad/tests/test_pandad.py"),
step("test camerad", "pytest system/camerad/test/test_camerad.py", [timeout: 90]),
])
},
'sensord': {
deviceStage("LSM + MMC", "tizi-lsmc", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test sensord", "pytest system/sensord/tests/test_sensord.py"),
])
},
'replay': {
deviceStage("model-replay", "tizi-replay", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py", [diffPaths: ["selfdrive/modeld/", "tinygrad_repo", "selfdrive/test/process_replay/model_replay.py"]]),
step("model replay", "selfdrive/test/process_replay/model_replay.py", [diffPaths: ["selfdrive/modeld/", "tinygrad_repo", "selfdrive/test/process_replay/model_replay.py"]]),
])
},
'tizi': {
deviceStage("tizi", "tizi", ["UNSAFE=1"], [
step("build openpilot", "cd system/manager && ./build.py"),
step("test pandad loopback", "pytest selfdrive/pandad/tests/test_pandad_loopback.py"),
step("test pandad spi", "pytest selfdrive/pandad/tests/test_pandad_spi.py"),
step("test amp", "pytest common/hardware/tici/tests/test_amplifier.py"),
])
},
)
}
} catch (Exception e) {
currentBuild.result = 'FAILED'
throw e
}
}
+62 -5
View File
@@ -3,18 +3,75 @@
## 🌞 What is sunnypilot?
[sunnypilot](https://github.com/sunnyhaibin/sunnypilot) is a fork of comma.ai's openpilot, an open source driver assistance system. sunnypilot offers the user a unique driving experience for over 300+ supported car makes and models with modified behaviors of driving assist engagements. sunnypilot complies with comma.ai's safety rules as accurately as possible.
## 💭 Join our Community Forum
Join the official sunnypilot community forum to stay up to date with all the latest features and be a part of shaping the future of sunnypilot!
* https://community.sunnypilot.ai/
## 💭 Join our Discord
Join the official sunnypilot Discord server to stay up to date with all the latest features and be a part of shaping the future of sunnypilot!
* https://discord.gg/sunnypilot
![](https://dcbadge.vercel.app/api/server/wRW3meAgtx?style=flat) ![Discord Shield](https://discordapp.com/api/guilds/880416502577266699/widget.png?style=shield)
## Documentation
https://docs.sunnypilot.ai/ is your one stop shop for everything from features to installation to FAQ about the sunnypilot
## 🚘 Running on a dedicated device in a car
First, check out this list of items you'll need to [get started](https://community.sunnypilot.ai/t/getting-started-using-sunnypilot-in-your-supported-car/251).
* A supported device to run this software
* a [comma three](https://comma.ai/shop/products/three) or a [C3X](https://comma.ai/shop/comma-3x)
* This software
* One of [the 300+ supported cars](https://github.com/commaai/openpilot/blob/master/docs/CARS.md). We support Honda, Toyota, Hyundai, Nissan, Kia, Chrysler, Lexus, Acura, Audi, VW, Ford and more. If your car is not supported but has adaptive cruise control and lane-keeping assist, it's likely able to run sunnypilot.
* A [car harness](https://comma.ai/shop/products/car-harness) to connect to your car
Detailed instructions for [how to mount the device in a car](https://comma.ai/setup).
## Installation
Next, refer to the sunnypilot community forum for [installation instructions](https://community.sunnypilot.ai/t/read-before-installing-sunnypilot/254), as well as a complete list of [Recommended Branch Installations](https://community.sunnypilot.ai/t/recommended-branch-installations/235).
Please refer to [Recommended Branches](#-recommended-branches) to find your preferred/supported branch. This guide will assume you want to install the latest `staging-tici` branch.
### If you want to use our newest branches (our rewrite)
> [!TIP]
>You can see the rewrite state on our [rewrite project board](https://github.com/orgs/sunnypilot/projects/2), and to install the new branches, you can use the following links
* sunnypilot not installed or you installed a version before 0.8.17?
1. [Factory reset/uninstall](https://github.com/commaai/openpilot/wiki/FAQ#how-can-i-reset-the-device) the previous software if you have another software/fork installed.
2. After factory reset/uninstall and upon reboot, select `Custom Software` when given the option.
3. Input the installation URL per [Recommended Branches](#-recommended-branches). Example: ```https://staging-tici.sunnypilot.ai```.
4. Complete the rest of the installation following the onscreen instructions.
* sunnypilot already installed and you installed a version after 0.8.17?
1. On the comma three, go to `Settings` ▶️ `Software`.
2. At the `Download` option, press `CHECK`. This will fetch the list of latest branches from sunnypilot.
3. At the `Target Branch` option, press `SELECT` to open the Target Branch selector.
4. Scroll to select the desired branch per Recommended Branches (see below). Example: `staging-tici`
| Branch | Installation URL |
|:---------------:|:---------------------------------------------:|
| `staging-tici` | `https://staging-tici.sunnypilot.ai` |
| `custom-branch` | `https://install.sunnypilot.ai/{branch_name}` |
| `release-tici` | **Not yet available**. |
> [!TIP]
> You can use sunnypilot/targetbranch as an install URL. Example: 'sunnypilot/staging-tici'.
> [!NOTE]
> Do you require further assistance with software installation? Join the [sunnypilot Discord server](https://discord.sunnypilot.com) and message us in the `#installation-help` channel.
<details>
<summary>Older legacy branches</summary>
### If you want to use our older legacy branches (*not recommended*)
> [**IMPORTANT**]
> It is recommended to [re-flash AGNOS](https://flash.comma.ai/) if you intend to downgrade from the new branches.
> You can still restore the latest sunnylink backup made on the old branches.
| Branch | Installation URL |
|:------------:|:--------------------------------:|
| `release-c3` | https://release-c3.sunnypilot.ai |
| `staging-c3` | https://staging-c3.sunnypilot.ai |
| `dev-c3` | https://dev-c3.sunnypilot.ai |
</details>
## 🎆 Pull Requests
We welcome both pull requests and issues on GitHub. Bug fixes are encouraged.
+2 -46
View File
@@ -1,51 +1,7 @@
Version 0.11.2 (2026-06-15)
========================
Version 0.11.1 (2026-05-18)
========================
* New driver monitoring model
* Improved image processing pipeline for driver camera
* Improved thermal policy for comma four
* Acura MDX 2022-24 support thanks to mvl-boston!
* Rivian R1S and R1T 2025 support thanks to lukasloetkolben!
Version 0.11.0 (2026-03-17)
========================
* New driving model #36798
* Fully trained using a learned simulator
* Improved longitudinal performance in Experimental mode
* Reduce comma four standby power usage by 77% to 52 mW
* Kia K7 2017 support thanks to royjr!
* Lexus LS 2018 support thanks to Hacheoy!
Version 0.10.3 (2025-12-17)
========================
* New driving model #36249
* New temporal policy architecture
* New on-policy training physics noise model
* New driver monitoring model #36409
* Trained on a new dataset, including comma four data
* Improved inter-process communication memory efficiency
Version 0.10.2 (2025-11-19)
========================
* comma four support
Version 0.10.1 (2025-09-08)
========================
* New driving model #36276
* World Model: removed global localization inputs
* World Model: 2x the number of parameters
* World Model: trained on 4x the number of segments
* VAE Compression Model: new architecture and training objective
* Driving Vision Model: trained on 4x the number of segments
* New Driver Monitoring model #36198
* Acura TLX 2021 support thanks to MVL!
* Honda City 2023 support thanks to vanillagorillaa and drFritz!
* Honda N-Box 2018 support thanks to miettal!
* Honda Odyssey 2021-25 support thanks to csouers and MVL!
* Honda Passport 2026 support thanks to vanillagorillaa and MVL!
* Record driving feedback using LKAS button
* Honda City 2023 support thanks to drFritz!
Version 0.10.0 (2025-08-05)
========================
-312
View File
@@ -1,312 +0,0 @@
import os
import subprocess
import sys
import sysconfig
import platform
import shlex
import importlib
import numpy as np
import SCons.Errors
from SCons.Defaults import _stripixes
TICI = os.path.isfile('/TICI')
SCons.Warnings.warningAsException(True)
Decider('MD5-timestamp')
SetOption('num_jobs', max(1, int(os.cpu_count()/(1 if "CI" in os.environ else 2))))
AddOption('--ccflags', action='store', type='string', default='', help='pass arbitrary flags over the command line')
AddOption('--verbose', action='store_true', default=False, help='show full build commands')
release = not os.path.exists(File('#.gitattributes').abspath) # file absent on release branch, see release_files.py
AddOption('--minimal',
action='store_false',
dest='extras',
default=(not TICI and not release),
help='the minimum build to run openpilot. no tests, tools, etc.')
# Detect platform
arch = subprocess.check_output(["uname", "-m"], encoding='utf8').rstrip()
if platform.system() == "Darwin":
arch = "Darwin"
elif arch == "aarch64" and TICI:
arch = "larch64"
assert arch in [
"larch64", # linux tici arm64
"aarch64", # linux pc arm64
"x86_64", # linux pc x64
"Darwin", # macOS arm64 (x86 not supported)
]
pkg_names = ['acados', 'bzip2', 'capnproto', 'catch2', 'eigen', 'ffmpeg', 'json11', 'libjpeg', 'libyuv', 'ncurses', 'zeromq', 'zstd']
pkgs = [importlib.import_module(name) for name in pkg_names]
acados = pkgs[pkg_names.index('acados')]
acados_include_dirs = [
acados.INCLUDE_DIR,
os.path.join(acados.INCLUDE_DIR, "blasfeo", "include"),
os.path.join(acados.INCLUDE_DIR, "hpipm", "include"),
]
# ***** enforce a whitelist of system libraries *****
# this prevents silently relying on a 3rd party package,
# e.g. apt-installed libusb. all libraries should either
# be distributed with all Linux distros and macOS, or
# vendored in commaai/dependencies.
allowed_system_libs = {
"EGL", "GLESv2", "GL",
"Qt5Charts", "Qt5Core", "Qt5Gui", "Qt5Widgets",
"dl", "drm", "gbm", "m", "pthread",
}
def _resolve_lib(env, name):
for d in env.Flatten(env.get('LIBPATH', [])):
p = Dir(str(d)).abspath
for ext in ('.a', '.so', '.dylib'):
f = File(os.path.join(p, f'lib{name}{ext}'))
if f.exists() or f.has_builder():
return name
if name in allowed_system_libs:
return name
raise SCons.Errors.UserError(f"Unexpected non-vendored library '{name}'")
def _libflags(target, source, env, for_signature):
libs = []
lp = env.subst('$LIBLITERALPREFIX')
for lib in env.Flatten(env.get('LIBS', [])):
if isinstance(lib, str):
if os.sep in lib or lib.startswith('#'):
libs.append(File(lib))
elif lib.startswith('-') or (lp and lib.startswith(lp)):
libs.append(lib)
else:
libs.append(_resolve_lib(env, lib))
else:
libs.append(lib)
return _stripixes(env['LIBLINKPREFIX'], libs, env['LIBLINKSUFFIX'],
env['LIBPREFIXES'], env['LIBSUFFIXES'], env, env['LIBLITERALPREFIX'])
env = Environment(
ENV={
"PATH": os.environ['PATH'],
"PYTHONPATH": Dir("#").abspath,
"ACADOS_SOURCE_DIR": acados.DIR,
"ACADOS_PYTHON_INTERFACE_PATH": acados.TEMPLATE_DIR,
"TERA_PATH": acados.TERA_PATH
},
CCFLAGS=[
"-g",
"-fPIC",
"-O2",
"-Wunused",
"-Werror",
"-Wshadow" if arch in ("Darwin", "larch64") else "-Wshadow=local",
"-Wno-unknown-warning-option",
"-Wno-inconsistent-missing-override",
"-Wno-c99-designator",
"-Wno-reorder-init-list",
"-Wno-vla-cxx-extension",
],
CFLAGS=["-std=gnu11"],
CXXFLAGS=["-std=c++1z"],
CPPPATH=[
"#",
"#msgq",
acados_include_dirs,
[x.INCLUDE_DIR for x in pkgs],
],
LIBPATH=[
"#common",
"#msgq_repo",
"#selfdrive/pandad",
"#rednose/helpers",
[x.LIB_DIR for x in pkgs],
],
RPATH=[],
CYTHONCFILESUFFIX=".cpp",
COMPILATIONDB_USE_ABSPATH=True,
REDNOSE_ROOT="#",
tools=["default", "cython", "compilation_db", "rednose_filter"],
toolpath=["#site_scons/site_tools", "#rednose_repo/site_scons/site_tools"],
)
if arch != "larch64":
env['_LIBFLAGS'] = _libflags
# Arch-specific flags and paths
if arch == "larch64":
env["CC"] = "clang"
env["CXX"] = "clang++"
env.Append(LIBPATH=[
"/usr/lib/aarch64-linux-gnu",
])
arch_flags = ["-D__TICI__", "-mcpu=cortex-a57", "-DQCOM2"]
env.Append(CCFLAGS=arch_flags)
env.Append(CXXFLAGS=arch_flags)
elif arch == "Darwin":
env.Append(LIBPATH=[
"/System/Library/Frameworks/OpenGL.framework/Libraries",
])
env.Append(CCFLAGS=["-DGL_SILENCE_DEPRECATION"])
env.Append(CXXFLAGS=["-DGL_SILENCE_DEPRECATION"])
_extra_cc = shlex.split(GetOption('ccflags') or '')
if _extra_cc:
env.Append(CCFLAGS=_extra_cc)
# no --as-needed on mac linker
if arch != "Darwin":
env.Append(LINKFLAGS=["-Wl,--as-needed", "-Wl,--no-undefined"])
# Shorter build output: show brief descriptions instead of full commands.
# Full command lines are still printed on failure by scons.
if not GetOption('verbose'):
for action, short in (
("CC", "CC"),
("CXX", "CXX"),
("LINK", "LINK"),
("SHCC", "CC"),
("SHCXX", "CXX"),
("SHLINK", "LINK"),
("AR", "AR"),
("RANLIB", "RANLIB"),
("AS", "AS"),
):
env[f"{action}COMSTR"] = f" [{short}] $TARGET"
# ********** Cython build environment **********
envCython = env.Clone()
envCython["CPPPATH"] += [sysconfig.get_paths()['include'], np.get_include()]
envCython["CCFLAGS"] += ["-Wno-#warnings", "-Wno-cpp", "-Wno-shadow", "-Wno-deprecated-declarations"]
envCython["CCFLAGS"].remove("-Werror")
envCython["LIBS"] = []
if arch == "Darwin":
envCython["LINKFLAGS"] = env["LINKFLAGS"] + ["-bundle", "-undefined", "dynamic_lookup"]
else:
envCython["LINKFLAGS"] = ["-pthread", "-shared"]
np_version = SCons.Script.Value(np.__version__)
Export('envCython', 'np_version')
Export('env', 'arch', 'acados', 'release')
# Setup cache dir
default_cache_dir = '/data/scons_cache' if arch == "larch64" else '/tmp/scons_cache'
cache_dir = ARGUMENTS.get('cache_dir', default_cache_dir)
cache_size_limit = 4e9 if "CI" in os.environ else 2e9
CacheDir(cache_dir)
Clean(["."], cache_dir)
def prune_cache_dir(target=None, source=None, env=None):
cache_files = sorted((os.path.join(root, f) for root, _, files in os.walk(cache_dir) for f in files), key=os.path.getmtime)
cache_size = sum(os.path.getsize(f) for f in cache_files)
for f in cache_files:
if cache_size < cache_size_limit:
break
cache_size -= os.path.getsize(f)
os.unlink(f)
# ********** start building stuff **********
# Build common module
SConscript(['common/SConscript'])
Import('_common')
common = [_common, 'json11', 'zmq']
Export('common')
# Build messaging (cereal + msgq + socketmaster + their dependencies)
# Enable swaglog include in submodules
env_swaglog = env.Clone()
env_swaglog['CXXFLAGS'].append('-DSWAGLOG="\\"common/swaglog.h\\""')
SConscript(['msgq_repo/SConscript'], exports={'env': env_swaglog})
SConscript(['cereal/SConscript'])
Import('socketmaster', 'msgq')
messaging = [socketmaster, msgq, 'capnp', 'kj',]
Export('messaging')
# Build other submodules
SConscript(['panda/SConscript'])
# Build rednose library
SConscript(['rednose/SConscript'])
# Build system services
SConscript([
'system/loggerd/SConscript',
])
if arch == "larch64":
SConscript(['system/camerad/SConscript'])
# Build selfdrive
SConscript([
'selfdrive/pandad/SConscript',
'selfdrive/controls/lib/lateral_mpc_lib/SConscript',
'selfdrive/controls/lib/longitudinal_mpc_lib/SConscript',
'selfdrive/locationd/SConscript',
'selfdrive/modeld/SConscript',
'selfdrive/ui/SConscript',
])
SConscript(['sunnypilot/SConscript'])
# Build desktop-only tools
if GetOption('extras') and arch != "larch64":
SConscript([
'tools/replay/SConscript',
'tools/cabana/SConscript',
'tools/jotpluggler/SConscript',
])
env.CompilationDatabase('compile_commands.json')
# progress output
def count_scons_nodes(nodes):
seen = set()
stack = list(nodes)
while stack:
node = stack.pop().disambiguate()
if node in seen:
continue
seen.add(node)
if hasattr(node, 'has_builder') and node.has_builder():
build_product_nodes.add(node)
executor = node.get_executor()
if executor is not None:
stack += executor.get_all_prerequisites() + executor.get_all_children()
return len(seen)
progress_interval = 5
progress_count = 0
build_product_nodes = set()
progress_total = max(1, count_scons_nodes(env.arg2nodes(BUILD_TARGETS or [Dir('.')], env.fs.Entry)))
def progress_function(node):
global progress_count
if progress_count >= progress_total:
return
progress_count = min(progress_count + progress_interval, progress_total)
progress = round(100. * progress_count / progress_total, 1)
sys.stderr.write("\rBuilding: %5.1f%%" % progress if sys.stderr.isatty() else "progress: %.1f\n" % progress)
if progress == 100. and sys.stderr.isatty():
sys.stderr.write("\n")
sys.stderr.flush()
Progress(progress_function, interval=progress_interval)
AddPostAction(BUILD_TARGETS or [Dir('.')], prune_cache_dir)
def check_build_product_size(target, source, env):
limit = 50 * 1024 * 1024 # GitHub max size
for t in target:
if hasattr(t, 'isfile') and t.isfile() and (size := os.path.getsize(t.abspath)) > limit:
raise SCons.Errors.UserError(f"{t} is {size / (1024 * 1024):.1f} MiB, exceeding the {limit / (1024 * 1024):.1f} MiB limit")
if not GetOption('extras'):
AddPostAction(list(build_product_nodes), Action(check_build_product_size, None))
-20
View File
@@ -1,20 +0,0 @@
Import('env', 'common', 'msgq')
cereal_dir = Dir('.')
gen_dir = Dir('gen')
# Build cereal
schema_files = ['log.capnp', 'car.capnp', 'deprecated.capnp', 'custom.capnp']
env.Command([f'gen/cpp/{s}.c++' for s in schema_files] + [f'gen/cpp/{s}.h' for s in schema_files],
schema_files,
f"capnpc --src-prefix={cereal_dir.path} $SOURCES -o c++:{gen_dir.path}/cpp/")
cereal = env.Library('cereal', [f'gen/cpp/{s}.c++' for s in schema_files])
# Build messaging
services_h = env.Command(['services.h'], ['services.py'], 'python3 ' + cereal_dir.path + '/services.py > $TARGET')
env.Program('messaging/bridge', ['messaging/bridge.cc', 'messaging/msgq_to_zmq.cc', 'messaging/bridge_zmq.cc'], LIBS=[msgq, common, 'pthread'])
socketmaster = env.Library('socketmaster', ['messaging/socketmaster.cc'])
Export('cereal', 'socketmaster')
+2 -175
View File
@@ -25,26 +25,6 @@ struct ModularAssistiveDrivingSystem {
}
}
struct IntelligentCruiseButtonManagement {
state @0 :IntelligentCruiseButtonManagementState;
sendButton @1 :SendButtonState;
vTarget @2 :Float32;
enum IntelligentCruiseButtonManagementState {
inactive @0; # No button press or default state
preActive @1; # Pre-active state before transitioning to increasing or decreasing
increasing @2; # Increasing speed
decreasing @3; # Decreasing speed
holding @4; # Holding steady speed
}
enum SendButtonState {
none @0;
increase @1;
decrease @2;
}
}
# Same struct as Log.RadarState.LeadData
struct LeadData {
dRel @0 :Float32;
@@ -68,49 +48,6 @@ struct LeadData {
struct SelfdriveStateSP @0x81c2f05a394cf4af {
mads @0 :ModularAssistiveDrivingSystem;
intelligentCruiseButtonManagement @1 :IntelligentCruiseButtonManagement;
enum AudibleAlert {
none @0;
engage @1;
disengage @2;
refuse @3;
warningSoft @4;
warningImmediate @5;
prompt @6;
promptRepeat @7;
promptDistracted @8;
# unused, these are reserved for upstream events so we don't collide
reserved9 @9;
reserved10 @10;
reserved11 @11;
reserved12 @12;
reserved13 @13;
reserved14 @14;
reserved15 @15;
reserved16 @16;
reserved17 @17;
reserved18 @18;
reserved19 @19;
reserved20 @20;
reserved21 @21;
reserved22 @22;
reserved23 @23;
reserved24 @24;
reserved25 @25;
reserved26 @26;
reserved27 @27;
reserved28 @28;
reserved29 @29;
reserved30 @30;
promptSingleLow @31;
promptSingleHigh @32;
}
}
struct ModelManagerSP @0xaedffd8f31e7b55d {
@@ -153,8 +90,6 @@ struct ModelManagerSP @0xaedffd8f31e7b55d {
navigation @1;
vision @2;
policy @3;
offPolicy @4;
onPolicy @5;
}
}
@@ -187,13 +122,6 @@ struct ModelManagerSP @0xaedffd8f31e7b55d {
struct LongitudinalPlanSP @0xf35cc4560bbf6ec2 {
dec @0 :DynamicExperimentalControl;
longitudinalPlanSource @1 :LongitudinalPlanSource;
smartCruiseControl @2 :SmartCruiseControl;
speedLimit @3 :SpeedLimit;
vTarget @4 :Float32;
aTarget @5 :Float32;
events @6 :List(OnroadEventSP.Event);
e2eAlerts @7 :E2eAlerts;
struct DynamicExperimentalControl {
state @0 :DynamicExperimentalControlState;
@@ -205,97 +133,6 @@ struct LongitudinalPlanSP @0xf35cc4560bbf6ec2 {
blended @1;
}
}
struct SmartCruiseControl {
vision @0 :Vision;
map @1 :Map;
struct Vision {
state @0 :VisionState;
vTarget @1 :Float32;
aTarget @2 :Float32;
currentLateralAccel @3 :Float32;
maxPredictedLateralAccel @4 :Float32;
enabled @5 :Bool;
active @6 :Bool;
}
struct Map {
state @0 :MapState;
vTarget @1 :Float32;
aTarget @2 :Float32;
enabled @3 :Bool;
active @4 :Bool;
}
enum VisionState {
disabled @0; # System disabled or inactive.
enabled @1; # No predicted substantial turn on vision range.
entering @2; # A substantial turn is predicted ahead, adapting speed to turn comfort levels.
turning @3; # Actively turning. Managing acceleration to provide a roll on turn feeling.
leaving @4; # Road ahead straightens. Start to allow positive acceleration.
overriding @5; # System overriding with manual control.
}
enum MapState {
disabled @0; # System disabled or inactive.
enabled @1; # No predicted substantial turn on map range.
turning @2; # Actively turning. Managing acceleration to provide a roll on turn feeling.
overriding @3; # System overriding with manual control.
}
}
struct SpeedLimit {
resolver @0 :Resolver;
assist @1 :Assist;
struct Resolver {
speedLimit @0 :Float32;
distToSpeedLimit @1 :Float32;
source @2 :Source;
speedLimitOffset @3 :Float32;
speedLimitLast @4 :Float32;
speedLimitFinal @5 :Float32;
speedLimitFinalLast @6 :Float32;
speedLimitValid @7 :Bool;
speedLimitLastValid @8 :Bool;
}
struct Assist {
state @0 :AssistState;
enabled @1 :Bool;
active @2 :Bool;
vTarget @3 :Float32;
aTarget @4 :Float32;
}
enum Source {
none @0;
car @1;
map @2;
}
enum AssistState {
disabled @0;
inactive @1; # No speed limit set or not enabled by parameter.
preActive @2;
pending @3; # Awaiting new speed limit.
adapting @4; # Reducing speed to match new speed limit.
active @5; # Cruising at speed limit.
}
}
enum LongitudinalPlanSource {
cruise @0;
sccVision @1;
sccMap @2;
speedLimitAssist @3;
}
struct E2eAlerts {
greenLightAlert @0 :Bool;
leadDepartAlert @1 :Bool;
}
}
struct OnroadEventSP @0xda96579883444c35 {
@@ -337,20 +174,12 @@ struct OnroadEventSP @0xda96579883444c35 {
pedalPressedAlertOnly @16;
laneTurnLeft @17;
laneTurnRight @18;
speedLimitPreActive @19;
speedLimitActive @20;
speedLimitChanged @21;
speedLimitPending @22;
e2eChime @23;
}
}
struct CarParamsSP @0x80ae746ee2596b11 {
flags @0 :UInt32; # flags for car specific quirks in sunnypilot
safetyParam @1 : Int16; # flags for sunnypilot's custom safety flags
pcmCruiseSpeed @3 :Bool;
intelligentCruiseButtonManagementAvailable @4 :Bool;
enableGasInterceptor @5 :Bool;
neuralNetworkLateralControl @2 :NeuralNetworkLateralControl;
@@ -370,7 +199,6 @@ struct CarControlSP @0xa5cd762cd951a455 {
params @1 :List(Param);
leadOne @2 :LeadData;
leadTwo @3 :LeadData;
intelligentCruiseButtonManagement @4 :IntelligentCruiseButtonManagement;
struct Param {
key @0 :Text;
@@ -379,7 +207,7 @@ struct CarControlSP @0xa5cd762cd951a455 {
valueDEPRECATED @1 :Text; # The data type change may cause issues with backwards compatibility.
}
enum ParamType {
string @0;
bool @1;
@@ -434,7 +262,6 @@ struct BackupManagerSP @0xf98d843bfd7004a3 {
}
struct CarStateSP @0xb86e6369214c01c8 {
speedLimit @0 :Float32;
}
struct LiveMapDataSP @0xf416ec09499d9d19 {
@@ -448,13 +275,13 @@ struct LiveMapDataSP @0xf416ec09499d9d19 {
struct ModelDataV2SP @0xa1680744031fdb2d {
laneTurnDirection @0 :TurnDirection;
}
enum TurnDirection {
none @0;
turnLeft @1;
turnRight @2;
}
}
struct CustomReserved10 @0xcb9fd56c7057593a {
}
-777
View File
@@ -1,777 +0,0 @@
using Cxx = import "./include/c++.capnp";
$Cxx.namespace("cereal");
@0x80ef1ec4889c2a63;
# deprecated.capnp: a home for deprecated structs
struct LogRotate @0x9811e1f38f62f2d1 {
segmentNum @0 :Int32;
path @1 :Text;
}
struct LiveUI @0xc08240f996aefced {
rearViewCam @0 :Bool;
alertText1 @1 :Text;
alertText2 @2 :Text;
awarenessStatus @3 :Float32;
}
struct UiLayoutState @0x88dcce08ad29dda0 {
activeApp @0 :App;
sidebarCollapsed @1 :Bool;
mapEnabled @2 :Bool;
mockEngaged @3 :Bool;
enum App @0x9917470acf94d285 {
home @0;
music @1;
nav @2;
settings @3;
none @4;
}
}
struct OrbslamCorrection @0x8afd33dc9b35e1aa {
correctionMonoTime @0 :UInt64;
prePositionECEF @1 :List(Float64);
postPositionECEF @2 :List(Float64);
prePoseQuatECEF @3 :List(Float32);
postPoseQuatECEF @4 :List(Float32);
numInliers @5 :UInt32;
}
struct EthernetPacket @0xa99a9d5b33cf5859 {
pkt @0 :Data;
ts @1 :Float32;
}
struct CellInfo @0xcff7566681c277ce {
timestamp @0 :UInt64;
repr @1 :Text; # android toString() for now
}
struct WifiScan @0xd4df5a192382ba0b {
bssid @0 :Text;
ssid @1 :Text;
capabilities @2 :Text;
frequency @3 :Int32;
level @4 :Int32;
timestamp @5 :Int64;
centerFreq0 @6 :Int32;
centerFreq1 @7 :Int32;
channelWidth @8 :ChannelWidth;
operatorFriendlyName @9 :Text;
venueName @10 :Text;
is80211mcResponder @11 :Bool;
passpoint @12 :Bool;
distanceCm @13 :Int32;
distanceSdCm @14 :Int32;
enum ChannelWidth @0xcb6a279f015f6b51 {
w20Mhz @0;
w40Mhz @1;
w80Mhz @2;
w160Mhz @3;
w80Plus80Mhz @4;
}
}
struct LiveEventData @0x94b7baa90c5c321e {
name @0 :Text;
value @1 :Int32;
}
struct ModelData @0xb8aad62cffef28a9 {
frameId @0 :UInt32;
frameAge @12 :UInt32;
frameDropPerc @13 :Float32;
timestampEof @9 :UInt64;
modelExecutionTime @14 :Float32;
gpuExecutionTime @16 :Float32;
rawPred @15 :Data;
path @1 :PathData;
leftLane @2 :PathData;
rightLane @3 :PathData;
lead @4 :LeadData;
freePath @6 :List(Float32);
settings @5 :ModelSettings;
leadFuture @7 :LeadData;
speed @8 :List(Float32);
meta @10 :MetaData;
longitudinal @11 :LongitudinalData;
struct PathData @0x8817eeea389e9f08 {
points @0 :List(Float32);
prob @1 :Float32;
std @2 :Float32;
stds @3 :List(Float32);
poly @4 :List(Float32);
validLen @5 :Float32;
}
struct LeadData @0xd1c9bef96d26fa91 {
dist @0 :Float32;
prob @1 :Float32;
std @2 :Float32;
relVel @3 :Float32;
relVelStd @4 :Float32;
relY @5 :Float32;
relYStd @6 :Float32;
relA @7 :Float32;
relAStd @8 :Float32;
}
struct ModelSettings @0xa26e3710efd3e914 {
bigBoxX @0 :UInt16;
bigBoxY @1 :UInt16;
bigBoxWidth @2 :UInt16;
bigBoxHeight @3 :UInt16;
boxProjection @4 :List(Float32);
yuvCorrection @5 :List(Float32);
inputTransform @6 :List(Float32);
}
struct MetaData @0x9744f25fb60f2bf8 {
engagedProb @0 :Float32;
desirePrediction @1 :List(Float32);
brakeDisengageProb @2 :Float32;
gasDisengageProb @3 :Float32;
steerOverrideProb @4 :Float32;
desireState @5 :List(Float32);
}
struct LongitudinalData @0xf98f999c6a071122 {
distances @2 :List(Float32);
speeds @0 :List(Float32);
accelerations @1 :List(Float32);
}
}
struct ECEFPoint @0xc25bbbd524983447 {
x @0 :Float64;
y @1 :Float64;
z @2 :Float64;
}
struct ECEFPointDEPRECATED @0xe10e21168db0c7f7 {
x @0 :Float32;
y @1 :Float32;
z @2 :Float32;
}
struct GPSPlannerPoints @0xab54c59699f8f9f3 {
curPosDEPRECATED @0 :ECEFPointDEPRECATED;
pointsDEPRECATED @1 :List(ECEFPointDEPRECATED);
curPos @6 :ECEFPoint;
points @7 :List(ECEFPoint);
valid @2 :Bool;
trackName @3 :Text;
speedLimit @4 :Float32;
accelTarget @5 :Float32;
}
struct GPSPlannerPlan @0xf5ad1d90cdc1dd6b {
valid @0 :Bool;
poly @1 :List(Float32);
trackName @2 :Text;
speed @3 :Float32;
acceleration @4 :Float32;
pointsDEPRECATED @5 :List(ECEFPointDEPRECATED);
points @6 :List(ECEFPoint);
xLookahead @7 :Float32;
}
struct UiNavigationEvent @0x90c8426c3eaddd3b {
type @0: Type;
status @1: Status;
distanceTo @2: Float32;
endRoadPointDEPRECATED @3: ECEFPointDEPRECATED;
endRoadPoint @4: ECEFPoint;
enum Type @0xe8db07dcf8fcea05 {
none @0;
laneChangeLeft @1;
laneChangeRight @2;
mergeLeft @3;
mergeRight @4;
turnLeft @5;
turnRight @6;
}
enum Status @0xb9aa88c75ef99a1f {
none @0;
passive @1;
approaching @2;
active @3;
}
}
struct LiveLocationData @0xb99b2bc7a57e8128 {
status @0 :UInt8;
# 3D fix
lat @1 :Float64;
lon @2 :Float64;
alt @3 :Float32; # m
# speed
speed @4 :Float32; # m/s
# NED velocity components
vNED @5 :List(Float32);
# roll, pitch, heading (x,y,z)
roll @6 :Float32; # WRT to center of earth?
pitch @7 :Float32; # WRT to center of earth?
heading @8 :Float32; # WRT to north?
# what are these?
wanderAngle @9 :Float32;
trackAngle @10 :Float32;
# car frame -- https://upload.wikimedia.org/wikipedia/commons/f/f5/RPY_angles_of_cars.png
# gyro, in car frame, deg/s
gyro @11 :List(Float32);
# accel, in car frame, m/s^2
accel @12 :List(Float32);
accuracy @13 :Accuracy;
source @14 :SensorSource;
# if we are fixing a location in the past
fixMonoTime @15 :UInt64;
gpsWeek @16 :Int32;
timeOfWeek @17 :Float64;
positionECEF @18 :List(Float64);
poseQuatECEF @19 :List(Float32);
pitchCalibration @20 :Float32;
yawCalibration @21 :Float32;
imuFrame @22 :List(Float32);
struct Accuracy @0x943dc4625473b03f {
pNEDError @0 :List(Float32);
vNEDError @1 :List(Float32);
rollError @2 :Float32;
pitchError @3 :Float32;
headingError @4 :Float32;
ellipsoidSemiMajorError @5 :Float32;
ellipsoidSemiMinorError @6 :Float32;
ellipsoidOrientationError @7 :Float32;
}
enum SensorSource @0xc871d3cc252af657 {
applanix @0;
kalman @1;
orbslam @2;
timing @3;
dummy @4;
}
}
struct OrbOdometry @0xd7700859ed1f5b76 {
# timing first
startMonoTime @0 :UInt64;
endMonoTime @1 :UInt64;
# fundamental matrix and error
f @2: List(Float64);
err @3: Float64;
# number of inlier points
inliers @4: Int32;
# for debug only
# indexed by endMonoTime features
# value is startMonoTime feature match
# -1 if no match
matches @5: List(Int16);
}
struct OrbFeatures @0xcd60164a8a0159ef {
timestampEof @0 :UInt64;
# transposed arrays of normalized image coordinates
# len(xs) == len(ys) == len(descriptors) * 32
xs @1 :List(Float32);
ys @2 :List(Float32);
descriptors @3 :Data;
octaves @4 :List(Int8);
# match index to last OrbFeatures
# -1 if no match
timestampLastEof @5 :UInt64;
matches @6: List(Int16);
}
struct OrbFeaturesSummary @0xd500d30c5803fa4f {
timestampEof @0 :UInt64;
timestampLastEof @1 :UInt64;
featureCount @2 :UInt16;
matchCount @3 :UInt16;
computeNs @4 :UInt64;
}
struct OrbKeyFrame @0xc8233c0345e27e24 {
# this is a globally unique id for the KeyFrame
id @0: UInt64;
# this is the location of the KeyFrame
pos @1: ECEFPoint;
# these are the features in the world
# len(dpos) == len(descriptors) * 32
dpos @2 :List(ECEFPoint);
descriptors @3 :Data;
}
struct KalmanOdometry @0x92e21bb7ea38793a {
trans @0 :List(Float32); # m/s in device frame
rot @1 :List(Float32); # rad/s in device frame
transStd @2 :List(Float32); # std m/s in device frame
rotStd @3 :List(Float32); # std rad/s in device frame
}
struct OrbObservation @0x9b326d4e436afec7 {
observationMonoTime @0 :UInt64;
normalizedCoordinates @1 :List(Float32);
locationECEF @2 :List(Float64);
matchDistance @3: UInt32;
}
struct CalibrationFeatures @0x8fdfadb254ea867a {
frameId @0 :UInt32;
p0 @1 :List(Float32);
p1 @2 :List(Float32);
status @3 :List(Int8);
}
struct NavStatus @0xbd8822120928120c {
isNavigating @0 :Bool;
currentAddress @1 :Address;
struct Address @0xce7cd672cacc7814 {
title @0 :Text;
lat @1 :Float64;
lng @2 :Float64;
house @3 :Text;
address @4 :Text;
street @5 :Text;
city @6 :Text;
state @7 :Text;
country @8 :Text;
}
}
struct NavUpdate @0xdb98be6565516acb {
isNavigating @0 :Bool;
curSegment @1 :Int32;
segments @2 :List(Segment);
struct LatLng @0x9eaef9187cadbb9b {
lat @0 :Float64;
lng @1 :Float64;
}
struct Segment @0xa5b39b4fc4d7da3f {
from @0 :LatLng;
to @1 :LatLng;
updateTime @2 :Int32;
distance @3 :Int32;
crossTime @4 :Int32;
exitNo @5 :Int32;
instruction @6 :Instruction;
parts @7 :List(LatLng);
enum Instruction @0xc5417a637451246f {
turnLeft @0;
turnRight @1;
keepLeft @2;
keepRight @3;
straight @4;
roundaboutExitNumber @5;
roundaboutExit @6;
roundaboutTurnLeft @7;
unkn8 @8;
roundaboutStraight @9;
unkn10 @10;
roundaboutTurnRight @11;
unkn12 @12;
roundaboutUturn @13;
unkn14 @14;
arrive @15;
exitLeft @16;
exitRight @17;
unkn18 @18;
uturn @19;
# ...
}
}
}
struct TrafficEvent @0xacfa74a094e62626 {
type @0 :Type;
distance @1 :Float32;
action @2 :Action;
resuming @3 :Bool;
enum Type @0xd85d75253435bf4b {
stopSign @0;
lightRed @1;
lightYellow @2;
lightGreen @3;
stopLight @4;
}
enum Action @0xa6f6ce72165ccb49 {
none @0;
yield @1;
stop @2;
resumeReady @3;
}
}
struct AndroidGnss @0xdfdf30d03fc485bd {
union {
measurements @0 :Measurements;
navigationMessage @1 :NavigationMessage;
}
struct Measurements @0xa20710d4f428d6cd {
clock @0 :Clock;
measurements @1 :List(Measurement);
struct Clock @0xa0e27b453a38f450 {
timeNanos @0 :Int64;
hardwareClockDiscontinuityCount @1 :Int32;
hasTimeUncertaintyNanos @2 :Bool;
timeUncertaintyNanos @3 :Float64;
hasLeapSecond @4 :Bool;
leapSecond @5 :Int32;
hasFullBiasNanos @6 :Bool;
fullBiasNanos @7 :Int64;
hasBiasNanos @8 :Bool;
biasNanos @9 :Float64;
hasBiasUncertaintyNanos @10 :Bool;
biasUncertaintyNanos @11 :Float64;
hasDriftNanosPerSecond @12 :Bool;
driftNanosPerSecond @13 :Float64;
hasDriftUncertaintyNanosPerSecond @14 :Bool;
driftUncertaintyNanosPerSecond @15 :Float64;
}
struct Measurement @0xd949bf717d77614d {
svId @0 :Int32;
constellation @1 :Constellation;
timeOffsetNanos @2 :Float64;
state @3 :Int32;
receivedSvTimeNanos @4 :Int64;
receivedSvTimeUncertaintyNanos @5 :Int64;
cn0DbHz @6 :Float64;
pseudorangeRateMetersPerSecond @7 :Float64;
pseudorangeRateUncertaintyMetersPerSecond @8 :Float64;
accumulatedDeltaRangeState @9 :Int32;
accumulatedDeltaRangeMeters @10 :Float64;
accumulatedDeltaRangeUncertaintyMeters @11 :Float64;
hasCarrierFrequencyHz @12 :Bool;
carrierFrequencyHz @13 :Float32;
hasCarrierCycles @14 :Bool;
carrierCycles @15 :Int64;
hasCarrierPhase @16 :Bool;
carrierPhase @17 :Float64;
hasCarrierPhaseUncertainty @18 :Bool;
carrierPhaseUncertainty @19 :Float64;
hasSnrInDb @20 :Bool;
snrInDb @21 :Float64;
multipathIndicator @22 :MultipathIndicator;
enum Constellation @0x9ef1f3ff0deb5ffb {
unknown @0;
gps @1;
sbas @2;
glonass @3;
qzss @4;
beidou @5;
galileo @6;
}
enum State @0xcbb9490adce12d72 {
unknown @0;
codeLock @1;
bitSync @2;
subframeSync @3;
towDecoded @4;
msecAmbiguous @5;
symbolSync @6;
gloStringSync @7;
gloTodDecoded @8;
bdsD2BitSync @9;
bdsD2SubframeSync @10;
galE1bcCodeLock @11;
galE1c2ndCodeLock @12;
galE1bPageSync @13;
sbasSync @14;
}
enum MultipathIndicator @0xc04e7b6231d4caa8 {
unknown @0;
detected @1;
notDetected @2;
}
}
}
struct NavigationMessage @0xe2517b083095fd4e {
type @0 :Int32;
svId @1 :Int32;
messageId @2 :Int32;
submessageId @3 :Int32;
data @4 :Data;
status @5 :Status;
enum Status @0xec1ff7996b35366f {
unknown @0;
parityPassed @1;
parityRebuilt @2;
}
}
}
struct LidarPts @0xe3d6685d4e9d8f7a {
r @0 :List(UInt16); # uint16 m*500.0
theta @1 :List(UInt16); # uint16 deg*100.0
reflect @2 :List(UInt8); # uint8 0-255
# For storing out of file.
idx @3 :UInt64;
# For storing in file
pkt @4 :Data;
}
struct LiveTracksDEPRECATED @0xb16f60103159415a {
trackId @0 :Int32;
dRel @1 :Float32;
yRel @2 :Float32;
vRel @3 :Float32;
aRel @4 :Float32;
timeStamp @5 :Float32;
status @6 :Float32;
currentTime @7 :Float32;
stationary @8 :Bool;
oncoming @9 :Bool;
}
struct LiveMpcData @0x92a5e332a85f32a0 {
x @0 :List(Float32);
y @1 :List(Float32);
psi @2 :List(Float32);
curvature @3 :List(Float32);
qpIterations @4 :UInt32;
calculationTime @5 :UInt64;
cost @6 :Float64;
}
struct LiveLongitudinalMpcData @0xe7e17c434f865ae2 {
xEgo @0 :List(Float32);
vEgo @1 :List(Float32);
aEgo @2 :List(Float32);
xLead @3 :List(Float32);
vLead @4 :List(Float32);
aLead @5 :List(Float32);
aLeadTau @6 :Float32; # lead accel time constant
qpIterations @7 :UInt32;
mpcId @8 :UInt32;
calculationTime @9 :UInt64;
cost @10 :Float64;
}
struct DriverStateDEPRECATED @0xb83c6cc593ed0a00 {
frameId @0 :UInt32;
modelExecutionTime @14 :Float32;
dspExecutionTime @16 :Float32;
rawPredictions @15 :Data;
faceOrientation @3 :List(Float32);
facePosition @4 :List(Float32);
faceProb @5 :Float32;
leftEyeProb @6 :Float32;
rightEyeProb @7 :Float32;
leftBlinkProb @8 :Float32;
rightBlinkProb @9 :Float32;
faceOrientationStd @11 :List(Float32);
facePositionStd @12 :List(Float32);
sunglassesProb @13 :Float32;
poorVision @17 :Float32;
partialFace @18 :Float32;
distractedPose @19 :Float32;
distractedEyes @20 :Float32;
eyesOnRoad @21 :Float32;
phoneUse @22 :Float32;
occludedProb @23 :Float32;
readyProb @24 :List(Float32);
notReadyProb @25 :List(Float32);
irPwrDEPRECATED @10 :Float32;
descriptorDEPRECATED @1 :List(Float32);
stdDEPRECATED @2 :Float32;
}
struct NavModelData @0xac3de5c437be057a {
frameId @0 :UInt32;
locationMonoTime @6 :UInt64;
modelExecutionTime @1 :Float32;
dspExecutionTime @2 :Float32;
features @3 :List(Float32);
# predicted future position
position @4 :XYData;
desirePrediction @5 :List(Float32);
# All SI units and in device frame
struct XYData @0xbe09e615b2507e26 {
x @0 :List(Float32);
y @1 :List(Float32);
xStd @2 :List(Float32);
yStd @3 :List(Float32);
}
}
struct AndroidBuildInfo @0xfe2919d5c21f426c {
board @0 :Text;
bootloader @1 :Text;
brand @2 :Text;
device @3 :Text;
display @4 :Text;
fingerprint @5 :Text;
hardware @6 :Text;
host @7 :Text;
id @8 :Text;
manufacturer @9 :Text;
model @10 :Text;
product @11 :Text;
radioVersion @12 :Text;
serial @13 :Text;
supportedAbis @14 :List(Text);
tags @15 :Text;
time @16 :Int64;
type @17 :Text;
user @18 :Text;
versionCodename @19 :Text;
versionRelease @20 :Text;
versionSdk @21 :Int32;
versionSecurityPatch @22 :Text;
}
struct AndroidSensor @0x9b513b93a887dbcd {
id @0 :Int32;
name @1 :Text;
vendor @2 :Text;
version @3 :Int32;
handle @4 :Int32;
type @5 :Int32;
maxRange @6 :Float32;
resolution @7 :Float32;
power @8 :Float32;
minDelay @9 :Int32;
fifoReservedEventCount @10 :UInt32;
fifoMaxEventCount @11 :UInt32;
stringType @12 :Text;
maxDelay @13 :Int32;
}
struct IosBuildInfo @0xd97e3b28239f5580 {
appVersion @0 :Text;
appBuild @1 :UInt32;
osVersion @2 :Text;
deviceModel @3 :Text;
}
enum FrameTypeDEPRECATED @0xa37f0d8558e193fd {
unknown @0;
neo @1;
chffrAndroid @2;
front @3;
}
struct AndroidCaptureResult @0xbcc3efbac41d2048 {
sensitivity @0 :Int32;
frameDuration @1 :Int64;
exposureTime @2 :Int64;
rollingShutterSkew @3 :UInt64;
colorCorrectionTransform @4 :List(Int32);
colorCorrectionGains @5 :List(Float32);
displayRotation @6 :Int8;
}
enum UsbPowerModeDEPRECATED @0xa8883583b32c9877 {
none @0;
client @1;
cdp @2;
dcp @3;
}
struct LateralINDIState @0x939463348632375e {
active @0 :Bool;
steeringAngleDeg @1 :Float32;
steeringRateDeg @2 :Float32;
steeringAccelDeg @3 :Float32;
rateSetPoint @4 :Float32;
accelSetPoint @5 :Float32;
accelError @6 :Float32;
delayedOutput @7 :Float32;
delta @8 :Float32;
output @9 :Float32;
saturated @10 :Bool;
steeringAngleDesiredDeg @11 :Float32;
steeringRateDesiredDeg @12 :Float32;
}
struct LateralLQRState @0x9024e2d790c82ade {
active @0 :Bool;
steeringAngleDeg @1 :Float32;
i @2 :Float32;
output @3 :Float32;
lqrOutput @4 :Float32;
saturated @5 :Bool;
steeringAngleDesiredDeg @6 :Float32;
}
struct LateralPlannerSolution @0x84caeca5a6b4acfe {
x @0 :List(Float32);
y @1 :List(Float32);
yaw @2 :List(Float32);
yawRate @3 :List(Float32);
xStd @4 :List(Float32);
yStd @5 :List(Float32);
yawStd @6 :List(Float32);
yawRateStd @7 :List(Float32);
}
struct GpsTrajectory @0x8cfeb072f5301000 {
x @0 :List(Float32);
y @1 :List(Float32);
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+574
View File
@@ -0,0 +1,574 @@
using Cxx = import "./include/c++.capnp";
$Cxx.namespace("cereal");
@0x80ef1ec4889c2a63;
# legacy.capnp: a home for deprecated structs
struct LogRotate @0x9811e1f38f62f2d1 {
segmentNum @0 :Int32;
path @1 :Text;
}
struct LiveUI @0xc08240f996aefced {
rearViewCam @0 :Bool;
alertText1 @1 :Text;
alertText2 @2 :Text;
awarenessStatus @3 :Float32;
}
struct UiLayoutState @0x88dcce08ad29dda0 {
activeApp @0 :App;
sidebarCollapsed @1 :Bool;
mapEnabled @2 :Bool;
mockEngaged @3 :Bool;
enum App @0x9917470acf94d285 {
home @0;
music @1;
nav @2;
settings @3;
none @4;
}
}
struct OrbslamCorrection @0x8afd33dc9b35e1aa {
correctionMonoTime @0 :UInt64;
prePositionECEF @1 :List(Float64);
postPositionECEF @2 :List(Float64);
prePoseQuatECEF @3 :List(Float32);
postPoseQuatECEF @4 :List(Float32);
numInliers @5 :UInt32;
}
struct EthernetPacket @0xa99a9d5b33cf5859 {
pkt @0 :Data;
ts @1 :Float32;
}
struct CellInfo @0xcff7566681c277ce {
timestamp @0 :UInt64;
repr @1 :Text; # android toString() for now
}
struct WifiScan @0xd4df5a192382ba0b {
bssid @0 :Text;
ssid @1 :Text;
capabilities @2 :Text;
frequency @3 :Int32;
level @4 :Int32;
timestamp @5 :Int64;
centerFreq0 @6 :Int32;
centerFreq1 @7 :Int32;
channelWidth @8 :ChannelWidth;
operatorFriendlyName @9 :Text;
venueName @10 :Text;
is80211mcResponder @11 :Bool;
passpoint @12 :Bool;
distanceCm @13 :Int32;
distanceSdCm @14 :Int32;
enum ChannelWidth @0xcb6a279f015f6b51 {
w20Mhz @0;
w40Mhz @1;
w80Mhz @2;
w160Mhz @3;
w80Plus80Mhz @4;
}
}
struct LiveEventData @0x94b7baa90c5c321e {
name @0 :Text;
value @1 :Int32;
}
struct ModelData @0xb8aad62cffef28a9 {
frameId @0 :UInt32;
frameAge @12 :UInt32;
frameDropPerc @13 :Float32;
timestampEof @9 :UInt64;
modelExecutionTime @14 :Float32;
gpuExecutionTime @16 :Float32;
rawPred @15 :Data;
path @1 :PathData;
leftLane @2 :PathData;
rightLane @3 :PathData;
lead @4 :LeadData;
freePath @6 :List(Float32);
settings @5 :ModelSettings;
leadFuture @7 :LeadData;
speed @8 :List(Float32);
meta @10 :MetaData;
longitudinal @11 :LongitudinalData;
struct PathData @0x8817eeea389e9f08 {
points @0 :List(Float32);
prob @1 :Float32;
std @2 :Float32;
stds @3 :List(Float32);
poly @4 :List(Float32);
validLen @5 :Float32;
}
struct LeadData @0xd1c9bef96d26fa91 {
dist @0 :Float32;
prob @1 :Float32;
std @2 :Float32;
relVel @3 :Float32;
relVelStd @4 :Float32;
relY @5 :Float32;
relYStd @6 :Float32;
relA @7 :Float32;
relAStd @8 :Float32;
}
struct ModelSettings @0xa26e3710efd3e914 {
bigBoxX @0 :UInt16;
bigBoxY @1 :UInt16;
bigBoxWidth @2 :UInt16;
bigBoxHeight @3 :UInt16;
boxProjection @4 :List(Float32);
yuvCorrection @5 :List(Float32);
inputTransform @6 :List(Float32);
}
struct MetaData @0x9744f25fb60f2bf8 {
engagedProb @0 :Float32;
desirePrediction @1 :List(Float32);
brakeDisengageProb @2 :Float32;
gasDisengageProb @3 :Float32;
steerOverrideProb @4 :Float32;
desireState @5 :List(Float32);
}
struct LongitudinalData @0xf98f999c6a071122 {
distances @2 :List(Float32);
speeds @0 :List(Float32);
accelerations @1 :List(Float32);
}
}
struct ECEFPoint @0xc25bbbd524983447 {
x @0 :Float64;
y @1 :Float64;
z @2 :Float64;
}
struct ECEFPointDEPRECATED @0xe10e21168db0c7f7 {
x @0 :Float32;
y @1 :Float32;
z @2 :Float32;
}
struct GPSPlannerPoints @0xab54c59699f8f9f3 {
curPosDEPRECATED @0 :ECEFPointDEPRECATED;
pointsDEPRECATED @1 :List(ECEFPointDEPRECATED);
curPos @6 :ECEFPoint;
points @7 :List(ECEFPoint);
valid @2 :Bool;
trackName @3 :Text;
speedLimit @4 :Float32;
accelTarget @5 :Float32;
}
struct GPSPlannerPlan @0xf5ad1d90cdc1dd6b {
valid @0 :Bool;
poly @1 :List(Float32);
trackName @2 :Text;
speed @3 :Float32;
acceleration @4 :Float32;
pointsDEPRECATED @5 :List(ECEFPointDEPRECATED);
points @6 :List(ECEFPoint);
xLookahead @7 :Float32;
}
struct UiNavigationEvent @0x90c8426c3eaddd3b {
type @0: Type;
status @1: Status;
distanceTo @2: Float32;
endRoadPointDEPRECATED @3: ECEFPointDEPRECATED;
endRoadPoint @4: ECEFPoint;
enum Type @0xe8db07dcf8fcea05 {
none @0;
laneChangeLeft @1;
laneChangeRight @2;
mergeLeft @3;
mergeRight @4;
turnLeft @5;
turnRight @6;
}
enum Status @0xb9aa88c75ef99a1f {
none @0;
passive @1;
approaching @2;
active @3;
}
}
struct LiveLocationData @0xb99b2bc7a57e8128 {
status @0 :UInt8;
# 3D fix
lat @1 :Float64;
lon @2 :Float64;
alt @3 :Float32; # m
# speed
speed @4 :Float32; # m/s
# NED velocity components
vNED @5 :List(Float32);
# roll, pitch, heading (x,y,z)
roll @6 :Float32; # WRT to center of earth?
pitch @7 :Float32; # WRT to center of earth?
heading @8 :Float32; # WRT to north?
# what are these?
wanderAngle @9 :Float32;
trackAngle @10 :Float32;
# car frame -- https://upload.wikimedia.org/wikipedia/commons/f/f5/RPY_angles_of_cars.png
# gyro, in car frame, deg/s
gyro @11 :List(Float32);
# accel, in car frame, m/s^2
accel @12 :List(Float32);
accuracy @13 :Accuracy;
source @14 :SensorSource;
# if we are fixing a location in the past
fixMonoTime @15 :UInt64;
gpsWeek @16 :Int32;
timeOfWeek @17 :Float64;
positionECEF @18 :List(Float64);
poseQuatECEF @19 :List(Float32);
pitchCalibration @20 :Float32;
yawCalibration @21 :Float32;
imuFrame @22 :List(Float32);
struct Accuracy @0x943dc4625473b03f {
pNEDError @0 :List(Float32);
vNEDError @1 :List(Float32);
rollError @2 :Float32;
pitchError @3 :Float32;
headingError @4 :Float32;
ellipsoidSemiMajorError @5 :Float32;
ellipsoidSemiMinorError @6 :Float32;
ellipsoidOrientationError @7 :Float32;
}
enum SensorSource @0xc871d3cc252af657 {
applanix @0;
kalman @1;
orbslam @2;
timing @3;
dummy @4;
}
}
struct OrbOdometry @0xd7700859ed1f5b76 {
# timing first
startMonoTime @0 :UInt64;
endMonoTime @1 :UInt64;
# fundamental matrix and error
f @2: List(Float64);
err @3: Float64;
# number of inlier points
inliers @4: Int32;
# for debug only
# indexed by endMonoTime features
# value is startMonoTime feature match
# -1 if no match
matches @5: List(Int16);
}
struct OrbFeatures @0xcd60164a8a0159ef {
timestampEof @0 :UInt64;
# transposed arrays of normalized image coordinates
# len(xs) == len(ys) == len(descriptors) * 32
xs @1 :List(Float32);
ys @2 :List(Float32);
descriptors @3 :Data;
octaves @4 :List(Int8);
# match index to last OrbFeatures
# -1 if no match
timestampLastEof @5 :UInt64;
matches @6: List(Int16);
}
struct OrbFeaturesSummary @0xd500d30c5803fa4f {
timestampEof @0 :UInt64;
timestampLastEof @1 :UInt64;
featureCount @2 :UInt16;
matchCount @3 :UInt16;
computeNs @4 :UInt64;
}
struct OrbKeyFrame @0xc8233c0345e27e24 {
# this is a globally unique id for the KeyFrame
id @0: UInt64;
# this is the location of the KeyFrame
pos @1: ECEFPoint;
# these are the features in the world
# len(dpos) == len(descriptors) * 32
dpos @2 :List(ECEFPoint);
descriptors @3 :Data;
}
struct KalmanOdometry @0x92e21bb7ea38793a {
trans @0 :List(Float32); # m/s in device frame
rot @1 :List(Float32); # rad/s in device frame
transStd @2 :List(Float32); # std m/s in device frame
rotStd @3 :List(Float32); # std rad/s in device frame
}
struct OrbObservation @0x9b326d4e436afec7 {
observationMonoTime @0 :UInt64;
normalizedCoordinates @1 :List(Float32);
locationECEF @2 :List(Float64);
matchDistance @3: UInt32;
}
struct CalibrationFeatures @0x8fdfadb254ea867a {
frameId @0 :UInt32;
p0 @1 :List(Float32);
p1 @2 :List(Float32);
status @3 :List(Int8);
}
struct NavStatus @0xbd8822120928120c {
isNavigating @0 :Bool;
currentAddress @1 :Address;
struct Address @0xce7cd672cacc7814 {
title @0 :Text;
lat @1 :Float64;
lng @2 :Float64;
house @3 :Text;
address @4 :Text;
street @5 :Text;
city @6 :Text;
state @7 :Text;
country @8 :Text;
}
}
struct NavUpdate @0xdb98be6565516acb {
isNavigating @0 :Bool;
curSegment @1 :Int32;
segments @2 :List(Segment);
struct LatLng @0x9eaef9187cadbb9b {
lat @0 :Float64;
lng @1 :Float64;
}
struct Segment @0xa5b39b4fc4d7da3f {
from @0 :LatLng;
to @1 :LatLng;
updateTime @2 :Int32;
distance @3 :Int32;
crossTime @4 :Int32;
exitNo @5 :Int32;
instruction @6 :Instruction;
parts @7 :List(LatLng);
enum Instruction @0xc5417a637451246f {
turnLeft @0;
turnRight @1;
keepLeft @2;
keepRight @3;
straight @4;
roundaboutExitNumber @5;
roundaboutExit @6;
roundaboutTurnLeft @7;
unkn8 @8;
roundaboutStraight @9;
unkn10 @10;
roundaboutTurnRight @11;
unkn12 @12;
roundaboutUturn @13;
unkn14 @14;
arrive @15;
exitLeft @16;
exitRight @17;
unkn18 @18;
uturn @19;
# ...
}
}
}
struct TrafficEvent @0xacfa74a094e62626 {
type @0 :Type;
distance @1 :Float32;
action @2 :Action;
resuming @3 :Bool;
enum Type @0xd85d75253435bf4b {
stopSign @0;
lightRed @1;
lightYellow @2;
lightGreen @3;
stopLight @4;
}
enum Action @0xa6f6ce72165ccb49 {
none @0;
yield @1;
stop @2;
resumeReady @3;
}
}
struct AndroidGnss @0xdfdf30d03fc485bd {
union {
measurements @0 :Measurements;
navigationMessage @1 :NavigationMessage;
}
struct Measurements @0xa20710d4f428d6cd {
clock @0 :Clock;
measurements @1 :List(Measurement);
struct Clock @0xa0e27b453a38f450 {
timeNanos @0 :Int64;
hardwareClockDiscontinuityCount @1 :Int32;
hasTimeUncertaintyNanos @2 :Bool;
timeUncertaintyNanos @3 :Float64;
hasLeapSecond @4 :Bool;
leapSecond @5 :Int32;
hasFullBiasNanos @6 :Bool;
fullBiasNanos @7 :Int64;
hasBiasNanos @8 :Bool;
biasNanos @9 :Float64;
hasBiasUncertaintyNanos @10 :Bool;
biasUncertaintyNanos @11 :Float64;
hasDriftNanosPerSecond @12 :Bool;
driftNanosPerSecond @13 :Float64;
hasDriftUncertaintyNanosPerSecond @14 :Bool;
driftUncertaintyNanosPerSecond @15 :Float64;
}
struct Measurement @0xd949bf717d77614d {
svId @0 :Int32;
constellation @1 :Constellation;
timeOffsetNanos @2 :Float64;
state @3 :Int32;
receivedSvTimeNanos @4 :Int64;
receivedSvTimeUncertaintyNanos @5 :Int64;
cn0DbHz @6 :Float64;
pseudorangeRateMetersPerSecond @7 :Float64;
pseudorangeRateUncertaintyMetersPerSecond @8 :Float64;
accumulatedDeltaRangeState @9 :Int32;
accumulatedDeltaRangeMeters @10 :Float64;
accumulatedDeltaRangeUncertaintyMeters @11 :Float64;
hasCarrierFrequencyHz @12 :Bool;
carrierFrequencyHz @13 :Float32;
hasCarrierCycles @14 :Bool;
carrierCycles @15 :Int64;
hasCarrierPhase @16 :Bool;
carrierPhase @17 :Float64;
hasCarrierPhaseUncertainty @18 :Bool;
carrierPhaseUncertainty @19 :Float64;
hasSnrInDb @20 :Bool;
snrInDb @21 :Float64;
multipathIndicator @22 :MultipathIndicator;
enum Constellation @0x9ef1f3ff0deb5ffb {
unknown @0;
gps @1;
sbas @2;
glonass @3;
qzss @4;
beidou @5;
galileo @6;
}
enum State @0xcbb9490adce12d72 {
unknown @0;
codeLock @1;
bitSync @2;
subframeSync @3;
towDecoded @4;
msecAmbiguous @5;
symbolSync @6;
gloStringSync @7;
gloTodDecoded @8;
bdsD2BitSync @9;
bdsD2SubframeSync @10;
galE1bcCodeLock @11;
galE1c2ndCodeLock @12;
galE1bPageSync @13;
sbasSync @14;
}
enum MultipathIndicator @0xc04e7b6231d4caa8 {
unknown @0;
detected @1;
notDetected @2;
}
}
}
struct NavigationMessage @0xe2517b083095fd4e {
type @0 :Int32;
svId @1 :Int32;
messageId @2 :Int32;
submessageId @3 :Int32;
data @4 :Data;
status @5 :Status;
enum Status @0xec1ff7996b35366f {
unknown @0;
parityPassed @1;
parityRebuilt @2;
}
}
}
struct LidarPts @0xe3d6685d4e9d8f7a {
r @0 :List(UInt16); # uint16 m*500.0
theta @1 :List(UInt16); # uint16 deg*100.0
reflect @2 :List(UInt8); # uint8 0-255
# For storing out of file.
idx @3 :UInt64;
# For storing in file
pkt @4 :Data;
}
+474 -424
View File
File diff suppressed because it is too large Load Diff
+12 -24
View File
@@ -1,8 +1,10 @@
# must be built with scons
from msgq import fake_event_handle, drain_sock_raw, MultiplePublishersError, IpcError, \
Context, Poller, SubSocket, PubSocket, SocketEventHandle, toggle_fake_events, \
set_fake_prefix, get_fake_prefix, delete_fake_prefix, wait_for_one_event
from msgq.ipc_pyx import Context, Poller, SubSocket, PubSocket, SocketEventHandle, toggle_fake_events, \
set_fake_prefix, get_fake_prefix, delete_fake_prefix, wait_for_one_event
from msgq.ipc_pyx import MultiplePublishersError, IpcError
from msgq import fake_event_handle, pub_sock, sub_sock, drain_sock_raw
import msgq
import os
import capnp
import time
@@ -11,25 +13,11 @@ from typing import Optional, List, Union, Dict
from cereal import log
from cereal.services import SERVICE_LIST
from openpilot.common.utils import MovingAverage
from openpilot.common.util import MovingAverage
NO_TRAVERSAL_LIMIT = 2**64-1
def pub_sock(endpoint: str) -> PubSocket:
service = SERVICE_LIST.get(endpoint)
segment_size = service.queue_size if service else 0
return msgq.pub_sock(endpoint, segment_size)
def sub_sock(endpoint: str, poller: Optional[Poller] = None, addr: str = "127.0.0.1",
conflate: bool = False, timeout: Optional[int] = None) -> SubSocket:
service = SERVICE_LIST.get(endpoint)
segment_size = service.queue_size if service else 0
return msgq.sub_sock(endpoint, poller=poller, addr=addr, conflate=conflate,
timeout=timeout, segment_size=segment_size)
def reset_context():
msgq.context = Context()
@@ -259,11 +247,11 @@ class PubMaster:
self.sock[s].send(dat)
def wait_for_readers_to_update(self, s: str, timeout: int, dt: float = 0.05) -> bool:
try:
self.sock[s].wait_for_readers(timeout=timeout, interval=dt)
return True
except TimeoutError:
return False
for _ in range(int(timeout*(1./dt))):
if self.sock[s].all_readers_updated():
return True
time.sleep(dt)
return False
def all_readers_updated(self, s: str) -> bool:
return self.sock[s].all_readers_updated()
return self.sock[s].all_readers_updated() # type: ignore
BIN
View File
Binary file not shown.
-72
View File
@@ -1,72 +0,0 @@
#include <cassert>
#include "cereal/messaging/msgq_to_zmq.h"
#include "cereal/services.h"
#include "common/util.h"
ExitHandler do_exit;
static std::vector<std::string> get_services(const std::string &whitelist_str, bool zmq_to_msgq) {
std::vector<std::string> service_list;
for (const auto& it : services) {
std::string name = it.second.name;
bool in_whitelist = whitelist_str.find(name) != std::string::npos;
if (zmq_to_msgq && !in_whitelist) {
continue;
}
service_list.push_back(name);
}
return service_list;
}
void msgq_to_zmq(const std::vector<std::string> &endpoints, const std::string &ip) {
MsgqToZmq bridge;
bridge.run(endpoints, ip);
}
void zmq_to_msgq(const std::vector<std::string> &endpoints, const std::string &ip) {
auto poller = std::make_unique<BridgeZmqPoller>();
auto pub_context = std::make_unique<Context>();
auto sub_context = std::make_unique<BridgeZmqContext>();
std::map<BridgeZmqSubSocket *, PubSocket *> sub2pub;
for (auto endpoint : endpoints) {
auto pub_sock = new PubSocket();
auto sub_sock = new BridgeZmqSubSocket();
size_t queue_size = services.at(endpoint).queue_size;
pub_sock->connect(pub_context.get(), endpoint, true, queue_size);
sub_sock->connect(sub_context.get(), endpoint, ip, false);
poller->registerSocket(sub_sock);
sub2pub[sub_sock] = pub_sock;
}
while (!do_exit) {
for (auto sub_sock : poller->poll(100)) {
std::unique_ptr<Message> msg(sub_sock->receive(true));
if (msg) {
sub2pub[sub_sock]->sendMessage(msg.get());
}
}
}
// Clean up allocated sockets
for (auto &[sub_sock, pub_sock] : sub2pub) {
delete sub_sock;
delete pub_sock;
}
}
int main(int argc, char **argv) {
bool is_zmq_to_msgq = argc > 2;
std::string ip = is_zmq_to_msgq ? argv[1] : "127.0.0.1";
std::string whitelist_str = is_zmq_to_msgq ? std::string(argv[2]) : "";
std::vector<std::string> endpoints = get_services(whitelist_str, is_zmq_to_msgq);
if (is_zmq_to_msgq) {
zmq_to_msgq(endpoints, ip);
} else {
msgq_to_zmq(endpoints, ip);
}
return 0;
}
-170
View File
@@ -1,170 +0,0 @@
#include "cereal/messaging/bridge_zmq.h"
#include <cassert>
#include <cstring>
#include <unistd.h>
static size_t fnv1a_hash(const std::string &str) {
const size_t fnv_prime = 0x100000001b3;
size_t hash_value = 0xcbf29ce484222325;
for (char c : str) {
hash_value ^= (unsigned char)c;
hash_value *= fnv_prime;
}
return hash_value;
}
// FIXME: This is a hack to get the port number from the socket name, might have collisions.
static int get_port(std::string endpoint) {
size_t hash_value = fnv1a_hash(endpoint);
int start_port = 8023;
int max_port = 65535;
return start_port + (hash_value % (max_port - start_port));
}
BridgeZmqContext::BridgeZmqContext() {
context = zmq_ctx_new();
}
BridgeZmqContext::~BridgeZmqContext() {
if (context != nullptr) {
zmq_ctx_term(context);
}
}
void BridgeZmqMessage::init(size_t sz) {
size = sz;
data = new char[size];
}
void BridgeZmqMessage::init(char *d, size_t sz) {
size = sz;
data = new char[size];
memcpy(data, d, size);
}
void BridgeZmqMessage::close() {
if (size > 0) {
delete[] data;
}
data = nullptr;
size = 0;
}
BridgeZmqMessage::~BridgeZmqMessage() {
close();
}
int BridgeZmqSubSocket::connect(BridgeZmqContext *context, std::string endpoint, std::string address, bool conflate, bool check_endpoint) {
sock = zmq_socket(context->getRawContext(), ZMQ_SUB);
if (sock == nullptr) {
return -1;
}
zmq_setsockopt(sock, ZMQ_SUBSCRIBE, "", 0);
if (conflate) {
int arg = 1;
zmq_setsockopt(sock, ZMQ_CONFLATE, &arg, sizeof(int));
}
int reconnect_ivl = 500;
zmq_setsockopt(sock, ZMQ_RECONNECT_IVL_MAX, &reconnect_ivl, sizeof(reconnect_ivl));
full_endpoint = "tcp://" + address + ":";
if (check_endpoint) {
full_endpoint += std::to_string(get_port(endpoint));
} else {
full_endpoint += endpoint;
}
return zmq_connect(sock, full_endpoint.c_str());
}
void BridgeZmqSubSocket::setTimeout(int timeout) {
zmq_setsockopt(sock, ZMQ_RCVTIMEO, &timeout, sizeof(int));
}
Message *BridgeZmqSubSocket::receive(bool non_blocking) {
zmq_msg_t msg;
assert(zmq_msg_init(&msg) == 0);
int flags = non_blocking ? ZMQ_DONTWAIT : 0;
int rc = zmq_msg_recv(&msg, sock, flags);
Message *ret = nullptr;
if (rc >= 0) {
ret = new BridgeZmqMessage;
ret->init((char *)zmq_msg_data(&msg), zmq_msg_size(&msg));
}
zmq_msg_close(&msg);
return ret;
}
BridgeZmqSubSocket::~BridgeZmqSubSocket() {
if (sock != nullptr) {
zmq_close(sock);
}
}
int BridgeZmqPubSocket::connect(BridgeZmqContext *context, std::string endpoint, bool check_endpoint) {
sock = zmq_socket(context->getRawContext(), ZMQ_PUB);
if (sock == nullptr) {
return -1;
}
full_endpoint = "tcp://*:";
if (check_endpoint) {
full_endpoint += std::to_string(get_port(endpoint));
} else {
full_endpoint += endpoint;
}
// ZMQ pub sockets cannot be shared between processes, so we need to ensure pid stays the same.
pid = getpid();
return zmq_bind(sock, full_endpoint.c_str());
}
int BridgeZmqPubSocket::sendMessage(Message *message) {
assert(pid == getpid());
return zmq_send(sock, message->getData(), message->getSize(), ZMQ_DONTWAIT);
}
int BridgeZmqPubSocket::send(char *data, size_t size) {
assert(pid == getpid());
return zmq_send(sock, data, size, ZMQ_DONTWAIT);
}
BridgeZmqPubSocket::~BridgeZmqPubSocket() {
if (sock != nullptr) {
zmq_close(sock);
}
}
void BridgeZmqPoller::registerSocket(BridgeZmqSubSocket *socket) {
assert(num_polls + 1 < (sizeof(polls) / sizeof(polls[0])));
polls[num_polls].socket = socket->getRawSocket();
polls[num_polls].events = ZMQ_POLLIN;
sockets.push_back(socket);
num_polls++;
}
std::vector<BridgeZmqSubSocket *> BridgeZmqPoller::poll(int timeout) {
std::vector<BridgeZmqSubSocket *> ret;
int rc = zmq_poll(polls, num_polls, timeout);
if (rc < 0) {
return ret;
}
for (size_t i = 0; i < num_polls; i++) {
if (polls[i].revents) {
ret.push_back(sockets[i]);
}
}
return ret;
}
-72
View File
@@ -1,72 +0,0 @@
#pragma once
#include <cstddef>
#include <string>
#include <vector>
#include <zmq.h>
#include "msgq/ipc.h"
class BridgeZmqContext {
public:
BridgeZmqContext();
void *getRawContext() { return context; }
~BridgeZmqContext();
private:
void *context = nullptr;
};
class BridgeZmqMessage : public Message {
public:
void init(size_t size);
void init(char *data, size_t size);
void close();
size_t getSize() { return size; }
char *getData() { return data; }
~BridgeZmqMessage();
private:
char *data = nullptr;
size_t size = 0;
};
class BridgeZmqSubSocket {
public:
int connect(BridgeZmqContext *context, std::string endpoint, std::string address, bool conflate = false, bool check_endpoint = true);
void setTimeout(int timeout);
Message *receive(bool non_blocking = false);
void *getRawSocket() { return sock; }
~BridgeZmqSubSocket();
private:
void *sock = nullptr;
std::string full_endpoint;
};
class BridgeZmqPubSocket {
public:
int connect(BridgeZmqContext *context, std::string endpoint, bool check_endpoint = true);
int sendMessage(Message *message);
int send(char *data, size_t size);
void *getRawSocket() { return sock; }
~BridgeZmqPubSocket();
private:
void *sock = nullptr;
std::string full_endpoint;
int pid = -1;
};
class BridgeZmqPoller {
public:
void registerSocket(BridgeZmqSubSocket *socket);
std::vector<BridgeZmqSubSocket *> poll(int timeout);
private:
static constexpr size_t MAX_BRIDGE_ZMQ_POLLERS = 128;
std::vector<BridgeZmqSubSocket *> sockets;
zmq_pollitem_t polls[MAX_BRIDGE_ZMQ_POLLERS] = {};
size_t num_polls = 0;
};
-146
View File
@@ -1,146 +0,0 @@
#include "cereal/messaging/msgq_to_zmq.h"
#include <cassert>
#include "cereal/services.h"
#include "common/util.h"
extern ExitHandler do_exit;
// Max messages to process per socket per poll
constexpr int MAX_MESSAGES_PER_SOCKET = 50;
static std::string recv_zmq_msg(void *sock) {
zmq_msg_t msg;
zmq_msg_init(&msg);
std::string ret;
if (zmq_msg_recv(&msg, sock, 0) > 0) {
ret.assign((char *)zmq_msg_data(&msg), zmq_msg_size(&msg));
}
zmq_msg_close(&msg);
return ret;
}
void MsgqToZmq::run(const std::vector<std::string> &endpoints, const std::string &ip) {
zmq_context = std::make_unique<BridgeZmqContext>();
msgq_context = std::make_unique<Context>();
// Create ZMQPubSockets for each endpoint
for (const auto &endpoint : endpoints) {
auto &socket_pair = socket_pairs.emplace_back();
socket_pair.endpoint = endpoint;
socket_pair.pub_sock = std::make_unique<BridgeZmqPubSocket>();
int ret = socket_pair.pub_sock->connect(zmq_context.get(), endpoint);
if (ret != 0) {
printf("Failed to create ZMQ publisher for [%s]: %s\n", endpoint.c_str(), zmq_strerror(zmq_errno()));
return;
}
}
// Start ZMQ monitoring thread to monitor socket events
std::thread thread(&MsgqToZmq::zmqMonitorThread, this);
// Main loop for processing messages
while (!do_exit) {
{
std::unique_lock lk(mutex);
cv.wait(lk, [this]() { return do_exit || !sub2pub.empty(); });
if (do_exit) break;
for (auto sub_sock : msgq_poller->poll(100)) {
// Process messages for each socket
BridgeZmqPubSocket *pub_sock = sub2pub.at(sub_sock);
for (int i = 0; i < MAX_MESSAGES_PER_SOCKET; ++i) {
auto msg = std::unique_ptr<Message>(sub_sock->receive(true));
if (!msg) break;
while (pub_sock->sendMessage(msg.get()) == -1) {
if (errno != EINTR) break;
}
}
}
}
util::sleep_for(1); // Give zmqMonitorThread a chance to acquire the mutex
}
thread.join();
}
void MsgqToZmq::zmqMonitorThread() {
std::vector<zmq_pollitem_t> pollitems;
// Set up ZMQ monitor for each pub socket
for (int i = 0; i < socket_pairs.size(); ++i) {
std::string addr = "inproc://op-bridge-monitor-" + std::to_string(i);
zmq_socket_monitor(socket_pairs[i].pub_sock->getRawSocket(), addr.c_str(), ZMQ_EVENT_ACCEPTED | ZMQ_EVENT_DISCONNECTED);
void *monitor_socket = zmq_socket(zmq_context->getRawContext(), ZMQ_PAIR);
zmq_connect(monitor_socket, addr.c_str());
pollitems.emplace_back(zmq_pollitem_t{.socket = monitor_socket, .events = ZMQ_POLLIN});
}
while (!do_exit) {
int ret = zmq_poll(pollitems.data(), pollitems.size(), 1000);
if (ret < 0) {
if (errno == EINTR) {
// Due to frequent EINTR signals from msgq, introduce a brief delay (200 ms)
// to reduce CPU usage during retry attempts.
util::sleep_for(200);
}
continue;
}
for (int i = 0; i < pollitems.size(); ++i) {
if (pollitems[i].revents & ZMQ_POLLIN) {
// First frame in message contains event number and value
std::string frame = recv_zmq_msg(pollitems[i].socket);
if (frame.empty()) continue;
uint16_t event_type = *(uint16_t *)(frame.data());
// Second frame in message contains event address
frame = recv_zmq_msg(pollitems[i].socket);
if (frame.empty()) continue;
std::unique_lock lk(mutex);
auto &pair = socket_pairs[i];
if (event_type & ZMQ_EVENT_ACCEPTED) {
printf("socket [%s] connected\n", pair.endpoint.c_str());
if (++pair.connected_clients == 1) {
// Create new MSGQ subscriber socket and map to ZMQ publisher
pair.sub_sock = std::make_unique<MSGQSubSocket>();
size_t queue_size = services.at(pair.endpoint).queue_size;
pair.sub_sock->connect(msgq_context.get(), pair.endpoint, "127.0.0.1", false, true, queue_size);
sub2pub[pair.sub_sock.get()] = pair.pub_sock.get();
registerSockets();
}
} else if (event_type & ZMQ_EVENT_DISCONNECTED) {
printf("socket [%s] disconnected\n", pair.endpoint.c_str());
if (pair.connected_clients == 0 || --pair.connected_clients == 0) {
// Remove MSGQ subscriber socket from mapping and reset it
sub2pub.erase(pair.sub_sock.get());
pair.sub_sock.reset(nullptr);
registerSockets();
}
}
cv.notify_one();
}
}
}
// Clean up monitor sockets
for (int i = 0; i < pollitems.size(); ++i) {
zmq_socket_monitor(socket_pairs[i].pub_sock->getRawSocket(), nullptr, 0);
zmq_close(pollitems[i].socket);
}
cv.notify_one();
}
void MsgqToZmq::registerSockets() {
msgq_poller = std::make_unique<MSGQPoller>();
for (const auto &socket_pair : socket_pairs) {
if (socket_pair.sub_sock) {
msgq_poller->registerSocket(socket_pair.sub_sock.get());
}
}
}
+6 -5
View File
@@ -7,8 +7,9 @@
#include <string>
#include <vector>
#define private public
#include "msgq/impl_msgq.h"
#include "cereal/messaging/bridge_zmq.h"
#include "msgq/impl_zmq.h"
class MsgqToZmq {
public:
@@ -21,16 +22,16 @@ protected:
struct SocketPair {
std::string endpoint;
std::unique_ptr<BridgeZmqPubSocket> pub_sock;
std::unique_ptr<ZMQPubSocket> pub_sock;
std::unique_ptr<MSGQSubSocket> sub_sock;
int connected_clients = 0;
};
std::unique_ptr<Context> msgq_context;
std::unique_ptr<BridgeZmqContext> zmq_context;
std::unique_ptr<MSGQContext> msgq_context;
std::unique_ptr<ZMQContext> zmq_context;
std::mutex mutex;
std::condition_variable cv;
std::unique_ptr<MSGQPoller> msgq_poller;
std::map<SubSocket *, BridgeZmqPubSocket *> sub2pub;
std::map<SubSocket *, ZMQPubSocket *> sub2pub;
std::vector<SocketPair> socket_pairs;
};
-204
View File
@@ -1,204 +0,0 @@
#include <assert.h>
#include <stdlib.h>
#include <string>
#include <mutex>
#include "cereal/services.h"
#include "cereal/messaging/messaging.h"
const bool SIMULATION = (getenv("SIMULATION") != nullptr) && (std::string(getenv("SIMULATION")) == "1");
static inline bool inList(const std::vector<const char *> &list, const char *value) {
for (auto &v : list) {
if (strcmp(value, v) == 0) return true;
}
return false;
}
class MessageContext {
public:
MessageContext() : ctx_(nullptr) {}
~MessageContext() { delete ctx_; }
inline Context *context() {
std::call_once(init_flag, [=]() { ctx_ = Context::create(); });
return ctx_;
}
private:
Context *ctx_;
std::once_flag init_flag;
};
MessageContext message_context;
struct SubMaster::SubMessage {
std::string name;
SubSocket *socket = nullptr;
float freq = 0.0f;
bool updated = false, alive = false, valid = false, ignore_alive;
uint64_t rcv_time = 0, rcv_frame = 0;
void *allocated_msg_reader = nullptr;
bool is_polled = false;
capnp::FlatArrayMessageReader *msg_reader = nullptr;
AlignedBuffer aligned_buf;
cereal::Event::Reader event;
};
SubMaster::SubMaster(const std::vector<const char *> &service_list, const std::vector<const char *> &poll,
const char *address, const std::vector<const char *> &ignore_alive) {
poller_ = Poller::create();
for (auto name : service_list) {
assert(services.count(std::string(name)) > 0);
service serv = services.at(std::string(name));
SubSocket *socket = SubSocket::create(message_context.context(), name, address ? address : "127.0.0.1", true, true, serv.queue_size);
assert(socket != 0);
bool is_polled = inList(poll, name) || poll.empty();
if (is_polled) poller_->registerSocket(socket);
SubMessage *m = new SubMessage{
.name = name,
.socket = socket,
.freq = serv.frequency,
.ignore_alive = inList(ignore_alive, name),
.allocated_msg_reader = malloc(sizeof(capnp::FlatArrayMessageReader)),
.is_polled = is_polled};
m->msg_reader = new (m->allocated_msg_reader) capnp::FlatArrayMessageReader({});
messages_[socket] = m;
services_[name] = m;
}
}
void SubMaster::update(int timeout) {
for (auto &kv : messages_) kv.second->updated = false;
auto sockets = poller_->poll(timeout);
// add non-polled sockets for non-blocking receive
for (auto &kv : messages_) {
SubMessage *m = kv.second;
SubSocket *s = kv.first;
if (!m->is_polled) sockets.push_back(s);
}
uint64_t current_time = nanos_since_boot();
std::vector<std::pair<std::string, cereal::Event::Reader>> messages;
for (auto s : sockets) {
Message *msg = s->receive(true);
if (msg == nullptr) continue;
SubMessage *m = messages_.at(s);
m->msg_reader->~FlatArrayMessageReader();
capnp::ReaderOptions options;
options.traversalLimitInWords = kj::maxValue; // Don't limit
m->msg_reader = new (m->allocated_msg_reader) capnp::FlatArrayMessageReader(m->aligned_buf.align(msg), options);
delete msg;
messages.push_back({m->name, m->msg_reader->getRoot<cereal::Event>()});
}
update_msgs(current_time, messages);
}
void SubMaster::update_msgs(uint64_t current_time, const std::vector<std::pair<std::string, cereal::Event::Reader>> &messages){
if (++frame == UINT64_MAX) frame = 1;
for (auto &kv : messages) {
auto m_find = services_.find(kv.first);
if (m_find == services_.end()){
continue;
}
SubMessage *m = m_find->second;
m->event = kv.second;
m->updated = true;
m->rcv_time = current_time;
m->rcv_frame = frame;
m->valid = m->event.getValid();
if (SIMULATION) m->alive = true;
}
if (!SIMULATION) {
for (auto &kv : messages_) {
SubMessage *m = kv.second;
m->alive = (m->freq <= (1e-5) || ((current_time - m->rcv_time) * (1e-9)) < (10.0 / m->freq));
}
}
}
bool SubMaster::all_(const std::vector<const char *> &service_list, bool valid, bool alive) {
int found = 0;
for (auto &kv : messages_) {
SubMessage *m = kv.second;
if (service_list.size() == 0 || inList(service_list, m->name.c_str())) {
found += (!valid || m->valid) && (!alive || (m->alive || m->ignore_alive));
}
}
return service_list.size() == 0 ? found == messages_.size() : found == service_list.size();
}
void SubMaster::drain() {
while (true) {
auto polls = poller_->poll(0);
if (polls.size() == 0)
break;
for (auto sock : polls) {
Message *msg = sock->receive(true);
delete msg;
}
}
}
bool SubMaster::updated(const char *name) const {
return services_.at(name)->updated;
}
bool SubMaster::alive(const char *name) const {
return services_.at(name)->alive;
}
bool SubMaster::valid(const char *name) const {
return services_.at(name)->valid;
}
uint64_t SubMaster::rcv_frame(const char *name) const {
return services_.at(name)->rcv_frame;
}
uint64_t SubMaster::rcv_time(const char *name) const {
return services_.at(name)->rcv_time;
}
cereal::Event::Reader &SubMaster::operator[](const char *name) const {
return services_.at(name)->event;
}
SubMaster::~SubMaster() {
delete poller_;
for (auto &kv : messages_) {
SubMessage *m = kv.second;
m->msg_reader->~FlatArrayMessageReader();
free(m->allocated_msg_reader);
delete m->socket;
delete m;
}
}
PubMaster::PubMaster(const std::vector<const char *> &service_list) {
for (auto name : service_list) {
assert(services.count(name) > 0);
service serv = services.at(std::string(name));
PubSocket *socket = PubSocket::create(message_context.context(), name, true, serv.queue_size);
assert(socket);
sockets_[name] = socket;
}
}
int PubMaster::send(const char *name, MessageBuilder &msg) {
auto bytes = msg.toBytes();
return send(name, bytes.begin(), bytes.size());
}
PubMaster::~PubMaster() {
for (auto s : sockets_) delete s.second;
}
+2 -2
View File
@@ -5,7 +5,7 @@ import numbers
import random
import threading
import time
from openpilot.common.parameterized import parameterized
from parameterized import parameterized
import pytest
from cereal import log, car
@@ -30,7 +30,7 @@ def zmq_sleep(t=1):
# TODO: this should take any capnp struct and returrn a msg with random populated data
def random_carstate():
fields = ["vEgo", "aEgo", "steeringTorque", "steeringAngleDeg"]
fields = ["vEgo", "aEgo", "brake", "steeringAngleDeg"]
msg = messaging.new_message("carState")
cs = msg.carState
for f in fields:
+1 -1
View File
@@ -1,7 +1,7 @@
import os
import tempfile
from typing import Dict
from openpilot.common.parameterized import parameterized
from parameterized import parameterized
import cereal.services as services
from cereal.services import SERVICE_LIST
@@ -1,165 +1,222 @@
#!/usr/bin/env python3
"""Validate sunnypilot routes are parseable by stock commaai/openpilot.
Cap'n Proto is wire-compatible across renames, type relocations, and
additive fields. The only breaking change is a union variant that
upstream doesn't recognize — an unknown discriminant makes the entire
union unreadable.
This script checks: for every struct with a union that exists in both
schemas, does sunnypilot introduce union variants upstream doesn't have?
"""
from __future__ import annotations
import argparse
import json
import os
import sys
from typing import Any
from typing import Any, List, Tuple
NO_DISCRIMINANT = 0xFFFF
DEBUG = False
def hex_id(value: int) -> str:
return f"0x{value:016x}"
def print_debug(string: str) -> None:
if DEBUG:
print(string)
def encode_struct(schema: Any) -> dict:
node = schema.node
fields = []
for name, field in schema.fields.items():
proto = field.proto
ordinal = proto.ordinal.explicit if proto.ordinal.which() == "explicit" else None
discriminant = proto.discriminantValue if proto.discriminantValue != NO_DISCRIMINANT else None
fields.append({"name": name, "ordinal": ordinal, "discriminant": discriminant})
return {
"typeId": hex_id(node.id),
"displayName": node.displayName,
"hasUnion": node.struct.discriminantCount > 0,
"fields": fields,
}
def create_schema_instance(struct: Any, prop: Tuple[str, Any]) -> Any:
"""
Create a new instance of a schema type, handling different field types.
Args:
struct: The Cap'n Proto schema structure
prop: A tuple containing the field name and field metadata
Returns:
A new initialized schema instance
"""
struct_instance = struct.new_message()
field_name, field_metadata = prop
try:
field_type = field_metadata.proto.slot.type.which()
# Initialize different types of fields
if field_type in ('list', 'text', 'data'):
struct_instance.init(field_name, 1)
print_debug(f"Initialized list/text/data field: {field_name}")
elif field_type in ('struct', 'object'):
struct_instance.init(field_name)
print_debug(f"Initialized struct/object field: {field_name}")
return struct_instance
except Exception as e:
print(f"Error creating instance for {field_name}: {e}")
return None
def _child_struct_schema(field: Any) -> Any:
proto = field.proto
if proto.which() == "group":
return field.schema
type_node = proto.slot.type
which = type_node.which()
if which == "struct":
return field.schema
if which == "list":
container = field.schema
element_type = type_node.list.elementType
while element_type.which() == "list":
container = container.elementType
element_type = element_type.list.elementType
if element_type.which() == "struct":
return container.elementType
return None
def get_schema_fields(schema_struct: Any) -> List[Tuple[str, Any]]:
"""
Retrieve all fields from a given schema structure.
Args:
schema_struct: The Cap'n Proto schema structure
Returns:
A list of field names and their metadata
"""
try:
# Get all fields from the schema
schema_fields = list(schema_struct.schema.fields.items())
print_debug("Discovered schema fields:")
for field_name, field_metadata in schema_fields:
print_debug(f"- {field_name}")
return schema_fields
except Exception as e:
print(f"Error retrieving schema fields: {e}")
return []
def collect_schema(root: Any) -> dict[str, dict]:
structs: dict[str, dict] = {}
stack = [root]
while stack:
schema = stack.pop()
type_id = hex_id(schema.node.id)
if type_id in structs:
continue
structs[type_id] = encode_struct(schema)
for _name, field in schema.fields.items():
try:
child = _child_struct_schema(field)
except Exception:
child = None
if child is not None:
stack.append(child)
return structs
def generate_schema_instances(schema_struct: Any) -> List[Any]:
"""
Generate instances for all fields in a given schema.
Args:
schema_struct: The Cap'n Proto schema structure
Returns:
A list of schema instances
"""
schema_fields = get_schema_fields(schema_struct)
instances = []
for field_prop in schema_fields:
try:
instance = create_schema_instance(schema_struct, field_prop)
if instance is not None:
instances.append(instance)
except Exception as e:
print(f"Skipping field due to error: {e}")
print(f"Generated {len(instances)} schema instances")
return instances
def load_log(cereal_dir: str, extra_imports: list[str] | None = None) -> Any:
import capnp
cereal_dir = os.path.abspath(cereal_dir)
capnp.remove_import_hook()
imports = [cereal_dir] + [os.path.abspath(p) for p in (extra_imports or [])]
return capnp.load(os.path.join(cereal_dir, "log.capnp"), imports=imports)
def persist_instances(instances: List[Any], filename: str) -> None:
"""
Write schema instances to a binary file.
Args:
instances: List of schema instances
filename: Output file path
"""
try:
with open(filename, 'wb') as f:
for instance in instances:
f.write(instance.to_bytes())
print(f"Successfully wrote {len(instances)} instances to {filename}")
except Exception as e:
print(f"Error persisting instances: {e}")
sys.exit(1)
def dump_schema(cereal_dir: str, path: str, extra_imports: list[str] | None = None) -> None:
log = load_log(cereal_dir, extra_imports)
payload = {
"root": hex_id(log.Event.schema.node.id),
"structs": collect_schema(log.Event.schema),
}
with open(path, "w", encoding="utf-8") as handle:
json.dump(payload, handle, indent=2, sort_keys=True)
print(f"wrote schema dump with {len(payload['structs'])} structs to {path}")
def read_instances(filename: str, schema_type: Any) -> List[Any]:
"""
Read schema instances from a binary file.
Args:
filename: Input file path
schema_type: The schema type to use for reading
Returns:
A list of read schema instances
"""
try:
with open(filename, 'rb') as f:
data = f.read()
instances = list(schema_type.read_multiple_bytes(data))
print(f"Read {len(instances)} instances from {filename}")
return instances
except Exception as e:
print(f"Error reading instances: {e}")
sys.exit(1)
def compare(sunnypilot_dump: dict, upstream_dump: dict) -> list[str]:
violations: list[str] = []
sunnypilot_structs = sunnypilot_dump["structs"]
upstream_structs = upstream_dump["structs"]
def compare_schemas(original_instances: List[Any], read_instances: List[Any]) -> bool:
"""
Compare original and read-back instances to detect potential breaking changes.
for type_id, sp_struct in sunnypilot_structs.items():
if not sp_struct["hasUnion"]:
continue
up_struct = upstream_structs.get(type_id)
if up_struct is None:
continue
Args:
original_instances: List of originally generated instances
read_instances: List of instances read back from file
up_ordinals = {f["ordinal"] for f in up_struct["fields"] if f.get("discriminant") is not None}
display = sp_struct["displayName"]
Returns:
Boolean indicating whether schemas appear compatible
"""
if len(original_instances) != len(read_instances):
print("❌ Schema Compatibility Warning: Instance count mismatch")
return False
for field in sp_struct["fields"]:
if field.get("discriminant") is None:
continue
if field["ordinal"] not in up_ordinals:
violations.append(
f"{display} @{field['ordinal']} '{field['name']}': "
f"union variant not present upstream (discriminant={field['discriminant']})"
)
compatible = True
for struct in read_instances:
try:
getattr(struct, struct.which()) # Attempting to access the field to validate readability
except Exception as e:
print(f"❌ Structural change detected: {struct.which()} is not readable.\nFull error: {e}")
compatible = False
return violations
return compatible
def run_read(cereal_dir: str, peer_path: str, extra_imports: list[str] | None = None) -> int:
log = load_log(cereal_dir, extra_imports)
with open(peer_path, "r", encoding="utf-8") as f:
peer_dump = json.load(f)
local_dump = {
"root": hex_id(log.Event.schema.node.id),
"structs": collect_schema(log.Event.schema),
}
violations = compare(sunnypilot_dump=peer_dump, upstream_dump=local_dump)
def main():
"""
CLI entry point for schema compatibility testing.
"""
# Setup argument parser
parser = argparse.ArgumentParser(
description='Cap\'n Proto Schema Compatibility Testing Tool',
epilog='Test schema compatibility by generating and reading back instances.'
)
if not violations:
print("cereal compat OK: upstream can parse sunnypilot routes.")
return 0
# Add mutually exclusive group for generation or reading mode
mode_group = parser.add_mutually_exclusive_group(required=True)
mode_group.add_argument('-g', '--generate', action='store_true',
help='Generate schema instances')
mode_group.add_argument('-r', '--read', action='store_true',
help='Read and validate schema instances')
print(f"cereal compat FAIL ({len(violations)} leaked union variant(s)):")
for v in violations:
print(f" {v}")
return 1
# Common arguments
parser.add_argument('-f', '--file',
default='schema_instances.bin',
help='Output/input binary file (default: schema_instances.bin)')
def main() -> int:
parser = argparse.ArgumentParser(description="sunnypilot cereal upstream compat check")
mode = parser.add_mutually_exclusive_group(required=True)
mode.add_argument("-g", "--generate", action="store_true", help="dump local schema to JSON")
mode.add_argument("-r", "--read", action="store_true", help="validate against peer schema")
parser.add_argument("-f", "--file", default="schema.json", help="JSON file path")
parser.add_argument("--cereal-dir", required=True, help="path to cereal directory")
parser.add_argument("-I", "--import-path", action="append", default=[], help="extra capnp import paths")
# Parse arguments
args = parser.parse_args()
# Import the schema dynamically
try:
from cereal import log
schema_type = log.Event
except ImportError:
print("Error: Unable to import schema. Ensure 'cereal' is installed.")
sys.exit(1)
# Execute based on mode
if args.generate:
dump_schema(args.cereal_dir, args.file, args.import_path)
return 0
return run_read(args.cereal_dir, args.file, args.import_path)
print("🔧 Generating Schema Instances")
instances = generate_schema_instances(schema_type)
persist_instances(instances, args.file)
print("✅ Instance generation complete")
elif args.read:
print("🔍 Reading and Validating Schema Instances")
generated_instances = generate_schema_instances(schema_type)
read_back_instances = read_instances(args.file, schema_type)
# Compare schemas
if compare_schemas(generated_instances, read_back_instances):
print("✅ Schema Compatibility: No breaking changes detected")
sys.exit(0)
else:
print("❌ Potential Schema Breaking Changes Detected")
sys.exit(1)
if __name__ == "__main__":
sys.exit(main())
main()
+101
View File
@@ -0,0 +1,101 @@
/* THIS IS AN AUTOGENERATED FILE, PLEASE EDIT services.py */
#ifndef __SERVICES_H
#define __SERVICES_H
#include <map>
#include <string>
struct service { std::string name; bool should_log; int frequency; int decimation; };
static std::map<std::string, service> services = {
{ "gyroscope", {"gyroscope", true, 104, 104}},
{ "gyroscope2", {"gyroscope2", true, 100, 100}},
{ "accelerometer", {"accelerometer", true, 104, 104}},
{ "accelerometer2", {"accelerometer2", true, 100, 100}},
{ "magnetometer", {"magnetometer", true, 25, -1}},
{ "lightSensor", {"lightSensor", true, 100, 100}},
{ "temperatureSensor", {"temperatureSensor", true, 2, 200}},
{ "temperatureSensor2", {"temperatureSensor2", true, 2, 200}},
{ "gpsNMEA", {"gpsNMEA", true, 9, -1}},
{ "deviceState", {"deviceState", true, 2, 1}},
{ "touch", {"touch", true, 20, 1}},
{ "can", {"can", true, 100, 2053}},
{ "controlsState", {"controlsState", true, 100, 10}},
{ "selfdriveState", {"selfdriveState", true, 100, 10}},
{ "pandaStates", {"pandaStates", true, 10, 1}},
{ "peripheralState", {"peripheralState", true, 2, 1}},
{ "radarState", {"radarState", true, 20, 5}},
{ "roadEncodeIdx", {"roadEncodeIdx", false, 20, 1}},
{ "liveTracks", {"liveTracks", true, 20, -1}},
{ "sendcan", {"sendcan", true, 100, 139}},
{ "logMessage", {"logMessage", true, 0, -1}},
{ "errorLogMessage", {"errorLogMessage", true, 0, 1}},
{ "liveCalibration", {"liveCalibration", true, 4, 4}},
{ "liveTorqueParameters", {"liveTorqueParameters", true, 4, 1}},
{ "liveDelay", {"liveDelay", true, 4, 1}},
{ "androidLog", {"androidLog", true, 0, -1}},
{ "carState", {"carState", true, 100, 10}},
{ "carControl", {"carControl", true, 100, 10}},
{ "carOutput", {"carOutput", true, 100, 10}},
{ "longitudinalPlan", {"longitudinalPlan", true, 20, 10}},
{ "driverAssistance", {"driverAssistance", true, 20, 20}},
{ "procLog", {"procLog", true, 0, 15}},
{ "gpsLocationExternal", {"gpsLocationExternal", true, 10, 10}},
{ "gpsLocation", {"gpsLocation", true, 1, 1}},
{ "ubloxGnss", {"ubloxGnss", true, 10, -1}},
{ "qcomGnss", {"qcomGnss", true, 2, -1}},
{ "gnssMeasurements", {"gnssMeasurements", true, 10, 10}},
{ "clocks", {"clocks", true, 0, 1}},
{ "ubloxRaw", {"ubloxRaw", true, 20, -1}},
{ "livePose", {"livePose", true, 20, 4}},
{ "liveParameters", {"liveParameters", true, 20, 5}},
{ "cameraOdometry", {"cameraOdometry", true, 20, 10}},
{ "thumbnail", {"thumbnail", true, 0, 1}},
{ "onroadEvents", {"onroadEvents", true, 1, 1}},
{ "carParams", {"carParams", true, 0, 1}},
{ "roadCameraState", {"roadCameraState", true, 20, 20}},
{ "driverCameraState", {"driverCameraState", true, 20, 20}},
{ "driverEncodeIdx", {"driverEncodeIdx", false, 20, 1}},
{ "driverStateV2", {"driverStateV2", true, 20, 10}},
{ "driverMonitoringState", {"driverMonitoringState", true, 20, 10}},
{ "wideRoadEncodeIdx", {"wideRoadEncodeIdx", false, 20, 1}},
{ "wideRoadCameraState", {"wideRoadCameraState", true, 20, 20}},
{ "drivingModelData", {"drivingModelData", true, 20, 10}},
{ "modelV2", {"modelV2", true, 20, -1}},
{ "managerState", {"managerState", true, 2, 1}},
{ "uploaderState", {"uploaderState", true, 0, 1}},
{ "navInstruction", {"navInstruction", true, 1, 10}},
{ "navRoute", {"navRoute", true, 0, -1}},
{ "navThumbnail", {"navThumbnail", true, 0, -1}},
{ "qRoadEncodeIdx", {"qRoadEncodeIdx", false, 20, -1}},
{ "userBookmark", {"userBookmark", true, 0, 1}},
{ "soundPressure", {"soundPressure", true, 10, 10}},
{ "rawAudioData", {"rawAudioData", false, 20, -1}},
{ "bookmarkButton", {"bookmarkButton", true, 0, 1}},
{ "audioFeedback", {"audioFeedback", true, 0, 1}},
{ "modelManagerSP", {"modelManagerSP", false, 1, 1}},
{ "backupManagerSP", {"backupManagerSP", false, 1, 1}},
{ "selfdriveStateSP", {"selfdriveStateSP", true, 100, 10}},
{ "longitudinalPlanSP", {"longitudinalPlanSP", true, 20, 10}},
{ "onroadEventsSP", {"onroadEventsSP", true, 1, 1}},
{ "carParamsSP", {"carParamsSP", true, 0, 1}},
{ "carControlSP", {"carControlSP", true, 100, 10}},
{ "carStateSP", {"carStateSP", true, 100, 10}},
{ "liveMapDataSP", {"liveMapDataSP", true, 1, 1}},
{ "modelDataV2SP", {"modelDataV2SP", true, 20, -1}},
{ "uiDebug", {"uiDebug", true, 0, 1}},
{ "testJoystick", {"testJoystick", true, 0, -1}},
{ "alertDebug", {"alertDebug", true, 20, 5}},
{ "roadEncodeData", {"roadEncodeData", false, 20, -1}},
{ "driverEncodeData", {"driverEncodeData", false, 20, -1}},
{ "wideRoadEncodeData", {"wideRoadEncodeData", false, 20, -1}},
{ "qRoadEncodeData", {"qRoadEncodeData", false, 20, -1}},
{ "livestreamWideRoadEncodeIdx", {"livestreamWideRoadEncodeIdx", false, 20, -1}},
{ "livestreamRoadEncodeIdx", {"livestreamRoadEncodeIdx", false, 20, -1}},
{ "livestreamDriverEncodeIdx", {"livestreamDriverEncodeIdx", false, 20, -1}},
{ "livestreamWideRoadEncodeData", {"livestreamWideRoadEncodeData", false, 20, -1}},
{ "livestreamRoadEncodeData", {"livestreamRoadEncodeData", false, 20, -1}},
{ "livestreamDriverEncodeData", {"livestreamDriverEncodeData", false, 20, -1}},
{ "customReservedRawData0", {"customReservedRawData0", true, 0, -1}},
{ "customReservedRawData1", {"customReservedRawData1", true, 0, -1}},
{ "customReservedRawData2", {"customReservedRawData2", true, 0, -1}},
};
#endif
+34 -33
View File
@@ -1,43 +1,39 @@
#!/usr/bin/env python3
from enum import IntEnum
from typing import Optional
# TODO: this should be automatically determined using the capnp schema
class QueueSize(IntEnum):
BIG = 10 * 1024 * 1024 # 10MB - video frames, large AI outputs
MEDIUM = 2 * 1024 * 1024 # 2MB - high freq (CAN), livestream
SMALL = 250 * 1024 # 250KB - most services
class Service:
def __init__(self, should_log: bool, frequency: float, decimation: Optional[int] = None,
queue_size: QueueSize = QueueSize.SMALL):
def __init__(self, should_log: bool, frequency: float, decimation: Optional[int] = None):
self.should_log = should_log
self.frequency = frequency
self.decimation = decimation
self.queue_size = queue_size
_services: dict[str, tuple] = {
# service: (should_log, frequency, qlog decimation (optional))
# note: the "EncodeIdx" packets will still be in the log
"gyroscope": (True, 104., 104),
"gyroscope2": (True, 100., 100),
"accelerometer": (True, 104., 104),
"accelerometer2": (True, 100., 100),
"magnetometer": (True, 25.),
"lightSensor": (True, 100., 100),
"temperatureSensor": (True, 2., 200),
"temperatureSensor2": (True, 2., 200),
"gpsNMEA": (True, 9.),
"deviceState": (True, 2., 1),
"touch": (True, 20., 1),
"can": (True, 100., 2053, QueueSize.BIG), # decimation gives ~3 msgs in a full segment
"controlsState": (True, 100., 10, QueueSize.MEDIUM),
"can": (True, 100., 2053), # decimation gives ~3 msgs in a full segment
"controlsState": (True, 100., 10),
"selfdriveState": (True, 100., 10),
"pandaStates": (True, 10., 1),
"peripheralState": (True, 2., 1),
"radarState": (True, 20., 5),
"roadEncodeIdx": (False, 20., 1),
"liveTracks": (True, 20.),
"sendcan": (True, 100., 139, QueueSize.MEDIUM),
"logMessage": (True, 0., None, QueueSize.BIG),
"errorLogMessage": (True, 0., 1, QueueSize.BIG),
"sendcan": (True, 100., 139),
"logMessage": (True, 0.),
"errorLogMessage": (True, 0., 1),
"liveCalibration": (True, 4., 4),
"liveTorqueParameters": (True, 4., 1),
"liveDelay": (True, 4., 1),
@@ -46,13 +42,13 @@ _services: dict[str, tuple] = {
"carControl": (True, 100., 10),
"carOutput": (True, 100., 10),
"longitudinalPlan": (True, 20., 10),
"lateralManeuverPlan": (True, 20.),
"driverAssistance": (True, 20., 20),
"procLog": (True, 0.5, 15, QueueSize.BIG),
"procLog": (True, 0.5, 15),
"gpsLocationExternal": (True, 10., 10),
"gpsLocation": (True, 1., 1),
"ubloxGnss": (True, 10.),
"qcomGnss": (True, 2.),
"gnssMeasurements": (True, 10., 10),
"clocks": (True, 0.1, 1),
"ubloxRaw": (True, 20.),
"livePose": (True, 20., 4),
@@ -69,22 +65,22 @@ _services: dict[str, tuple] = {
"wideRoadEncodeIdx": (False, 20., 1),
"wideRoadCameraState": (True, 20., 20),
"drivingModelData": (True, 20., 10),
"modelV2": (True, 20., None, QueueSize.BIG),
"modelV2": (True, 20.),
"managerState": (True, 2., 1),
"uploaderState": (True, 0., 1),
"navInstruction": (True, 1., 10),
"navRoute": (True, 0.),
"navThumbnail": (True, 0.),
"qRoadEncodeIdx": (False, 20.),
"userBookmark": (True, 0., 1),
"soundPressure": (True, 10., 10),
"rawAudioData": (False, 20.),
"bookmarkButton": (True, 0., 1),
"audioFeedback": (True, 0., 1),
"roadEncodeData": (False, 20., None, QueueSize.BIG),
"driverEncodeData": (False, 20., None, QueueSize.BIG),
"wideRoadEncodeData": (False, 20., None, QueueSize.BIG),
"qRoadEncodeData": (False, 20., None, QueueSize.BIG),
# sunnypilot
"modelManagerSP": (False, 1., 1, QueueSize.BIG),
"backupManagerSP": (False, 1., 1, QueueSize.BIG),
"modelManagerSP": (False, 1., 1),
"backupManagerSP": (False, 1., 1),
"selfdriveStateSP": (True, 100., 10),
"longitudinalPlanSP": (True, 20., 10),
"onroadEventsSP": (True, 1., 1),
@@ -92,20 +88,25 @@ _services: dict[str, tuple] = {
"carControlSP": (True, 100., 10),
"carStateSP": (True, 100., 10),
"liveMapDataSP": (True, 1., 1),
"modelDataV2SP": (True, 20., None, QueueSize.BIG),
"liveLocationKalman": (True, 20.),
"modelDataV2SP": (True, 20.),
# debug
"uiDebug": (True, 0., 1),
"testJoystick": (True, 0.),
"alertDebug": (True, 20., 5),
"roadEncodeData": (False, 20.),
"driverEncodeData": (False, 20.),
"wideRoadEncodeData": (False, 20.),
"qRoadEncodeData": (False, 20.),
"livestreamWideRoadEncodeIdx": (False, 20.),
"livestreamRoadEncodeIdx": (False, 20.),
"livestreamDriverEncodeIdx": (False, 20.),
"livestreamWideRoadEncodeData": (False, 20., None, QueueSize.MEDIUM),
"livestreamRoadEncodeData": (False, 20., None, QueueSize.MEDIUM),
"livestreamDriverEncodeData": (False, 20., None, QueueSize.MEDIUM),
"livestreamWideRoadEncodeData": (False, 20.),
"livestreamRoadEncodeData": (False, 20.),
"livestreamDriverEncodeData": (False, 20.),
"customReservedRawData0": (True, 0.),
"customReservedRawData1": (True, 0.),
"customReservedRawData2": (True, 0.),
}
SERVICE_LIST = {name: Service(*vals) for
idx, (name, vals) in enumerate(_services.items())}
@@ -120,13 +121,13 @@ def build_header():
h += "#include <map>\n"
h += "#include <string>\n"
h += "struct service { std::string name; bool should_log; float frequency; int decimation; size_t queue_size; };\n"
h += "struct service { std::string name; bool should_log; int frequency; int decimation; };\n"
h += "static std::map<std::string, service> services = {\n"
for k, v in SERVICE_LIST.items():
should_log = "true" if v.should_log else "false"
decimation = -1 if v.decimation is None else v.decimation
h += ' { "%s", {"%s", %s, %f, %d, %d}},\n' % \
(k, k, should_log, v.frequency, decimation, v.queue_size)
h += ' { "%s", {"%s", %s, %d, %d}},\n' % \
(k, k, should_log, v.frequency, decimation)
h += "};\n"
h += "#endif\n"
+1
View File
@@ -0,0 +1 @@
*.cpp
-23
View File
@@ -1,23 +0,0 @@
Import('env', 'envCython')
common_libs = [
'params.cc',
'swaglog.cc',
'util.cc',
'ratekeeper.cc',
]
_common = env.Library('common', common_libs, LIBS="json11")
Export('_common')
if GetOption('extras'):
env.Program('tests/test_common',
['tests/test_runner.cc', 'tests/test_params.cc', 'tests/test_util.cc', 'tests/test_swaglog.cc'],
LIBS=[_common, 'json11', 'zmq', 'pthread'])
# Cython bindings
params_python = envCython.Program('params_pyx.so', 'params_pyx.pyx', LIBS=envCython['LIBS'] + [_common, 'zmq', 'json11'])
common_python = [params_python]
Export('common_python')
+4 -8
View File
@@ -14,13 +14,9 @@ class Api:
def post(self, *args, **kwargs):
return self.service.post(*args, **kwargs)
def get_token(self, payload_extra=None, expiry_hours=1):
return self.service.get_token(payload_extra, expiry_hours)
def get_token(self, expiry_hours=1):
return self.service.get_token(expiry_hours)
def api_get(endpoint, method='GET', timeout=None, access_token=None, session=None, **params):
return CommaConnectApi(None).api_get(endpoint, method, timeout, access_token, session, **params)
def get_key_pair() -> tuple[str, str, str] | tuple[None, None, None]:
return CommaConnectApi(None).get_key_pair()
def api_get(endpoint, method='GET', timeout=None, access_token=None, **params):
return CommaConnectApi(None).api_get(endpoint, method, timeout, access_token, **params)
+9 -25
View File
@@ -1,22 +1,18 @@
import jwt
import os
import requests
import unicodedata
from datetime import datetime, timedelta, UTC
from openpilot.common.hardware.hw import Paths
from openpilot.system.hardware.hw import Paths
from openpilot.system.version import get_version
# name: jwt signature algorithm
KEYS = {"id_rsa": "RS256",
"id_ecdsa": "ES256"}
class BaseApi:
def __init__(self, dongle_id, api_host, user_agent="openpilot-"):
self.dongle_id = dongle_id
self.api_host = api_host
self.user_agent = user_agent
self.jwt_algorithm, self.private_key, _ = self.get_key_pair()
with open(f'{Paths.persist_root()}/comma/id_rsa') as f:
self.private_key = f.read()
def get(self, *args, **kwargs):
return self.request('GET', *args, **kwargs)
@@ -27,7 +23,7 @@ class BaseApi:
def request(self, method, endpoint, timeout=None, access_token=None, **params):
return self.api_get(endpoint, method=method, timeout=timeout, access_token=access_token, **params)
def _get_token(self, payload_extra=None, expiry_hours=1, **extra_payload):
def _get_token(self, expiry_hours=1, **extra_payload):
now = datetime.now(UTC).replace(tzinfo=None)
payload = {
'identity': self.dongle_id,
@@ -36,22 +32,20 @@ class BaseApi:
'exp': now + timedelta(hours=expiry_hours),
**extra_payload
}
if payload_extra is not None:
payload.update(payload_extra)
token = jwt.encode(payload, self.private_key, algorithm=self.jwt_algorithm)
token = jwt.encode(payload, self.private_key, algorithm='RS256')
if isinstance(token, bytes):
token = token.decode('utf8')
return token
def get_token(self, payload_extra=None, expiry_hours=1):
return self._get_token(payload_extra, expiry_hours)
def get_token(self, expiry_hours=1):
return self._get_token(expiry_hours)
def remove_non_ascii_chars(self, text):
normalized_text = unicodedata.normalize('NFD', text)
ascii_encoded_text = normalized_text.encode('ascii', 'ignore')
return ascii_encoded_text.decode()
def api_get(self, endpoint, method='GET', timeout=None, access_token=None, session=None, json=None, **params):
def api_get(self, endpoint, method='GET', timeout=None, access_token=None, json=None, **params):
headers = {}
if access_token is not None:
headers['Authorization'] = "JWT " + access_token
@@ -59,14 +53,4 @@ class BaseApi:
version = self.remove_non_ascii_chars(get_version())
headers['User-Agent'] = self.user_agent + version
# TODO: add session to Api
req = requests if session is None else session
return req.request(method, f"{self.api_host}/{endpoint}", timeout=timeout, headers=headers, json=json, params=params)
@staticmethod
def get_key_pair() -> tuple[str, str, str] | tuple[None, None, None]:
for key in KEYS:
if os.path.isfile(Paths.persist_root() + f'/comma/{key}') and os.path.isfile(Paths.persist_root() + f'/comma/{key}.pub'):
with open(Paths.persist_root() + f'/comma/{key}') as private, open(Paths.persist_root() + f'/comma/{key}.pub') as public:
return KEYS[key], private.read(), public.read()
return None, None, None
return requests.request(method, f"{self.api_host}/{endpoint}", timeout=timeout, headers=headers, json=json, params=params)
+28
View File
@@ -0,0 +1,28 @@
#pragma once
#ifdef __APPLE__
#include <OpenCL/cl.h>
#else
#include <CL/cl.h>
#endif
#include <string>
#define CL_CHECK(_expr) \
do { \
assert(CL_SUCCESS == (_expr)); \
} while (0)
#define CL_CHECK_ERR(_expr) \
({ \
cl_int err = CL_INVALID_VALUE; \
__typeof__(_expr) _ret = _expr; \
assert(_ret&& err == CL_SUCCESS); \
_ret; \
})
cl_device_id cl_get_device_id(cl_device_type device_type);
cl_context cl_create_context(cl_device_id device_id);
void cl_release_context(cl_context context);
cl_program cl_program_from_source(cl_context ctx, cl_device_id device_id, const std::string& src, const char* args = nullptr);
cl_program cl_program_from_file(cl_context ctx, cl_device_id device_id, const char* path, const char* args);
+9
View File
@@ -0,0 +1,9 @@
# remove all keys that end in DEPRECATED
def strip_deprecated_keys(d):
for k in list(d.keys()):
if isinstance(k, str):
if k.endswith('DEPRECATED'):
d.pop(k)
elif isinstance(d[k], dict):
strip_deprecated_keys(d[k])
return d
-3
View File
@@ -1,3 +0,0 @@
from openpilot.common.esim.base import LPABase, LPAError, LPAProfileNotFoundError, Profile
__all__ = ["LPABase", "LPAError", "LPAProfileNotFoundError", "Profile"]
-55
View File
@@ -1,55 +0,0 @@
from abc import ABC, abstractmethod
from dataclasses import dataclass
class LPAError(RuntimeError):
pass
class LPAProfileNotFoundError(LPAError):
pass
@dataclass
class Profile:
iccid: str
nickname: str
enabled: bool
provider: str
@property
def is_comma(self) -> bool:
return self.provider == 'Webbing' and self.iccid.startswith('8985235')
class LPABase(ABC):
@abstractmethod
def list_profiles(self) -> list[Profile]:
pass
@abstractmethod
def get_active_profile(self) -> Profile | None:
pass
@abstractmethod
def delete_profile(self, iccid: str) -> None:
pass
@abstractmethod
def download_profile(self, qr: str, nickname: str | None = None) -> None:
pass
@abstractmethod
def nickname_profile(self, iccid: str, nickname: str) -> None:
pass
@abstractmethod
def switch_profile(self, iccid: str) -> None:
pass
def process_notifications(self) -> None:
pass
@abstractmethod
def is_euicc(self) -> bool:
pass
-70
View File
@@ -1,70 +0,0 @@
#!/usr/bin/env python3
import argparse
from openpilot.common.hardware import HARDWARE
from openpilot.common.esim.base import LPABase, Profile
def sorted_profiles(lpa: LPABase) -> list[Profile]:
return sorted(lpa.list_profiles(), key=lambda p: p.iccid)
def resolve_iccid(lpa: LPABase, ref: str) -> str:
# ref is either a 1-based index into the sorted list, or a literal iccid
if ref.isdigit():
profiles = sorted_profiles(lpa)
idx = int(ref) - 1
if not 0 <= idx < len(profiles):
raise SystemExit(f'no profile at index {ref} (have {len(profiles)})')
return profiles[idx].iccid
return ref
def print_profiles(lpa: LPABase) -> None:
profiles = sorted_profiles(lpa)
print(f'\n{len(profiles)} profile{"s" if len(profiles) != 1 else ""}:')
for i, p in enumerate(profiles, start=1):
print(f'{i}. {p.iccid} (nickname: {p.nickname or "<none provided>"}) (provider: {p.provider}) - {"enabled" if p.enabled else "disabled"}')
if __name__ == '__main__':
parser = argparse.ArgumentParser(prog='esim.py', description='manage eSIM profiles on your comma device', epilog='comma.ai')
sub = parser.add_subparsers(dest='cmd')
sub.add_parser('list', help='list profiles')
p_switch = sub.add_parser('switch', help='switch to profile')
p_switch.add_argument('profile', help='iccid or 1-based index from `list`')
p_delete = sub.add_parser('delete', help='delete profile (warning: this cannot be undone)')
p_delete.add_argument('profile', help='iccid or 1-based index from `list`')
p_download = sub.add_parser('download', help='download a profile using QR code (format: LPA:1$rsp.truphone.com$QRF-SPEEDTEST)')
p_download.add_argument('qr')
p_download.add_argument('name')
p_nickname = sub.add_parser('nickname', help='update the nickname for a profile')
p_nickname.add_argument('profile', help='iccid or 1-based index from `list`')
p_nickname.add_argument('name')
args = parser.parse_args()
lpa = HARDWARE.get_sim_lpa()
if args.cmd == 'switch':
lpa.switch_profile(resolve_iccid(lpa, args.profile))
elif args.cmd == 'delete':
iccid = resolve_iccid(lpa, args.profile)
confirm = input(f'are you sure you want to delete profile {iccid}? (y/N) ')
if confirm == 'y':
lpa.delete_profile(iccid)
else:
print('cancelled')
exit(0)
elif args.cmd == 'download':
lpa.download_profile(args.qr, args.name)
elif args.cmd == 'nickname':
lpa.nickname_profile(resolve_iccid(lpa, args.profile), args.name)
else:
if args.cmd is None:
parser.print_help()
print_profiles(lpa)
-133
View File
@@ -1,133 +0,0 @@
# GSMA Certificate Issuer (CI) bundle for eSIM RSP
# Source: https://euicc-manual.osmocom.org/docs/pki/ci/bundle.pem
issuer=
countryName = CH
organizationName = OISTE Foundation
commonName = OISTE GSMA CI G1
notBefore=2024-01-16 23:17:39Z
notAfter=2059-01-07 23:17:38Z
-----BEGIN CERTIFICATE-----
MIIB9zCCAZ2gAwIBAgIUSpBSCCDYPOEG/IFHUCKpZ2pIAQMwCgYIKoZIzj0EAwIw
QzELMAkGA1UEBhMCQ0gxGTAXBgNVBAoMEE9JU1RFIEZvdW5kYXRpb24xGTAXBgNV
BAMMEE9JU1RFIEdTTUEgQ0kgRzEwIBcNMjQwMTE2MjMxNzM5WhgPMjA1OTAxMDcy
MzE3MzhaMEMxCzAJBgNVBAYTAkNIMRkwFwYDVQQKDBBPSVNURSBGb3VuZGF0aW9u
MRkwFwYDVQQDDBBPSVNURSBHU01BIENJIEcxMFkwEwYHKoZIzj0CAQYIKoZIzj0D
AQcDQgAEvZ3s3PFC4NgrCcCMmHJ6DJ66uzAHuLcvjJnOn+TtBNThS7YHLDyHCa2v
7D+zTP+XTtgqgcLoB56Gha9EQQQ4xKNtMGswDwYDVR0TAQH/BAUwAwEB/zAQBgNV
HREECTAHiAVghXQFDjAXBgNVHSABAf8EDTALMAkGB2eBEgECAQAwHQYDVR0OBBYE
FEwnlnrSDBSzkelgHkHmBK1XwCIvMA4GA1UdDwEB/wQEAwIBBjAKBggqhkjOPQQD
AgNIADBFAiBVcywTj017jKpAQ+gwy4MqK2hQvzve6lkvQkgSP6ykHwIhAI0KFwCD
jnPbmcJsG41hUrWNlf+IcrMvFuYii0DasBNi
-----END CERTIFICATE-----
issuer=
organizationName = GSM Association
commonName = GSM Association - RSP2 Root CI1
notBefore=2017-02-22 00:00:00Z
notAfter=2052-02-21 23:59:59Z
-----BEGIN CERTIFICATE-----
MIICSTCCAe+gAwIBAgIQbmhWeneg7nyF7hg5Y9+qejAKBggqhkjOPQQDAjBEMRgw
FgYDVQQKEw9HU00gQXNzb2NpYXRpb24xKDAmBgNVBAMTH0dTTSBBc3NvY2lhdGlv
biAtIFJTUDIgUm9vdCBDSTEwIBcNMTcwMjIyMDAwMDAwWhgPMjA1MjAyMjEyMzU5
NTlaMEQxGDAWBgNVBAoTD0dTTSBBc3NvY2lhdGlvbjEoMCYGA1UEAxMfR1NNIEFz
c29jaWF0aW9uIC0gUlNQMiBSb290IENJMTBZMBMGByqGSM49AgEGCCqGSM49AwEH
A0IABJ1qutL0HCMX52GJ6/jeibsAqZfULWj/X10p/Min6seZN+hf5llovbCNuB2n
unLz+O8UD0SUCBUVo8e6n9X1TuajgcAwgb0wDgYDVR0PAQH/BAQDAgEGMA8GA1Ud
EwEB/wQFMAMBAf8wEwYDVR0RBAwwCogIKwYBBAGC6WAwFwYDVR0gAQH/BA0wCzAJ
BgdngRIBAgEAME0GA1UdHwRGMEQwQqBAoD6GPGh0dHA6Ly9nc21hLWNybC5zeW1h
dXRoLmNvbS9vZmZsaW5lY2EvZ3NtYS1yc3AyLXJvb3QtY2kxLmNybDAdBgNVHQ4E
FgQUgTcPUSXQsdQI1MOyMubSXnlb6/swCgYIKoZIzj0EAwIDSAAwRQIgIJdYsOMF
WziPK7l8nh5mu0qiRiVf25oa9ullG/OIASwCIQDqCmDrYf+GziHXBOiwJwnBaeBO
aFsiLzIEOaUuZwdNUw==
-----END CERTIFICATE-----
issuer=
countryName = US
organizationName = Entrust, Inc.
organizationalUnitName = See www.entrust.net/legal-terms
organizationalUnitName = (c) 2016 Entrust, Inc. - for authorized use only
commonName = Entrust eSIM Certification Authority
notBefore=2016-11-16 16:04:02Z
notAfter=2051-10-16 16:34:02Z
-----BEGIN CERTIFICATE-----
MIIC6DCCAo2gAwIBAgIRAIy4GT7M5nHsAAAAAFgsinowCgYIKoZIzj0EAwIwgbkx
CzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1FbnRydXN0LCBJbmMuMSgwJgYDVQQLEx9T
ZWUgd3d3LmVudHJ1c3QubmV0L2xlZ2FsLXRlcm1zMTkwNwYDVQQLEzAoYykgMjAx
NiBFbnRydXN0LCBJbmMuIC0gZm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxLTArBgNV
BAMTJEVudHJ1c3QgZVNJTSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAgFw0xNjEx
MTYxNjA0MDJaGA8yMDUxMTAxNjE2MzQwMlowgbkxCzAJBgNVBAYTAlVTMRYwFAYD
VQQKEw1FbnRydXN0LCBJbmMuMSgwJgYDVQQLEx9TZWUgd3d3LmVudHJ1c3QubmV0
L2xlZ2FsLXRlcm1zMTkwNwYDVQQLEzAoYykgMjAxNiBFbnRydXN0LCBJbmMuIC0g
Zm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxLTArBgNVBAMTJEVudHJ1c3QgZVNJTSBD
ZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IA
BAdzwGHeQ1Wb2f4DmHTByR5/IWL3JugQ1U3908a++bHdlt+TTA7K4c5cYZ+51Yz/
hg/bacxguPDh9uQUK6Wg3a6jcjBwMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/
BAQDAgEGMBcGA1UdIAEB/wQNMAswCQYHZ4ESAQIBADAVBgNVHREEDjAMiApghkgB
hvpsFAoAMB0GA1UdDgQWBBQWcEt/NR42B/GMS3AAXDoAPf1BSjAKBggqhkjOPQQD
AgNJADBGAiEAspjXMvaBZyAg86Z0AAtT0yBRAi1EyaAfNz9kDJeAE04CIQC3efj8
ATL7/tDBOhANy3cK8PS/1NIlu9vqMLCZsZvJ0Q==
-----END CERTIFICATE-----
issuer=
countryName = FR
organizationName = OBERTHUR TECHNOLOGIES
organizationalUnitName = TELECOM
commonName = MC4 OT ROOT CI v1
notBefore=2016-11-15 00:00:01Z
notAfter=2046-11-08 23:59:59Z
-----BEGIN CERTIFICATE-----
MIICOjCCAeGgAwIBAgIBATAKBggqhkjOPQQDAjBbMQswCQYDVQQGEwJGUjEeMBwG
A1UEChMVT0JFUlRIVVIgVEVDSE5PTE9HSUVTMRAwDgYDVQQLEwdURUxFQ09NMRow
GAYDVQQDExFNQzQgT1QgUk9PVCBDSSB2MTAeFw0xNjExMTUwMDAwMDFaFw00NjEx
MDgyMzU5NTlaMFsxCzAJBgNVBAYTAkZSMR4wHAYDVQQKExVPQkVSVEhVUiBURUNI
Tk9MT0dJRVMxEDAOBgNVBAsTB1RFTEVDT00xGjAYBgNVBAMTEU1DNCBPVCBST09U
IENJIHYxMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEHb/Gajt3OZxuaDSklBQE
D4lOd6PGPLSvtfkM952ubdyy45tJwAeA0eEii0CLrFT6tcfXkW+H/5mQyMRXaAUk
T6OBlTCBkjAfBgNVHSMEGDAWgBTNbmC3LXoGPLyEYluR6A/jBAbhPjAdBgNVHQ4E
FgQUzW5gty16Bjy8hGJbkegP4wQG4T4wDgYDVR0PAQH/BAQDAgAGMBcGA1UdIAEB
/wQNMAswCQYHZ4ESAQIBADAWBgNVHREEDzANiAsrBgEEAYHvb7OITTAPBgNVHRMB
Af8EBTADAQH/MAoGCCqGSM49BAMCA0cAMEQCIEw4Nc7f2fDtoH+6ON/bknfDQxmT
ikThXjhpLtSrSKN2AiAxHxgC87L0FDnH8dJNlkdGX9c0JIx6oLheIplfS6k+jg==
-----END CERTIFICATE-----
issuer=
commonName = SubMan V4.2 CI Google Pixel
organizationName = Giesecke and Devrient GmbH
organizationalUnitName = Mobile Security
countryName = DE
notBefore=2017-05-10 00:00:00Z
notAfter=2027-05-10 00:00:00Z
-----BEGIN CERTIFICATE-----
MIICaTCCAg6gAwIBAgICASwwCgYIKoZIzj0EAwIwczElMCMGA1UEAxMcIFN1Yk1h
biBWNC4yIENJIEdvb2dsZSBQaXhlbDEjMCEGA1UEChMaR2llc2Vja2UgYW5kIERl
dnJpZW50IEdtYkgxGDAWBgNVBAsTD01vYmlsZSBTZWN1cml0eTELMAkGA1UEBhMC
REUwHhcNMTcwNTEwMDAwMDAwWhcNMjcwNTEwMDAwMDAwWjBzMSUwIwYDVQQDExwg
U3ViTWFuIFY0LjIgQ0kgR29vZ2xlIFBpeGVsMSMwIQYDVQQKExpHaWVzZWNrZSBh
bmQgRGV2cmllbnQgR21iSDEYMBYGA1UECxMPTW9iaWxlIFNlY3VyaXR5MQswCQYD
VQQGEwJERTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABHNorfaJsGzqWNawyAhl
IAv9QL2/+b9RsUoso06t/dKX1MRr5CUJ51acvv5TAFhQKIml+dwLbFnV5aO+8W6Z
wxajgZEwgY4wHwYDVR0jBBgwFoAUtg8LiX/WMLiM/tYWH46oCMU4KsMwHQYDVR0O
BBYEFLYPC4l/1jC4jP7WFh+OqAjFOCrDMA4GA1UdDwEB/wQEAwIBBjAXBgNVHSAB
Af8EDTALMAkGB2eBEgECAQAwDwYDVR0TAQH/BAUwAwEB/zASBgNVHREECzAJiAcr
BgEEAdwPMAoGCCqGSM49BAMCA0kAMEYCIQDpoZcuAQrjATW8U+AWqMUJ0dY6nWW1
R1QmFzVZ1yMXSwIhALCvRqkCtgiavdeFeSgsSNbY5Fhd+QoCltuSh1U4TE7A
-----END CERTIFICATE-----
issuer=
countryName = DE
commonName = SubMan V4.2 CI
organizationName = Giesecke and Devrient
organizationalUnitName = Mobile Security
notBefore=2016-08-12 13:51:48Z
notAfter=2026-08-12 13:51:48Z
-----BEGIN CERTIFICATE-----
MIICUjCCAfigAwIBAgIDQgAAMAoGCCqGSM49BAMCMGAxCzAJBgNVBAYTAkRFMRcw
FQYDVQQDEw5TdWJNYW4gVjQuMiBDSTEeMBwGA1UEChMVR2llc2Vja2UgYW5kIERl
dnJpZW50MRgwFgYDVQQLEw9Nb2JpbGUgU2VjdXJpdHkwHhcNMTYwODEyMTM1MTQ4
WhcNMjYwODEyMTM1MTQ4WjBgMQswCQYDVQQGEwJERTEXMBUGA1UEAxMOU3ViTWFu
IFY0LjIgQ0kxHjAcBgNVBAoTFUdpZXNlY2tlIGFuZCBEZXZyaWVudDEYMBYGA1UE
CxMPTW9iaWxlIFNlY3VyaXR5MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEYIgl
VQr9wbXOlwPp8qMg5Df08Cli9Mc+lpr3Lwa9PlVA3QWlLeX4GfD4H3phLBqVIa17
yHttmtheTxi0KoEqhKOBoDCBnTAdBgNVHQ4EFgQU6lOt7zMpuVCa/XVf1Ei4LcG8
7P8wDgYDVR0PAQH/BAQDAgEGMBcGA1UdIAEB/wQNMAswCQYHZ4ESAQIBADAPBgNV
HRMBAf8EBTADAQH/MBIGA1UdEQQLMAmIBysGAQQB3A8wLgYDVR0fBCcwJTAjoCGg
H4YdaHR0cDovL2dpLWRlLmNvbS90ZXN0LmNybC5wZW0wCgYIKoZIzj0EAwIDSAAw
RQIhAMMx2L/VHDiOW+Fl/OuFmhCdizYM17Yn9zAVieKO2T0iAiANWtCMmY+DzkqK
yHxBFX0U2tBd682zP4DpgRt8j3Ylew==
-----END CERTIFICATE-----
-791
View File
@@ -1,791 +0,0 @@
# SGP.22 v2.3: https://www.gsma.com/solutions-and-impact/technologies/esim/wp-content/uploads/2021/07/SGP.22-v2.3.pdf
import atexit
import base64
import fcntl
import hashlib
import os
import requests
import serial
import subprocess
import sys
import termios
import time
from collections.abc import Callable, Generator
from contextlib import contextmanager
from typing import Any
from pathlib import Path
from openpilot.common.time_helpers import system_time_valid
from openpilot.common.esim.base import LPABase, LPAError, LPAProfileNotFoundError, Profile
GSMA_CI_BUNDLE = str(Path(__file__).parent / "gsma_ci_bundle.pem")
DEFAULT_DEVICE = "/dev/modem_at0"
DEFAULT_BAUD = 9600
DEFAULT_TIMEOUT = 5.0
# https://euicc-manual.osmocom.org/docs/lpa/applet-id/
ISDR_AID = "A0000005591010FFFFFFFF8900000100"
ES10X_MSS = 120
HTTP_TIMEOUT = 30
OPEN_ISDR_RETRIES = 10
OPEN_ISDR_RETRY_DELAY_S = 0.25
OPEN_ISDR_RESET_ATTEMPT = 5
SEND_APDU_RETRIES = 3
LOCK_FILE = '/dev/shm/modem.lock'
DEBUG = os.environ.get("DEBUG") == "1"
# TLV Tags
TAG_ICCID = 0x5A
TAG_STATUS = 0x80
TAG_EUICC_INFO = 0xBF20
TAG_PREPARE_DOWNLOAD = 0xBF21
TAG_BPP_COMMAND = 0xBF23
TAG_PROFILE_METADATA = 0xBF25
TAG_INSTALL_RESULT_DATA = 0xBF27
TAG_LIST_NOTIFICATION = 0xBF28
TAG_SET_NICKNAME = 0xBF29
TAG_RETRIEVE_NOTIFICATION = 0xBF2B
TAG_PROFILE_INFO_LIST = 0xBF2D
TAG_EUICC_CHALLENGE = 0xBF2E
TAG_NOTIFICATION_METADATA = 0xBF2F
TAG_NOTIFICATION_SENT = 0xBF30
TAG_ENABLE_PROFILE = 0xBF31
TAG_DELETE_PROFILE = 0xBF33
TAG_BPP = 0xBF36
TAG_PROFILE_INSTALL_RESULT = 0xBF37
TAG_AUTH_SERVER = 0xBF38
TAG_CANCEL_SESSION = 0xBF41
TAG_OK = 0xA0
PROFILE_OK = 0x00
PROFILE_NOT_IN_DISABLED_STATE = 0x02
PROFILE_CAT_BUSY = 0x05
PROFILE_ERROR_CODES = {
0x01: "iccidOrAidNotFound", PROFILE_NOT_IN_DISABLED_STATE: "profileNotInDisabledState",
0x03: "disallowedByPolicy", 0x04: "wrongProfileReenabling",
PROFILE_CAT_BUSY: "catBusy", 0x06: "undefinedError",
}
AUTH_SERVER_ERROR_CODES = {
0x01: "eUICCVerificationFailed", 0x02: "eUICCCertificateExpired",
0x03: "eUICCCertificateRevoked", 0x05: "invalidServerSignature",
0x06: "euiccCiPKUnknown", 0x0A: "matchingIdRefused",
0x10: "insufficientMemory",
}
BPP_COMMAND_NAMES = {
0: "initialiseSecureChannel", 1: "configureISDP", 2: "storeMetadata",
3: "storeMetadata2", 4: "replaceSessionKeys", 5: "loadProfileElements",
}
BPP_ERROR_REASONS = {
1: "incorrectInputValues", 2: "invalidSignature", 3: "invalidTransactionId",
4: "unsupportedCrtValues", 5: "unsupportedRemoteOperationType",
6: "unsupportedProfileClass", 7: "scp03tStructureError", 8: "scp03tSecurityError",
9: "iccidAlreadyExistsOnEuicc", 10: "insufficientMemoryForProfile",
11: "installInterrupted", 12: "peProcessingError", 13: "dataMismatch",
14: "invalidNAA",
}
BPP_ERROR_MESSAGES = {
9: "This eSIM profile is already installed on this device.",
10: "Not enough memory on the eUICC to install this profile.",
12: "Profile installation failed. The QR code may have already been used.",
}
# SGP.22 §5.2.6 SM-DP+ reason/subject codes mapped to user-friendly messages
ES9P_ERROR_MESSAGES: dict[tuple[str, str], str] = {
('3.8', '8.2.6'): "This eSIM profile is already installed on another device. Please use a new QR code.",
('3.8', '8.2.1'): "This eSIM profile has expired. Please request a new QR code.",
('3.8', '8.1'): "The SM-DP+ server refused this request.",
('3.1', '8.2.6'): "This eSIM profile has been revoked by the carrier.",
('3.9', '8.2.6'): "This eSIM profile download has already been completed.",
('2.1', '8.8'): "The device is not compatible with this eSIM profile.",
('1.2', '8.1'): "The SM-DP+ server is temporarily unavailable. Try again later.",
}
NOTIFICATION_OPERATIONS = {0x80: "install", 0x40: "enable", 0x20: "disable", 0x10: "delete"}
STATE_LABELS = {0: "disabled", 1: "enabled", 255: "unknown"}
ICON_LABELS = {0: "jpeg", 1: "png", 255: "unknown"}
CLASS_LABELS = {0: "test", 1: "provisioning", 2: "operational", 255: "unknown"}
# TLV tag -> (field_name, decoder)
FieldMap = dict[int, tuple[str, Callable[[bytes], Any]]]
def b64e(data: bytes) -> str:
return base64.b64encode(data).decode("ascii")
def base64_trim(s: str) -> str:
return "".join(c for c in s if c not in "\n\r \t")
def b64d(s: str) -> bytes:
return base64.b64decode(base64_trim(s))
class AtClient:
def __init__(self, device: str, baud: int, timeout: float) -> None:
self.channel: str | None = None
self._device = device
self._baud = baud
self._timeout = timeout
self._serial: serial.Serial | None = None
def send_raw(self, data: bytes) -> None:
self._ensure_serial()
self._serial.reset_input_buffer()
self._serial.write(data)
self._serial.flush()
def close(self) -> None:
try:
if self.channel:
try:
self.query(f"AT+CCHC={self.channel}")
except (RuntimeError, TimeoutError):
pass
self.channel = None
finally:
if self._serial:
self._serial.close()
def _send(self, cmd: str) -> None:
if DEBUG:
print(f"SER >> {cmd}", file=sys.stderr)
self._serial.write((cmd + "\r").encode("ascii"))
def _expect(self) -> list[str]:
lines: list[str] = []
while True:
raw = self._serial.readline()
if not raw:
raise TimeoutError("AT command timed out")
line = raw.decode(errors="ignore").strip()
if not line:
continue
if DEBUG:
print(f"SER << {line}", file=sys.stderr)
if line == "OK":
return lines
if line == "ERROR" or line.startswith("+CME ERROR"):
raise RuntimeError(f"AT command failed: {line}")
lines.append(line)
def _ensure_serial(self, reconnect: bool = False) -> None:
if reconnect:
self.channel = None
try:
if self._serial:
self._serial.close()
except Exception:
pass
self._serial = None
if self._serial is None:
self._serial = serial.Serial(self._device, baudrate=self._baud, timeout=self._timeout)
def query(self, cmd: str) -> list[str]:
self._ensure_serial()
try:
self._send(cmd)
return self._expect()
except serial.SerialException:
self._ensure_serial(reconnect=True)
self._send(cmd)
return self._expect()
def _open_isdr_once(self) -> None:
if self.channel:
try:
self.query(f"AT+CCHC={self.channel}")
except RuntimeError:
pass
self.channel = None
# drain any unsolicited responses before opening
if self._serial:
try:
self._serial.reset_input_buffer()
except (OSError, serial.SerialException, termios.error):
self._ensure_serial(reconnect=True)
for line in self.query(f'AT+CCHO="{ISDR_AID}"'):
if line.startswith("+CCHO:") and (ch := line.split(":", 1)[1].strip()):
self.channel = ch
return
raise RuntimeError("Failed to open ISD-R application")
def _reset_modem(self) -> None:
if self._serial:
try:
self._serial.close()
except Exception:
pass
self._serial = None
subprocess.run(['/usr/comma/lte/lte.sh', 'start'], capture_output=True)
def open_isdr(self) -> None:
for attempt in range(OPEN_ISDR_RETRIES):
try:
self._open_isdr_once()
return
except (RuntimeError, TimeoutError, termios.error, serial.SerialException):
time.sleep(OPEN_ISDR_RETRY_DELAY_S)
if attempt == OPEN_ISDR_RESET_ATTEMPT:
self._reset_modem()
raise RuntimeError("Failed to open ISD-R after retries")
def send_apdu(self, apdu: bytes) -> tuple[bytes, int, int]:
for attempt in range(SEND_APDU_RETRIES):
try:
if not self.channel:
self.open_isdr()
hex_payload = apdu.hex().upper()
for line in self.query(f'AT+CGLA={self.channel},{len(hex_payload)},"{hex_payload}"'):
if line.startswith("+CGLA:"):
parts = line.split(":", 1)[1].split(",", 1)
if len(parts) == 2:
data = bytes.fromhex(parts[1].strip().strip('"'))
if len(data) >= 2:
return data[:-2], data[-2], data[-1]
raise RuntimeError("Missing +CGLA response")
except (RuntimeError, ValueError):
self.channel = None
if attempt == SEND_APDU_RETRIES - 1:
raise
raise RuntimeError("send_apdu failed")
# --- TLV utilities ---
def iter_tlv(data: bytes, with_positions: bool = False) -> Generator:
idx, length = 0, len(data)
while idx < length:
start_pos = idx
tag = data[idx]
idx += 1
if tag & 0x1F == 0x1F: # Multi-byte tag
tag_value = tag
while idx < length:
next_byte = data[idx]
idx += 1
tag_value = (tag_value << 8) | next_byte
if not (next_byte & 0x80):
break
else:
tag_value = tag
if idx >= length:
break
size = data[idx]
idx += 1
if size & 0x80: # Multi-byte length
num_bytes = size & 0x7F
if idx + num_bytes > length:
break
size = int.from_bytes(data[idx : idx + num_bytes], "big")
idx += num_bytes
if idx + size > length:
break
value = data[idx : idx + size]
idx += size
yield (tag_value, value, start_pos, idx) if with_positions else (tag_value, value)
def find_tag(data: bytes, target: int) -> bytes | None:
return next((v for t, v in iter_tlv(data) if t == target), None)
def require_tag(data: bytes, target: int, label: str = "") -> bytes:
v = find_tag(data, target)
if v is None:
raise RuntimeError(f"Missing {label or f'tag 0x{target:X}'}")
return v
def tbcd_to_string(raw: bytes) -> str:
return "".join(str(n) for b in raw for n in (b & 0x0F, b >> 4) if n <= 9)
def string_to_tbcd(s: str) -> bytes:
digits = [int(c) for c in s if c.isdigit()]
return bytes(digits[i] | ((digits[i + 1] if i + 1 < len(digits) else 0xF) << 4) for i in range(0, len(digits), 2))
def encode_tlv(tag: int, value: bytes) -> bytes:
tag_bytes = bytes([(tag >> 8) & 0xFF, tag & 0xFF]) if tag > 255 else bytes([tag])
vlen = len(value)
if vlen <= 127:
return tag_bytes + bytes([vlen]) + value
length_bytes = vlen.to_bytes((vlen.bit_length() + 7) // 8, "big")
return tag_bytes + bytes([0x80 | len(length_bytes)]) + length_bytes + value
def int_bytes(n: int) -> bytes:
"""Encode a positive integer as minimal big-endian bytes (at least 1 byte)."""
return n.to_bytes((n.bit_length() + 7) // 8 or 1, "big")
PROFILE: FieldMap = {
TAG_ICCID: ("iccid", tbcd_to_string),
0x4F: ("isdpAid", lambda v: v.hex().upper()),
0x9F70: ("profileState", lambda v: STATE_LABELS.get(v[0], "unknown")),
0x90: ("profileNickname", lambda v: v.decode("utf-8", errors="ignore") or None),
0x91: ("serviceProviderName", lambda v: v.decode("utf-8", errors="ignore") or None),
0x92: ("profileName", lambda v: v.decode("utf-8", errors="ignore") or None),
0x93: ("iconType", lambda v: ICON_LABELS.get(v[0], "unknown")),
0x94: ("icon", b64e),
0x95: ("profileClass", lambda v: CLASS_LABELS.get(v[0], "unknown")),
}
def decode_struct(data: bytes, field_map: FieldMap) -> dict[str, Any]:
"""Parse TLV data using a {tag: (field_name, decoder)} map into a dict."""
result: dict[str, Any] = {name: None for name, _ in field_map.values()}
for tag, value in iter_tlv(data):
if (field := field_map.get(tag)):
result[field[0]] = field[1](value)
return result
# --- ES10x command transport ---
def es10x_command(client: AtClient, data: bytes) -> bytes:
response = bytearray()
sequence = 0
offset = 0
while offset < len(data):
chunk = data[offset : offset + ES10X_MSS]
offset += len(chunk)
is_last = offset == len(data)
apdu = bytes([0x80, 0xE2, 0x91 if is_last else 0x11, sequence & 0xFF, len(chunk)]) + chunk
segment, sw1, sw2 = client.send_apdu(apdu)
response.extend(segment)
while True:
if sw1 == 0x61: # More data available
segment, sw1, sw2 = client.send_apdu(bytes([0x80, 0xC0, 0x00, 0x00, sw2 or 0]))
response.extend(segment)
continue
if (sw1 & 0xF0) == 0x90:
break
raise RuntimeError(f"APDU failed with SW={sw1:02X}{sw2:02X}")
sequence += 1
return bytes(response)
# --- Profile operations ---
NOTIFICATION: FieldMap = {
TAG_STATUS: ("seqNumber", lambda v: int.from_bytes(v, "big")),
0x81: ("profileManagementOperation",
lambda v: NOTIFICATION_OPERATIONS.get(next((m for m in NOTIFICATION_OPERATIONS if len(v) >= 2 and v[1] & m), 0), "unknown")),
0x0C: ("notificationAddress", lambda v: v.decode("utf-8", errors="ignore")),
TAG_ICCID: ("iccid", tbcd_to_string),
}
def decode_profiles(blob: bytes) -> list[dict]:
root = require_tag(blob, TAG_PROFILE_INFO_LIST, "ProfileInfoList")
list_ok = find_tag(root, TAG_OK)
if list_ok is None:
return []
return [decode_struct(value, PROFILE) for tag, value in iter_tlv(list_ok) if tag == 0xE3]
def list_profiles(client: AtClient) -> list[dict]:
return decode_profiles(es10x_command(client, TAG_PROFILE_INFO_LIST.to_bytes(2, "big") + b"\x00"))
def set_profile_nickname(client: AtClient, iccid: str, nickname: str) -> None:
nickname_bytes = nickname.encode("utf-8")
if len(nickname_bytes) > 64:
raise ValueError("Profile nickname must be 64 bytes or less")
content = encode_tlv(TAG_ICCID, string_to_tbcd(iccid)) + encode_tlv(0x90, nickname_bytes)
response = es10x_command(client, encode_tlv(TAG_SET_NICKNAME, content))
code = require_tag(require_tag(response, TAG_SET_NICKNAME, "SetNicknameResponse"), TAG_STATUS, "SetNickname status")[0]
if code == 0x01:
raise LPAError(f"profile {iccid} not found")
if code != 0x00:
raise RuntimeError(f"SetNickname failed with status 0x{code:02X}")
# --- ES9P HTTP ---
def es9p_request(smdp_address: str, endpoint: str, payload: dict, error_prefix: str = "Request", session: requests.Session | None = None) -> dict:
url = f"https://{smdp_address}/gsma/rsp2/es9plus/{endpoint}"
headers = {"User-Agent": "gsma-rsp-lpad", "X-Admin-Protocol": "gsma/rsp/v2.3.0", "Content-Type": "application/json"}
http = session or requests
resp = http.post(url, json=payload, headers=headers, timeout=HTTP_TIMEOUT, verify=GSMA_CI_BUNDLE)
resp.raise_for_status()
if not resp.content:
return {}
data = resp.json()
if "header" in data and "functionExecutionStatus" in data["header"]:
status = data["header"]["functionExecutionStatus"]
if status.get("status") == "Failed":
sd = status.get("statusCodeData", {})
reason = sd.get("reasonCode", "unknown")
subject = sd.get("subjectCode", "unknown")
msg = ES9P_ERROR_MESSAGES.get((reason, subject),
f"{error_prefix} failed: {reason}/{subject} - {sd.get('message', 'unknown')}")
raise RuntimeError(msg)
return data
# --- Notifications ---
def list_notifications(client: AtClient) -> list[dict]:
response = es10x_command(client, encode_tlv(TAG_LIST_NOTIFICATION, b""))
root = require_tag(response, TAG_LIST_NOTIFICATION, "ListNotificationResponse")
metadata_list = find_tag(root, TAG_OK)
if metadata_list is None:
return []
return [decode_struct(value, NOTIFICATION) for tag, value in iter_tlv(metadata_list) if tag == TAG_NOTIFICATION_METADATA]
def process_notifications(client: AtClient) -> None:
for notification in list_notifications(client):
seq_number, smdp_address = notification["seqNumber"], notification["notificationAddress"]
try:
request = encode_tlv(TAG_RETRIEVE_NOTIFICATION, encode_tlv(TAG_OK, encode_tlv(TAG_STATUS, int_bytes(seq_number))))
response = es10x_command(client, request)
content = require_tag(require_tag(response, TAG_RETRIEVE_NOTIFICATION, "RetrieveNotificationsListResponse"),
TAG_OK, "RetrieveNotificationsListResponse")
pending_notif = next((v for t, v in iter_tlv(content) if t in (TAG_PROFILE_INSTALL_RESULT, 0x30)), None)
if pending_notif is None:
raise RuntimeError("Missing PendingNotification")
es9p_request(smdp_address, "handleNotification", {"pendingNotification": b64e(pending_notif)}, "HandleNotification")
response = es10x_command(client, encode_tlv(TAG_NOTIFICATION_SENT, encode_tlv(TAG_STATUS, int_bytes(seq_number))))
root = require_tag(response, TAG_NOTIFICATION_SENT, "NotificationSentResponse")
if int.from_bytes(require_tag(root, TAG_STATUS, "RemoveNotificationFromList status"), "big") != 0:
raise RuntimeError("RemoveNotificationFromList failed")
except Exception as e:
print(f"notification {seq_number} failed: {e}", file=sys.stderr)
# --- Authentication & Download ---
def get_challenge_and_info(client: AtClient) -> tuple[bytes, bytes]:
challenge_resp = es10x_command(client, encode_tlv(TAG_EUICC_CHALLENGE, b""))
challenge = require_tag(require_tag(challenge_resp, TAG_EUICC_CHALLENGE, "GetEuiccDataResponse"),
TAG_STATUS, "challenge in response")
info_resp = es10x_command(client, encode_tlv(TAG_EUICC_INFO, b""))
require_tag(info_resp, TAG_EUICC_INFO, "GetEuiccInfo1Response")
return challenge, info_resp
def authenticate_server(client: AtClient, b64_signed1: str, b64_sig1: str, b64_pk_id: str, b64_cert: str, matching_id: str) -> str:
tac = bytes([0x35, 0x29, 0x06, 0x11])
device_info = encode_tlv(TAG_STATUS, tac) + encode_tlv(0xA1, b"")
ctx_inner = encode_tlv(TAG_STATUS, matching_id.encode("utf-8")) + encode_tlv(0xA1, device_info)
content = b64d(b64_signed1) + b64d(b64_sig1) + b64d(b64_pk_id) + b64d(b64_cert) + encode_tlv(0xA0, ctx_inner)
response = es10x_command(client, encode_tlv(TAG_AUTH_SERVER, content))
root = require_tag(response, TAG_AUTH_SERVER, "AuthenticateServerResponse")
error_tag = find_tag(root, 0xA1)
if error_tag is not None:
code = int.from_bytes(error_tag, "big") if error_tag else 0
raise RuntimeError(f"AuthenticateServer rejected by eUICC: {AUTH_SERVER_ERROR_CODES.get(code, 'unknown')} (0x{code:02X})")
return b64e(response)
def prepare_download(client: AtClient, b64_signed2: str, b64_sig2: str, b64_cert: str, cc: str | None = None) -> str:
smdp_signed2 = b64d(b64_signed2)
smdp_signature2 = b64d(b64_sig2)
smdp_certificate = b64d(b64_cert)
smdp_signed2_root = find_tag(smdp_signed2, 0x30)
if smdp_signed2_root is None:
raise RuntimeError("Invalid smdpSigned2")
transaction_id = find_tag(smdp_signed2_root, TAG_STATUS)
cc_required_flag = find_tag(smdp_signed2_root, 0x01)
if transaction_id is None or cc_required_flag is None:
raise RuntimeError("Invalid smdpSigned2")
content = smdp_signed2 + smdp_signature2
if int.from_bytes(cc_required_flag, "big") != 0:
if not cc:
raise RuntimeError("Confirmation code required but not provided")
content += encode_tlv(0x04, hashlib.sha256(hashlib.sha256(cc.encode("utf-8")).digest() + transaction_id).digest())
content += smdp_certificate
response = es10x_command(client, encode_tlv(TAG_PREPARE_DOWNLOAD, content))
require_tag(response, TAG_PREPARE_DOWNLOAD, "PrepareDownloadResponse")
return b64e(response)
def _parse_tlv_header_len(data: bytes) -> int:
tag_len = 2 if data[0] & 0x1F == 0x1F else 1
length_byte = data[tag_len]
return tag_len + (1 + (length_byte & 0x7F) if length_byte & 0x80 else 1)
def _split_bpp(bpp: bytes) -> list[bytes]:
"""Split a BoundProfilePackage into APDU chunks per SGP.22 §5.7.6."""
root_value = None
for tag, value, start, end in iter_tlv(bpp, with_positions=True):
if tag == TAG_BPP:
root_value = value
val_start = start + _parse_tlv_header_len(bpp[start:end])
break
if root_value is None:
raise RuntimeError("Invalid BoundProfilePackage")
chunks: list[bytes] = []
for tag, value, start, end in iter_tlv(root_value, with_positions=True):
if tag == TAG_BPP_COMMAND:
chunks.append(bpp[0 : val_start + end])
elif tag in (0xA0, 0xA2):
chunks.append(bpp[val_start + start : val_start + end])
elif tag in (0xA1, 0xA3):
hdr_len = _parse_tlv_header_len(root_value[start:end])
chunks.append(bpp[val_start + start : val_start + start + hdr_len])
for _, _, cs, ce in iter_tlv(value, with_positions=True):
chunks.append(value[cs:ce])
return chunks
def _parse_install_result(response: bytes) -> dict[str, Any] | None:
"""Parse a ProfileInstallResult from an APDU response, or None if not present."""
root = find_tag(response, TAG_PROFILE_INSTALL_RESULT)
if not root:
return None
result_data = find_tag(root, TAG_INSTALL_RESULT_DATA)
if not result_data:
return None
result: dict[str, Any] = {"seqNumber": 0, "success": False, "bppCommandId": None, "errorReason": None}
notif_meta = find_tag(result_data, TAG_NOTIFICATION_METADATA)
if notif_meta:
seq_num = find_tag(notif_meta, TAG_STATUS)
if seq_num:
result["seqNumber"] = int.from_bytes(seq_num, "big")
final_result = find_tag(result_data, 0xA2)
if final_result:
for tag, value in iter_tlv(final_result):
if tag == 0xA0:
result["success"] = True
elif tag == 0xA1:
bpp_cmd = find_tag(value, TAG_STATUS)
if bpp_cmd:
result["bppCommandId"] = int.from_bytes(bpp_cmd, "big")
err = find_tag(value, 0x81)
if err:
result["errorReason"] = int.from_bytes(err, "big")
return result
def load_bpp(client: AtClient, b64_bpp: str) -> dict:
bpp = b64d(b64_bpp)
result = None
for chunk in _split_bpp(bpp):
response = es10x_command(client, chunk)
if response and (parsed := _parse_install_result(response)):
result = parsed
break
if result is None:
raise RuntimeError("Profile installation failed: no result from eUICC")
if not result["success"] and result["errorReason"] is not None:
msg = BPP_ERROR_MESSAGES.get(result["errorReason"])
if not msg:
cmd_name = BPP_COMMAND_NAMES.get(result["bppCommandId"], f"unknown({result['bppCommandId']})")
err_name = BPP_ERROR_REASONS.get(result["errorReason"], f"unknown({result['errorReason']})")
msg = f"Profile installation failed at {cmd_name}: {err_name}"
raise RuntimeError(msg)
if not result["success"]:
raise RuntimeError("Profile installation failed: no result from eUICC")
return result
def parse_metadata(b64_metadata: str) -> dict:
root = find_tag(b64d(b64_metadata), TAG_PROFILE_METADATA)
if root is None:
raise RuntimeError("Invalid profileMetadata")
return decode_struct(root, PROFILE)
def cancel_session(client: AtClient, transaction_id: bytes, reason: int = 127) -> str:
content = encode_tlv(0x80, transaction_id) + encode_tlv(0x81, bytes([reason]))
response = es10x_command(client, encode_tlv(TAG_CANCEL_SESSION, content))
return b64e(response)
def parse_lpa_activation_code(activation_code: str) -> tuple[str, str]:
"""Parse 'LPA:1$smdp.example.com$MATCHING-ID' into (smdp_address, matching_id)."""
if not activation_code.startswith("LPA:"):
raise ValueError("Invalid activation code format")
parts = activation_code[4:].split("$")
if len(parts) != 3:
raise ValueError("Invalid activation code format")
return parts[1], parts[2]
def _b64_field(data: dict, key: str) -> str:
return base64_trim(data[key])
def _cancel_session_safe(client: AtClient, smdp: str, tx_id: str, session: requests.Session) -> None:
b64_cancel = ""
try:
b64_cancel = cancel_session(client, b64d(tx_id))
except Exception:
pass
try:
es9p_request(smdp, "cancelSession", {"transactionId": tx_id, "cancelSessionResponse": b64_cancel}, "CancelSession", session=session)
except Exception:
pass
def download_profile(client: AtClient, activation_code: str) -> str:
"""Download and install an eSIM profile. Returns the ICCID of the installed profile."""
if not system_time_valid():
raise RuntimeError("System time is not set; TLS certificate validation requires a valid clock")
smdp, matching_id = parse_lpa_activation_code(activation_code)
challenge, euicc_info = get_challenge_and_info(client)
session = requests.Session()
tx_id = None
try:
# step 1: initiate authentication
auth = es9p_request(smdp, "initiateAuthentication", {
"smdpAddress": smdp, "euiccChallenge": b64e(challenge),
"euiccInfo1": b64e(euicc_info), "matchingId": matching_id,
}, "Authentication", session=session)
tx_id = _b64_field(auth, "transactionId")
# step 2: authenticate server
b64_auth = authenticate_server(client,
_b64_field(auth, "serverSigned1"), _b64_field(auth, "serverSignature1"),
_b64_field(auth, "euiccCiPKIdToBeUsed"), _b64_field(auth, "serverCertificate"),
matching_id)
# step 3: authenticate client + get metadata
cli = es9p_request(smdp, "authenticateClient", {
"transactionId": tx_id, "authenticateServerResponse": b64_auth,
}, "Authentication", session=session)
iccid = parse_metadata(_b64_field(cli, "profileMetadata"))["iccid"]
# step 4: prepare download
b64_prep = prepare_download(client,
_b64_field(cli, "smdpSigned2"), _b64_field(cli, "smdpSignature2"),
_b64_field(cli, "smdpCertificate"))
# step 5: get and install bound profile package
bpp = es9p_request(smdp, "getBoundProfilePackage", {
"transactionId": tx_id, "prepareDownloadResponse": b64_prep,
}, "GetBoundProfilePackage", session=session)
load_bpp(client, _b64_field(bpp, "boundProfilePackage"))
return iccid
except Exception:
if tx_id:
_cancel_session_safe(client, smdp, tx_id, session)
raise
finally:
session.close()
class TiciLPA(LPABase):
def __init__(self):
if hasattr(self, '_client'):
return
self._client = AtClient(DEFAULT_DEVICE, DEFAULT_BAUD, DEFAULT_TIMEOUT)
atexit.register(self._client.close)
@contextmanager
def _acquire_lock(self):
fd = os.open(LOCK_FILE, os.O_CREAT | os.O_RDWR)
try:
fcntl.flock(fd, fcntl.LOCK_EX)
yield
finally:
fcntl.flock(fd, fcntl.LOCK_UN)
os.close(fd)
@contextmanager
def _acquire_channel(self):
with self._acquire_lock():
try:
self._client.open_isdr()
yield
finally:
if self._client.channel:
try:
self._client.query(f"AT+CCHC={self._client.channel}")
except (RuntimeError, TimeoutError):
pass
self._client.channel = None
def list_profiles(self) -> list[Profile]:
with self._acquire_channel():
return [
Profile(
iccid=p.get("iccid", ""),
nickname=p.get("profileNickname") or "",
enabled=p.get("profileState") == "enabled",
provider=p.get("serviceProviderName") or "",
)
for p in list_profiles(self._client)
]
def get_active_profile(self) -> Profile | None:
return None
def process_notifications(self) -> None:
if not system_time_valid():
raise RuntimeError("System time is not set; TLS certificate validation requires a valid clock")
with self._acquire_channel():
process_notifications(self._client)
def delete_profile(self, iccid: str) -> None:
profile = next((p for p in self.list_profiles() if p.iccid == iccid), None)
if profile is None:
raise LPAProfileNotFoundError(f"profile not found: {iccid}")
if profile.is_comma:
raise LPAError("refusing to delete a comma profile")
with self._acquire_channel():
request = encode_tlv(TAG_DELETE_PROFILE, encode_tlv(TAG_ICCID, string_to_tbcd(iccid)))
response = es10x_command(self._client, request)
code = require_tag(require_tag(response, TAG_DELETE_PROFILE, "DeleteProfileResponse"), TAG_STATUS, "DeleteProfile status")[0]
if code != PROFILE_OK:
raise LPAError(f"DeleteProfile failed: {PROFILE_ERROR_CODES.get(code, 'unknown')} (0x{code:02X})")
def download_profile(self, qr: str, nickname: str | None = None) -> None:
with self._acquire_channel():
iccid = download_profile(self._client, qr)
if nickname and iccid:
set_profile_nickname(self._client, iccid, nickname)
def nickname_profile(self, iccid: str, nickname: str) -> None:
with self._acquire_channel():
set_profile_nickname(self._client, iccid, nickname)
def _enable_profile(self, iccid: str) -> int:
inner = encode_tlv(TAG_OK, encode_tlv(TAG_ICCID, string_to_tbcd(iccid)))
inner += b'\x01\x01\x01' # refreshFlag=1
response = es10x_command(self._client, encode_tlv(TAG_ENABLE_PROFILE, inner))
return require_tag(require_tag(response, TAG_ENABLE_PROFILE, "EnableProfileResponse"), TAG_STATUS, "EnableProfile status")[0]
def switch_profile(self, iccid: str) -> None:
with self._acquire_channel():
code = self._enable_profile(iccid)
if code == PROFILE_CAT_BUSY: # stale eUICC transaction, reset and retry
self._client._reset_modem()
self._client.open_isdr()
code = self._enable_profile(iccid)
if code not in (PROFILE_OK, PROFILE_NOT_IN_DISABLED_STATE):
raise LPAError(f"EnableProfile failed: {PROFILE_ERROR_CODES.get(code, 'unknown')} (0x{code:02X})")
def is_euicc(self) -> bool:
# +CCHO:<n> -> ISD-R applet present, eUICC. Any error -> non-eUICC.
with self._acquire_lock():
try:
lines = self._client.query(f'AT+CCHO="{ISDR_AID}"')
except RuntimeError:
return False
for line in lines:
if line.startswith("+CCHO:") and (ch := line.split(":", 1)[1].strip()):
try:
self._client.query(f"AT+CCHC={ch}")
except (RuntimeError, TimeoutError):
pass
self._client.channel = None
return True
return False
-55
View File
@@ -1,55 +0,0 @@
#!/usr/bin/env python3
import sys
import math
import os
from pathlib import Path
CHUNK_SIZE = 45 * 1024 * 1024 # 45MB, under GitHub's 50MB limit
def get_chunk_name(name, idx, num_chunks):
return f"{name}.chunk{idx+1:02d}of{num_chunks:02d}"
def get_manifest_path(name):
return f"{name}.chunkmanifest"
def _chunk_paths(path, num_chunks):
return [get_manifest_path(path)] + [get_chunk_name(path, i, num_chunks) for i in range(num_chunks)]
def get_chunk_targets(path, file_size):
num_chunks = math.ceil(file_size / CHUNK_SIZE)
return _chunk_paths(path, num_chunks)
def chunk_file(path, targets):
manifest_path, *chunk_paths = targets
with open(path, 'rb') as f:
data = f.read()
actual_num_chunks = max(1, math.ceil(len(data) / CHUNK_SIZE))
assert len(chunk_paths) >= actual_num_chunks, f"Allowed {len(chunk_paths)} chunks but needs at least {actual_num_chunks}, for path {path}"
for i, chunk_path in enumerate(chunk_paths):
with open(chunk_path, 'wb') as f:
f.write(data[i * CHUNK_SIZE:(i + 1) * CHUNK_SIZE])
Path(manifest_path).write_text(str(len(chunk_paths)))
os.remove(path)
def get_existing_chunks(path):
if os.path.isfile(path):
return [path]
if os.path.isfile(manifest := get_manifest_path(path)):
num_chunks = int(Path(manifest).read_text().strip())
return _chunk_paths(path, num_chunks)
raise FileNotFoundError(path)
def read_file_chunked(path):
manifest_path = get_manifest_path(path)
if os.path.isfile(manifest_path):
num_chunks = int(Path(manifest_path).read_text().strip())
return b''.join(Path(get_chunk_name(path, i, num_chunks)).read_bytes() for i in range(num_chunks))
if os.path.isfile(path):
return Path(path).read_bytes()
raise FileNotFoundError(path)
if __name__ == "__main__":
path = sys.argv[1]
chunk_paths = get_chunk_targets(path, os.path.getsize(path))
chunk_file(path, chunk_paths)
+58
View File
@@ -0,0 +1,58 @@
import io
import os
import tempfile
import contextlib
import zstandard as zstd
LOG_COMPRESSION_LEVEL = 10 # little benefit up to level 15. level ~17 is a small step change
class CallbackReader:
"""Wraps a file, but overrides the read method to also
call a callback function with the number of bytes read so far."""
def __init__(self, f, callback, *args):
self.f = f
self.callback = callback
self.cb_args = args
self.total_read = 0
def __getattr__(self, attr):
return getattr(self.f, attr)
def read(self, *args, **kwargs):
chunk = self.f.read(*args, **kwargs)
self.total_read += len(chunk)
self.callback(*self.cb_args, self.total_read)
return chunk
@contextlib.contextmanager
def atomic_write_in_dir(path: str, mode: str = 'w', buffering: int = -1, encoding: str = None, newline: str = None,
overwrite: bool = False):
"""Write to a file atomically using a temporary file in the same directory as the destination file."""
dir_name = os.path.dirname(path)
if not overwrite and os.path.exists(path):
raise FileExistsError(f"File '{path}' already exists. To overwrite it, set 'overwrite' to True.")
with tempfile.NamedTemporaryFile(mode=mode, buffering=buffering, encoding=encoding, newline=newline, dir=dir_name, delete=False) as tmp_file:
yield tmp_file
tmp_file_name = tmp_file.name
os.replace(tmp_file_name, path)
def get_upload_stream(filepath: str, should_compress: bool) -> tuple[io.BufferedIOBase, int]:
if not should_compress:
file_size = os.path.getsize(filepath)
file_stream = open(filepath, "rb")
return file_stream, file_size
# Compress the file on the fly
compressed_stream = io.BytesIO()
compressor = zstd.ZstdCompressor(level=LOG_COMPRESSION_LEVEL)
with open(filepath, "rb") as f:
compressor.copy_stream(f, compressed_stream)
compressed_size = compressed_stream.tell()
compressed_stream.seek(0)
return compressed_stream, compressed_size
-17
View File
@@ -15,20 +15,3 @@ class FirstOrderFilter:
self.initialized = True
self.x = x
return self.x
class BounceFilter(FirstOrderFilter):
def __init__(self, x0, rc, dt, initialized=True, bounce=2):
self.velocity = FirstOrderFilter(0.0, 0.15, dt)
self.bounce = bounce
super().__init__(x0, rc, dt, initialized)
def update(self, x):
super().update(x)
scale = self.dt / (1.0 / 60.0) # tuned at 60 fps
self.velocity.x += (x - self.x) * self.bounce * scale * self.dt
self.velocity.update(0.0)
if abs(self.velocity.x) < 1e-3:
self.velocity.x = 0.0
self.x += self.velocity.x
return self.x
+7 -7
View File
@@ -1,30 +1,30 @@
from functools import cache
import subprocess
from openpilot.common.utils import run_cmd, run_cmd_default
from openpilot.common.run import run_cmd, run_cmd_default
@cache
def get_commit(cwd: str | None = None, branch: str = "HEAD") -> str:
def get_commit(cwd: str = None, branch: str = "HEAD") -> str:
return run_cmd_default(["git", "rev-parse", branch], cwd=cwd)
@cache
def get_commit_date(cwd: str | None = None, commit: str = "HEAD") -> str:
def get_commit_date(cwd: str = None, commit: str = "HEAD") -> str:
return run_cmd_default(["git", "show", "--no-patch", "--format='%ct %ci'", commit], cwd=cwd)
@cache
def get_short_branch(cwd: str | None = None) -> str:
def get_short_branch(cwd: str = None) -> str:
return run_cmd_default(["git", "rev-parse", "--abbrev-ref", "HEAD"], cwd=cwd)
@cache
def get_branch(cwd: str | None = None) -> str:
def get_branch(cwd: str = None) -> str:
return run_cmd_default(["git", "rev-parse", "--abbrev-ref", "--symbolic-full-name", "@{u}"], cwd=cwd)
@cache
def get_origin(cwd: str | None = None) -> str:
def get_origin(cwd: str = None) -> str:
try:
local_branch = run_cmd(["git", "name-rev", "--name-only", "HEAD"], cwd=cwd)
tracking_remote = run_cmd(["git", "config", "branch." + local_branch + ".remote"], cwd=cwd)
@@ -34,7 +34,7 @@ def get_origin(cwd: str | None = None) -> str:
@cache
def get_normalized_origin(cwd: str | None = None) -> str:
def get_normalized_origin(cwd: str = None) -> str:
return get_origin(cwd) \
.replace("git@", "", 1) \
.replace(".git", "", 1) \
-16
View File
@@ -1,16 +0,0 @@
import os
from typing import cast
from openpilot.common.hardware.base import HardwareBase
from openpilot.common.hardware.tici.hardware import Tici
from openpilot.common.hardware.pc.hardware import Pc
TICI = os.path.isfile('/TICI')
AGNOS = os.path.isfile('/AGNOS')
PC = not TICI
if TICI:
HARDWARE = cast(HardwareBase, Tici())
else:
HARDWARE = cast(HardwareBase, Pc())
-25
View File
@@ -1,25 +0,0 @@
#pragma once
#include <cstdlib>
#include <fstream>
#include <map>
#include <string>
#include "cereal/gen/cpp/log.capnp.h"
// no-op base hw class
class HardwareNone {
public:
static std::string get_name() { return ""; }
static cereal::InitData::DeviceType get_device_type() { return cereal::InitData::DeviceType::UNKNOWN; }
static std::string get_serial() { return "cccccc"; }
static std::map<std::string, std::string> get_init_logs() {
return {};
}
static void set_ir_power(int percentage) {}
static bool PC() { return false; }
};
-161
View File
@@ -1,161 +0,0 @@
import os
from abc import abstractmethod, ABC
from dataclasses import dataclass, fields
from cereal import log
from openpilot.common.esim.base import LPABase
NetworkType = log.DeviceState.NetworkType
NetworkStrength = log.DeviceState.NetworkStrength
@dataclass
class ThermalZone:
# a zone from /sys/class/thermal/thermal_zone*
name: str # a.k.a type
scale: float = 1000. # scale to get degrees in C
zone_number = -1
def read(self) -> float:
if self.zone_number < 0:
for n in os.listdir("/sys/devices/virtual/thermal"):
if not n.startswith("thermal_zone"):
continue
with open(os.path.join("/sys/devices/virtual/thermal", n, "type")) as f:
if f.read().strip() == self.name:
self.zone_number = int(n.removeprefix("thermal_zone"))
break
try:
with open(f"/sys/devices/virtual/thermal/thermal_zone{self.zone_number}/temp") as f:
return int(f.read()) / self.scale
except FileNotFoundError:
return 0
@dataclass
class ThermalConfig:
cpu: list[ThermalZone] | None = None
gpu: list[ThermalZone] | None = None
dsp: ThermalZone | None = None
pmic: list[ThermalZone] | None = None
memory: ThermalZone | None = None
intake: ThermalZone | None = None
exhaust: ThermalZone | None = None
gnss: ThermalZone | None = None
bottomSoc: ThermalZone | None = None
def get_msg(self):
ret = {}
for f in fields(ThermalConfig):
v = getattr(self, f.name)
if v is not None:
if isinstance(v, list):
ret[f.name + "TempC"] = [x.read() for x in v]
else:
ret[f.name + "TempC"] = v.read()
return ret
class HardwareBase(ABC):
@staticmethod
def get_cmdline() -> dict[str, str]:
with open('/proc/cmdline') as f:
cmdline = f.read()
return {kv[0]: kv[1] for kv in [s.split('=') for s in cmdline.split(' ')] if len(kv) == 2}
@staticmethod
def read_param_file(path, parser, default=0):
try:
with open(path) as f:
return parser(f.read())
except Exception:
return default
def booted(self) -> bool:
return True
def reboot(self, reason=None):
print("REBOOT!")
def uninstall(self):
print("uninstall")
def get_os_version(self):
return None
@abstractmethod
def get_device_type(self):
pass
def get_imei(self) -> str:
return ""
def get_serial(self):
return ""
def get_network_info(self):
return None
def get_network_type(self):
return NetworkType.none
def get_sim_info(self):
return {
'sim_id': '',
'mcc_mnc': None,
'network_type': ["Unknown"],
'sim_state': ["ABSENT"],
'data_connected': False
}
def get_sim_lpa(self) -> LPABase:
raise NotImplementedError("SIM LPA not available")
def get_network_strength(self, network_type):
return NetworkStrength.unknown
def get_network_metered(self, network_type) -> bool:
return network_type not in (NetworkType.none, NetworkType.wifi, NetworkType.ethernet)
def get_current_power_draw(self):
return 0
def get_som_power_draw(self):
return 0
def shutdown(self):
print("SHUTDOWN!")
def get_thermal_config(self):
return ThermalConfig()
def set_display_power(self, on: bool):
pass
def set_screen_brightness(self, percentage):
pass
def get_screen_brightness(self):
return 0
def set_power_save(self, powersave_enabled):
pass
def get_gpu_usage_percent(self):
return 0
def get_modem_temperatures(self):
return []
def initialize_hardware(self):
pass
def reset_internal_panda(self):
pass
def recover_internal_panda(self):
pass
def get_modem_data_usage(self):
return -1, -1
def set_ir_power(self, percent: int):
pass

Some files were not shown because too many files have changed in this diff Show More