Compare commits

..

5 Commits

Author SHA1 Message Date
DevTekVE a3201845f2 Merge branch 'temp-squash-long-distance-btn-exp-mode' into master-dev-c3-new-test 2025-01-18 19:39:08 +01:00
DevTekVE 9567c074c1 Longitudinal: Distance button hold to toggle Chill/Experimental Mode (#576)
Source commit: 13cb725652

Original commits:

* Sync: `commaai/openpilot:master` into `sunnypilot/sunnypilot:master-new` (#580)
* test_processes: update ref logs to afde277
* # Conflicts:
* #    .github/workflows/ui_preview.yaml
* #    opendbc_repo
* #    panda
* #    release/release_files.py
* #    selfdrive/test/process_replay/ref_commit
* #    selfdrive/ui/tests/test_ui/run.py
* bring back ui.py (#34396)
* * bring back uipy
* * fix it
* * fix
* Log satellite count in GpsLocationData (#34395)
* * Log satellite count in GpsLocationData
* * update refs
* * forgot to build
* Toyota: allow brake hold (#34394)
* toyota okay
* fix uv (#34393)
* uv fix
* Update RELEASES.md
* release this month
* no need
* fix
* unused
* Longitudinal: Distance button hold to toggle Chill/Experimental Mode
* Hyundai: match cancel button panda safety logic (#34390)
* * rising edge
* * should work
* * TODO
* * fix
* card: fix cruise speed initialization w/ buttons (#34386)
* * fix initialize w/ buttons
* * what
* * comment
* VW: switch to common pcmCruise check (#34389)
* Update car_specific.py
* card: vcruise all in one place (#34387)
* * no reason to not be here
* * oh this was off by a frame!
* * ref
* Fix missing visual car dash alerts (#34385)
* * fix missing visual dash alerts
* * update refs
* process_replay: add cast in migrate_longitudinalPlan (#34383)
* * Cast
* * Something
* * Remove newline
* process_replay: ignore unknown members in the migration code (#34382)
* * Fix the migration for the events
* * clean up
* clean up
* clean up
* * no continue
* ---------
* Co-authored-by: Shane Smiskol <shane@smiskol.com>
* Getting rid of openpilot.common.numpy_fast (#34368)
* * Got rid openpilot.common.numpy_fast
* * fixed some data type erros
* * importing numpy instead of importing specific functions
* * fixing some numpy importing mistakes
* * Update selfdrive/car/cruise.py
* ---------
* Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
* Toyota: remove longitudinal derivative (#34378)
* * Update Python packages
* * update refs
* ---------
* Co-authored-by: Vehicle Researcher <user@comma.ai>
* Co-authored-by: Shane Smiskol <shane@smiskol.com>
* PlotJuggler: Layout for debugging locationdTemporaryError (#34381)
* * PlotJuggler: layout for debugging locationd
* * not needed, and codespell doesn't like it
* raylib: revert auto fit screen size (#34380)
* revert auto fit screen size
* Remove release files (conceptually) (#34379)
* * Remove release files (conceptually)
* * fix
* raylib: refactor to implement new App class (#34375)
* refactor to implement new App class
* [bot] Update Python packages (#34376)
* * Update Python packages
* * time -> time_helpers
* ---------
* Co-authored-by: Vehicle Researcher <user@comma.ai>
* Co-authored-by: Maxime Desroches <desroches.maxime@gmail.com>
* CI: Add missing uppercase_keyboard UI view (#34347)
* * add-uppercase-preview
* * testing-UI-on-fork
* * change back
* * add-my-branch
* * needs to see sha from fork for test
* * fix missing sha
* * change back
* * get correct named artifact
* * try this
* * experimenting coordinates
* * try these coordinates
* * try these coordinates
* * draws circles to see touches - changes coordinates
* * try these changes
* * better coordinates
* * click is more centered
* * try again
* * revert back
* * revert these too
* * last revert...
* Keyboard: add missing control btns to uppercase (#34344)
* * add-slash-to uppercase
* * Trigger UI preview workflow
* Move uds.py (#34374)
* * rm uds pt. 1
* * rm uds pt. 2
* * imports
* oops need raw
* third_party: add raygui (#34369)
* swaglog: fix locale dependence (#34367)
* encoderd: refactor VideoEncoder::publisher_publish to standardize member variable access (#34342)
* remove redundant pointer parameter
* modeld: properly release OpenCL context in  `__dealloc__` method (#34353)
* release OpenCL context in __dealloc__
* release soon
* loggerd: typing and remove unused default arg (#34349)
* deleter cleanups (#34345)
2025-01-18 19:39:07 +01:00
DevTekVE a79a2ac494 Merge branch 'temp-squash-long-distance-btn-exp-mode' into master-dev-c3-new-test 2025-01-18 19:35:46 +01:00
DevTekVE e6055d68be Longitudinal: Distance button hold to toggle Chill/Experimental Mode (#576)
Original commits:

* Sync: `commaai/openpilot:master` into `sunnypilot/sunnypilot:master-new` (#580)
* test_processes: update ref logs to afde277
* # Conflicts:
* #    .github/workflows/ui_preview.yaml
* #    opendbc_repo
* #    panda
* #    release/release_files.py
* #    selfdrive/test/process_replay/ref_commit
* #    selfdrive/ui/tests/test_ui/run.py
* bring back ui.py (#34396)
* * bring back uipy
* * fix it
* * fix
* Log satellite count in GpsLocationData (#34395)
* * Log satellite count in GpsLocationData
* * update refs
* * forgot to build
* Toyota: allow brake hold (#34394)
* toyota okay
* fix uv (#34393)
* uv fix
* Update RELEASES.md
* release this month
* no need
* fix
* unused
* Longitudinal: Distance button hold to toggle Chill/Experimental Mode
* Hyundai: match cancel button panda safety logic (#34390)
* * rising edge
* * should work
* * TODO
* * fix
* card: fix cruise speed initialization w/ buttons (#34386)
* * fix initialize w/ buttons
* * what
* * comment
* VW: switch to common pcmCruise check (#34389)
* Update car_specific.py
* card: vcruise all in one place (#34387)
* * no reason to not be here
* * oh this was off by a frame!
* * ref
* Fix missing visual car dash alerts (#34385)
* * fix missing visual dash alerts
* * update refs
* process_replay: add cast in migrate_longitudinalPlan (#34383)
* * Cast
* * Something
* * Remove newline
* process_replay: ignore unknown members in the migration code (#34382)
* * Fix the migration for the events
* * clean up
* clean up
* clean up
* * no continue
* ---------
* Co-authored-by: Shane Smiskol <shane@smiskol.com>
* Getting rid of openpilot.common.numpy_fast (#34368)
* * Got rid openpilot.common.numpy_fast
* * fixed some data type erros
* * importing numpy instead of importing specific functions
* * fixing some numpy importing mistakes
* * Update selfdrive/car/cruise.py
* ---------
* Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
* Toyota: remove longitudinal derivative (#34378)
* * Update Python packages
* * update refs
* ---------
* Co-authored-by: Vehicle Researcher <user@comma.ai>
* Co-authored-by: Shane Smiskol <shane@smiskol.com>
* PlotJuggler: Layout for debugging locationdTemporaryError (#34381)
* * PlotJuggler: layout for debugging locationd
* * not needed, and codespell doesn't like it
* raylib: revert auto fit screen size (#34380)
* revert auto fit screen size
* Remove release files (conceptually) (#34379)
* * Remove release files (conceptually)
* * fix
* raylib: refactor to implement new App class (#34375)
* refactor to implement new App class
* [bot] Update Python packages (#34376)
* * Update Python packages
* * time -> time_helpers
* ---------
* Co-authored-by: Vehicle Researcher <user@comma.ai>
* Co-authored-by: Maxime Desroches <desroches.maxime@gmail.com>
* CI: Add missing uppercase_keyboard UI view (#34347)
* * add-uppercase-preview
* * testing-UI-on-fork
* * change back
* * add-my-branch
* * needs to see sha from fork for test
* * fix missing sha
* * change back
* * get correct named artifact
* * try this
* * experimenting coordinates
* * try these coordinates
* * try these coordinates
* * draws circles to see touches - changes coordinates
* * try these changes
* * better coordinates
* * click is more centered
* * try again
* * revert back
* * revert these too
* * last revert...
* Keyboard: add missing control btns to uppercase (#34344)
* * add-slash-to uppercase
* * Trigger UI preview workflow
* Move uds.py (#34374)
* * rm uds pt. 1
* * rm uds pt. 2
* * imports
* oops need raw
* third_party: add raygui (#34369)
* swaglog: fix locale dependence (#34367)
* encoderd: refactor VideoEncoder::publisher_publish to standardize member variable access (#34342)
* remove redundant pointer parameter
* modeld: properly release OpenCL context in  `__dealloc__` method (#34353)
* release OpenCL context in __dealloc__
* release soon
* loggerd: typing and remove unused default arg (#34349)
* deleter cleanups (#34345)
2025-01-18 19:35:46 +01:00
DevTekVE 3b907acfe7 Longitudinal: Distance button hold to toggle Chill/Experimental Mode
Original commits:

* no need
* fix
* unused
* Longitudinal: Distance button hold to toggle Chill/Experimental Mode
2025-01-15 21:42:38 +01:00
2725 changed files with 197854 additions and 141186 deletions
+19
View File
@@ -0,0 +1,19 @@
---
Checks: '
bugprone-*,
-bugprone-integer-division,
-bugprone-narrowing-conversions,
performance-*,
clang-analyzer-*,
misc-*,
-misc-unused-parameters,
modernize-*,
-modernize-avoid-c-arrays,
-modernize-deprecated-headers,
-modernize-use-auto,
-modernize-use-using,
-modernize-use-nullptr,
-modernize-use-trailing-return-type,
'
CheckOptions:
...
-4
View File
@@ -1,6 +1,2 @@
Wen
REGIST
PullRequest
cancelled
FOF
NoO
+39
View File
@@ -0,0 +1,39 @@
**/.git
.DS_Store
*.dylib
*.DSYM
*.d
*.pyc
*.pyo
.*.swp
.*.swo
.*.un~
*.tmp
*.o
*.o-*
*.os
*.os-*
*.so
*.a
venv/
.venv/
notebooks
phone
massivemap
neos
installer
chffr/app2
chffr/backend/env
selfdrive/nav
selfdrive/baseui
selfdrive/test/simulator2
**/cache_data
xx/plus
xx/community
xx/projects
!xx/projects/eon_testing_master
!xx/projects/map3d
xx/ops
xx/junk
+1 -1
View File
@@ -5,7 +5,7 @@ end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
[*.{py,pyx,pxd}]
[{*.py, *.pyx, *.pxd}]
charset = utf-8
indent_style = space
indent_size = 2
+13 -3
View File
@@ -7,8 +7,18 @@
*.png filter=lfs diff=lfs merge=lfs -text
*.gif filter=lfs diff=lfs merge=lfs -text
*.ttf filter=lfs diff=lfs merge=lfs -text
*.otf filter=lfs diff=lfs merge=lfs -text
*.wav filter=lfs diff=lfs merge=lfs -text
openpilot/selfdrive/car/tests/test_models_segs.txt filter=lfs diff=lfs merge=lfs -text
openpilot/common/hardware/tici/updater filter=lfs diff=lfs merge=lfs -text
selfdrive/test/process_replay/fakedata/*.zst filter=lfs diff=lfs merge=lfs -text
selfdrive/car/tests/test_models_segs.txt filter=lfs diff=lfs merge=lfs -text
system/hardware/tici/updater filter=lfs diff=lfs merge=lfs -text
selfdrive/ui/qt/spinner_larch64 filter=lfs diff=lfs merge=lfs -text
selfdrive/ui/qt/text_larch64 filter=lfs diff=lfs merge=lfs -text
third_party/**/*.a filter=lfs diff=lfs merge=lfs -text
third_party/**/*.so filter=lfs diff=lfs merge=lfs -text
third_party/**/*.so.* filter=lfs diff=lfs merge=lfs -text
third_party/**/*.dylib filter=lfs diff=lfs merge=lfs -text
third_party/acados/*/t_renderer filter=lfs diff=lfs merge=lfs -text
third_party/qt5/larch64/bin/lrelease filter=lfs diff=lfs merge=lfs -text
third_party/qt5/larch64/bin/lupdate filter=lfs diff=lfs merge=lfs -text
third_party/catch2/include/catch2/catch.hpp filter=lfs diff=lfs merge=lfs -text
-11
View File
@@ -1,11 +0,0 @@
* @sunnypilot/dev-internal
/.github/ @devtekve @sunnyhaibin
/release/ci/ @devtekve @sunnyhaibin
/tinygrad_repo @devtekve @Discountchubbs
/tinygrad/ @devtekve @Discountchubbs
/selfdrive/controls/lib/longitudinal_planner.py @devtekve @Discountchubbs
/selfdrive/controls/lib/longitudinal_mpc_lib/long_mpc.py @devtekve @Discountchubbs
/selfdrive/modeld/ @devtekve @Discountchubbs
/sunnypilot/model* @devtekve @Discountchubbs
/sunnypilot/sunnylink/ @devtekve
/system/athena/ @devtekve
+53
View File
@@ -0,0 +1,53 @@
name: Car bug report
description: For issues with a particular car make or model
labels: ["car", "bug"]
body:
- type: markdown
attributes:
value: >
Before creating a **bug report**, please check the following:
* Ensure you're running the latest openpilot release.
* Ensure you're using officially supported hardware. Issues running on PCs have a different issue template.
* Ensure there isn't an existing issue for your bug. If there is, leave a comment on the existing issue.
* Ensure you're running stock openpilot. We cannot look into bug reports from forks.
If you're unsure whether you've hit a bug, check out the #installation-help channel in the [community Discord server](https://discord.comma.ai).
- type: textarea
attributes:
label: Describe the bug
description: Also include a description of how to reproduce the bug
validations:
required: true
- type: input
id: car
attributes:
label: Which car does this affect?
placeholder: Toyota Prius 2017
validations:
required: true
- type: input
id: route
attributes:
label: Provide a route where the issue occurs
description: Ensure the route is fully uploaded at https://useradmin.comma.ai
placeholder: 77611a1fac303767|2020-05-11--16-37-07
validations:
required: true
- type: input
id: version
attributes:
label: openpilot version
description: If you're not on release, provide the commit hash
placeholder: 0.8.10
validations:
required: true
- type: textarea
attributes:
label: Additional info
+1 -4
View File
@@ -1,12 +1,9 @@
blank_issues_enabled: false
contact_links:
- name: Car bug report
url: https://github.com/commaai/opendbc/issues/new
about: For issues with a particular car make or model
- name: Join the Discord
url: https://discord.comma.ai
about: The community Discord is for both openpilot development and experience discussion
- name: Report driving behavior feedback
- name: Report model bugs
url: https://discord.com/channels/469524606043160576/1254834193066623017
about: Feedback for the driving and driver monitoring models goes in the #driving-feedback in Discord
- name: Community Wiki
+8
View File
@@ -0,0 +1,8 @@
---
name: Enhancement
about: For openpilot enhancement suggestions
title: ''
labels: 'enhancement'
assignees: ''
---
+8 -12
View File
@@ -1,31 +1,27 @@
ci:
CI / testing:
- changed-files:
- any-glob-to-all-files: "{.github/**,**/test_*,**/test/**,Jenkinsfile}"
chore:
- changed-files:
- any-glob-to-all-files: "{.github/**}"
- any-glob-to-all-files: "{.github/**,**/test_*,Jenkinsfile}"
car:
- changed-files:
- any-glob-to-all-files: '{openpilot/selfdrive/car/**,opendbc_repo}'
- any-glob-to-all-files: '{selfdrive/car/**,opendbc_repo}'
simulation:
- changed-files:
- any-glob-to-all-files: 'openpilot/tools/sim/**'
- any-glob-to-all-files: 'tools/sim/**'
ui:
- changed-files:
- any-glob-to-all-files: '{openpilot/selfdrive/assets/**,openpilot/selfdrive/ui/**,openpilot/system/ui/**}'
- any-glob-to-all-files: 'selfdrive/ui/**'
tools:
- changed-files:
- any-glob-to-all-files: 'openpilot/tools/**'
- any-glob-to-all-files: 'tools/**'
multilanguage:
- changed-files:
- any-glob-to-all-files: 'openpilot/selfdrive/ui/translations/**'
- any-glob-to-all-files: 'selfdrive/ui/translations/**'
autonomy:
- changed-files:
- any-glob-to-all-files: "{openpilot/selfdrive/modeld/models/**,openpilot/selfdrive/test/process_replay/model_replay_ref_commit,openpilot/sunnypilot/modeld*/models/**}"
- any-glob-to-all-files: "{selfdrive/modeld/models/**,selfdrive/test/process_replay/model_replay_ref_commit}"
+2 -2
View File
@@ -44,8 +44,8 @@ Explain how you tested this bug fix.
**Checklist**
- [ ] added entry to CAR in openpilot/selfdrive/car/*/values.py and ran `openpilot/selfdrive/car/docs.py` to generate new docs
- [ ] test route added to [routes.py](https://github.com/commaai/openpilot/blob/master/openpilot/selfdrive/car/tests/routes.py)
- [ ] added entry to CAR in selfdrive/car/*/values.py and ran `selfdrive/car/docs.py` to generate new docs
- [ ] test route added to [routes.py](https://github.com/commaai/openpilot/blob/master/selfdrive/car/tests/routes.py)
- [ ] route with openpilot:
- [ ] route with stock system:
- [ ] car harness used (if comma doesn't sell it, put N/A):
-43
View File
@@ -1,43 +0,0 @@
exclude-labels:
- 'no-changelog'
categories:
- title: '🚀 Features'
labels:
- 'feature'
- 'enhancement'
- title: '🐛 Bug Fixes'
collapse-after: 5
labels:
- 'fix'
- 'bugfix'
- 'bug'
- title: '🧰 Maintenance'
collapse-after: 5
label: 'chore'
change-template: '- $TITLE @$AUTHOR (#$NUMBER)'
change-title-escapes: '\<*_&'
replacers:
- search: '/[Ss][Uu][Nn][Nn][Yy][Pp][Ii][Ll][Oo][Tt]/g'
replace: 'sunnypilot'
- search: '/\b[Ss][Pp]\b/g'
replace: 'SP'
version-resolver:
major:
labels:
- 'major'
minor:
labels:
- 'minor'
patch:
labels:
- 'patch'
default: patch
name-template: 'v$RESOLVED_VERSION 🚀'
tag-template: 'v$RESOLVED_VERSION'
version-template: "0.$MAJOR.$MINOR.$PATCH" # The day OP becomes v1, we need to bump this
tag-prefix: "v0." # The day OP becomes v1, we need to bump this
prerelease-identifier: "staging"
template: |
## Changes
$CHANGES
+58
View File
@@ -0,0 +1,58 @@
name: 'automatically cache based on current runner'
inputs:
path:
description: 'path to cache'
required: true
key:
description: 'key'
required: true
restore-keys:
description: 'restore-keys'
required: true
save:
description: 'whether to save the cache'
default: 'false'
required: false
outputs:
cache-hit:
description: 'cache hit occurred'
value: ${{ (contains(runner.name, 'nsc') && steps.ns-cache.outputs.cache-hit) ||
(!contains(runner.name, 'nsc') && inputs.save != 'false' && steps.gha-cache.outputs.cache-hit) ||
(!contains(runner.name, 'nsc') && inputs.save == 'false' && steps.gha-cache-ro.outputs.cache-hit) }}
runs:
using: "composite"
steps:
- name: setup namespace cache
id: ns-cache
if: ${{ contains(runner.name, 'nsc') }}
uses: namespacelabs/nscloud-cache-action@v1
with:
path: ${{ inputs.path }}
- name: setup github cache
id: gha-cache
if: ${{ !contains(runner.name, 'nsc') && inputs.save != 'false' }}
uses: 'actions/cache@v4'
with:
path: ${{ inputs.path }}
key: ${{ inputs.key }}
restore-keys: ${{ inputs.restore-keys }}
- name: setup github cache
id: gha-cache-ro
if: ${{ !contains(runner.name, 'nsc') && inputs.save == 'false' }}
uses: 'actions/cache/restore@v4'
with:
path: ${{ inputs.path }}
key: ${{ inputs.key }}
restore-keys: ${{ inputs.restore-keys }}
# make the directory manually in case we didn't get a hit, so it doesn't fail on future steps
- id: scons-cache-setup
shell: bash
run: |
mkdir -p ${{ inputs.path }}
sudo chmod -R 777 ${{ inputs.path }}
sudo chown -R $USER ${{ inputs.path }}
+21 -59
View File
@@ -1,7 +1,7 @@
name: "PR review"
on:
pull_request_target:
types: [ opened, reopened, synchronize, edited ]
types: [opened, reopened, synchronize, edited, edited]
jobs:
labeler:
@@ -9,83 +9,45 @@ jobs:
permissions:
contents: read
pull-requests: write
issues: write
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v4
with:
submodules: false
# Label PRs
- uses: actions/labeler@v6
- uses: actions/labeler@v5.0.0
with:
dot: true
configuration-path: .github/labeler.yaml
# Check PR target branch
- name: check branch
uses: Vankka/pr-target-branch-action@5da68a42bcb7b43d39104295a876a6f3f8d7908b
uses: Vankka/pr-target-branch-action@def32ec9d93514138d6ac0132ee62e120a72aed5
if: github.repository == 'sunnypilot/sunnypilot'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
target: /^(?!master$).*/
exclude: /sunnypilot:.*/
exclude: /commaai:.*/
change-to: ${{ github.base_ref }}
already-exists-action: close_this
already-exists-comment: "Your PR should be made against the `master` branch"
update-pr-labels:
name: Update fork's PR Labels
runs-on: ubuntu-latest
if: (github.event.pull_request.head.repo.fork && (contains(github.event_name, 'pull_request') && github.event.action == 'synchronize'))
env:
PR_LABEL: 'dev'
TRUST_FORK_PR_LABEL: 'trust-fork-pr'
steps:
- name: Check if PR has dev label
id: check-labels
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const prNumber = context.payload.pull_request.number;
const { data: labels } = await github.rest.issues.listLabelsOnIssue({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNumber
});
# Welcome comment
- name: "First timers PR"
uses: actions/first-interaction@v1
if: github.event.pull_request.head.repo.full_name != 'sunnypilot/sunnypilot'
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
pr-message: |
<!-- _(run_id **${{ github.run_id }}**)_ -->
Thanks for contributing to openpilot! In order for us to review your PR as quickly as possible, check the following:
* Convert your PR to a draft unless it's ready to review
* Read the [contributing docs](https://github.com/sunnypilot/sunnypilot/blob/master/docs/CONTRIBUTING.md)
* Before marking as "ready for review", ensure:
* the goal is clearly stated in the description
* all the tests are passing
* the change is [something we merge](https://github.com/sunnypilot/sunnypilot/blob/master/docs/CONTRIBUTING.md#what-gets-merged)
* include a route or your device' dongle ID if relevant
const hasDevC3Label = labels.some(label => label.name === process.env.PR_LABEL);
const hasTrustLabel = labels.some(label => label.name === process.env.TRUST_FORK_PR_LABEL);
console.log(`PR #${prNumber} has ${process.env.PR_LABEL} label: ${hasDevC3Label}`);
console.log(`PR #${prNumber} has ${process.env.TRUST_FORK_PR_LABEL} label: ${hasTrustLabel}`);
core.setOutput('has-dev', hasDevC3Label ? 'true' : 'false');
core.setOutput('has-trust', hasTrustLabel ? 'true' : 'false');
- name: Remove trust-fork-pr label if present
if: steps.check-labels.outputs.has-dev == 'true' && steps.check-labels.outputs.has-trust == 'true'
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const prNumber = context.payload.pull_request.number;
await github.rest.issues.removeLabel({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNumber,
name: process.env.TRUST_FORK_PR_LABEL
});
console.log(`Removed '${process.env.TRUST_FORK_PR_LABEL}' label from PR #${prNumber} as it received new commits`);
// Add a comment to the PR
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNumber,
body: `The \`${process.env.TRUST_FORK_PR_LABEL}\` label has been automatically removed because new commits were pushed to this PR. This PR will need to be re-reviewed before the label can be applied again.`
});
+6 -4
View File
@@ -5,7 +5,9 @@ on:
workflow_dispatch:
env:
PYTHONPATH: ${{ github.workspace }}
BASE_IMAGE: openpilot-base
DOCKER_REGISTRY: ghcr.io/commaai
RUN: docker run --shm-size 2G -v $PWD:/tmp/openpilot -w /tmp/openpilot -e PYTHONPATH=/tmp/openpilot -e NUM_JOBS -e JOB_ID -e GITHUB_ACTION -e GITHUB_REF -e GITHUB_HEAD_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_RUN_ID -v $GITHUB_WORKSPACE/.ci_cache/scons_cache:/tmp/scons_cache -v $GITHUB_WORKSPACE/.ci_cache/comma_download_cache:/tmp/comma_download_cache -v $GITHUB_WORKSPACE/.ci_cache/openpilot_cache:/tmp/openpilot_cache $DOCKER_REGISTRY/$BASE_IMAGE:latest /bin/bash -c
jobs:
badges:
@@ -15,13 +17,13 @@ jobs:
permissions:
contents: write
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v4
with:
submodules: true
- run: ./tools/op.sh setup
- uses: ./.github/workflows/setup-with-retry
- name: Push badges
run: |
python3 selfdrive/ui/translations/create_badges.py
${{ env.RUN }} "scons -j$(nproc) && python3 selfdrive/ui/translations/create_badges.py"
rm .gitattributes
@@ -1,306 +0,0 @@
name: Build and push all tinygrad models
on:
workflow_dispatch:
inputs:
set_min_version:
description: 'Minimum selector version required for the models (see helpers.py or readme.md)'
required: true
type: string
jobs:
setup:
runs-on: ubuntu-latest
outputs:
json_version: ${{ steps.get-json.outputs.json_version }}
recompiled_dir: ${{ steps.create-recompiled-dir.outputs.recompiled_dir }}
json_file: ${{ steps.get-json.outputs.json_file }}
model_matrix: ${{ steps.set-matrix.outputs.model_matrix }}
tinygrad_ref: ${{ steps.get-tinygrad-ref.outputs.tinygrad_ref }}
steps:
- name: Checkout sunnypilot repo
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot
path: sunnypilot
submodules: recursive
- name: Get tinygrad_repo ref
id: get-tinygrad-ref
run: |
cd sunnypilot
export PYTHONPATH=$(pwd)
ref=$(python3 sunnypilot/models/tinygrad_ref.py)
echo "tinygrad_ref=$ref" >> $GITHUB_OUTPUT
echo "tinygrad_ref is $ref"
- name: Checkout docs repo (sunnypilot-models, gh-pages)
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot-models
ref: gh-pages
path: docs
ssh-key: ${{ secrets.CI_SUNNYPILOT_DOCS_PRIVATE_KEY }}
- name: Get next JSON version to use (from GitHub docs repo)
id: get-json
run: |
cd docs/docs
latest=$(ls driving_models_v*.json | sed -E 's/.*_v([0-9]+)\.json/\1/' | sort -n | tail -1)
next=$((latest+1))
json_file="driving_models_v${next}.json"
cp "driving_models_v${latest}.json" "$json_file"
echo "json_file=docs/docs/$json_file" >> $GITHUB_OUTPUT
echo "json_version=$((next+0))" >> $GITHUB_OUTPUT
echo "SRC_JSON_FILE=docs/docs/driving_models_v${latest}.json" >> $GITHUB_ENV
- name: Extract tinygrad models
id: set-matrix
working-directory: docs/docs
run: |
jq -c '[.bundles[] | select(.runner=="tinygrad") | {ref, display_name: (.display_name | gsub(" \\([^)]*\\)"; "")), is_20hz}]' "$(basename "${SRC_JSON_FILE}")" > matrix.json
echo "model_matrix=$(cat matrix.json)" >> $GITHUB_OUTPUT
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.GITLAB_SSH_PRIVATE_KEY }}
- run: |
mkdir -p ~/.ssh
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Clone GitLab docs repo and create new recompiled dir
id: create-recompiled-dir
env:
GIT_SSH_COMMAND: 'ssh -o UserKnownHostsFile=~/.ssh/known_hosts'
run: |
git clone --depth 1 --filter=tree:0 --sparse git@gitlab.com:sunnypilot/public/${{ vars.MODELS_GITLAB }} gitlab_docs
cd gitlab_docs
git checkout main
git sparse-checkout set --no-cone models/
cd models
latest_dir=$(ls -d recompiled* 2>/dev/null | sed -E 's/recompiled([0-9]+)/\1/' | sort -n | tail -1)
if [[ -z "$latest_dir" ]]; then
next_dir=1
else
next_dir=$((latest_dir+1))
fi
recompiled_dir="${next_dir}"
mkdir -p "recompiled${recompiled_dir}"
touch "recompiled${recompiled_dir}/.gitkeep"
cd ../..
echo "recompiled_dir=$recompiled_dir" >> $GITHUB_OUTPUT
- name: Push empty recompiled dir to GitLab
run: |
cd gitlab_docs
git add models/recompiled${{ steps.create-recompiled-dir.outputs.recompiled_dir }}
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git commit -m "Add recompiled${{ steps.create-recompiled-dir.outputs.recompiled_dir }} for build-all" || echo "No changes to commit"
git push origin main
- name: Push new JSON to GitHub docs repo
run: |
cd docs
git pull origin gh-pages
git add docs/"$(basename ${{ steps.get-json.outputs.json_file }})"
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git commit -m "Add new ${{ steps.get-json.outputs.json_file }} for build-all" || echo "No changes to commit"
git push origin gh-pages
get_and_build:
needs: [setup]
strategy:
matrix:
model: ${{ fromJson(needs.setup.outputs.model_matrix) }}
fail-fast: false
uses: ./.github/workflows/build-single-tinygrad-model.yaml
with:
upstream_branch: ${{ matrix.model.ref }}
custom_name: ${{ matrix.model.display_name }}
is_20hz: ${{ matrix.model.is_20hz }}
recompiled_dir: ${{ needs.setup.outputs.recompiled_dir }}
json_version: ${{ needs.setup.outputs.json_version }}
secrets: inherit
retry_failed_models:
needs: [setup, get_and_build]
runs-on: ubuntu-latest
if: ${{ needs.setup.result != 'failure' && !cancelled() }}
outputs:
retry_matrix: ${{ steps.set-retry-matrix.outputs.retry_matrix }}
steps:
- uses: actions/download-artifact@v4
with:
pattern: model-*
path: output
- id: set-retry-matrix
run: |
echo '${{ needs.setup.outputs.model_matrix }}' > matrix.json
built=(); while IFS= read -r line; do built+=("$line"); done < <(
find output -maxdepth 1 -name 'model-*' -printf "%f\n" | sed -E 's/^model-//' | sed -E 's/-[0-9]+$//' | sed -E 's/ \([^)]*\)//' | awk '{gsub(/^ +| +$/, ""); print}'
)
jq -c --argjson built "$(printf '%s\n' "${built[@]}" | jq -R . | jq -s .)" \
'map(select(.display_name as $n | ($built | index($n | gsub("^ +| +$"; "")) | not)))' matrix.json > retry_matrix.json
echo "retry_matrix=$(cat retry_matrix.json)" >> $GITHUB_OUTPUT
retry_get_and_build:
needs: [setup, get_and_build, retry_failed_models]
if: ${{ needs.get_and_build.result == 'failure' || (needs.retry_failed_models.outputs.retry_matrix != '[]' && needs.retry_failed_models.outputs.retry_matrix != '') }}
strategy:
matrix:
model: ${{ fromJson(needs.retry_failed_models.outputs.retry_matrix) }}
fail-fast: false
uses: ./.github/workflows/build-single-tinygrad-model.yaml
with:
upstream_branch: ${{ matrix.model.ref }}
custom_name: ${{ matrix.model.display_name }}
is_20hz: ${{ matrix.model.is_20hz }}
recompiled_dir: ${{ needs.setup.outputs.recompiled_dir }}
json_version: ${{ needs.setup.outputs.json_version }}
artifact_suffix: -retry
secrets: inherit
publish_models:
name: Publish models sequentially
needs: [setup, get_and_build, retry_failed_models, retry_get_and_build]
if: ${{ !cancelled() && (needs.get_and_build.result != 'failure' || needs.retry_get_and_build.result == 'success' || (needs.retry_failed_models.outputs.retry_matrix != '[]' && needs.retry_failed_models.outputs.retry_matrix != '')) }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
max-parallel: 1
matrix:
model: ${{ fromJson(needs.setup.outputs.model_matrix) }}
env:
RECOMPILED_DIR: recompiled${{ needs.setup.outputs.recompiled_dir }}
JSON_FILE: ${{ needs.setup.outputs.json_file }}
ARTIFACT_NAME_INPUT: ${{ matrix.model.display_name }}
steps:
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.GITLAB_SSH_PRIVATE_KEY }}
- name: Add GitLab.com SSH key to known_hosts
run: |
mkdir -p ~/.ssh
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Clone GitLab docs repo
env:
GIT_SSH_COMMAND: 'ssh -o UserKnownHostsFile=~/.ssh/known_hosts'
run: |
echo "Cloning GitLab"
git clone --depth 1 --filter=tree:0 --sparse git@gitlab.com:sunnypilot/public/${{ vars.MODELS_GITLAB }} gitlab_docs
cd gitlab_docs
echo "checkout models/${RECOMPILED_DIR}"
git sparse-checkout set --no-cone models/${RECOMPILED_DIR}
git checkout main
cd ..
- name: Checkout docs repo
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot-models
ref: gh-pages
path: docs
ssh-key: ${{ secrets.CI_SUNNYPILOT_DOCS_PRIVATE_KEY }}
- name: Validate recompiled dir and JSON version
run: |
if [ ! -d "gitlab_docs/models/$RECOMPILED_DIR" ]; then
echo "Recompiled dir $RECOMPILED_DIR does not exist in GitLab repo"
exit 1
fi
if [ ! -f "$JSON_FILE" ]; then
echo "JSON file $JSON_FILE does not exist!"
exit 1
fi
- name: Download artifact name file
uses: actions/download-artifact@v4
with:
name: artifact-name-${{ env.ARTIFACT_NAME_INPUT }}
path: artifact_name
- name: Read artifact name
id: read-artifact-name
run: |
ARTIFACT_NAME=$(cat artifact_name/artifact_name.txt)
echo "artifact_name=$ARTIFACT_NAME" >> $GITHUB_OUTPUT
- name: Download model artifact
uses: actions/download-artifact@v4
with:
name: ${{ steps.read-artifact-name.outputs.artifact_name }}
path: output
- name: Remove onnx files bc not needed for recompiled dir since they already exist from single build
run: |
find output -type f -name '*.onnx' -delete
find output -type f -name 'big_*.pkl' -delete
find output -type f -name 'dmonitoring_model_tinygrad.pkl' -delete
- name: Copy model artifacts to gitlab
env:
ARTIFACT_NAME: ${{ steps.read-artifact-name.outputs.artifact_name }}
run: |
ARTIFACT_DIR="gitlab_docs/models/${RECOMPILED_DIR}/${ARTIFACT_NAME}"
mkdir -p "$ARTIFACT_DIR"
for path in output/*; do
if [ "$(basename "$path")" = "artifact_name.txt" ]; then
continue
fi
name="$(basename "$path")"
if [ -d "$path" ]; then
mkdir -p "$ARTIFACT_DIR/$name"
cp -r "$path"/* "$ARTIFACT_DIR/$name/"
echo "Copied dir $name -> $ARTIFACT_DIR/$name"
else
cp "$path" "$ARTIFACT_DIR/"
echo "Copied file $name -> $ARTIFACT_DIR/"
fi
done
- name: Push recompiled dir to GitLab
env:
GITLAB_SSH_PRIVATE_KEY: ${{ secrets.GITLAB_SSH_PRIVATE_KEY }}
run: |
cd gitlab_docs
git checkout main
git pull origin main
for d in models/"$RECOMPILED_DIR"/*/; do
git sparse-checkout add "$d"
done
git add models/"$RECOMPILED_DIR"
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git commit -m "Update $RECOMPILED_DIR with model from build-all-tinygrad-models" || echo "No changes to commit"
git push origin main
- run: |
cd docs
git pull origin gh-pages
- name: update json
run: |
ARGS=""
[ -n "${{ inputs.set_min_version }}" ] && ARGS="$ARGS --set-min-version \"${{ inputs.set_min_version }}\""
ARGS="$ARGS --sort-by-date"
ARGS="$ARGS --tinygrad-ref \"${{ needs.setup.outputs.tinygrad_ref }}\""
eval python3 docs/json_parser.py \
--json-path "$JSON_FILE" \
--recompiled-dir "gitlab_docs/models/$RECOMPILED_DIR" \
$ARGS
- name: Push updated json to GitHub
run: |
cd docs
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git checkout gh-pages
git add docs/"$(basename $JSON_FILE)"
git commit -m "Update $(basename $JSON_FILE) after recompiling model" || echo "No changes to commit"
git push origin gh-pages
@@ -1,238 +0,0 @@
name: Build Single Tinygrad Model and Push
on:
workflow_call:
inputs:
upstream_branch:
description: 'Upstream commit to build from'
required: true
type: string
custom_name:
description: 'Custom name for the model (no date, only name)'
required: false
type: string
recompiled_dir:
description: 'Existing recompiled directory number (e.g. 3 for recompiled3)'
required: true
type: string
json_version:
description: 'driving_models version number to update (e.g. 5 for driving_models_v5.json)'
required: true
type: string
artifact_suffix:
description: 'Suffix for artifact name'
required: false
type: string
default: ''
is_20hz:
description: 'Is this a 20Hz model'
required: false
type: boolean
default: true
bypass_push:
description: 'Bypass pushing to GitLab for build-all'
required: false
default: true
type: boolean
workflow_dispatch:
inputs:
upstream_branch:
description: 'Upstream commit to build from'
required: true
type: string
custom_name:
description: 'Custom name for the model (no date, only name)'
required: false
type: string
is_20hz:
description: 'Is this a 20Hz model'
required: false
type: boolean
default: true
recompiled_dir:
description: 'Existing recompiled directory number (e.g. 3 for recompiled3)'
required: true
type: string
json_version:
description: 'driving_models version number to update (e.g. 5 for driving_models_v5.json)'
required: true
type: string
model_folder:
description: 'Model folder'
type: choice
default: 'None'
options:
- None
- Simple Plan Models
- Space Lab Models
- TR Models
- DTR Models
- Custom Merge Models
- FOF series models
- Other
custom_model_folder:
description: 'Custom model folder name (if "Other" selected)'
required: false
type: string
generation:
description: 'Model generation'
required: false
type: string
version:
description: 'Minimum selector version'
required: false
type: string
env:
RECOMPILED_DIR: recompiled${{ inputs.recompiled_dir }}
JSON_FILE: docs/docs/driving_models_v${{ inputs.json_version }}.json
jobs:
build_model:
uses: ./.github/workflows/sunnypilot-build-model.yaml
with:
upstream_branch: ${{ inputs.upstream_branch }}
custom_name: ${{ inputs.custom_name || inputs.upstream_branch }}
is_20hz: ${{ inputs.is_20hz }}
artifact_suffix: ${{ inputs.artifact_suffix }}
secrets: inherit
publish_model:
if: ${{ !inputs.bypass_push && !cancelled() }}
concurrency:
group: gitlab-push-${{ inputs.recompiled_dir }}
cancel-in-progress: false
needs: build_model
runs-on: ubuntu-latest
steps:
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.GITLAB_SSH_PRIVATE_KEY }}
- name: Add GitLab.com SSH key to known_hosts
run: |
mkdir -p ~/.ssh
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Clone GitLab docs repo
env:
GIT_SSH_COMMAND: 'ssh -o UserKnownHostsFile=~/.ssh/known_hosts'
run: |
echo "Cloning GitLab"
git clone --depth 1 --filter=tree:0 --sparse git@gitlab.com:sunnypilot/public/${{ vars.MODELS_GITLAB }} gitlab_docs
cd gitlab_docs
echo "checkout models/${RECOMPILED_DIR}"
git sparse-checkout set --no-cone models/${RECOMPILED_DIR}
git checkout main
cd ..
- name: Checkout docs repo
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot-models
ref: gh-pages
path: docs
ssh-key: ${{ secrets.CI_SUNNYPILOT_DOCS_PRIVATE_KEY }}
- name: Validate recompiled dir and JSON version
run: |
if [ ! -d "gitlab_docs/models/$RECOMPILED_DIR" ]; then
echo "Recompiled dir $RECOMPILED_DIR does not exist in GitLab repo"
exit 1
fi
if [ ! -f "$JSON_FILE" ]; then
echo "JSON file $JSON_FILE does not exist!"
exit 1
fi
- name: Download artifact name file
uses: actions/download-artifact@v4
with:
name: artifact-name-${{ inputs.custom_name || inputs.upstream_branch }}
path: artifact_name
- name: Read artifact name
id: read-artifact-name
run: |
ARTIFACT_NAME=$(cat artifact_name/artifact_name.txt)
echo "artifact_name=$ARTIFACT_NAME" >> $GITHUB_OUTPUT
- name: Download and extract model artifact
uses: actions/download-artifact@v4
with:
name: ${{ steps.read-artifact-name.outputs.artifact_name }}
path: output
- name: Remove unwanted files
run: |
find output -type f -name 'dmonitoring_model_tinygrad.pkl' -delete
find output -type f -name 'dmonitoring_model.onnx' -delete
- name: Copy model artifact(s) to GitLab recompiled dir
env:
ARTIFACT_NAME: ${{ steps.read-artifact-name.outputs.artifact_name }}
run: |
ARTIFACT_DIR="gitlab_docs/models/${RECOMPILED_DIR}/${ARTIFACT_NAME}"
mkdir -p "$ARTIFACT_DIR"
for path in output/*; do
if [ "$(basename "$path")" = "artifact_name.txt" ]; then
continue
fi
name="$(basename "$path")"
if [ -d "$path" ]; then
mkdir -p "$ARTIFACT_DIR/$name"
cp -r "$path"/* "$ARTIFACT_DIR/$name/"
echo "Copied dir $name -> $ARTIFACT_DIR/$name"
else
cp "$path" "$ARTIFACT_DIR/"
echo "Copied file $name -> $ARTIFACT_DIR/"
fi
done
- name: Push recompiled dir to GitLab
env:
GITLAB_SSH_PRIVATE_KEY: ${{ secrets.GITLAB_SSH_PRIVATE_KEY }}
run: |
cd gitlab_docs
git checkout main
git pull origin main
for d in models/"$RECOMPILED_DIR"/*/; do
git sparse-checkout add "$d"
done
git add models/"$RECOMPILED_DIR"
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git commit -m "Create/Update $RECOMPILED_DIR with new/updated model from build-single-tinygrad-model" || echo "No changes to commit"
git push origin main
- run: |
cd docs
git pull origin gh-pages
- name: Run json_parser.py to update JSON
run: |
FOLDER="${{ inputs.model_folder }}"
if [ "$FOLDER" = "Other" ]; then
FOLDER="${{ inputs.custom_model_folder }}"
fi
ARGS=""
if [ "$FOLDER" != "None" ] && [ -n "$FOLDER" ]; then
ARGS="$ARGS --model-folder \"$FOLDER\""
fi
[ -n "${{ inputs.generation }}" ] && ARGS="$ARGS --generation \"${{ inputs.generation }}\""
[ -n "${{ inputs.version }}" ] && ARGS="$ARGS --version \"${{ inputs.version }}\""
eval python3 docs/json_parser.py \
--json-path "$JSON_FILE" \
--recompiled-dir "gitlab_docs/models/$RECOMPILED_DIR" \
--sort-by-date \
$ARGS
- name: Push updated JSON to GitHub docs repo
run: |
cd docs
git config --global user.name "GitHub Action"
git config --global user.email "action@github.com"
git checkout gh-pages
git add docs/"$(basename $JSON_FILE)"
git commit -m "Update $(basename $JSON_FILE) after recompiling model" || echo "No changes to commit"
git push origin gh-pages
-84
View File
@@ -1,84 +0,0 @@
name: cereal validation
on:
push:
branches:
- master
pull_request:
paths:
- 'cereal/**'
workflow_dispatch:
workflow_call:
inputs:
run_number:
default: '1'
required: true
type: string
concurrency:
group: cereal-validation-ci-run-${{ inputs.run_number }}-${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && github.run_id || github.head_ref || github.ref }}-${{ github.workflow }}-${{ github.event_name }}
cancel-in-progress: true
env:
CI: 1
jobs:
validate_cereal_with_upstream:
name: Validate cereal with Upstream
runs-on: ubuntu-24.04
steps:
- name: Checkout sunnypilot cereal
uses: actions/checkout@v6
with:
sparse-checkout: cereal
- name: Init sunnypilot opendbc submodule
run: git submodule update --init --depth 1 opendbc_repo
- name: Checkout upstream openpilot
uses: actions/checkout@v6
with:
repository: 'commaai/openpilot'
path: upstream_openpilot
ref: "refs/heads/master"
- name: Init upstream opendbc submodule
working-directory: upstream_openpilot
run: git submodule update --init --depth 1 opendbc_repo
- name: Locate upstream capnp paths
id: locate-capnp
run: |
CEREAL_DIR=$(find upstream_openpilot -maxdepth 4 -name log.capnp -path '*/cereal/log.capnp' -printf '%h\n' -quit)
if [ -z "$CEREAL_DIR" ]; then
echo "::error::Could not locate cereal/log.capnp in upstream openpilot"
exit 1
fi
echo "cereal_dir=$CEREAL_DIR" >> "$GITHUB_OUTPUT"
echo "Found upstream cereal at: $CEREAL_DIR"
IMPORT_ARGS=""
CAR_CAPNP=$(find upstream_openpilot -maxdepth 5 -name car.capnp -path '*/opendbc/car/car.capnp' -printf '%h\n' -quit)
if [ -n "$CAR_CAPNP" ]; then
IMPORT_ARGS="-I $CAR_CAPNP"
echo "Found car.capnp at: $CAR_CAPNP"
fi
echo "import_args=$IMPORT_ARGS" >> "$GITHUB_OUTPUT"
- name: Install uv
run: pip install uv
- name: Generate sunnypilot schema
run: |
PYCAPNP_VER=$(python3 -c "import re; m=re.search(r'name = \"pycapnp\"\nversion = \"([^\"]+)\"', open('uv.lock').read()); print(m.group(1))")
uv run --isolated --with "pycapnp==${PYCAPNP_VER}" \
python3 cereal/messaging/tests/validate_sp_cereal_upstream.py \
-g -f /tmp/sp_schema.json --cereal-dir cereal
- name: Validate against upstream
run: |
PYCAPNP_VER=$(python3 -c "import re; m=re.search(r'name = \"pycapnp\"\nversion = \"([^\"]+)\"', open('uv.lock').read()); print(m.group(1))")
uv run --isolated --with "pycapnp==${PYCAPNP_VER}" \
python3 cereal/messaging/tests/validate_sp_cereal_upstream.py \
-r -f /tmp/sp_schema.json --cereal-dir ${{ steps.locate-capnp.outputs.cereal_dir }} \
${{ steps.locate-capnp.outputs.import_args }}
+101
View File
@@ -0,0 +1,101 @@
name: weekly CI test report
on:
schedule:
- cron: '37 9 * * 1' # 9:37AM UTC -> 2:37AM PST every monday
workflow_dispatch:
inputs:
ci_runs:
description: 'The amount of runs to trigger in CI test report'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
env:
CI_RUNS: ${{ github.event.inputs.ci_runs || '50' }}
jobs:
setup:
if: github.repository == 'sunnypilot/sunnypilot'
runs-on: ubuntu-latest
outputs:
ci_runs: ${{ steps.ci_runs_setup.outputs.matrix }}
steps:
- id: ci_runs_setup
name: CI_RUNS=${{ env.CI_RUNS }}
run: |
matrix=$(python3 -c "import json; print(json.dumps({ 'run_number' : list(range(${{ env.CI_RUNS }})) }))")
echo "matrix=$matrix" >> $GITHUB_OUTPUT
ci_matrix_run:
needs: [ setup ]
strategy:
fail-fast: false
matrix: ${{fromJSON(needs.setup.outputs.ci_runs)}}
uses: sunnypilot/sunnypilot/.github/workflows/ci_weekly_run.yaml@master
with:
run_number: ${{ matrix.run_number }}
report:
needs: [ci_matrix_run]
runs-on: ubuntu-latest
if: always()
steps:
- name: Get job results
uses: actions/github-script@v7
id: get-job-results
with:
script: |
const jobs = await github
.paginate("GET /repos/{owner}/{repo}/actions/runs/{run_id}/attempts/{attempt}/jobs", {
owner: "commaai",
repo: "${{ github.event.repository.name }}",
run_id: "${{ github.run_id }}",
attempt: "${{ github.run_attempt }}",
})
var report = {}
jobs.slice(1, jobs.length-1).forEach(job => {
if (job.conclusion === "skipped") return;
const jobName = job.name.split(" / ")[2];
const runRegex = /\((.*?)\)/;
const run = job.name.match(runRegex)[1];
report[jobName] = report[jobName] || { successes: [], failures: [], canceled: [] };
switch (job.conclusion) {
case "success":
report[jobName].successes.push({ "run_number": run, "link": job.html_url}); break;
case "failure":
report[jobName].failures.push({ "run_number": run, "link": job.html_url }); break;
case "canceled":
report[jobName].canceled.push({ "run_number": run, "link": job.html_url }); break;
}
});
return JSON.stringify({"jobs": report});
- name: Add job results to summary
env:
JOB_RESULTS: ${{ fromJSON(steps.get-job-results.outputs.result) }}
run: |
cat <<EOF >> template.html
<table>
<thead>
<tr>
<th></th>
<th>Job</th>
<th>✅ Passing</th>
<th>❌ Failure Details</th>
</tr>
</thead>
<tbody>
{% for key in jobs.keys() %}<tr>
<td>{% for i in range(5) %}{% if i+1 <= (5 * jobs[key]["successes"]|length // ${{ env.CI_RUNS }}) %}🟩{% else %}🟥{% endif %}{% endfor%}</td>
<td>{{ key }}</td>
<td>{{ 100 * jobs[key]["successes"]|length // ${{ env.CI_RUNS }} }}%</td>
<td>{% if jobs[key]["failures"]|length > 0 %}<details>{% for failure in jobs[key]["failures"] %}<a href="{{ failure['link'] }}">Log for run #{{ failure['run_number'] }}</a><br>{% endfor %}</details>{% else %}{% endif %}</td>
</td>
</tr>{% endfor %}
</table>
EOF
pip install jinja2-cli
echo $JOB_RESULTS | jinja2 template.html > report.html
echo "# CI Test Report - ${{ env.CI_RUNS }} Runs" >> $GITHUB_STEP_SUMMARY
cat report.html >> $GITHUB_STEP_SUMMARY
+17
View File
@@ -0,0 +1,17 @@
name: weekly CI test run
on:
workflow_call:
inputs:
run_number:
required: true
type: string
concurrency:
group: ci-run-${{ inputs.run_number }}-${{ github.ref }}
cancel-in-progress: true
jobs:
selfdrive_tests:
uses: sunnypilot/sunnypilot/.github/workflows/selfdrive_tests.yaml@master
with:
run_number: ${{ inputs.run_number }}
@@ -0,0 +1,21 @@
name: 'compile openpilot'
runs:
using: "composite"
steps:
- shell: bash
name: Build openpilot with all flags
run: |
${{ env.RUN }} "scons -j$(nproc)"
${{ env.RUN }} "release/check-dirty.sh"
- shell: bash
name: Cleanup scons cache and rebuild
run: |
${{ env.RUN }} "rm -rf /tmp/scons_cache/* && \
scons -j$(nproc) --cache-populate"
- name: Save scons cache
uses: actions/cache/save@v4
if: (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/master-new')
with:
path: .ci_cache/scons_cache
key: scons-${{ runner.arch }}-${{ env.CACHE_COMMIT_DATE }}-${{ github.sha }}
-57
View File
@@ -1,57 +0,0 @@
name: diff report
on:
pull_request_target:
types: [opened, synchronize, reopened]
jobs:
comment:
name: comment
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
pull-requests: write
actions: read
steps:
- name: Wait for process replay
id: wait
continue-on-error: true
uses: lewagon/wait-on-check-action@1d57e2c51a58d812d2765e036a028b6bdb5a6154
with:
ref: ${{ github.event.pull_request.head.sha }}
check-name: process replay
repo-token: ${{ secrets.GITHUB_TOKEN }}
allowed-conclusions: success,failure
wait-interval: 20
- name: Get tests run ID
if: steps.wait.outcome == 'success'
id: tests-run
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
run_id=$(gh api \
"repos/${{ github.repository }}/actions/workflows/tests.yaml/runs?event=pull_request&head_sha=${{ github.event.pull_request.head.sha }}&per_page=1" \
--jq '.workflow_runs[0].id // empty')
if [ -z "$run_id" ]; then
echo "No tests.yaml run found for ${{ github.event.pull_request.head.sha }}" >&2
exit 1
fi
echo "run-id=$run_id" >> "$GITHUB_OUTPUT"
- name: Download diff
if: steps.wait.outcome == 'success'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
repository: ${{ github.repository }}
run-id: ${{ steps.tests-run.outputs.run-id }}
name: diff_report_${{ github.event.number }}
path: .
- name: Comment on PR
if: steps.wait.outcome == 'success'
uses: thollander/actions-comment-pull-request@24bffb9b452ba05a4f3f77933840a6a841d1b32b
with:
file-path: diff_report.txt
comment-tag: diff_report
pr-number: ${{ github.event.number }}
github-token: ${{ secrets.GITHUB_TOKEN }}
+6 -6
View File
@@ -22,19 +22,19 @@ jobs:
steps:
- uses: commaai/timeout@v1
- uses: actions/checkout@v7
- uses: actions/checkout@v4
with:
submodules: true
# Build
- name: Build docs
run: |
git lfs pull
pip install zensical
python scripts/docs.py build
# TODO: can we install just the "docs" dependency group without the normal deps?
pip install mkdocs
mkdocs build
# Push to docs.comma.ai
- uses: actions/checkout@v7
- uses: actions/checkout@v4
if: github.ref == 'refs/heads/master' && github.repository == 'sunnypilot/sunnypilot'
with:
path: openpilot-docs
@@ -45,7 +45,7 @@ jobs:
run: |
set -x
source tools/release/identity.sh
source release/identity.sh
cd openpilot-docs
git checkout --orphan tmp
+3 -54
View File
@@ -5,54 +5,14 @@ on:
types: [created, edited]
jobs:
cleanup-branches:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Delete stale Jenkins branches
uses: actions/github-script@v9
with:
script: |
const cutoff = Date.now() - 24 * 60 * 60 * 1000;
const prefixes = ['tmp-jenkins', '__jenkins'];
for await (const response of github.paginate.iterator(github.rest.repos.listBranches, {
owner: context.repo.owner,
repo: context.repo.repo,
per_page: 100,
})) {
for (const branch of response.data) {
if (!prefixes.some(p => branch.name.startsWith(p))) continue;
const { data: commit } = await github.rest.repos.getCommit({
owner: context.repo.owner,
repo: context.repo.repo,
ref: branch.commit.sha,
});
const commitDate = new Date(commit.commit.committer.date).getTime();
if (commitDate < cutoff) {
console.log(`Deleting branch: ${branch.name} (last commit: ${commit.commit.committer.date})`);
await github.rest.git.deleteRef({
owner: context.repo.owner,
repo: context.repo.repo,
ref: `heads/${branch.name}`,
});
}
}
}
# TODO: gc old branches in a separate job in this workflow
scan-comments:
runs-on: ubuntu-latest
if: ${{ github.event.issue.pull_request }}
permissions:
contents: write
issues: write
steps:
- name: Check for trigger phrase
id: check_comment
uses: actions/github-script@v9
uses: actions/github-script@v7
with:
script: |
const triggerPhrase = "trigger-jenkins";
@@ -72,7 +32,7 @@ jobs:
- name: Checkout repository
if: steps.check_comment.outputs.result == 'true'
uses: actions/checkout@v7
uses: actions/checkout@v4
with:
ref: refs/pull/${{ github.event.issue.number }}/head
@@ -83,14 +43,3 @@ jobs:
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git checkout -b tmp-jenkins-${{ github.event.issue.number }}
GIT_LFS_SKIP_PUSH=1 git push -f origin tmp-jenkins-${{ github.event.issue.number }}
- name: Delete trigger comment
if: steps.check_comment.outputs.result == 'true' && always()
uses: actions/github-script@v9
with:
script: |
await github.rest.issues.deleteComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: context.payload.comment.id,
});
+6 -6
View File
@@ -9,11 +9,11 @@ on:
- cron: '0 0 * * *' # Runs at 00:00 UTC every day
push:
branches:
- 'master'
- 'master-new'
pull_request:
branches:
- 'master'
workflow_dispatch: # enables manual triggering
- 'master-new'
workflow_dispatch: # enables manual triggering
inputs:
upstream_branch:
default: 'master'
@@ -23,14 +23,14 @@ jobs:
sync:
runs-on: ubuntu-latest
# Skip if PR is in draft mode
if: (github.event_name != 'pull_request' || (github.event_name == 'pull_request' && github.event.pull_request.draft == false)) && !github.event.pull_request.head.repo.fork
if: github.event_name != 'pull_request' || (github.event_name == 'pull_request' && github.event.pull_request.draft == false)
steps:
- name: Checkout Repository
uses: actions/checkout@v4
with:
repository: 'commaai/openpilot'
ref: ${{ inputs.upstream_branch }}
- name: LFS Fetch
run: |
git lfs fetch
@@ -48,7 +48,7 @@ jobs:
- name: Add GitLab public keys
run: |
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Ensure branch
run: |
if git symbolic-ref -q HEAD >/dev/null; then
-42
View File
@@ -1,42 +0,0 @@
name: "model review"
on:
pull_request:
types: [opened, reopened, synchronize]
paths:
- 'openpilot/selfdrive/modeld/models/*.onnx'
workflow_dispatch:
jobs:
comment:
permissions:
contents: read
pull-requests: write
runs-on: ubuntu-latest
if: github.repository == 'commaai/openpilot'
steps:
- name: Checkout
uses: actions/checkout@v7
with:
submodules: true
- name: Checkout master
uses: actions/checkout@v7
with:
ref: master
path: base
- run: git lfs pull
- run: cd base && git lfs pull
- name: scripts/reporter.py
id: report
run: |
echo "content<<EOF" >> $GITHUB_OUTPUT
echo "## Model Review" >> $GITHUB_OUTPUT
PYTHONPATH=${{ github.workspace }} MASTER_PATH=${{ github.workspace }}/base python scripts/reporter.py >> $GITHUB_OUTPUT
echo "EOF" >> $GITHUB_OUTPUT
- name: Post model report comment
uses: marocchino/sticky-pull-request-comment@0ea0beb66eb9baf113663a64ec522f60e49231c0
with:
header: model-review
message: ${{ steps.report.outputs.content }}
@@ -1,105 +0,0 @@
name: 'Post to Discourse'
description: 'Posts a message to a Discourse topic (existing or new)'
inputs:
discourse-url:
description: 'Discourse instance URL (e.g., https://discourse.example.com)'
required: true
api-key:
description: 'Discourse API key'
required: true
api-username:
description: 'Discourse API username'
required: true
topic-id:
description: 'Discourse topic ID to post to (use this OR category-id + title)'
required: false
category-id:
description: 'Category ID for new topic (required if topic-id not provided)'
required: false
title:
description: 'Title for new topic (required if topic-id not provided)'
required: false
message:
description: 'Message content (markdown supported)'
required: true
outputs:
post-number:
description: 'The post number in the topic'
value: ${{ steps.post.outputs.post_number }}
post-url:
description: 'Direct URL to the post'
value: ${{ steps.post.outputs.post_url }}
topic-id:
description: 'The topic ID (useful when creating a new topic)'
value: ${{ steps.post.outputs.topic_id }}
runs:
using: "composite"
steps:
- name: Post to Discourse
id: post
shell: bash
run: |
# Validate inputs
if [ -z "${{ inputs.topic-id }}" ] && ([ -z "${{ inputs.category-id }}" ] || [ -z "${{ inputs.title }}" ]); then
echo "❌ Error: Must provide either topic-id OR both category-id and title"
exit 1
fi
if [ -n "${{ inputs.topic-id }}" ] && ([ -n "${{ inputs.category-id }}" ] || [ -n "${{ inputs.title }}" ]); then
echo "⚠️ Warning: Both topic-id and category-id/title provided. Will post to existing topic."
fi
# Determine if creating new topic or posting to existing
if [ -n "${{ inputs.topic-id }}" ]; then
echo "📝 Posting to existing topic ID: ${{ inputs.topic-id }}"
# Create JSON payload for posting to existing topic
PAYLOAD=$(jq -n \
--arg content '${{ inputs.message }}' \
--arg topic_id "${{ inputs.topic-id }}" \
'{topic_id: $topic_id, raw: $content}')
else
echo "✨ Creating new topic: ${{ inputs.title }}"
# Create JSON payload for new topic
PAYLOAD=$(jq -n \
--arg content '${{ inputs.message }}' \
--arg title "${{ inputs.title }}" \
--arg category "${{ inputs.category-id }}" \
'{title: $title, category: ($category | tonumber), raw: $content}')
fi
# Post to Discourse
RESPONSE=$(curl -s -w "\n%{http_code}" \
-X POST "${{ inputs.discourse-url }}/posts.json" \
-H "Content-Type: application/json" \
-H "Api-Key: ${{ inputs.api-key }}" \
-H "Api-Username: ${{ inputs.api-username }}" \
-d "$PAYLOAD")
HTTP_CODE=$(echo "$RESPONSE" | tail -n1)
BODY=$(echo "$RESPONSE" | sed '$d')
if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then
echo "✅ Successfully posted to Discourse!"
POST_NUMBER=$(echo "$BODY" | jq -r '.post_number // "unknown"')
TOPIC_ID=$(echo "$BODY" | jq -r '.topic_id // "${{ inputs.topic-id }}"')
POST_URL="${{ inputs.discourse-url }}/t/${TOPIC_ID}/${POST_NUMBER}"
echo "post_number=${POST_NUMBER}" >> $GITHUB_OUTPUT
echo "post_url=${POST_URL}" >> $GITHUB_OUTPUT
echo "topic_id=${TOPIC_ID}" >> $GITHUB_OUTPUT
echo "Topic ID: ${TOPIC_ID}"
echo "Post number: ${POST_NUMBER}"
echo "URL: ${POST_URL}"
else
echo "❌ Failed to post to Discourse"
echo "HTTP Code: ${HTTP_CODE}"
echo "Response: ${BODY}"
exit 1
fi
+3 -3
View File
@@ -6,7 +6,7 @@ on:
env:
DOCKER_LOGIN: docker login ghcr.io -u ${{ github.actor }} -p ${{ secrets.GITHUB_TOKEN }}
BUILD: release/ci/docker_build_sp.sh
BUILD: selfdrive/test/docker_build.sh prebuilt
jobs:
build_prebuilt:
@@ -28,8 +28,8 @@ jobs:
wait-interval: 30
running-workflow-name: 'build prebuilt'
repo-token: ${{ secrets.GITHUB_TOKEN }}
check-regexp: ^((?!.*(build master-ci|create badges).*).)*$
- uses: actions/checkout@v6
check-regexp: ^((?!.*(build master-ci).*).)*$
- uses: actions/checkout@v4
with:
submodules: true
- run: git lfs pull
-28
View File
@@ -1,28 +0,0 @@
name: Release Drafter
on:
push:
branches:
- master
tags:
- 'v*'
pull_request_target:
types: [opened, reopened, synchronize]
workflow_dispatch:
permissions:
contents: read
jobs:
update_release_draft:
permissions:
contents: write
pull-requests: write
runs-on: ubuntu-latest
steps:
- uses: release-drafter/release-drafter@v6
with:
config-name: release-drafter.yml
prerelease: ${{ !startsWith(github.ref, 'refs/tags/v') }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+33 -10
View File
@@ -5,27 +5,50 @@ on:
workflow_dispatch:
jobs:
build___nightly:
name: build __nightly
build_masterci:
name: build master-ci
env:
TARGET_DIR: /tmp/openpilot
ImageOS: ubuntu20
container:
image: ghcr.io/commaai/openpilot-base:latest
runs-on: ubuntu-latest
if: github.repository == 'sunnypilot/sunnypilot'
permissions:
checks: read
contents: write
steps:
- name: Install wait-on-check-action dependencies
run: |
sudo apt-get update
sudo apt-get install -y libyaml-dev
- name: Wait for green check mark
if: ${{ github.event_name == 'schedule' }}
uses: lewagon/wait-on-check-action@1d57e2c51a58d812d2765e036a028b6bdb5a6154
if: ${{ github.event_name != 'workflow_dispatch' }}
uses: lewagon/wait-on-check-action@ccfb013c15c8afb7bf2b7c028fb74dc5a068cccc
with:
ref: master
wait-interval: 30
running-workflow-name: 'build __nightly'
running-workflow-name: 'build master-ci'
repo-token: ${{ secrets.GITHUB_TOKEN }}
check-regexp: ^((?!.*(build prebuilt|create badges).*).)*$
- uses: actions/checkout@v4 # checkout version > v4 breaks nightly release script
check-regexp: ^((?!.*(build prebuilt).*).)*$
- uses: actions/checkout@v4
with:
submodules: true
fetch-depth: 0
- run: ./tools/op.sh setup
- name: Push __nightly
run: BRANCH=__nightly tools/release/build_stripped.sh
- name: Pull LFS
run: |
git config --global --add safe.directory '*'
git lfs pull
- name: Build master-ci
run: |
release/build_devel.sh
- name: Run tests
run: |
export PYTHONPATH=$TARGET_DIR
cd $TARGET_DIR
scons -j$(nproc)
pytest -n logical selfdrive/car/tests/test_car_interfaces.py
- name: Push master-ci
run: |
unset TARGET_DIR
BRANCH=master-ci release/build_devel.sh
+15 -91
View File
@@ -3,119 +3,43 @@ name: repo maintenance
on:
schedule:
- cron: "0 14 * * 1" # every Monday at 2am UTC (6am PST)
pull_request:
types: [closed]
workflow_dispatch:
env:
PYTHONPATH: ${{ github.workspace }}
jobs:
package_updates:
name: package_updates
runs-on: ubuntu-latest
if: >-
github.repository == 'sunnypilot/sunnypilot' &&
(github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')
container:
image: ghcr.io/commaai/openpilot-base:latest
if: github.repository == 'sunnypilot/sunnypilot'
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v4
with:
submodules: true
- run: ./tools/op.sh setup
- name: uv lock
run: uv lock --upgrade
- name: uv pip tree
id: pip_tree
run: |
echo 'PIP_TREE<<EOF' >> $GITHUB_OUTPUT
uv pip tree >> $GITHUB_OUTPUT
echo 'EOF' >> $GITHUB_OUTPUT
- name: venv size
id: venv_size
run: |
echo 'VENV_SIZE<<EOF' >> $GITHUB_OUTPUT
echo "Total: $(du -sh .venv | cut -f1)" >> $GITHUB_OUTPUT
echo "" >> $GITHUB_OUTPUT
echo "Top 10 by size:" >> $GITHUB_OUTPUT
du -sh .venv/lib/python*/site-packages/* 2>/dev/null \
| grep -v '\.dist-info' \
| grep -v '__pycache__' \
| sort -rh \
| head -10 \
| while IFS=$'\t' read size path; do echo "$size ${path##*/}"; done >> $GITHUB_OUTPUT
echo 'EOF' >> $GITHUB_OUTPUT
python3 -m ensurepip --upgrade
pip3 install uv
uv lock --upgrade
- name: bump submodules
run: |
git config submodule.msgq.update none
git config submodule.rednose_repo.update none
git config submodule.teleoprtc_repo.update none
git config submodule.tinygrad.update none
git submodule update --remote
git config --global --add safe.directory '*'
git -c submodule."tinygrad".update=none submodule update --remote
git add .
- name: update car docs
run: |
python openpilot/selfdrive/car/docs.py
scons -j$(nproc) --minimal opendbc_repo
PYTHONPATH=. python selfdrive/car/docs.py
git add docs/CARS.md
- name: Create Pull Request
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1
uses: peter-evans/create-pull-request@9153d834b60caba6d51c9b9510b087acf9f33f83
with:
author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
token: ${{ github.repository == 'commaai/openpilot' && secrets.ACTIONS_CREATE_PR_PAT || secrets.GITHUB_TOKEN }}
author: Vehicle Researcher <user@comma.ai>
token: ${{ secrets.ACTIONS_CREATE_PR_PAT }}
commit-message: Update Python packages
title: '[bot] Update Python packages'
branch: auto-package-updates
base: master
delete-branch: true
body: |
Automatic PR from repo-maintenance -> package_updates
```
$ du -sh .venv && du -sh .venv/lib/python*/site-packages/* | sort -rh | head -10
${{ steps.venv_size.outputs.VENV_SIZE }}
```
```
$ uv pip tree
${{ steps.pip_tree.outputs.PIP_TREE }}
```
body: 'Automatic PR from repo-maintenance -> package_updates'
labels: bot
cleanup_closed_branches:
if: github.repository == 'commaai/openpilot'
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: read
steps:
- uses: actions/github-script@v9
with:
script: |
const { owner, repo } = context.repo;
const upstream = `${owner}/${repo}`;
for await (const response of github.paginate.iterator(github.rest.pulls.list, {
owner,
repo,
state: 'closed',
per_page: 100,
})) {
for (const pr of response.data) {
if (pr.head.repo?.full_name !== upstream) continue;
const branch = pr.head.ref;
try {
await github.rest.git.deleteRef({
owner,
repo,
ref: `heads/${branch}`,
});
console.log(`Deleted branch ${branch} (PR #${pr.number})`);
} catch (error) {
if (error.status === 422 || error.status === 403) {
console.log(`Skipping branch ${branch} (PR #${pr.number}): ${error.message}`);
continue;
}
throw error;
}
}
}
+357
View File
@@ -0,0 +1,357 @@
name: selfdrive
on:
push:
branches:
- master
- master-new
pull_request:
workflow_dispatch:
workflow_call:
inputs:
run_number:
default: '1'
required: true
type: string
concurrency:
group: selfdrive-tests-ci-run-${{ inputs.run_number }}-${{ github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/master-new') && github.run_id || github.head_ref || github.ref }}-${{ github.workflow }}-${{ github.event_name }}
cancel-in-progress: true
env:
REPORT_NAME: report-${{ inputs.run_number || '1' }}-${{ github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/master-new') && 'master' || github.event.number }}
PYTHONWARNINGS: error
BASE_IMAGE: openpilot-base
AZURE_TOKEN: ${{ secrets.AZURE_COMMADATACI_OPENPILOTCI_TOKEN }}
DOCKER_LOGIN: docker login ghcr.io -u ${{ github.actor }} -p ${{ secrets.GITHUB_TOKEN }}
BUILD: selfdrive/test/docker_build.sh base
RUN: docker run --shm-size 2G -v $PWD:/tmp/openpilot -w /tmp/openpilot -e CI=1 -e PYTHONWARNINGS=error -e FILEREADER_CACHE=1 -e PYTHONPATH=/tmp/openpilot -e NUM_JOBS -e JOB_ID -e GITHUB_ACTION -e GITHUB_REF -e GITHUB_HEAD_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_RUN_ID -v $GITHUB_WORKSPACE/.ci_cache/scons_cache:/tmp/scons_cache -v $GITHUB_WORKSPACE/.ci_cache/comma_download_cache:/tmp/comma_download_cache -v $GITHUB_WORKSPACE/.ci_cache/openpilot_cache:/tmp/openpilot_cache $BASE_IMAGE /bin/bash -c
PYTEST: pytest --continue-on-collection-errors --cov --cov-report=xml --cov-append --durations=0 --durations-min=5 --hypothesis-seed 0 -n logical
jobs:
build_release:
if: github.repository == 'commaai/openpilot' # build_release blocked for the time being to only comma as we may have a different process.
name: build release
runs-on:
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
env:
STRIPPED_DIR: /tmp/releasepilot
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Getting LFS files
uses: nick-fields/retry@7152eba30c6575329ac0576536151aca5a72780e
with:
timeout_minutes: 2
max_attempts: 3
command: git lfs pull
- name: Build devel
timeout-minutes: 1
run: TARGET_DIR=$STRIPPED_DIR release/build_devel.sh
- uses: ./.github/workflows/setup-with-retry
- name: Check submodules
if: github.repository == 'sunnypilot/sunnypilot'
timeout-minutes: 3
run: release/check-submodules.sh
- name: Build openpilot and run checks
timeout-minutes: ${{ ((steps.restore-scons-cache.outputs.cache-hit == 'true') && 10 || 30) }} # allow more time when we missed the scons cache
run: |
cd $STRIPPED_DIR
${{ env.RUN }} "python3 system/manager/build.py"
- name: Run tests
timeout-minutes: 1
run: |
cd $STRIPPED_DIR
${{ env.RUN }} "release/check-dirty.sh"
build:
runs-on:
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Setup docker push
if: github.ref == 'refs/heads/master' && github.event_name != 'pull_request' && github.repository == 'commaai/openpilot'
run: |
echo "PUSH_IMAGE=true" >> "$GITHUB_ENV"
$DOCKER_LOGIN
- uses: ./.github/workflows/setup-with-retry
- uses: ./.github/workflows/compile-openpilot
timeout-minutes: 30
build_mac:
name: build macOS
runs-on: ${{ ((github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-macos-8x14' || 'macos-latest' }}
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Homebrew cache
uses: ./.github/workflows/auto-cache
with:
path: ~/Library/Caches/Homebrew
key: build_macos_${{ hashFiles('.github/workflows/selfdrive_tests.yaml') }}
- name: Install dependencies
run: ./tools/mac_setup.sh
env:
# package install has DeprecationWarnings
PYTHONWARNINGS: default
- run: git lfs pull
- run: echo "CACHE_COMMIT_DATE=$(git log -1 --pretty='format:%cd' --date=format:'%Y-%m-%d-%H:%M')" >> $GITHUB_ENV
- name: Getting scons cache
uses: ./.github/workflows/auto-cache
with:
path: /tmp/scons_cache
key: build_macos_${{ hashFiles('.github/workflows/selfdrive_tests.yaml') }}
- name: Building openpilot
run: . .venv/bin/activate && scons -j$(nproc)
static_analysis:
name: static analysis
runs-on:
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
env:
PYTHONWARNINGS: default
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: ./.github/workflows/setup-with-retry
- name: Static analysis
timeout-minutes: 1
run: ${{ env.RUN }} "scripts/lint/lint.sh"
unit_tests:
name: unit tests
runs-on:
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: ./.github/workflows/setup-with-retry
- name: Build openpilot
run: ${{ env.RUN }} "scons -j$(nproc)"
- name: Run unit tests
timeout-minutes: ${{ contains(runner.name, 'nsc') && 1 || 20 }}
run: |
${{ env.RUN }} "$PYTEST --collect-only -m 'not slow' &> /dev/null && \
MAX_EXAMPLES=1 $PYTEST -m 'not slow' && \
./selfdrive/ui/tests/create_test_translations.sh && \
QT_QPA_PLATFORM=offscreen ./selfdrive/ui/tests/test_translations && \
chmod -R 777 /tmp/comma_download_cache"
- name: "Upload coverage to Codecov"
uses: codecov/codecov-action@v4
with:
name: ${{ github.job }}
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
process_replay:
name: process replay
runs-on:
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: ./.github/workflows/setup-with-retry
- name: Cache test routes
id: dependency-cache
uses: actions/cache@v4
with:
path: .ci_cache/comma_download_cache
key: proc-replay-${{ hashFiles('selfdrive/test/process_replay/ref_commit', 'selfdrive/test/process_replay/test_processes.py') }}
- name: Build openpilot
run: |
${{ env.RUN }} "scons -j$(nproc)"
- name: Run replay
timeout-minutes: ${{ contains(runner.name, 'nsc') && (steps.dependency-cache.outputs.cache-hit == 'true') && 1 || 20 }}
run: |
${{ env.RUN }} "coverage run selfdrive/test/process_replay/test_processes.py -j$(nproc) && \
chmod -R 777 /tmp/comma_download_cache && \
coverage combine && \
coverage xml"
- name: Print diff
id: print-diff
if: always()
run: cat selfdrive/test/process_replay/diff.txt
- uses: actions/upload-artifact@v4
if: always()
continue-on-error: true
with:
name: process_replay_diff.txt
path: selfdrive/test/process_replay/diff.txt
- name: Upload reference logs
if: ${{ failure() && steps.print-diff.outcome == 'success' && github.repository == 'commaai/openpilot' && env.AZURE_TOKEN != '' }}
run: |
${{ env.RUN }} "unset PYTHONWARNINGS && AZURE_TOKEN='$AZURE_TOKEN' python3 selfdrive/test/process_replay/test_processes.py -j$(nproc) --upload-only"
- name: Run regen
if: false
timeout-minutes: 4
run: |
${{ env.RUN }} "ONNXCPU=1 $PYTEST selfdrive/test/process_replay/test_regen.py && \
chmod -R 777 /tmp/comma_download_cache"
- name: "Upload coverage to Codecov"
uses: codecov/codecov-action@v4
with:
name: ${{ github.job }}
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
test_cars:
name: cars
runs-on:
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
strategy:
fail-fast: false
matrix:
job: [0, 1, 2, 3]
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: ./.github/workflows/setup-with-retry
- name: Cache test routes
id: routes-cache
uses: actions/cache@v4
with:
path: .ci_cache/comma_download_cache
key: car_models-${{ hashFiles('selfdrive/car/tests/test_models.py', 'opendbc/car/tests/routes.py') }}-${{ matrix.job }}
- name: Build openpilot
run: ${{ env.RUN }} "scons -j$(nproc)"
- name: Test car models
timeout-minutes: ${{ contains(runner.name, 'nsc') && (steps.routes-cache.outputs.cache-hit == 'true') && 1 || 20 }}
run: |
${{ env.RUN }} "MAX_EXAMPLES=1 $PYTEST selfdrive/car/tests/test_models.py && \
chmod -R 777 /tmp/comma_download_cache"
env:
NUM_JOBS: 4
JOB_ID: ${{ matrix.job }}
- name: "Upload coverage to Codecov"
uses: codecov/codecov-action@v4
with:
name: ${{ github.job }}-${{ matrix.job }}
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
car_docs_diff:
name: PR comments
runs-on: ubuntu-latest
#if: github.event_name == 'pull_request'
if: false # TODO: run this in opendbc?
steps:
- uses: actions/checkout@v4
with:
submodules: true
ref: ${{ github.event.pull_request.base.ref }}
- run: git lfs pull
- uses: ./.github/workflows/setup-with-retry
- name: Get base car info
run: |
${{ env.RUN }} "scons -j$(nproc) && python3 selfdrive/debug/dump_car_docs.py --path /tmp/openpilot_cache/base_car_docs"
sudo chown -R $USER:$USER ${{ github.workspace }}
- uses: actions/checkout@v4
with:
submodules: true
path: current
- run: cd current && git lfs pull
- name: Save car docs diff
id: save_diff
run: |
cd current
${{ env.RUN }} "scons -j$(nproc)"
output=$(${{ env.RUN }} "python3 selfdrive/debug/print_docs_diff.py --path /tmp/openpilot_cache/base_car_docs")
output="${output//$'\n'/'%0A'}"
echo "::set-output name=diff::$output"
- name: Find comment
if: ${{ env.AZURE_TOKEN != '' }}
uses: peter-evans/find-comment@3eae4d37986fb5a8592848f6a574fdf654e61f9e
id: fc
with:
issue-number: ${{ github.event.pull_request.number }}
body-includes: This PR makes changes to
- name: Update comment
if: ${{ steps.save_diff.outputs.diff != '' && env.AZURE_TOKEN != '' }}
uses: peter-evans/create-or-update-comment@71345be0265236311c031f5c7866368bd1eff043
with:
comment-id: ${{ steps.fc.outputs.comment-id }}
issue-number: ${{ github.event.pull_request.number }}
body: "${{ steps.save_diff.outputs.diff }}"
edit-mode: replace
- name: Delete comment
if: ${{ steps.fc.outputs.comment-id != '' && steps.save_diff.outputs.diff == '' && env.AZURE_TOKEN != '' }}
uses: actions/github-script@v7
with:
script: |
github.rest.issues.deleteComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: ${{ steps.fc.outputs.comment-id }}
})
simulator_driving:
name: simulator driving
runs-on:
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
if: (github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: ./.github/workflows/setup-with-retry
- name: Build openpilot
run: |
${{ env.RUN }} "scons -j$(nproc)"
- name: Driving test
timeout-minutes: 1
run: |
${{ env.RUN }} "source selfdrive/test/setup_xvfb.sh && \
source selfdrive/test/setup_vsound.sh && \
CI=1 pytest tools/sim/tests/test_metadrive_bridge.py"
create_ui_report:
# This job name needs to be the same as UI_JOB_NAME in ui_preview.yaml
name: Create UI Report
runs-on:
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: ./.github/workflows/setup-with-retry
- name: caching frames
id: frames-cache
uses: actions/cache@v4
with:
path: .ci_cache/comma_download_cache
key: ui_screenshots_test_${{ hashFiles('selfdrive/ui/tests/test_ui/run.py') }}
- name: Build openpilot
run: ${{ env.RUN }} "scons -j$(nproc)"
- name: Create Test Report
timeout-minutes: ${{ ((steps.frames-cache.outputs.cache-hit == 'true') && 2 || 4) }}
run: >
${{ env.RUN }} "PYTHONWARNINGS=ignore &&
source selfdrive/test/setup_xvfb.sh &&
CACHE_ROOT=/tmp/comma_download_cache python3 selfdrive/ui/tests/test_ui/run.py &&
chmod -R 777 /tmp/comma_download_cache"
- name: Upload Test Report
uses: actions/upload-artifact@v4
with:
name: ${{ env.REPORT_NAME }}
path: selfdrive/ui/tests/test_ui/report_1/screenshots
@@ -0,0 +1,37 @@
name: 'openpilot env setup, with retry on failure'
inputs:
docker_hub_pat:
description: 'Auth token for Docker Hub, required for BuildJet jobs'
required: false
default: ''
sleep_time:
description: 'Time to sleep between retries'
required: false
default: 30
runs:
using: "composite"
steps:
- id: setup1
uses: ./.github/workflows/setup
continue-on-error: true
with:
is_retried: true
- if: steps.setup1.outcome == 'failure'
shell: bash
run: sleep ${{ inputs.sleep_time }}
- id: setup2
if: steps.setup1.outcome == 'failure'
uses: ./.github/workflows/setup
continue-on-error: true
with:
is_retried: true
- if: steps.setup2.outcome == 'failure'
shell: bash
run: sleep ${{ inputs.sleep_time }}
- id: setup3
if: steps.setup2.outcome == 'failure'
uses: ./.github/workflows/setup
with:
is_retried: true
+56
View File
@@ -0,0 +1,56 @@
name: 'openpilot env setup'
inputs:
is_retried:
description: 'A mock param that asserts that we use the setup-with-retry instead of this action directly'
required: false
default: 'false'
runs:
using: "composite"
steps:
# assert that this action is retried using the setup-with-retry
- shell: bash
if: ${{ inputs.is_retried == 'false' }}
run: |
echo "You should not run this action directly. Use setup-with-retry instead"
exit 1
- shell: bash
name: No retries!
run: |
if [ "${{ github.run_attempt }}" -gt 1 ]; then
echo -e "\033[0;31m##################################################"
echo -e "\033[0;31m Retries not allowed! Fix the flaky test! "
echo -e "\033[0;31m##################################################\033[0m"
exit 1
fi
# do this after checkout to ensure our custom LFS config is used to pull from GitLab
- shell: bash
run: git lfs pull
# build cache
- id: date
shell: bash
run: echo "CACHE_COMMIT_DATE=$(git log -1 --pretty='format:%cd' --date=format:'%Y-%m-%d-%H:%M')" >> $GITHUB_ENV
- shell: bash
run: echo "$CACHE_COMMIT_DATE"
- id: scons-cache
uses: ./.github/workflows/auto-cache
with:
path: .ci_cache/scons_cache
key: scons-${{ runner.arch }}-${{ env.CACHE_COMMIT_DATE }}-${{ github.sha }}
restore-keys: |
scons-${{ runner.arch }}-${{ env.CACHE_COMMIT_DATE }}
scons-${{ runner.arch }}
# as suggested here: https://github.com/moby/moby/issues/32816#issuecomment-910030001
- id: normalize-file-permissions
shell: bash
name: Normalize file permissions to ensure a consistent docker build cache
run: |
find . -type f -executable -not -perm 755 -exec chmod 755 {} \;
find . -type f -not -executable -not -perm 644 -exec chmod 644 {} \;
# build our docker image
- shell: bash
run: eval ${{ env.BUILD }}
+3 -26
View File
@@ -5,15 +5,14 @@ on:
workflow_dispatch:
env:
DAYS_BEFORE_PR_CLOSE: 7
DAYS_BEFORE_PR_STALE: 24
DAYS_BEFORE_PR_STALE_DRAFT: 30
DAYS_BEFORE_PR_CLOSE: 2
DAYS_BEFORE_PR_STALE: 9
jobs:
stale:
runs-on: ubuntu-latest
steps:
- uses: actions/stale@v10
- uses: actions/stale@v9
with:
exempt-all-milestones: true
@@ -25,28 +24,6 @@ jobs:
exempt-pr-labels: "ignore stale,needs testing" # if wip or it needs testing from the community, don't mark as stale
days-before-pr-stale: ${{ env.DAYS_BEFORE_PR_STALE }}
days-before-pr-close: ${{ env.DAYS_BEFORE_PR_CLOSE }}
exempt-draft-pr: false
# issue config
days-before-issue-stale: -1 # ignore issues for now
# same as above, but give draft PRs more time
stale_drafts:
runs-on: ubuntu-latest
steps:
- uses: actions/stale@v10
with:
exempt-all-milestones: true
# pull request config
stale-pr-message: 'This PR has had no activity for ${{ env.DAYS_BEFORE_PR_STALE_DRAFT }} days. It will be automatically closed in ${{ env.DAYS_BEFORE_PR_CLOSE }} days if there is no activity.'
close-pr-message: 'This PR has been automatically closed due to inactivity. Feel free to re-open once activity resumes.'
stale-pr-label: stale
delete-branch: ${{ github.event.pull_request.head.repo.full_name == 'commaai/openpilot' }} # only delete branches on the main repo
exempt-pr-labels: "ignore stale,needs testing" # if wip or it needs testing from the community, don't mark as stale
days-before-pr-stale: ${{ env.DAYS_BEFORE_PR_STALE_DRAFT }}
days-before-pr-close: ${{ env.DAYS_BEFORE_PR_CLOSE }}
exempt-draft-pr: true
# issue config
days-before-issue-stale: -1 # ignore issues for now
+56 -190
View File
@@ -5,31 +5,8 @@ env:
OUTPUT_DIR: ${{ github.workspace }}/output
SCONS_CACHE_DIR: ${{ github.workspace }}/release/ci/scons_cache
UPSTREAM_REPO: "commaai/openpilot"
TINYGRAD_PATH: ${{ github.workspace }}/tinygrad_repo
MODELS_DIR: ${{ github.workspace }}/selfdrive/modeld/models
on:
workflow_call:
inputs:
upstream_branch:
description: 'Upstream branch to build from'
required: true
default: 'master'
type: string
custom_name:
description: 'Custom name for the model (no date, only name)'
required: false
type: string
is_20hz:
description: 'Is this a 20Hz model'
required: false
type: boolean
default: true
artifact_suffix:
description: 'Suffix for artifact name'
required: false
type: string
default: ''
workflow_dispatch:
inputs:
upstream_branch:
@@ -38,80 +15,41 @@ on:
default: 'master'
type: string
custom_name:
description: 'Custom name for the model (no date, only name)'
description: 'Custom name for the model'
required: false
type: string
file_name:
description: 'File name prefix for the model files'
required: false
type: string
is_20hz:
description: 'Is this a 20Hz model'
required: false
type: boolean
default: true
default: false
run-name: Build model [${{ inputs.custom_name || inputs.upstream_branch }}] from ref [${{ inputs.upstream_branch }}]
jobs:
get_model:
runs-on: ubuntu-latest
env:
REF: ${{ inputs.upstream_branch }}
outputs:
model_date: ${{ steps.commit-date.outputs.model_date }}
steps:
# Note: To allow dynamic models from both openpilot and sunnypilot (merges/mashups), we try commaai as default,
# and fallback to sunnypilot if the ref checkout fails.
- name: Checkout commaai/openpilot
id: checkout_upstream
continue-on-error: true
uses: actions/checkout@v4
with:
repository: commaai/openpilot
ref: ${{ inputs.upstream_branch }}
submodules: recursive
path: openpilot
- name: Fallback to sunnypilot/sunnypilot
if: steps.checkout_upstream.outcome == 'failure'
uses: actions/checkout@v4
with:
repository: sunnypilot/sunnypilot
ref: ${{ inputs.upstream_branch }}
submodules: recursive
path: openpilot
- name: Get commit date
id: commit-date
run: |
cd ${{ github.workspace }}/openpilot
commit_date=$(git log -1 --format=%cd --date=format:'%B %d, %Y')
echo "model_date=${commit_date}" >> $GITHUB_OUTPUT
cat $GITHUB_OUTPUT
- run: |
cd ${{ github.workspace }}/openpilot
git lfs pull
- name: 'Upload Artifact'
uses: actions/upload-artifact@v4
with:
name: models-${{ env.REF }}${{ inputs.artifact_suffix }}
path: ${{ github.workspace }}/openpilot/selfdrive/modeld/models/*.onnx
build_model:
runs-on: [self-hosted, tici]
needs: get_model
env:
MODEL_NAME: ${{ inputs.custom_name || inputs.upstream_branch }} (${{ needs.get_model.outputs.model_date }})
REF: ${{ inputs.upstream_branch }}
runs-on: self-hosted
steps:
- uses: actions/checkout@v4
with:
repository: ${{ env.UPSTREAM_REPO }}
ref: ${{ github.event.inputs.upstream_branch }}
submodules: recursive
- run: git lfs pull
- name: Cache SCons
uses: actions/cache@v4
with:
path: ${{env.SCONS_CACHE_DIR}}
key: scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-model-${{ github.sha }}
# Note: GitHub Actions enforces cache isolation between different build sources (PR builds, workflow dispatches, etc.)
# Note: GitHub Actions enforces cache isolation between different build sources (PR builds, workflow dispatches, etc.)
# for security. Only caches from the default branch are shared across all builds. This is by design and cannot be overridden.
restore-keys: |
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-model
@@ -122,146 +60,74 @@ jobs:
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}
- name: Set environment variables
id: set-env
run: |
# Set up common environment
source /etc/profile;
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
printenv >> $GITHUB_ENV
if [[ "${{ runner.debug }}" == "1" ]]; then
cat $GITHUB_OUTPUT
fi
- name: Setup build environment
run: |
mkdir -p "${BUILD_DIR}/"
sudo find $BUILD_DIR/ -mindepth 1 -delete
echo "Starting build stage..."
echo "BUILD_DIR: ${BUILD_DIR}"
echo "CI_DIR: ${CI_DIR}"
echo "VERSION: ${{ steps.set-env.outputs.version }}"
echo "UV_PROJECT_ENVIRONMENT: ${UV_PROJECT_ENVIRONMENT}"
echo "VIRTUAL_ENV: ${VIRTUAL_ENV}"
echo "-------"
if [[ "${{ runner.debug }}" == "1" ]]; then
printenv
fi
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
rm -rf ${{ env.MODELS_DIR }}/*.onnx
- name: Download model artifacts
uses: actions/download-artifact@v4
with:
name: models-${{ env.REF }}${{ inputs.artifact_suffix }}
path: ${{ github.workspace }}/selfdrive/modeld/models
- run: |
rm -f ${{ github.workspace }}/selfdrive/modeld/models/{dmonitoring_model,big_driving_policy,big_driving_vision}.onnx
echo "Building from: ${{ env.UPSTREAM_REPO }} branch: ${{ github.event.inputs.upstream_branch }}"
- name: Patch SConstruct to pass arbitrary cache
run: |
sed -i.bak 's#cache_dir =#default_cache_dir =#' ${{ github.workspace }}/SConstruct
printf '/default_cache_dir/a\\\ncache_dir = ARGUMENTS.get("cache_dir", default_cache_dir)\n' | sed -i.bak -f - ${{ github.workspace }}/SConstruct
cat ${{ github.workspace }}/SConstruct
- name: Build Model
run: |
source /etc/profile
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
export PYTHONPATH="${PYTHONPATH}:${{ env.TINYGRAD_PATH }}:${{ github.workspace }}"
COMPILE_MODELD="${{ github.workspace }}/sunnypilot/modeld_v2/compile_modeld.py"
MODEL_SIZE=$(python3 -c "from openpilot.common.transformations.model import MEDMODEL_INPUT_SIZE as s; print(f'{s[0]}x{s[1]}')")
CAMERA_RES=$(python3 -c "from openpilot.common.transformations.camera import _ar_ox_fisheye as a, _os_fisheye as o; print(f'{a.width}x{a.height} {o.width}x{o.height}')")
TG_FLAGS="DEV=QCOM IMAGE=1 FLOAT16=1 NOLOCALS=1 JIT_BATCH_SIZE=0 OPENPILOT_HACKS=1"
# Generate metadata for all ONNX files
find "${{ env.MODELS_DIR }}" -maxdepth 1 -name '*.onnx' | while IFS= read -r onnx_file; do
echo "Generating metadata: $onnx_file"
env ${TG_FLAGS} python3 "${{ env.MODELS_DIR }}/../get_model_metadata.py" "$onnx_file" || true
done
# Detect model type and build compile args
VISION_ONNX="${{ env.MODELS_DIR }}/driving_vision.onnx"
POLICY_ONNX="${{ env.MODELS_DIR }}/driving_policy.onnx"
OFF_POLICY_ONNX="${{ env.MODELS_DIR }}/driving_off_policy.onnx"
ON_POLICY_ONNX="${{ env.MODELS_DIR }}/driving_on_policy.onnx"
SUPERCOMBO_ONNX="${{ env.MODELS_DIR }}/supercombo.onnx"
MODEL_TYPE="" ONNX_ARGS="" OUTPUT_NAME=""
if [ -f "$VISION_ONNX" ]; then
ONNX_ARGS="--vision-onnx $VISION_ONNX"
if [ -f "$ON_POLICY_ONNX" ] && [ -f "$OFF_POLICY_ONNX" ]; then
MODEL_TYPE=vision_multi_policy
ONNX_ARGS="$ONNX_ARGS --off-policy-onnx $OFF_POLICY_ONNX --on-policy-onnx $ON_POLICY_ONNX"
elif [ -f "$OFF_POLICY_ONNX" ] && [ -f "$POLICY_ONNX" ]; then
MODEL_TYPE=vision_multi_policy
ONNX_ARGS="$ONNX_ARGS --policy-onnx $POLICY_ONNX --off-policy-onnx $OFF_POLICY_ONNX"
elif [ -f "$POLICY_ONNX" ]; then
MODEL_TYPE=vision_policy
ONNX_ARGS="$ONNX_ARGS --policy-onnx $POLICY_ONNX"
fi
elif [ -f "$SUPERCOMBO_ONNX" ]; then
MODEL_TYPE=supercombo
ONNX_ARGS="--supercombo-onnx $SUPERCOMBO_ONNX"
fi
if [ -n "$MODEL_TYPE" ]; then
echo "Detected: $MODEL_TYPE -> driving_tinygrad.pkl"
env ${TG_FLAGS} python3 "$COMPILE_MODELD" \
--model-type $MODEL_TYPE \
--model-size $MODEL_SIZE \
--camera-resolutions $CAMERA_RES \
$ONNX_ARGS \
--output "${{ env.MODELS_DIR }}/driving_tinygrad.pkl"
fi
- name: Validate Model Outputs
run: |
source /etc/profile
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
python3 "${{ github.workspace }}/release/ci/model_generator.py" \
--validate-only \
--model-dir "${{ env.MODELS_DIR }}"
scons -j$(nproc) cache_dir=${{ env.SCONS_CACHE_DIR }} ${{ github.workspace }}/selfdrive/modeld
- name: Prepare Output
run: |
sudo rm -rf ${{ env.OUTPUT_DIR }}
mkdir -p ${{ env.OUTPUT_DIR }}
sudo rm -rf ${OUTPUT_DIR}
mkdir -p ${OUTPUT_DIR}
# Copy the model files
rsync -avm \
--include='*.dlc' \
--include='*.thneed' \
--include='*.pkl' \
--include='*.chunk*' \
--include='*.chunkmanifest' \
--include='*.onnx' \
--exclude='*' \
--delete-excluded \
--chown=comma:comma \
${{ env.MODELS_DIR }}/ ${{ env.OUTPUT_DIR }}/
python3 "${{ github.workspace }}/release/ci/model_generator.py" \
--model-dir "${{ env.MODELS_DIR }}" \
--output-dir "${{ env.OUTPUT_DIR }}" \
--custom-name "${{ env.MODEL_NAME }}" \
--upstream-branch "${{ inputs.upstream_branch }}" \
${{ inputs.is_20hz && '--is-20hz' || '' }}
- name: Write artifact name to file
run: echo "model-${{ env.MODEL_NAME }}${{ inputs.artifact_suffix }}-${{ github.run_number }}" > ${{ env.OUTPUT_DIR }}/artifact_name.txt
./selfdrive/modeld/models/ ${OUTPUT_DIR}/
# Rename files if file_name is provided
if [ ! -z "${{ inputs.file_name }}" ]; then
mv ${OUTPUT_DIR}/supercombo.thneed ${OUTPUT_DIR}/supercombo-${{ inputs.file_name }}.thneed
mv ${OUTPUT_DIR}/supercombo_metadata.pkl ${OUTPUT_DIR}/supercombo-${{ inputs.file_name }}_metadata.pkl
fi
# Calculate SHA256 hashes
HASH=$(sha256sum ${OUTPUT_DIR}/supercombo*.thneed | cut -d' ' -f1)
METADATA_HASH=$(sha256sum ${OUTPUT_DIR}/supercombo*_metadata.pkl | cut -d' ' -f1)
# Create metadata.json
cat > ${OUTPUT_DIR}/metadata.json << EOF
{
"display_name": "${{ inputs.custom_name || inputs.upstream_branch }}",
"full_name": "${{ inputs.file_name || 'default' }}",
"is_20hz": ${{ inputs.is_20hz || false }},
"files": {
"drive_model": {
"file_name": "$(basename ${OUTPUT_DIR}/supercombo*.thneed)",
"sha256": "${HASH}"
},
"metadata": {
"file_name": "$(basename ${OUTPUT_DIR}/supercombo*_metadata.pkl)",
"sha256": "${METADATA_HASH}"
}
},
"ref": "${{ inputs.upstream_branch }}",
"build_time": "$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
}
- name: Upload Build Artifacts
id: upload-artifact
uses: actions/upload-artifact@v4
with:
name: model-${{ env.MODEL_NAME }}${{ inputs.artifact_suffix }}-${{ github.run_number }}
name: model-${{ github.event.inputs.custom_name || github.event.inputs.upstream_branch }}-${{ github.run_number }}
path: ${{ env.OUTPUT_DIR }}
- name: Upload artifact name file
uses: actions/upload-artifact@v4
with:
name: artifact-name-${{ inputs.custom_name || inputs.upstream_branch }}
path: ${{ env.OUTPUT_DIR }}/artifact_name.txt
- name: Re-enable powersave
if: always()
run: |
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
+113 -207
View File
@@ -6,149 +6,97 @@ env:
CI_DIR: ${{ github.workspace }}/release/ci
SCONS_CACHE_DIR: ${{ github.workspace }}/release/ci/scons_cache
PUBLIC_REPO_URL: "https://github.com/sunnypilot/sunnypilot"
# Branch configurations
STAGING_SOURCE_BRANCH: 'master'
# Runtime configuration
SOURCE_BRANCH: "${{ github.head_ref || github.ref_name }}"
MASTER_BRANCH: "master"
MASTER_NEW_BRANCH: "master-new"
DEV_C3_SOURCE_BRANCH: "master-dev-c3-new"
# Target branch configurations
STAGING_TARGET_BRANCH: "staging-c3-new"
DEV_TARGET_BRANCH: "dev-c3-new"
RELEASE_TARGET_BRANCH: "release-c3-new"
on:
push:
branches: [ master, master-dev ]
tags: [ 'release/*' ]
pull_request_target:
types: [ labeled ]
branches: [ master, master-new, master-dev-c3-new ]
tags: [ '*' ]
pull_request:
branches: [ master, master-new ]
workflow_dispatch:
inputs:
wait_for_tests:
description: 'Wait for tests to finish'
extra_version:
description: 'Extra version identifier'
required: false
type: boolean
default: false
default: ''
jobs:
prepare_strategy:
runs-on: ubuntu-24.04
if: (!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
outputs:
environment: ${{ steps.strategy.outputs.environment }}
new_branch: ${{ steps.strategy.outputs.new_branch }}
extra_version_identifier: ${{ steps.strategy.outputs.extra_version_identifier }}
version: ${{ steps.strategy.outputs.version }}
cancel_publish_in_progress: ${{ steps.strategy.outputs.cancel_publish_in_progress }}
publish_concurrency_group: ${{ steps.strategy.outputs.publish_concurrency_group }}
is_stable_branch: ${{ steps.strategy.outputs.is_stable_branch }}
build: ${{ steps.strategy.outputs.build }}
steps:
- uses: actions/checkout@v4
- name: Extract deploy strategy
id: strategy
run: |
echo '::group::Strategy Extraction'
BRANCH="${{ github.head_ref || github.ref_name }}"
echo "Current branch: $BRANCH"
STRATEGY_JSON='${{ vars.DEPLOY_STRATEGY }}'
CONFIG=$(echo "$STRATEGY_JSON" | jq -r --arg branch "$BRANCH" '
.configs[] | select(.branch == $branch)
')
BUILD="$(date '+%Y.%m.%d')-${{ github.run_number }}"
if [[ -z "$CONFIG" || "$CONFIG" == "null" ]]; then
echo "No exact strategy match found. Falling back to feature/fork logic."
IS_FORK="${{ github.event.pull_request.head.repo.fork && 'true' || 'false' }}"
FORK_SUFFIX=$( [[ "$IS_FORK" == "true" ]] && echo "-fork" || echo "" )
NEW_BRANCH="${BRANCH}${FORK_SUFFIX}-prebuilt"
echo "new_branch=$NEW_BRANCH" >> $GITHUB_OUTPUT
echo "version=$BUILD" >> $GITHUB_OUTPUT
echo "cancel_publish_in_progress=true" >> $GITHUB_OUTPUT
echo "publish_concurrency_group=publish-${BRANCH}" >> $GITHUB_OUTPUT
echo "environment=feature-branch" >> $GITHUB_OUTPUT
echo "extra_version_identifier=feature-branch" >> $GITHUB_OUTPUT
else
echo "Matched config: $CONFIG"
environment=$(echo "$CONFIG" | jq -r '.environment')
echo "environment=$environment" >> $GITHUB_OUTPUT
echo "new_branch=$(echo "$CONFIG" | jq -r '.target_branch')" >> $GITHUB_OUTPUT
cancel="$(echo "$CONFIG" | jq -r '.cancel_publish_in_progress')";
echo "cancel_publish_in_progress=$( [ "$cancel" = "null" ] && echo "true" || echo $cancel)" >> $GITHUB_OUTPUT
echo "publish_concurrency_group=publish-${BRANCH}$( [ "$cancel" = "null" ] || [ "$cancel" = "true" ] || echo "${{ github.sha }}" )" >> $GITHUB_OUTPUT
is_stable_branch="$(echo "$CONFIG" | jq -r '.stable_branch // false')";
echo "is_stable_branch=$is_stable_branch" >> $GITHUB_OUTPUT
stable_version=$(cat openpilot/sunnypilot/common/version.h | grep SUNNYPILOT_VERSION | sed -e 's/[^0-9|.]//g');
echo "version=$([ "$is_stable_branch" = "true" ] && echo "$stable_version" || echo "$BUILD")" >> $GITHUB_OUTPUT
echo "extra_version_identifier=${environment}" >> $GITHUB_OUTPUT
fi
echo "build=$BUILD" >> $GITHUB_OUTPUT
cat $GITHUB_OUTPUT
validate_tests:
runs-on: ubuntu-24.04
needs: [ prepare_strategy ]
if: ${{
((github.event_name == 'workflow_dispatch' && inputs.wait_for_tests) ||
(github.event_name == 'push' && needs.prepare_strategy.outputs.is_stable_branch == 'true') ||
contains(github.event_name, 'pull_request') && (github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
}}
steps:
- uses: actions/checkout@v4
- name: Wait for Tests
uses: ./.github/workflows/wait-for-action # Path to where you place the action
with:
workflow: tests.yaml # The workflow file to monitor
github-token: ${{ secrets.GITHUB_TOKEN }}
should-wait-for-start: ${{ github.event_name == 'push' && 'true' || 'false' }}
build:
needs: [ validate_tests, prepare_strategy ]
concurrency:
group: build-${{ github.head_ref || github.ref_name }}
cancel-in-progress: false
runs-on: [self-hosted, tici]
runs-on: self-hosted
outputs:
new_branch: ${{ needs.prepare_strategy.outputs.new_branch }}
version: ${{ needs.prepare_strategy.outputs.version }}
extra_version_identifier: ${{ needs.prepare_strategy.outputs.extra_version_identifier }}
commit_sha: ${{ github.sha }}
if: ${{
(always() && !cancelled() && !failure()) &&
needs.prepare_strategy.result == 'success' &&
(needs.validate_tests.result == 'success' || needs.validate_tests.result == 'skipped') &&
(!contains(github.event_name, 'pull_request') ||
(github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
}}
new_branch: ${{ steps.set-env.outputs.new_branch }}
version: ${{ steps.set-env.outputs.version }}
extra_version_identifier: ${{ steps.set-env.outputs.extra_version_identifier }}
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
ref: ${{ env.SOURCE_BRANCH }}
repository: ${{ github.event.pull_request.head.repo.fork && github.event.pull_request.head.repo.full_name || github.repository }}
- run: git lfs pull
- name: Cache SCons
uses: actions/cache@v4
with:
path: ${{env.SCONS_CACHE_DIR}}
key: scons-${{ runner.os }}-${{ runner.arch }}-${{ env.SOURCE_BRANCH }}-${{ github.sha }}
# Note: GitHub Actions enforces cache isolation between different build sources (PR builds, workflow dispatches, etc.)
key: scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-${{ github.sha }}
# Note: GitHub Actions enforces cache isolation between different build sources (PR builds, workflow dispatches, etc.)
# for security. Only caches from the default branch are shared across all builds. This is by design and cannot be overridden.
restore-keys: |
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.SOURCE_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.STAGING_SOURCE_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_NEW_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}
- name: Set Configuration
run: |
if [[ "${{ github.ref_name }}" == "${{ env.DEV_C3_SOURCE_BRANCH }}" ]]; then
# Dev configuration
echo "BRANCH_TYPE=dev" >> $GITHUB_ENV
echo "NEW_BRANCH=${{ env.DEV_TARGET_BRANCH }}" >> $GITHUB_ENV
echo "VERSION=$(date '+%Y.%m.%d')-${{ github.run_number }}" >> $GITHUB_ENV
echo "EXTRA_VERSION_IDENTIFIER=${{ github.run_number }}" >> $GITHUB_ENV
elif [[ "${{ github.ref_name }}" == "${{ env.MASTER_BRANCH }}" || "${{ github.ref_name }}" == "${{ env.MASTER_NEW_BRANCH }}" ]]; then
# Master configuration
echo "BRANCH_TYPE=master" >> $GITHUB_ENV
echo "NEW_BRANCH=${{ env.STAGING_TARGET_BRANCH }}" >> $GITHUB_ENV
echo "EXTRA_VERSION_IDENTIFIER=staging" >> $GITHUB_ENV
echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')-staging" >> $GITHUB_ENV
elif [[ "${{ github.ref }}" == refs/tags/* ]]; then
# Tag configuration
echo "BRANCH_TYPE=tag" >> $GITHUB_ENV
echo "NEW_BRANCH=${{ env.RELEASE_TARGET_BRANCH }}" >> $GITHUB_ENV
echo "EXTRA_VERSION_IDENTIFIER=release" >> $GITHUB_ENV
echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')-release" >> $GITHUB_ENV
else
# Feature branch configuration
echo "BRANCH_TYPE=dispatch" >> $GITHUB_ENV
echo "NEW_BRANCH=${{ github.head_ref || github.ref_name }}-prebuilt" >> $GITHUB_ENV
echo "VERSION=$(date '+%Y.%m.%d')-${{ github.run_number }}" >> $GITHUB_ENV
fi
- name: Set environment variables
id: set-env
run: |
echo "new_branch=${{ needs.prepare_strategy.outputs.new_branch }}" >> $GITHUB_OUTPUT
echo "version=${{ needs.prepare_strategy.outputs.version }}" >> $GITHUB_OUTPUT
echo "extra_version_identifier=${{ needs.prepare_strategy.outputs.extra_version_identifier }}" >> $GITHUB_OUTPUT
echo "commit_sha=${{ github.sha }}" >> $GITHUB_OUTPUT
# Write to GITHUB_OUTPUT from environment variables
echo "new_branch=$NEW_BRANCH" >> $GITHUB_OUTPUT
[[ ! -z "$EXTRA_VERSION_IDENTIFIER" ]] && echo "extra_version_identifier=$EXTRA_VERSION_IDENTIFIER" >> $GITHUB_OUTPUT
[[ ! -z "$VERSION" ]] && echo "version=$VERSION" >> $GITHUB_OUTPUT
# Set up common environment
source /etc/profile;
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
@@ -174,19 +122,16 @@ jobs:
fi
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --disable
- name: Build Panda
run: |
scons -j$(nproc) cache_dir=${{env.SCONS_CACHE_DIR}} ${{ github.workspace }}/panda
- name: Build Main Project
run: |
export PYTHONPATH="$BUILD_DIR"
./tools/release/release_files.py | sort | uniq | rsync -rRl${RUNNER_DEBUG:+v} --files-from=- . $BUILD_DIR/
./release/release_files.py | sort | uniq | rsync -rRl${RUNNER_DEBUG:+v} --files-from=- . $BUILD_DIR/
cd $BUILD_DIR
sed -i '/from .board.jungle import PandaJungle, PandaJungleDFU/s/^/#/' panda/__init__.py
echo "Building sunnypilot's modeld_v2..."
scons -j$(nproc) cache_dir=${{env.SCONS_CACHE_DIR}} --minimal sunnypilot/modeld_v2
echo "Building sunnypilot's locationd..."
scons -j2 cache_dir=${{env.SCONS_CACHE_DIR}} --minimal sunnypilot/selfdrive/locationd
echo "Building openpilot's locationd..."
scons -j1 cache_dir=${{env.SCONS_CACHE_DIR}} --minimal selfdrive/locationd
echo "Building rest of sunnypilot"
scons -j$(nproc) cache_dir=${{env.SCONS_CACHE_DIR}} --minimal
touch ${BUILD_DIR}/prebuilt
if [[ "${{ runner.debug }}" == "1" ]]; then
@@ -198,27 +143,37 @@ jobs:
sudo rm -rf ${OUTPUT_DIR}
mkdir -p ${OUTPUT_DIR}
rsync -am${RUNNER_DEBUG:+v} \
--include='**/panda/board/' \
--include='**/panda/board/obj' \
--include='**/panda/board/obj/panda.bin.signed' \
--include='**/panda/board/obj/panda_h7.bin.signed' \
--include='**/panda/board/obj/bootstub.panda.bin' \
--include='**/panda/board/obj/bootstub.panda_h7.bin' \
--exclude='.sconsign.dblite' \
--exclude='*.a' \
--exclude='*.o' \
--exclude='*.os' \
--exclude='*.pyc' \
--exclude='moc_*' \
--exclude='__pycache__' \
--exclude='*.cc' \
--exclude='Jenkinsfile' \
--exclude='supercombo.onnx' \
--exclude='**/panda/board/*' \
--exclude='**/panda/board/obj/**' \
--exclude='**/panda/certs/' \
--exclude='**/panda/crypto/' \
--exclude='**/release/' \
--exclude='**/.github/' \
--exclude='**/openpilot/selfdrive/ui/replay/' \
--exclude='**/selfdrive/ui/replay/' \
--exclude='**/__pycache__/' \
--exclude='**/selfdrive/ui/*.h' \
--exclude='**/selfdrive/ui/**/*.h' \
--exclude='**/selfdrive/ui/qt/offroad/sunnypilot/' \
--exclude='${{env.SCONS_CACHE_DIR}}' \
--exclude='**/.git/' \
--exclude='**/SConstruct' \
--exclude='**/SConscript' \
--exclude='**/.venv/' \
--exclude='openpilot/selfdrive/modeld/models/*.onnx*' \
--exclude='openpilot/sunnypilot/modeld*/models/*.onnx*' \
--exclude='openpilot/third_party/*x86*' \
--exclude='openpilot/third_party/*Darwin*' \
--delete-excluded \
--chown=comma:comma \
${BUILD_DIR}/ ${OUTPUT_DIR}/
@@ -239,18 +194,14 @@ jobs:
run: |
PYTHONPATH=$PYTHONPATH:${{ github.workspace }}/ ${{ github.workspace }}/scripts/manage-powersave.py --enable
publish:
concurrency:
# We do a bit of a hack here to avoid canceling the publishing job if a new commit comes in while we're publishing by adding the sha to the group name.
# This means that if multiple commits come in while we're publishing, they will be queued up and publish one after the other.
# Otherwise, if a job is waiting to be published due to environment wait time, it would be canceled by a new commit and restart the wait time.
group: ${{ needs.prepare_strategy.outputs.publish_concurrency_group }}
cancel-in-progress: ${{ needs.prepare_strategy.outputs.cancel_publish_in_progress == 'true' }}
if: ${{ (always() && !cancelled() && !failure()) && needs.build.result == 'success' && needs.prepare_strategy.result == 'success' && (!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt')) }}
needs: [ build, prepare_strategy ]
group: publish-${{ github.head_ref || github.ref_name }}
cancel-in-progress: true
if: ${{ github.event_name != 'pull_request' || github.event_name == 'pull_request' && github.event.pull_request.draft }}
needs: build
runs-on: ubuntu-24.04
environment: ${{ needs.prepare_strategy.outputs.environment }}
environment: ${{ contains(fromJSON(vars.AUTO_DEPLOY_PREBUILT_BRANCHES), github.head_ref || github.ref_name) && 'auto-deploy' || 'feature-branch' }}
steps:
- uses: actions/checkout@v4
@@ -282,8 +233,8 @@ jobs:
"${{ needs.build.outputs.new_branch }}" \
"${{ needs.build.outputs.version }}" \
"https://x-access-token:${{github.token}}@github.com/sunnypilot/sunnypilot.git" \
"${{ needs.build.outputs.extra_version_identifier }}"
"-${{ needs.build.outputs.extra_version_identifier }}"
echo ""
echo "---- ️ To update the list of branches that auto deploy prebuilts -----"
echo ""
@@ -291,79 +242,34 @@ jobs:
echo "2. Current value: ${{ vars.AUTO_DEPLOY_PREBUILT_BRANCHES }}"
echo "3. Update as needed (JSON array with no spaces)"
- name: Tag ${{ needs.prepare_strategy.outputs.environment }}
if: ${{ needs.prepare_strategy.outputs.is_stable_branch == 'true' && (github.event_name != 'push' || !startsWith(github.ref, 'refs/tags/')) }}
run: |
TAG="${{ needs.prepare_strategy.outputs.environment }}/${{ needs.prepare_strategy.outputs.version }}/${{ needs.prepare_strategy.outputs.build }}"
git tag -f -a ${TAG} -m "${{ needs.prepare_strategy.outputs.environment }} @ ${{ needs.prepare_strategy.outputs.version }} of build ${{ needs.build.outputs.build }}."
git push -f origin ${TAG}
notify:
needs:
- prepare_strategy
- build
- publish
needs: [ build, publish ]
runs-on: ubuntu-24.04
if: ${{ (always() && !cancelled() && !failure())
&& needs.publish.result == 'success'
&& (!contains(github.event_name, 'pull_request') || (github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
&& (fromJSON(vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES_V2)[github.head_ref || github.ref_name] != null) }}
if: success()
steps:
- uses: actions/checkout@v4
- name: Setup Alpine Linux environment
uses: jirutka/setup-alpine@v1.2.0
with:
packages: 'jq gettext curl'
- name: Prepare notification message
id: message
- name: Send Discord Notification
env:
DISCORD_WEBHOOK: ${{ contains(fromJSON(vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES), github.head_ref || github.ref_name) && secrets.DISCORD_DEV_FEEDBACK_CHANNEL_WEBHOOK || secrets.DISCORD_DEV_PRIVATE_CHANNEL_WEBHOOK }}
run: |
TEMPLATE='${{ vars.DISCOURSE_GENERAL_UPDATE_NOTICE }}'
export VERSION="${{ needs.prepare_strategy.outputs.version }}"
export branch_name="${{ env.SOURCE_BRANCH }}"
export new_branch="${{ needs.prepare_strategy.outputs.new_branch }}"
export commit_sha="${{ github.sha }}"
export commit_short_sha="${{ github.sha }}"
export commit_short_sha="${commit_short_sha:0:7}"
export extra_version_identifier="${{ needs.prepare_strategy.outputs.extra_version_identifier || github.run_number }}"
export PUBLIC_REPO_URL="${{ env.PUBLIC_REPO_URL }}"
MESSAGE=$(cat << 'EOF' | envsubst
${{ vars.DISCOURSE_GENERAL_UPDATE_NOTICE }}
EOF
)
{
echo 'content<<EOFMARKER'
echo "$MESSAGE"
echo 'EOFMARKER'
} >> $GITHUB_OUTPUT
shell: bash
- name: Post to Discourse
uses: ./.github/workflows/post-to-discourse
with:
discourse-url: ${{ vars.DISCOURSE_URL }}
api-key: ${{ secrets.DISCOURSE_API_KEY }}
api-username: "system"
topic-id: ${{ fromJSON(vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES_V2)[github.head_ref || github.ref_name].topic_id }}
message: ${{ steps.message.outputs.content }}
manage-pr-labels:
name: Remove prebuilt label
runs-on: ubuntu-latest
if: (always() && contains(github.event_name, 'pull_request') && (github.event.action == 'labeled' && github.event.label.name == 'prebuilt'))
env:
LABEL: prebuilt
steps:
- name: Remove trust-fork-pr label if present
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const prNumber = context.payload.pull_request.number;
await github.rest.issues.removeLabel({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNumber,
name: process.env.LABEL
});
console.log(`Removed '${process.env.LABEL}' label from PR #${prNumber}`);
TEMPLATE='${{ vars.DISCORD_GENERAL_UPDATE_NOTICE }}'
export EXTRA_VERSION_IDENTIFIER="${{ needs.build.outputs.extra_version_identifier }}"
export VERSION="${{ needs.build.outputs.version }}"
export branch_name=${{ github.head_ref || github.ref_name }}
export new_branch=${{ needs.build.outputs.new_branch }}
export extra_version_identifier=${{ needs.build.outputs.extra_version_identifier || github.run_number}}
echo ${TEMPLATE} | envsubst | jq -c '.' | tee payload.json
curl -X POST -H "Content-Type: application/json" -d @payload.json $DISCORD_WEBHOOK
echo ""
echo "---- ️ To update the list of branches that notify to dev-feedback -----"
echo ""
echo "1. Go to: ${{ github.server_url }}/${{ github.repository }}/settings/variables/actions/DEV_FEEDBACK_NOTIFICATION_BRANCHES"
echo "2. Current value: ${{ vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES }}"
echo "3. Update as needed (JSON array with no spaces)"
shell: alpine.sh {0}
@@ -1,243 +0,0 @@
name: Build dev
env:
DEFAULT_SOURCE_BRANCH: "master"
DEFAULT_TARGET_BRANCH: "master-dev"
LFS_URL: 'https://gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git/info/lfs'
LFS_PUSH_URL: 'ssh://git@gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git'
on:
push:
branches:
- master
pull_request_target:
types: [ labeled ]
branches:
- 'master'
workflow_dispatch:
inputs:
source_branch:
description: 'Source branch to reset from'
required: true
default: 'master'
type: string
target_branch:
description: 'Target branch to reset and squash into'
required: true
default: 'master-dev'
type: string
cancel_in_progress:
description: 'Cancel any in-progress runs of this workflow'
required: false
default: true
type: boolean
concurrency:
group: ${{ github.workflow }}
cancel-in-progress: ${{ inputs.cancel_in_progress || github.event_name == 'push' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}
jobs:
reset-and-squash:
runs-on: ubuntu-latest
if: (
(github.event_name == 'workflow_dispatch')
|| (github.event_name == 'push' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch))
|| (contains(github.event_name, 'pull_request') && ((github.event.action == 'labeled' && (github.event.label.name == vars.PREBUILT_PR_LABEL || github.event.label.name == 'trust-fork-pr') && contains(github.event.pull_request.labels.*.name, vars.PREBUILT_PR_LABEL))))
)
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Fetch all history for all branches
token: ${{ secrets.GITHUB_TOKEN }}
persist-credentials: false
- name: Wait for Tests
uses: ./.github/workflows/wait-for-action # Path to where you place the action
if: (
(github.event_name == 'push' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch))
|| (contains(github.event_name, 'pull_request') && ((github.event.action == 'labeled' && (github.event.label.name == vars.PREBUILT_PR_LABEL || github.event.label.name == 'trust-fork-pr') && contains(github.event.pull_request.labels.*.name, vars.PREBUILT_PR_LABEL))))
)
with:
workflow: tests.yaml # The workflow file to monitor
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Configure Git
run: |
git config --global user.name 'github-actions[bot]'
git config --global user.email 'github-actions[bot]@users.noreply.github.com'
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
- name: Add GitLab public keys
run: |
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install PyGithub
- name: Check branches exist
run: |
# Check if source branch exists
if ! git ls-remote --heads origin ${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }} | grep -q "${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}"; then
echo "Source branch ${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }} does not exist!"
exit 1
fi
# Make sure we have the latest source branch
git fetch origin ${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}
# Check if target branch exists
if ! git ls-remote --heads origin ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} | grep -q "${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}"; then
echo "Target branch ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} does not exist, creating it from ${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}"
git checkout -b ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} origin/${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}
git push origin ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}
else
# Fetch target branch if it exists
git fetch origin ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}
fi
- name: Reset target branch
run: |
echo "Resetting ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} to match ${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}"
# Delete if exists and recreate pointing to source
git branch -D ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} || true
git branch ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} origin/${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}
- name: Get PRs to squash
id: get-prs
run: |
# Use GitHub API to get PRs with specific label, ordered by creation date
PR_LIST=$(gh api graphql -f query='
query($search_query:String!) {
search(query: $search_query, type:ISSUE, first:40) {
nodes {
... on PullRequest {
number
headRefName
title
createdAt
labels(last:10) {
nodes {
name
}
}
headRepository {
name
nameWithOwner
url
isFork
}
commits(last: 1) {
nodes {
commit {
statusCheckRollup {
state
}
}
}
}
}
}
}
}' -F search_query="repo:${{ github.repository }} is:pr is:open label:${{ vars.PREBUILT_PR_LABEL }},${{ vars.PREBUILT_PR_LABEL }}-c3 draft:false sort:created-asc")
PR_LIST=${PR_LIST//\'/}
echo "PR_LIST=${PR_LIST}" >> $GITHUB_OUTPUT
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Process PRs
run: |
cp ${{ github.workspace }}/release/ci/squash_and_merge.py /tmp/squash_and_merge.py && \
chmod +x /tmp/squash_and_merge.py && \
python3 ${{ github.workspace }}/release/ci/squash_and_merge_prs.py \
--pr-data '${{ steps.get-prs.outputs.PR_LIST }}' \
--target-branch ${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }} \
--squash-script-path '/tmp/squash_and_merge.py'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Update LFS Config
run: |
echo '[lfs]' > .lfsconfig
echo ' url = ${{ env.LFS_URL }}' >> .lfsconfig
echo ' pushurl = ${{ env.LFS_PUSH_URL }}' >> .lfsconfig
echo ' locksverify = false' >> .lfsconfig
- name: Restore workflows from source
run: |
TARGET_BRANCH="${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}"
SOURCE_BRANCH="${{ inputs.source_branch || env.DEFAULT_SOURCE_BRANCH }}"
# Ensure we are on the target branch
git checkout $TARGET_BRANCH
echo "Restoring .github/workflows from $SOURCE_BRANCH"
git checkout origin/$SOURCE_BRANCH -- .github/workflows
if ! git diff --cached --quiet; then
echo "Workflows differ. Committing restoration."
git commit -m "chore: restore .github/workflows from $SOURCE_BRANCH"
else
echo "Workflows match $SOURCE_BRANCH."
fi
- uses: actions/create-github-app-token@v2
id: ci-token
with:
app-id: ${{ secrets.CI_GITHUB_ACTIONS_TOKEN_APP_ID }}
private-key: ${{ secrets.CI_GITHUB_ACTIONS_TOKEN_APP_PRIVATE_KEY }}
- name: Push changes if there are diffs
id: push-changes
run: |
TARGET_BRANCH="${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}"
# Use the App Token to set the remote URL with authentication
git remote set-url origin "https://x-access-token:${{ steps.ci-token.outputs.token }}@github.com/${{ github.repository }}.git"
# Fetch the latest from remote
git fetch origin $TARGET_BRANCH
# Check for diffs between local and remote
if git diff $TARGET_BRANCH origin/$TARGET_BRANCH --quiet; then
echo "No changes to push - local and remote branches are identical"
echo "has_changes=false" >> $GITHUB_OUTPUT
exit 0
fi
# Push with the authenticated origin
if ! git push origin $TARGET_BRANCH --force; then
echo "Failed to push changes to $TARGET_BRANCH"
exit 1
fi
echo "Branch $TARGET_BRANCH has been reset and updated with squashed PRs"
echo "has_changes=true" >> $GITHUB_OUTPUT
- name: Trigger and wait for selfdrive tests
if: steps.push-changes.outputs.has_changes == 'true'
run: |
echo "Triggering selfdrive tests..."
gh workflow run tests.yaml --ref "${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}"
echo "Sleeping for 120s to give plenty of time for the action to start and then we wait"
sleep 120
echo "Getting latest run ID..."
RUN_ID=$(gh run list --workflow=tests.yaml --branch="${{ inputs.target_branch || env.DEFAULT_TARGET_BRANCH }}" --limit=1 --json databaseId --jq '.[0].databaseId')
echo "Watching run ID: $RUN_ID"
gh run watch "$RUN_ID"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-78
View File
@@ -1,78 +0,0 @@
name: Debug Discourse Posting
on:
push:
jobs:
test-discourse-post:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Post test message to Discourse
uses: ./.github/workflows/post-to-discourse
with:
discourse-url: ${{ vars.DISCOURSE_URL }}
api-key: ${{ secrets.DISCOURSE_API_KEY }}
api-username: ${{ secrets.DISCOURSE_API_USERNAME }}
topic-id: ${{ vars.DISCOURSE_UPDATES_TOPIC_ID }}
message: |
## 🧪 Test Post from GitHub Actions
**This is a test post to verify Discourse integration**
- **Workflow**: ${{ github.workflow }}
- **Run Number**: #${{ github.run_number }}
- **Branch**: `${{ github.ref_name }}`
- **Commit**: ${{ github.sha }}
- **Actor**: @${{ github.actor }}
- **Timestamp**: ${{ github.event.head_commit.timestamp }}
---
### Fake Build Info (for testing)
- **Version**: 0.9.8-test
- **Build**: #42
- **Branch**: release-test
[View workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})
*This is an automated test message. Drive safe! 🚗💨*
- name: Create topic on Discourse
uses: ./.github/workflows/post-to-discourse
with:
discourse-url: ${{ vars.DISCOURSE_URL }}
api-key: ${{ secrets.DISCOURSE_API_KEY }}
api-username: ${{ secrets.DISCOURSE_API_USERNAME }}
#topic-id: ${{ vars.DISCOURSE_UPDATES_TOPIC_ID }}
category-id: 4
title: "This is a test of a new topic instead of a reply"
message: |
## 🧪 Test Post from GitHub Actions
**This is a test post to verify Discourse integration**
- **Workflow**: ${{ github.workflow }}
- **Run Number**: #${{ github.run_number }}
- **Branch**: `${{ github.ref_name }}`
- **Commit**: ${{ github.sha }}
- **Actor**: @${{ github.actor }}
- **Timestamp**: ${{ github.event.head_commit.timestamp }}
---
### Fake Build Info (for testing)
- **Version**: 0.9.8-test
- **Build**: #42
- **Branch**: release-test
[View workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})
*This is an automated test message. Drive safe! 🚗💨*
- name: Display results
if: always()
run: |
echo "::notice::Discourse post test completed"
echo "Check your Discourse topic to verify the post appeared correctly"
-245
View File
@@ -1,245 +0,0 @@
name: tests
on:
push:
branches:
- master
pull_request:
workflow_dispatch:
workflow_call:
inputs:
run_number:
default: '1'
required: true
type: string
concurrency:
group: tests-ci-run-${{ inputs.run_number }}-${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && github.run_id || github.head_ref || github.ref }}-${{ github.workflow }}-${{ github.event_name }}
cancel-in-progress: true
env:
CI: 1
PYTHONPATH: ${{ github.workspace }}
PYTEST: pytest --continue-on-collection-errors --durations=0 -n logical
jobs:
build_release:
name: build release
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16"]')
|| fromJSON('["ubuntu-24.04"]') }}
env:
STRIPPED_DIR: /tmp/releasepilot
PYTHONPATH: /tmp/releasepilot
steps:
- uses: actions/checkout@v7
with:
submodules: true
- name: Getting LFS files
uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60
with:
timeout_minutes: 2
max_attempts: 3
command: git lfs pull
- name: Build devel
timeout-minutes: 1
run: TARGET_DIR=$STRIPPED_DIR tools/release/build_stripped.sh
- run: ./tools/op.sh setup
- name: Build openpilot and run checks
timeout-minutes: 30
working-directory: ${{ env.STRIPPED_DIR }}
run: python3 openpilot/system/manager/build.py
- name: Run tests
timeout-minutes: 1
working-directory: ${{ env.STRIPPED_DIR }}
run: tools/release/check-dirty.sh
- name: Check submodules
if: github.repository == 'sunnypilot/sunnypilot'
timeout-minutes: 3
run: |
if [ "${{ github.ref }}" != "refs/heads/master" ]; then
git fetch origin master:refs/remotes/origin/master
SUBMODULE_PATHS=$(git diff origin/master HEAD --name-only | grep -E '^[^/]+$' | while read path; do
if git ls-files --stage "$path" | grep -q "^160000"; then
echo "$path"
fi
done | tr '\n' ' ')
if [ -n "$SUBMODULE_PATHS" ]; then
echo "Changed submodule paths: $SUBMODULE_PATHS"
export SUBMODULE_PATHS="$SUBMODULE_PATHS"
export CHECK_PR_REFS=true
fi
fi
tools/release/check-submodules.sh
build_mac:
name: build macOS
runs-on: ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-macos-8x14' || 'macos-latest' }}
steps:
- uses: actions/checkout@v7
- name: Remove Homebrew from environment
run: |
FILTERED=$(echo "$PATH" | tr ':' '\n' | grep -v '/opt/homebrew' | tr '\n' ':')
echo "PATH=${FILTERED}/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin" >> $GITHUB_ENV
- run: ./tools/op.sh setup
- name: Building openpilot
run: scons
static_analysis:
name: static analysis
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16"]')
|| fromJSON('["ubuntu-24.04"]') }}
steps:
- uses: actions/checkout@v7
- run: ./tools/op.sh setup
- name: Static analysis
timeout-minutes: 1
run: scripts/lint/lint.sh
unit_tests:
name: unit tests
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16"]')
|| fromJSON('["ubuntu-24.04"]') }}
steps:
- uses: actions/checkout@v7
- run: ./tools/op.sh setup
- name: Build openpilot
run: scons
- name: Run unit tests
timeout-minutes: ${{ contains(runner.name, 'nsc') && 2 || 999 }}
env:
RAYLIB_BACKEND: headless
run: |
# Pre-compile Python bytecode so each pytest worker doesn't need to
$PYTEST --collect-only -m 'not slow' -qq
MAX_EXAMPLES=1 $PYTEST -m 'not slow'
process_replay:
name: process replay
if: false # disable process_replay for forks
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16"]')
|| fromJSON('["ubuntu-24.04"]') }}
steps:
- uses: actions/checkout@v7
- run: ./tools/op.sh setup
- name: Build openpilot
run: scons
- name: Run replay
timeout-minutes: ${{ contains(runner.name, 'nsc') && 2 || 20 }}
continue-on-error: ${{ github.ref == 'refs/heads/master' }}
run: openpilot/selfdrive/test/process_replay/test_processes.py -j$(nproc)
- name: Print diff
id: print-diff
if: always()
run: cat openpilot/selfdrive/test/process_replay/diff.txt
- name: Print diff report
if: always()
run: cat openpilot/selfdrive/test/process_replay/diff_report.txt
- uses: actions/upload-artifact@v7
if: always()
continue-on-error: true
with:
name: process_replay_diff.txt
path: openpilot/selfdrive/test/process_replay/diff.txt
- name: Upload diff report
uses: actions/upload-artifact@v7
if: always() && github.event_name == 'pull_request'
continue-on-error: true
with:
name: diff_report_${{ github.event.number }}
path: openpilot/selfdrive/test/process_replay/diff_report.txt
- name: Checkout ci-artifacts
if: github.repository == 'commaai/openpilot' && github.ref == 'refs/heads/master'
uses: actions/checkout@v7
with:
repository: commaai/ci-artifacts
ssh-key: ${{ secrets.CI_ARTIFACTS_DEPLOY_KEY }}
path: ${{ github.workspace }}/ci-artifacts
- name: Prepare refs
if: github.repository == 'commaai/openpilot' && github.ref == 'refs/heads/master'
working-directory: ${{ github.workspace }}/ci-artifacts
run: |
git config user.name "GitHub Actions Bot"
git config user.email "<>"
git fetch origin process-replay || true
git checkout process-replay 2>/dev/null || git checkout --orphan process-replay
cp ${{ github.workspace }}/openpilot/selfdrive/test/process_replay/fakedata/*.zst .
echo "${{ github.sha }}" > ref_commit
git add .
git commit -m "process-replay refs for ${{ github.repository }}@${{ github.sha }}" || echo "No changes to commit"
- name: Push refs
if: github.repository == 'commaai/openpilot' && github.ref == 'refs/heads/master'
uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60
with:
timeout_minutes: 2
max_attempts: 3
command: cd ${{ github.workspace }}/ci-artifacts && git push origin process-replay --force
- name: Run regen
if: false
timeout-minutes: 4
env:
ONNXCPU: 1
run: $PYTEST openpilot/selfdrive/test/process_replay/test_regen.py
simulator_driving:
name: simulator driving
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16"]')
|| fromJSON('["ubuntu-24.04"]') }}
if: false # FIXME: Started to timeout recently
steps:
- uses: actions/checkout@v7
- run: ./tools/op.sh setup
- name: Build openpilot
run: scons
- name: Driving test
timeout-minutes: 2
env:
# MetaDrive renders offscreen through panda3d's EGL pipe on llvmpipe
EGL_PLATFORM: surfaceless
run: pytest -s openpilot/tools/sim/tests/test_metadrive_bridge.py
create_ui_report:
name: Create UI Report
runs-on: ${{
(github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
&& fromJSON('["namespace-profile-amd64-8x16"]')
|| fromJSON('["ubuntu-24.04"]') }}
steps:
- uses: actions/checkout@v7
- run: ./tools/op.sh setup
- name: Build openpilot
run: scons
- name: Create UI Report
env:
RAYLIB_BACKEND: headless
run: |
python3 openpilot/selfdrive/ui/tests/diff/replay.py
python3 openpilot/selfdrive/ui/tests/diff/replay.py --big
- name: Upload UI Report
uses: actions/upload-artifact@v7
with:
name: ui-report-${{ inputs.run_number || '1' }}-${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && 'master' || github.event.number }}
path: openpilot/selfdrive/ui/tests/diff/report
+82 -94
View File
@@ -3,25 +3,21 @@ on:
push:
branches:
- master
- master-new
pull_request_target:
types: [assigned, opened, synchronize, reopened, edited]
branches:
- 'master'
- 'master-new'
paths:
- 'openpilot/selfdrive/assets/**'
- 'openpilot/selfdrive/ui/**'
- 'openpilot/system/ui/**'
- 'selfdrive/ui/**'
workflow_dispatch:
env:
UI_JOB_NAME: "Create UI Report"
REPORT_NAME: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && 'master' || github.event.number }}
SHA: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && github.sha || github.event.pull_request.head.sha }}
BRANCH_NAME: "openpilot/pr-${{ github.event.number }}-ui-preview"
REPORT_FILES_BRANCH_NAME: "mici-raylib-ui-reports"
# variant:video_prefix:master_branch
VARIANTS: "mici:mici_ui_replay:openpilot_master_ui_mici_raylib big:tizi_ui_replay:openpilot_master_ui_big_raylib"
REPORT_NAME: ${{ github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/master-new') && 'master' || github.event.number }}
SHA: ${{ github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/master-new') && github.sha || github.event.pull_request.head.sha }}
BRANCH_NAME: "openpilot/pr-${{ github.event.number }}"
jobs:
preview:
@@ -34,12 +30,11 @@ jobs:
pull-requests: write
actions: read
steps:
- uses: actions/checkout@v7
with:
submodules: true
- name: Waiting for ui generation to start
run: sleep 30
- name: Waiting for ui generation to end
uses: lewagon/wait-on-check-action@1d57e2c51a58d812d2765e036a028b6bdb5a6154
uses: lewagon/wait-on-check-action@v1.3.4
with:
ref: ${{ env.SHA }}
check-name: ${{ env.UI_JOB_NAME }}
@@ -53,93 +48,98 @@ jobs:
echo "run_id=$(curl https://api.github.com/repos/${{ github.repository }}/commits/${{ env.SHA }}/check-runs | jq -r '.check_runs[] | select(.name == "${{ env.UI_JOB_NAME }}") | .html_url | capture("(?<number>[0-9]+)") | .number')" >> $GITHUB_OUTPUT
- name: Getting proposed ui
uses: dawidd6/action-download-artifact@b6e2e70617bc3265edd6dab6c906732b2f1ae151
id: download-artifact
uses: dawidd6/action-download-artifact@v6
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
run_id: ${{ steps.get_run_id.outputs.run_id }}
search_artifacts: true
name: ui-report-1-${{ env.REPORT_NAME }}
name: report-1-${{ env.REPORT_NAME }}
path: ${{ github.workspace }}/pr_ui
- name: Getting mici master ui
uses: actions/checkout@v7
- name: Getting master ui
uses: actions/checkout@v4
with:
repository: sunnypilot/ci-artifacts
ssh-key: ${{ secrets.CI_ARTIFACTS_DEPLOY_KEY }}
path: ${{ github.workspace }}/master_mici
ref: openpilot_master_ui_mici_raylib
- name: Getting big master ui
uses: actions/checkout@v7
with:
repository: sunnypilot/ci-artifacts
ssh-key: ${{ secrets.CI_ARTIFACTS_DEPLOY_KEY }}
path: ${{ github.workspace }}/master_big
ref: openpilot_master_ui_big_raylib
path: ${{ github.workspace }}/master_ui
ref: openpilot_master_ui
- name: Saving new master ui
if: github.ref == 'refs/heads/master' && github.event_name == 'push'
if: (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/master-new') && github.event_name == 'push'
working-directory: ${{ github.workspace }}/master_ui
run: |
for variant in $VARIANTS; do
IFS=':' read -r name video branch <<< "$variant"
master_dir="${{ github.workspace }}/master_${name}"
cd "$master_dir"
git checkout --orphan=new_branch
git rm -rf *
git branch -D "$branch"
git branch -m "$branch"
git config user.name "GitHub Actions Bot"
git config user.email "<>"
cp "${{ github.workspace }}/pr_ui/${video}.mp4" .
git add .
git commit -m "${name} video for commit ${{ env.SHA }}"
git push origin "$branch" --force
done
git checkout --orphan=new_master_ui
git rm -rf *
git branch -D openpilot_master_ui
git branch -m openpilot_master_ui
git config user.name "GitHub Actions Bot"
git config user.email "<>"
mv ${{ github.workspace }}/pr_ui/*.png .
git add .
git commit -m "screenshots for commit ${{ env.SHA }}"
git push origin openpilot_master_ui --force
- name: Setup FFmpeg
uses: AnimMouse/setup-ffmpeg@178e0b4a408f06ce40d896c3cd79f50fa6f8b0c3
- name: Finding diffs
- name: Finding diff
if: github.event_name == 'pull_request_target'
id: find_diff
run: |
export PYTHONPATH=${{ github.workspace }}
baseurl="https://github.com/sunnypilot/ci-artifacts/raw/refs/heads/${{ env.BRANCH_NAME }}"
run: >-
sudo apt-get install -y imagemagick
COMMENT=""
for variant in $VARIANTS; do
IFS=':' read -r name video _ <<< "$variant"
diff_name="${name}_diff"
scenes="homescreen settings_device settings_software settings_sunnylink settings_toggles settings_sunnypilot settings_sunnypilot_mads settings_trips settings_developer offroad_alert update_available prime onroad onroad_disengaged onroad_override onroad_sidebar onroad_wide onroad_wide_sidebar onroad_alert_small onroad_alert_mid onroad_alert_full driver_camera body keyboard keyboard_uppercase"
A=($scenes)
mv "${{ github.workspace }}/pr_ui/${video}.mp4" "${{ github.workspace }}/pr_ui/${video}_proposed.mp4"
cp "${{ github.workspace }}/master_${name}/${video}.mp4" "${{ github.workspace }}/pr_ui/${video}_master.mp4"
DIFF=""
TABLE="<details><summary>All Screenshots</summary>"
TABLE="${TABLE}<table>"
diff_exit_code=0
python3 ${{ github.workspace }}/openpilot/selfdrive/ui/tests/diff/diff.py \
"${{ github.workspace }}/pr_ui/${video}_master.mp4" \
"${{ github.workspace }}/pr_ui/${video}_proposed.mp4" \
"${diff_name}.html" --basedir "$baseurl" --no-open || diff_exit_code=$?
for ((i=0; i<${#A[*]}; i=i+1));
do
cp "${{ github.workspace }}/openpilot/selfdrive/ui/tests/diff/report/${diff_name}.html" "${{ github.workspace }}/pr_ui/"
cp "${{ github.workspace }}/openpilot/selfdrive/ui/tests/diff/report/${diff_name}.mp4" "${{ github.workspace }}/pr_ui/"
if ! compare -fuzz 2% -highlight-color DeepSkyBlue1 -lowlight-color Black -compose Src ${{ github.workspace }}/master_ui/${A[$i]}.png ${{ github.workspace }}/pr_ui/${A[$i]}.png ${{ github.workspace }}/pr_ui/${A[$i]}_diff.png; then
convert ${{ github.workspace }}/pr_ui/${A[$i]}_diff.png -transparent black mask.png
composite mask.png ${{ github.workspace }}/master_ui/${A[$i]}.png composite_diff.png
convert -delay 100 ${{ github.workspace }}/master_ui/${A[$i]}.png composite_diff.png -loop 0 ${{ github.workspace }}/pr_ui/${A[$i]}_diff.gif
REPORT_URL="https://sunnypilot.github.io/ci-artifacts/${diff_name}_pr_${{ github.event.number }}.html"
if [ $diff_exit_code -eq 0 ]; then
COMMENT+="**${name}**: Videos are identical! [View Diff Report]($REPORT_URL)"$'\n'
mv ${{ github.workspace }}/master_ui/${A[$i]}.png ${{ github.workspace }}/pr_ui/${A[$i]}_master_ref.png
DIFF="${DIFF}<details open>"
DIFF="${DIFF}<summary>${A[$i]} : \$\${\\color{red}\\text{DIFFERENT}}\$\$</summary>"
DIFF="${DIFF}<table>"
DIFF="${DIFF}<tr>"
DIFF="${DIFF} <td> master <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}_master_ref.png\"> </td>"
DIFF="${DIFF} <td> proposed <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}.png\"> </td>"
DIFF="${DIFF}</tr>"
DIFF="${DIFF}<tr>"
DIFF="${DIFF} <td> diff <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}_diff.png\"> </td>"
DIFF="${DIFF} <td> composite diff <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}_diff.gif\"> </td>"
DIFF="${DIFF}</tr>"
DIFF="${DIFF}</table>"
DIFF="${DIFF}</details>"
else
COMMENT+="**${name}**: ⚠️ <strong>Videos differ!</strong> [View Diff Report]($REPORT_URL)"$'\n'
rm -f ${{ github.workspace }}/pr_ui/${A[$i]}_diff.png
fi
INDEX=$(($i % 2))
if [[ $INDEX -eq 0 ]]; then
TABLE="${TABLE}<tr>"
fi
TABLE="${TABLE} <td> <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}.png\"> </td>"
if [[ $INDEX -eq 1 || $(($i + 1)) -eq ${#A[*]} ]]; then
TABLE="${TABLE}</tr>"
fi
done
{
echo "COMMENT<<EOF"
echo "$COMMENT"
echo "EOF"
} >> "$GITHUB_OUTPUT"
TABLE="${TABLE}</table></details>"
echo "DIFF=$DIFF$TABLE" >> "$GITHUB_OUTPUT"
- name: Saving proposed ui
if: github.event_name == 'pull_request_target'
working-directory: ${{ github.workspace }}/master_mici
working-directory: ${{ github.workspace }}/master_ui
run: |
git config user.name "GitHub Actions Bot"
git config user.email "<>"
@@ -147,29 +147,17 @@ jobs:
git rm -rf *
mv ${{ github.workspace }}/pr_ui/* .
git add .
git commit -m "ui videos for PR #${{ github.event.number }}"
git commit -m "screenshots for PR #${{ github.event.number }}"
git push origin ${{ env.BRANCH_NAME }} --force
# Append diff reports to report files branch
git fetch origin ${{ env.REPORT_FILES_BRANCH_NAME }}
git checkout ${{ env.REPORT_FILES_BRANCH_NAME }}
for variant in $VARIANTS; do
IFS=':' read -r name _ _ <<< "$variant"
diff_name="${name}_diff"
cp "${{ github.workspace }}/openpilot/selfdrive/ui/tests/diff/report/${diff_name}.html" "${diff_name}_pr_${{ github.event.number }}.html"
git add "${diff_name}_pr_${{ github.event.number }}.html"
done
git commit -m "ui diff reports for PR #${{ github.event.number }}" || echo "No changes to commit"
git push origin ${{ env.REPORT_FILES_BRANCH_NAME }}
- name: Comment on PR
- name: Comment Screenshots on PR
if: github.event_name == 'pull_request_target'
uses: thollander/actions-comment-pull-request@24bffb9b452ba05a4f3f77933840a6a841d1b32b
uses: thollander/actions-comment-pull-request@v2
with:
message: |
<!-- _(run_id_ui_preview **${{ github.run_id }}**)_ -->
<!-- _(run_id_screenshots **${{ github.run_id }}**)_ -->
## UI Preview
${{ steps.find_diff.outputs.COMMENT }}
comment-tag: run_id_ui_preview
pr-number: ${{ github.event.number }}
github-token: ${{ secrets.GITHUB_TOKEN }}
${{ steps.find_diff.outputs.DIFF }}
comment_tag: run_id_screenshots
pr_number: ${{ github.event.number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -1,52 +0,0 @@
name: 'Wait for Tests'
description: 'Action to wait for workflow tests to start and complete'
inputs:
workflow:
description: 'The workflow file name to monitor'
required: true
default: 'tests.yaml'
branch:
description: 'The branch to monitor (defaults to current branch)'
required: false
default: ''
github-token:
description: 'GitHub token for API access'
required: true
wait-time:
description: 'Initial sleep time in seconds before monitoring starts'
required: false
default: '30'
should-wait-for-start:
description: 'Whether to wait for tests to start'
required: false
default: false
runs:
using: 'composite'
steps:
- name: Wait for tests to start
if: inputs.should-wait-for-start == 'true'
shell: bash
run: |
echo "Sleeping for ${{ inputs.wait-time }} seconds to give some time for the action to start and then we'll wait"
sleep ${{ inputs.wait-time }}
- name: Wait for tests to finish
shell: bash
run: |
BRANCH="${{ inputs.branch || github.head_ref || github.ref_name }}"
echo "Looking for workflow runs of ${{ inputs.workflow }} on branch $BRANCH"
RUN_ID=$(gh run list --workflow=${{ inputs.workflow }} --branch="$BRANCH" --limit=1 --json databaseId --jq '.[0].databaseId')
echo "Watching run ID: $RUN_ID"
gh run watch "$RUN_ID"
CONCLUSION=$(gh run view "$RUN_ID" --json conclusion --jq '.conclusion')
echo "Run concluded with: $CONCLUSION"
if [[ "$CONCLUSION" != "success" ]]; then
echo "❌ Workflow run failed with conclusion: $CONCLUSION"
exit 1
fi
env:
GITHUB_TOKEN: ${{ inputs.github-token }}
+48 -39
View File
@@ -10,23 +10,15 @@ venv/
.overlay_init
.overlay_consistent
.sconsign.dblite
model2.png
a.out
.hypothesis
.cache/
bin/
# created at launch for TICI PYTHONPATH (PC uses editable installs via pyproject.toml)
/msgq
/opendbc
/rednose
/teleoprtc
/tinygrad
/docs_site/
*.mp4
*.dylib
*.DSYM
*.d
*.pem
*.pyc
*.pyo
.*.swp
@@ -43,60 +35,77 @@ bin/
*.class
*.pyxbldc
*.vcd
*.mo
*_pyx.cpp
*.stats
*.pkl
*.pkl*
*.qm
config.json
clcache
compile_commands.json
compare_runtime*.html
openpilot/selfdrive/modeld/models/tg_input_devices.json
# build artifacts
docs_site/
openpilot/selfdrive/pandad/pandad
openpilot/cereal/services.h
openpilot/cereal/gen
openpilot/cereal/messaging/bridge
openpilot/selfdrive/ui/translations/tmp
openpilot/selfdrive/car/tests/cars_dump
openpilot/system/camerad/camerad
openpilot/system/camerad/test/ae_gray_test
persist
selfdrive/pandad/pandad
cereal/services.h
cereal/gen
cereal/messaging/bridge
selfdrive/logcatd/logcatd
selfdrive/mapd/default_speeds_by_region.json
system/proclogd/proclogd
selfdrive/ui/translations/alerts_generated.h
selfdrive/ui/translations/tmp
selfdrive/test/longitudinal_maneuvers/out
selfdrive/car/tests/cars_dump
system/camerad/camerad
system/camerad/test/ae_gray_test
notebooks
hyperthneed
provisioning
.coverage*
coverage.xml
htmlcov
pandaextra
.mypy_cache/
flycheck_*
cppcheck_report.txt
comma*.sh
selfdrive/modeld/thneed/compile
selfdrive/modeld/models/*.thneed
selfdrive/modeld/models/*.pkl
sunnypilot/modeld/thneed/compile
sunnypilot/modeld/models/*.thneed
sunnypilot/modeld/models/*.pkl
# openpilot log files
*.bz2
*.zst
*.rlog
build/
!**/.gitkeep
poetry.toml
Pipfile
### VisualStudioCode ###
*.vsix
.history
.ionide
.vscode/*
.history/
!.vscode/settings.json
!.vscode/tasks.json
!.vscode/launch.json
!.vscode/extensions.json
!.vscode/*.code-snippets
# agents
.claude/
.context/
PLAN.md
TASK.md
CLAUDE.md
SKILL.md
# Local History for Visual Studio Code
.history/
# Built Visual Studio Code Extensions
*.vsix
### VisualStudioCode Patch ###
# Ignore all local history of files
.history
.ionide
### JetBrains ###
!.idea/customTargets.xml
+1 -4
View File
@@ -15,7 +15,4 @@
url = https://github.com/commaai/teleoprtc
[submodule "tinygrad"]
path = tinygrad_repo
url = https://github.com/sunnypilot/tinygrad.git
[submodule "sunnypilot/neural_network_data"]
path = openpilot/sunnypilot/neural_network_data
url = https://github.com/sunnypilot/neural-network-data.git
url = https://github.com/commaai/tinygrad.git
+2 -2
View File
@@ -2,7 +2,7 @@
<tool name="uv Scons Build Debug" showInMainMenu="false" showInEditor="false" showInProject="false" showInSearchPopup="false" disabled="false" useConsole="true" showConsoleOnStdOut="false" showConsoleOnStdErr="false" synchronizeAfterRun="true">
<exec>
<option name="COMMAND" value="bash" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -u -j$(nproc) --ccflags=\&quot;-fno-inline\&quot;&quot;" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -u -j$(nproc) --compile_db --ccflags=\&quot;-fno-inline\&quot;&quot;" />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
@@ -16,7 +16,7 @@
<tool name="uv Scons Build Release" showInMainMenu="false" showInEditor="false" showInProject="false" showInSearchPopup="false" disabled="false" useConsole="true" showConsoleOnStdOut="false" showConsoleOnStdErr="false" synchronizeAfterRun="true">
<exec>
<option name="COMMAND" value="bash" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -u -j$(nproc)&quot; " />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -u -j$(nproc) --compile_db&quot; " />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
-1
View File
@@ -1 +0,0 @@
3.12.13
-26
View File
@@ -1,26 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Build_BIG_UI" type="PythonConfigurationType" factoryName="Python">
<module name="sunnypilot" />
<option name="ENV_FILES" value="" />
<option name="INTERPRETER_OPTIONS" value="" />
<option name="PARENT_ENVS" value="true" />
<envs>
<env name="BIG" value="1" />
</envs>
<option name="SDK_HOME" value="" />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$/" />
<option name="IS_MODULE_SDK" value="true" />
<option name="ADD_CONTENT_ROOTS" value="true" />
<option name="ADD_SOURCE_ROOTS" value="true" />
<option name="SCRIPT_NAME" value="$ProjectFileDir$/selfdrive/ui/ui.py" />
<option name="PARAMETERS" value="" />
<option name="SHOW_COMMAND_LINE" value="false" />
<option name="EMULATE_TERMINAL" value="false" />
<option name="MODULE_MODE" value="false" />
<option name="REDIRECT_INPUT" value="false" />
<option name="INPUT_FILE" value="" />
<method v="2">
<option name="ToolBeforeRunTask" enabled="true" actionId="Tool_External Tools_uv Scons Build Debug" />
</method>
</configuration>
</component>
-23
View File
@@ -1,23 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Build_SMALL_UI" type="PythonConfigurationType" factoryName="Python">
<module name="sunnypilot" />
<option name="ENV_FILES" value="" />
<option name="INTERPRETER_OPTIONS" value="" />
<option name="PARENT_ENVS" value="true" />
<option name="SDK_HOME" value="" />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$/" />
<option name="IS_MODULE_SDK" value="true" />
<option name="ADD_CONTENT_ROOTS" value="true" />
<option name="ADD_SOURCE_ROOTS" value="true" />
<option name="SCRIPT_NAME" value="$ProjectFileDir$/selfdrive/ui/ui.py" />
<option name="PARAMETERS" value="" />
<option name="SHOW_COMMAND_LINE" value="false" />
<option name="EMULATE_TERMINAL" value="false" />
<option name="MODULE_MODE" value="false" />
<option name="REDIRECT_INPUT" value="false" />
<option name="INPUT_FILE" value="" />
<method v="2">
<option name="ToolBeforeRunTask" enabled="true" actionId="Tool_External Tools_uv Scons Build Debug" />
</method>
</configuration>
</component>
+5 -47
View File
@@ -6,9 +6,9 @@
"type": "pickString",
"description": "Select the process to debug",
"options": [
"openpilot/selfdrive/controls/controlsd.py",
"openpilot/system/timed/timed.py",
"openpilot/tools/sim/run_bridge.py"
"selfdrive/controls/controlsd.py",
"system/timed/timed.py",
"tools/sim/run_bridge.py"
]
},
{
@@ -16,18 +16,13 @@
"type": "pickString",
"description": "Select the process to debug",
"options": [
"openpilot/selfdrive/ui/ui"
"selfdrive/ui/ui"
]
},
{
"id": "args",
"description": "Arguments to pass to the process",
"type": "promptString"
},
{
"id": "replayArg",
"type": "promptString",
"description": "Enter route or segment to replay."
}
],
"configurations": [
@@ -45,44 +40,7 @@
"type": "cppdbg",
"request": "launch",
"program": "${workspaceFolder}/${input:cpp_process}",
"cwd": "${workspaceFolder}"
},
{
"name": "Attach LLDB to Replay drive",
"type": "lldb",
"request": "attach",
"pid": "${command:pickMyProcess}",
"sourceMap": {
".": "${workspaceFolder}/opendbc/safety"
},
"initCommands": [
"script import time; time.sleep(3)"
]
},
{
"name": "Replay drive",
"type": "debugpy",
"request": "launch",
"program": "${workspaceFolder}/opendbc/safety/tests/safety_replay/replay_drive.py",
"args": [
"${input:replayArg}"
],
"console": "integratedTerminal",
"justMyCode": false,
"env": {
"PYTHONPATH": "${workspaceFolder}"
},
"subProcess": true,
"stopOnEntry": false
}
],
"compounds": [
{
"name": "Replay drive + Safety LLDB",
"configurations": [
"Replay drive",
"Attach LLDB to Replay drive"
]
"cwd": "${workspaceFolder}",
}
]
}
+6
View File
@@ -17,5 +17,11 @@
"**/.git",
"**/.venv",
"**/__pycache__",
// exclude directories that should be using the symlinked version
"common/**",
"selfdrive/**",
"system/**",
"third_party/**",
"tools/**",
]
}
-1320
View File
File diff suppressed because it is too large Load Diff
+13
View File
@@ -0,0 +1,13 @@
FROM ghcr.io/commaai/openpilot-base:latest
ENV PYTHONUNBUFFERED=1
ENV OPENPILOT_PATH=/home/batman/openpilot
ENV PYTHONPATH=${OPENPILOT_PATH}:${PYTHONPATH}
RUN mkdir -p ${OPENPILOT_PATH}
WORKDIR ${OPENPILOT_PATH}
COPY . ${OPENPILOT_PATH}/
RUN scons --cache-readonly -j$(nproc)
+81
View File
@@ -0,0 +1,81 @@
FROM ubuntu:24.04
ENV PYTHONUNBUFFERED=1
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -y --no-install-recommends sudo tzdata locales ssh pulseaudio xvfb x11-xserver-utils gnome-screenshot python3-tk python3-dev && \
rm -rf /var/lib/apt/lists/*
RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && locale-gen
ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US:en
ENV LC_ALL=en_US.UTF-8
COPY tools/install_ubuntu_dependencies.sh /tmp/tools/
RUN /tmp/tools/install_ubuntu_dependencies.sh && \
rm -rf /var/lib/apt/lists/* /tmp/* && \
cd /usr/lib/gcc/arm-none-eabi/* && \
rm -rf arm/ thumb/nofp thumb/v6* thumb/v8* thumb/v7+fp thumb/v7-r+fp.sp
# Add OpenCL
RUN apt-get update && apt-get install -y --no-install-recommends \
apt-utils \
alien \
unzip \
tar \
curl \
xz-utils \
dbus \
gcc-arm-none-eabi \
tmux \
vim \
libx11-6 \
wget \
&& rm -rf /var/lib/apt/lists/*
RUN mkdir -p /tmp/opencl-driver-intel && \
cd /tmp/opencl-driver-intel && \
wget https://github.com/intel/llvm/releases/download/2024-WW14/oclcpuexp-2024.17.3.0.09_rel.tar.gz && \
wget https://github.com/oneapi-src/oneTBB/releases/download/v2021.12.0/oneapi-tbb-2021.12.0-lin.tgz && \
mkdir -p /opt/intel/oclcpuexp_2024.17.3.0.09_rel && \
cd /opt/intel/oclcpuexp_2024.17.3.0.09_rel && \
tar -zxvf /tmp/opencl-driver-intel/oclcpuexp-2024.17.3.0.09_rel.tar.gz && \
mkdir -p /etc/OpenCL/vendors && \
echo /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64/libintelocl.so > /etc/OpenCL/vendors/intel_expcpu.icd && \
cd /opt/intel && \
tar -zxvf /tmp/opencl-driver-intel/oneapi-tbb-2021.12.0-lin.tgz && \
ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbb.so /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbbmalloc.so /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbb.so.12 /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
ln -s /opt/intel/oneapi-tbb-2021.12.0/lib/intel64/gcc4.8/libtbbmalloc.so.2 /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 && \
mkdir -p /etc/ld.so.conf.d && \
echo /opt/intel/oclcpuexp_2024.17.3.0.09_rel/x64 > /etc/ld.so.conf.d/libintelopenclexp.conf && \
ldconfig -f /etc/ld.so.conf.d/libintelopenclexp.conf && \
cd / && \
rm -rf /tmp/opencl-driver-intel
ENV NVIDIA_VISIBLE_DEVICES=all
ENV NVIDIA_DRIVER_CAPABILITIES=graphics,utility,compute
ENV QTWEBENGINE_DISABLE_SANDBOX=1
RUN dbus-uuidgen > /etc/machine-id
ARG USER=batman
ARG USER_UID=1001
RUN useradd -m -s /bin/bash -u $USER_UID $USER
RUN usermod -aG sudo $USER
RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
USER $USER
COPY --chown=$USER pyproject.toml uv.lock /home/$USER
COPY --chown=$USER tools/install_python_dependencies.sh /home/$USER/tools/
ENV VIRTUAL_ENV=/home/$USER/.venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
RUN cd /home/$USER && \
tools/install_python_dependencies.sh && \
rm -rf tools/ pyproject.toml uv.lock .cache
USER root
RUN sudo git config --global --add safe.directory /tmp/openpilot
Vendored
+60 -46
View File
@@ -12,7 +12,7 @@ def retryWithDelay(int maxRetries, int delay, Closure body) {
def device(String ip, String step_label, String cmd) {
withCredentials([file(credentialsId: 'id_rsa', variable: 'key_file')]) {
def ssh_cmd = """
ssh -o ControlMaster=auto -o ControlPath=/tmp/ssh_control_%C -o ControlPersist=yes -o ConnectTimeout=5 -o ServerAliveInterval=5 -o ServerAliveCountMax=2 -o BatchMode=yes -o StrictHostKeyChecking=no -i ${key_file} 'comma@${ip}' exec /usr/bin/bash <<'END'
ssh -o ConnectTimeout=5 -o ServerAliveInterval=5 -o ServerAliveCountMax=2 -o BatchMode=yes -o StrictHostKeyChecking=no -i ${key_file} 'comma@${ip}' exec /usr/bin/bash <<'END'
set -e
@@ -22,7 +22,7 @@ shopt -s huponexit # kill all child processes when the shell exits
export CI=1
export PYTHONWARNINGS=error
#export LOGPRINT=debug # this has gotten too spammy...
export LOGPRINT=debug
export TEST_DIR=${env.TEST_DIR}
export SOURCE_DIR=${env.SOURCE_DIR}
export GIT_BRANCH=${env.GIT_BRANCH}
@@ -31,7 +31,7 @@ export CI_ARTIFACTS_TOKEN=${env.CI_ARTIFACTS_TOKEN}
export GITHUB_COMMENTS_TOKEN=${env.GITHUB_COMMENTS_TOKEN}
export AZURE_TOKEN='${env.AZURE_TOKEN}'
# only use 1 thread for tici tests since most require HIL
export PYTEST_ADDOPTS="-n0 -s"
export PYTEST_ADDOPTS="-n 0"
export GIT_SSH_COMMAND="ssh -i /data/gitkey"
@@ -63,8 +63,6 @@ if [ -f /data/openpilot/launch_env.sh ]; then
source /data/openpilot/launch_env.sh
fi
export LD_LIBRARY_PATH="\$(python -c 'import ffmpeg; print(ffmpeg.LIB_DIR)'):/usr/local/lib:\${LD_LIBRARY_PATH:-}"
ln -snf ${env.TEST_DIR} /data/pythonpath
cd ${env.TEST_DIR} || true
@@ -95,7 +93,7 @@ def deviceStage(String stageName, String deviceType, List extra_env, def steps)
retry (3) {
def date = sh(script: 'date', returnStdout: true).trim();
device(device_ip, "set time", "date -s '" + date + "'")
device(device_ip, "git checkout", extra + "\n" + readFile("openpilot/selfdrive/test/setup_device_ci.sh"))
device(device_ip, "git checkout", extra + "\n" + readFile("selfdrive/test/setup_device_ci.sh"))
}
steps.each { item ->
def name = item[0]
@@ -105,7 +103,7 @@ def deviceStage(String stageName, String deviceType, List extra_env, def steps)
def diffPaths = args.diffPaths ?: []
def cmdTimeout = args.timeout ?: 9999
if (branch != "master" && !branch.contains("__jenkins_loop_") && diffPaths && !hasPathChanged(gitDiff, diffPaths)) {
if (branch != "master" && diffPaths && !hasPathChanged(gitDiff, diffPaths)) {
println "Skipping ${name}: no changes in ${diffPaths}."
return
} else {
@@ -168,11 +166,11 @@ node {
env.GIT_BRANCH = checkout(scm).GIT_BRANCH
env.GIT_COMMIT = checkout(scm).GIT_COMMIT
def excludeBranches = ['__nightly', 'devel', 'devel-staging',
'release-tizi', 'release-tizi-staging', 'release-mici', 'release-mici-staging', 'testing-closet*', 'hotfix-*']
def excludeBranches = ['master-ci', 'devel', 'devel-staging', 'release3', 'release3-staging',
'testing-closet*', 'hotfix-*']
def excludeRegex = excludeBranches.join('|').replaceAll('\\*', '.*')
if (env.BRANCH_NAME != 'master' && !env.BRANCH_NAME.contains('__jenkins_loop_')) {
if (env.BRANCH_NAME != 'master') {
properties([
disableConcurrentBuilds(abortPrevious: true)
])
@@ -180,21 +178,21 @@ node {
try {
if (env.BRANCH_NAME == 'devel-staging') {
deviceStage("build release-tizi-staging", "tizi-needs-can", [], [
step("build release-tizi-staging", "RELEASE_BRANCH=release-tizi-staging,release-mici-staging $SOURCE_DIR/tools/release/build_release.sh"),
deviceStage("build release3-staging", "tici-needs-can", [], [
step("build release3-staging", "RELEASE_BRANCH=release3-staging $SOURCE_DIR/release/build_release.sh"),
])
}
if (env.BRANCH_NAME == '__nightly') {
if (env.BRANCH_NAME == 'master-ci') {
parallel (
'nightly': {
deviceStage("build nightly", "tizi-needs-can", [], [
step("build nightly", "RELEASE_BRANCH=nightly $SOURCE_DIR/tools/release/build_release.sh"),
deviceStage("build nightly", "tici-needs-can", [], [
step("build nightly", "RELEASE_BRANCH=nightly $SOURCE_DIR/release/build_release.sh"),
])
},
'nightly-dev': {
deviceStage("build nightly-dev", "tizi-needs-can", [], [
step("build nightly-dev", "PANDA_DEBUG_BUILD=1 RELEASE_BRANCH=nightly-dev $SOURCE_DIR/tools/release/build_release.sh"),
deviceStage("build nightly-dev", "tici-needs-can", [], [
step("build nightly-dev", "PANDA_DEBUG_BUILD=1 RELEASE_BRANCH=nightly-dev $SOURCE_DIR/release/build_release.sh"),
])
},
)
@@ -202,53 +200,69 @@ node {
if (!env.BRANCH_NAME.matches(excludeRegex)) {
parallel (
// tici tests
'onroad tests': {
deviceStage("onroad", "tizi-needs-can", ["UNSAFE=1"], [
step("build openpilot", "cd openpilot/system/manager && ./build.py"),
step("check dirty", "tools/release/check-dirty.sh"),
step("onroad tests", "pytest openpilot/selfdrive/test/test_onroad.py -s", [timeout: 60]),
deviceStage("onroad", "tici-needs-can", ["UNSAFE=1"], [
// TODO: ideally, this test runs in master-ci, but it takes 5+m to build it
//["build master-ci", "cd $SOURCE_DIR/release && TARGET_DIR=$TEST_DIR $SOURCE_DIR/scripts/retry.sh ./build_devel.sh"],
step("build openpilot", "cd system/manager && ./build.py"),
step("check dirty", "release/check-dirty.sh"),
step("onroad tests", "pytest selfdrive/test/test_onroad.py -s", [timeout: 60]),
//["time to onroad", "pytest selfdrive/test/test_time_to_onroad.py"],
])
},
'HW + Unit Tests': {
deviceStage("tizi-hardware", "tizi-common", ["UNSAFE=1"], [
step("build", "cd openpilot/system/manager && ./build.py"),
step("test power draw", "pytest -s openpilot/selfdrive/test//test_power_draw.py"),
step("test encoder", "pytest openpilot/system/loggerd/tests/test_encoder.py", [diffPaths: ["openpilot/system/loggerd/"]]),
step("test manager", "pytest openpilot/system/manager/test/test_manager.py"),
deviceStage("tici-hardware", "tici-common", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test pandad", "pytest selfdrive/pandad/tests/test_pandad.py", [diffPaths: ["panda", "selfdrive/pandad/"]]),
step("test power draw", "pytest -s system/hardware/tici/tests/test_power_draw.py"),
step("test encoder", "LD_LIBRARY_PATH=/usr/local/lib pytest system/loggerd/tests/test_encoder.py", [diffPaths: ["system/loggerd/"]]),
step("test pigeond", "pytest system/ubloxd/tests/test_pigeond.py", [diffPaths: ["system/ubloxd/"]]),
step("test manager", "pytest system/manager/test/test_manager.py"),
])
},
'camerad OX03C10': {
deviceStage("OX03C10", "tizi-ox03c10", ["UNSAFE=1"], [
step("build", "cd openpilot/system/manager && ./build.py"),
step("test pandad", "pytest openpilot/selfdrive/pandad/tests/test_pandad.py"),
step("test camerad", "pytest openpilot/system/camerad/test/test_camerad.py", [timeout: 90]),
'loopback': {
deviceStage("loopback", "tici-loopback", ["UNSAFE=1"], [
step("build openpilot", "cd system/manager && ./build.py"),
step("test pandad loopback", "pytest selfdrive/pandad/tests/test_pandad_loopback.py"),
])
},
'camerad OS04C10': {
deviceStage("OS04C10", "tici-os04c10", ["UNSAFE=1"], [
step("build", "cd openpilot/system/manager && ./build.py"),
step("test pandad", "pytest openpilot/selfdrive/pandad/tests/test_pandad.py"),
step("test camerad", "pytest openpilot/system/camerad/test/test_camerad.py", [timeout: 90]),
'camerad': {
deviceStage("AR0231", "tici-ar0231", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test camerad", "pytest system/camerad/test/test_camerad.py", [timeout: 60]),
step("test exposure", "pytest system/camerad/test/test_exposure.py"),
])
deviceStage("OX03C10", "tici-ox03c10", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test camerad", "pytest system/camerad/test/test_camerad.py", [timeout: 60]),
step("test exposure", "pytest system/camerad/test/test_exposure.py"),
])
},
'sensord': {
deviceStage("LSM + MMC", "tizi-lsmc", ["UNSAFE=1"], [
step("build", "cd openpilot/system/manager && ./build.py"),
step("test sensord", "pytest openpilot/system/sensord/tests/test_sensord.py"),
deviceStage("LSM + MMC", "tici-lsmc", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test sensord", "pytest system/sensord/tests/test_sensord.py"),
])
deviceStage("BMX + LSM", "tici-bmx-lsm", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test sensord", "pytest system/sensord/tests/test_sensord.py"),
])
},
'replay': {
deviceStage("model-replay", "tizi-replay", ["UNSAFE=1"], [
step("build", "cd openpilot/system/manager && ./build.py", [diffPaths: ["openpilot/selfdrive/modeld/", "tinygrad_repo", "openpilot/selfdrive/test/process_replay/model_replay.py"]]),
step("model replay", "openpilot/selfdrive/test/process_replay/model_replay.py", [diffPaths: ["openpilot/selfdrive/modeld/", "tinygrad_repo", "openpilot/selfdrive/test/process_replay/model_replay.py"]]),
deviceStage("model-replay", "tici-replay", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py", [diffPaths: ["selfdrive/modeld/", "tinygrad_repo", "selfdrive/test/process_replay/model_replay.py"]]),
step("model replay", "selfdrive/test/process_replay/model_replay.py", [diffPaths: ["selfdrive/modeld/", "tinygrad_repo", "selfdrive/test/process_replay/model_replay.py"]]),
])
},
'tizi': {
deviceStage("tizi", "tizi", ["UNSAFE=1"], [
step("build openpilot", "cd openpilot/system/manager && ./build.py"),
step("test pandad loopback", "pytest openpilot/selfdrive/pandad/tests/test_pandad_loopback.py"),
step("test pandad spi", "pytest openpilot/selfdrive/pandad/tests/test_pandad_spi.py"),
step("test amp", "pytest openpilot/common/hardware/tici/tests/test_amplifier.py"),
step("build openpilot", "cd system/manager && ./build.py"),
step("test pandad loopback", "SINGLE_PANDA=1 pytest selfdrive/pandad/tests/test_pandad_loopback.py"),
step("test pandad spi", "pytest selfdrive/pandad/tests/test_pandad_spi.py"),
step("test pandad", "pytest selfdrive/pandad/tests/test_pandad.py", [diffPaths: ["panda", "selfdrive/pandad/"]]),
step("test amp", "pytest system/hardware/tici/tests/test_amplifier.py"),
step("test qcomgpsd", "pytest system/qcomgpsd/tests/test_qcomgpsd.py", [diffPaths: ["system/qcomgpsd/"]]),
])
},
+91 -58
View File
@@ -1,74 +1,107 @@
![](https://user-images.githubusercontent.com/47793918/233812617-beab2e71-57b9-479e-8bff-c3931347ca40.png)
<div align="center" style="text-align: center;">
## 🌞 What is sunnypilot?
[sunnypilot](https://github.com/sunnyhaibin/sunnypilot) is a fork of comma.ai's openpilot, an open source driver assistance system. sunnypilot offers the user a unique driving experience for over 300+ supported car makes and models with modified behaviors of driving assist engagements. sunnypilot complies with comma.ai's safety rules as accurately as possible.
<h1>openpilot</h1>
## 💭 Join our Community Forum
Join the official sunnypilot community forum to stay up to date with all the latest features and be a part of shaping the future of sunnypilot!
* https://community.sunnypilot.ai/
<p>
<b>openpilot is an operating system for robotics.</b>
<br>
Currently, it upgrades the driver assistance system in 275+ supported cars.
</p>
## Documentation
https://docs.sunnypilot.ai/ is your one stop shop for everything from features to installation to FAQ about the sunnypilot
<h3>
<a href="https://docs.comma.ai">Docs</a>
<span> · </span>
<a href="https://docs.comma.ai/contributing/roadmap/">Roadmap</a>
<span> · </span>
<a href="https://github.com/commaai/openpilot/blob/master/docs/CONTRIBUTING.md">Contribute</a>
<span> · </span>
<a href="https://discord.comma.ai">Community</a>
<span> · </span>
<a href="https://comma.ai/shop">Try it on a comma 3X</a>
</h3>
## 🚘 Running on a dedicated device in a car
First, check out this list of items you'll need to [get started](https://community.sunnypilot.ai/t/getting-started-using-sunnypilot-in-your-supported-car/251).
Quick start: `bash <(curl -fsSL openpilot.comma.ai)`
## Installation
Next, refer to the sunnypilot community forum for [installation instructions](https://community.sunnypilot.ai/t/read-before-installing-sunnypilot/254), as well as a complete list of [Recommended Branch Installations](https://community.sunnypilot.ai/t/recommended-branch-installations/235).
![openpilot tests](https://github.com/commaai/openpilot/actions/workflows/selfdrive_tests.yaml/badge.svg)
[![codecov](https://codecov.io/gh/commaai/openpilot/branch/master/graph/badge.svg)](https://codecov.io/gh/commaai/openpilot)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![X Follow](https://img.shields.io/twitter/follow/comma_ai)](https://x.com/comma_ai)
[![Discord](https://img.shields.io/discord/469524606043160576)](https://discord.comma.ai)
## 🎆 Pull Requests
We welcome both pull requests and issues on GitHub. Bug fixes are encouraged.
</div>
Pull requests should be against the most current `master` branch.
## 📊 User Data
By default, sunnypilot uploads the driving data to comma servers. You can also access your data through [comma connect](https://connect.comma.ai/).
sunnypilot is open source software. The user is free to disable data collection if they wish to do so.
sunnypilot logs the road-facing camera, CAN, GPS, IMU, magnetometer, thermal sensors, crashes, and operating system logs.
The driver-facing camera and microphone are only logged if you explicitly opt-in in settings.
By using this software, you understand that use of this software or its related services will generate certain types of user data, which may be logged and stored at the sole discretion of comma. By accepting this agreement, you grant an irrevocable, perpetual, worldwide right to comma for the use of this data.
## Licensing
sunnypilot is released under the [MIT License](LICENSE). This repository includes original work as well as significant portions of code derived from [openpilot by comma.ai](https://github.com/commaai/openpilot), which is also released under the MIT license with additional disclaimers.
The original openpilot license notice, including comma.ais indemnification and alpha software disclaimer, is reproduced below as required:
> openpilot is released under the MIT license. Some parts of the software are released under other licenses as specified.
>
> Any user of this software shall indemnify and hold harmless Comma.ai, Inc. and its directors, officers, employees, agents, stockholders, affiliates, subcontractors and customers from and against all allegations, claims, actions, suits, demands, damages, liabilities, obligations, losses, settlements, judgments, costs and expenses (including without limitation attorneys fees and costs) which arise out of, relate to or result from any use of this software by user.
>
> **THIS IS ALPHA QUALITY SOFTWARE FOR RESEARCH PURPOSES ONLY. THIS IS NOT A PRODUCT.
> YOU ARE RESPONSIBLE FOR COMPLYING WITH LOCAL LAWS AND REGULATIONS.
> NO WARRANTY EXPRESSED OR IMPLIED.**
For full license terms, please see the [`LICENSE`](LICENSE) file.
## 💰 Support sunnypilot
If you find any of the features useful, consider becoming a [sponsor on GitHub](https://github.com/sponsors/sunnyhaibin) to support future feature development and improvements.
<table>
<tr>
<td><a href="https://youtu.be/NmBfgOanCyk" title="Video By Greer Viau"><img src="https://github.com/commaai/openpilot/assets/8762862/2f7112ae-f748-4f39-b617-fabd689c3772"></a></td>
<td><a href="https://youtu.be/VHKyqZ7t8Gw" title="Video By Logan LeGrand"><img src="https://github.com/commaai/openpilot/assets/8762862/92351544-2833-40d7-9e0b-7ef7ae37ec4c"></a></td>
<td><a href="https://youtu.be/SUIZYzxtMQs" title="A drive to Taco Bell"><img src="https://github.com/commaai/openpilot/assets/8762862/05ceefc5-2628-439c-a9b2-89ce77dc6f63"></a></td>
</tr>
</table>
By becoming a sponsor, you will gain access to exclusive content, early access to new features, and the opportunity to directly influence the project's development.
Using openpilot in a car
------
To use openpilot in a car, you need four things:
1. **Supported Device:** a comma 3/3X, available at [comma.ai/shop](https://comma.ai/shop/comma-3x).
2. **Software:** The setup procedure for the comma 3/3X allows users to enter a URL for custom software. Use the URL `openpilot.comma.ai` to install the release version.
3. **Supported Car:** Ensure that you have one of [the 275+ supported cars](docs/CARS.md).
4. **Car Harness:** You will also need a [car harness](https://comma.ai/shop/car-harness) to connect your comma 3/3X to your car.
<h3>GitHub Sponsor</h3>
We have detailed instructions for [how to install the harness and device in a car](https://comma.ai/setup). Note that it's possible to run openpilot on [other hardware](https://blog.comma.ai/self-driving-car-for-free/), although it's not plug-and-play.
<a href="https://github.com/sponsors/sunnyhaibin">
<img src="https://user-images.githubusercontent.com/47793918/244135584-9800acbd-69fd-4b2b-bec9-e5fa2d85c817.png" alt="Become a Sponsor" width="300" style="max-width: 100%; height: auto;">
</a>
<br>
### Branches
| branch | URL | description |
|------------------|----------------------------------------|-------------------------------------------------------------------------------------|
| `release3` | openpilot.comma.ai | This is openpilot's release branch. |
| `release3-staging` | openpilot-test.comma.ai | This is the staging branch for releases. Use it to get new releases slightly early. |
| `nightly` | openpilot-nightly.comma.ai | This is the bleeding edge development branch. Do not expect this to be stable. |
| `nightly-dev` | installer.comma.ai/commaai/nightly-dev | Same as nightly, but includes experimental development features for some cars. |
<h3>PayPal</h3>
To start developing openpilot
------
<a href="https://paypal.me/sunnyhaibin0850" target="_blank">
<img src="https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif" alt="PayPal this" title="PayPal - The safer, easier way to pay online!" border="0" />
</a>
<br></br>
openpilot is developed by [comma](https://comma.ai/) and by users like you. We welcome both pull requests and issues on [GitHub](http://github.com/commaai/openpilot).
Your continuous love and support are greatly appreciated! Enjoy 🥰
* Join the [community Discord](https://discord.comma.ai)
* Check out [the contributing docs](docs/CONTRIBUTING.md)
* Check out the [openpilot tools](tools/)
* Read about the [development workflow](docs/WORKFLOW.md)
* Code documentation lives at https://docs.comma.ai
* Information about running openpilot lives on the [community wiki](https://github.com/commaai/openpilot/wiki)
<span>-</span> Jason, Founder of sunnypilot
Want to get paid to work on openpilot? [comma is hiring](https://comma.ai/jobs#open-positions) and offers lots of [bounties](https://comma.ai/bounties) for external contributors.
Safety and Testing
----
* openpilot observes [ISO26262](https://en.wikipedia.org/wiki/ISO_26262) guidelines, see [SAFETY.md](docs/SAFETY.md) for more details.
* openpilot has software-in-the-loop [tests](.github/workflows/selfdrive_tests.yaml) that run on every commit.
* The code enforcing the safety model lives in panda and is written in C, see [code rigor](https://github.com/commaai/panda#code-rigor) for more details.
* panda has software-in-the-loop [safety tests](https://github.com/commaai/panda/tree/master/tests/safety).
* Internally, we have a hardware-in-the-loop Jenkins test suite that builds and unit tests the various processes.
* panda has additional hardware-in-the-loop [tests](https://github.com/commaai/panda/blob/master/Jenkinsfile).
* We run the latest openpilot in a testing closet containing 10 comma devices continuously replaying routes.
Licensing
------
openpilot is released under the MIT license. Some parts of the software are released under other licenses as specified.
Any user of this software shall indemnify and hold harmless Comma.ai, Inc. and its directors, officers, employees, agents, stockholders, affiliates, subcontractors and customers from and against all allegations, claims, actions, suits, demands, damages, liabilities, obligations, losses, settlements, judgments, costs and expenses (including without limitation attorneys fees and costs) which arise out of, relate to or result from any use of this software by user.
**THIS IS ALPHA QUALITY SOFTWARE FOR RESEARCH PURPOSES ONLY. THIS IS NOT A PRODUCT.
YOU ARE RESPONSIBLE FOR COMPLYING WITH LOCAL LAWS AND REGULATIONS.
NO WARRANTY EXPRESSED OR IMPLIED.**
User Data and comma Account
------
By default, openpilot uploads the driving data to our servers. You can also access your data through [comma connect](https://connect.comma.ai/). We use your data to train better models and improve openpilot for everyone.
openpilot is open source software: the user is free to disable data collection if they wish to do so.
openpilot logs the road-facing cameras, CAN, GPS, IMU, magnetometer, thermal sensors, crashes, and operating system logs.
The driver-facing camera is only logged if you explicitly opt-in in settings. The microphone is not recorded.
By using openpilot, you agree to [our Privacy Policy](https://comma.ai/privacy). You understand that use of this software or its related services will generate certain types of user data, which may be logged and stored at the sole discretion of comma. By accepting this agreement, you grant an irrevocable, perpetual, worldwide right to comma for the use of this data.
+12 -83
View File
@@ -1,94 +1,23 @@
Version 0.11.2 (2026-06-15)
Version 0.9.9 (2025-03-30)
========================
* Coming soon
* Rivian support
* F-150 & Mach-E support
* Tesla Model 3 support
Version 0.11.1 (2026-05-18)
Version 0.9.8 (2025-01-30)
========================
* New driver monitoring model
* Improved image processing pipeline for driver camera
* Improved thermal policy for comma four
* Acura MDX 2022-24 support thanks to mvl-boston!
* Rivian R1S and R1T 2025 support thanks to lukasloetkolben!
Version 0.11.0 (2026-03-17)
========================
* New driving model #36798
* Fully trained using a learned simulator
* Improved longitudinal performance in Experimental mode
* Reduce comma four standby power usage by 77% to 52 mW
* Kia K7 2017 support thanks to royjr!
* Lexus LS 2018 support thanks to Hacheoy!
Version 0.10.3 (2025-12-17)
========================
* New driving model #36249
* New temporal policy architecture
* New on-policy training physics noise model
* New driver monitoring model #36409
* Trained on a new dataset, including comma four data
* Improved inter-process communication memory efficiency
Version 0.10.2 (2025-11-19)
========================
* comma four support
Version 0.10.1 (2025-09-08)
========================
* New driving model #36276
* World Model: removed global localization inputs
* World Model: 2x the number of parameters
* World Model: trained on 4x the number of segments
* VAE Compression Model: new architecture and training objective
* Driving Vision Model: trained on 4x the number of segments
* New Driver Monitoring model #36198
* Acura TLX 2021 support thanks to MVL!
* Honda City 2023 support thanks to vanillagorillaa and drFritz!
* Honda N-Box 2018 support thanks to miettal!
* Honda Odyssey 2021-25 support thanks to csouers and MVL!
* Honda Passport 2026 support thanks to vanillagorillaa and MVL!
Version 0.10.0 (2025-08-05)
========================
* New driving model
* New training architecture
* Described in our CVPR paper: "Learning to Drive from a World Model"
* Longitudinal MPC replaced by E2E planning from World Model in Experimental Mode
* Action from lateral MPC as training objective replaced by E2E planning from World Model
* Low-speed lead car ground-truth fixes
* Enable live-learned steering actuation delay
* Opt-in audio recording for dashcam video
* Acura MDX 2025 support thanks to vanillagorillaa and MVL!
* Honda Accord 2023-25 support thanks to vanillagorillaa and MVL!
* Honda CR-V 2023-25 support thanks to vanillagorillaa and MVL!
* Honda Pilot 2023-25 support thanks to vanillagorillaa and MVL!
Version 0.9.9 (2025-05-23)
========================
* New driving model
* New training architecture using parts from MLSIM
* Steering actuation delay is now learned online
* Ford Escape 2023-24 support thanks to incognitojam!
* Ford Kuga 2024 support thanks to incognitojam!
* Hyundai Nexo 2021 support thanks to sunnyhaibin!
* Tesla Model 3 and Y support thanks to lukasloetkolben!
* Lexus RC 2023 support thanks to nelsonjchen!
Version 0.9.8 (2025-02-28)
========================
* New driving model
* Model now gates applying positive acceleration in Chill mode
* New driver monitoring model
* New driving monitoring model
* Reduced false positives related to passengers
* Image processing pipeline moved to the ISP
* More GPU time for bigger driving models
* More GPU time for driving models
* Power draw reduced 0.5W, which means your device runs cooler
* Added toggle to enable driver monitoring even when openpilot is not engaged
* Localizer rewritten to remove GPS dependency at runtime
* Firehose Mode for maximizing your training data uploads
* Enable openpilot longitudinal control for Ford Q3 vehicles
* New Toyota TSS2 longitudinal tune
* Rivian R1S and R1T support thanks to lukasloetkolben!
* Ford F-150, F-150 Hybrid, Mach-E, and Ranger support
* Coming soon
* New driving model with gas gating
* Training data upload mode
Version 0.9.7 (2024-06-13)
========================
@@ -1104,7 +1033,7 @@ Version 0.2.1 (2016-12-14)
Version 0.2 (2016-12-12)
=========================
* Car/Radar abstraction layers have shipped, see openpilot/cereal/car.capnp
* Car/Radar abstraction layers have shipped, see cereal/car.capnp
* controlsd has been refactored
* Shipped plant model and testing maneuvers
* visiond exits more gracefully now
+304 -246
View File
@@ -3,264 +3,365 @@ import subprocess
import sys
import sysconfig
import platform
import shlex
import importlib
import numpy as np
import SCons.Errors
from SCons.Defaults import _stripixes
TICI = os.path.isfile('/TICI')
SCons.Warnings.warningAsException(True)
# pending upstream fix - https://github.com/SCons/scons/issues/4461
#SetOption('warn', 'all')
TICI = os.path.isfile('/TICI')
AGNOS = TICI
Decider('MD5-timestamp')
SetOption('num_jobs', max(1, int(os.cpu_count()/(1 if "CI" in os.environ else 2))))
SetOption('num_jobs', int(os.cpu_count()/2))
AddOption('--kaitai',
action='store_true',
help='Regenerate kaitai struct parsers')
AddOption('--asan',
action='store_true',
help='turn on ASAN')
AddOption('--ubsan',
action='store_true',
help='turn on UBSan')
AddOption('--coverage',
action='store_true',
help='build with test coverage options')
AddOption('--clazy',
action='store_true',
help='build with clazy')
AddOption('--compile_db',
action='store_true',
help='build clang compilation database')
AddOption('--ccflags',
action='store',
type='string',
default='',
help='pass arbitrary flags over the command line')
AddOption('--external-sconscript',
action='store',
metavar='FILE',
dest='external_sconscript',
help='add an external SConscript to the build')
AddOption('--pc-thneed',
action='store_true',
dest='pc_thneed',
help='use thneed on pc')
AddOption('--mutation',
action='store_true',
help='generate mutation-ready code')
AddOption('--ccflags', action='store', type='string', default='', help='pass arbitrary flags over the command line')
AddOption('--verbose', action='store_true', default=False, help='show full build commands')
release = not os.path.exists(File('#.gitattributes').abspath) # file absent on release branch, see release_files.py
AddOption('--minimal',
action='store_false',
dest='extras',
default=(not TICI and not release),
default=os.path.exists(File('#.lfsconfig').abspath), # minimal by default on release branch (where there's no LFS)
help='the minimum build to run openpilot. no tests, tools, etc.')
submodule_python_paths = [
Dir("#").abspath,
Dir("#msgq_repo").abspath,
Dir("#opendbc_repo").abspath,
Dir("#rednose_repo").abspath,
Dir("#teleoprtc_repo").abspath,
Dir("#tinygrad_repo").abspath,
]
for p in reversed(submodule_python_paths):
if p not in sys.path:
sys.path.insert(0, p)
AddOption('--stock-ui',
action='store_true',
dest='stock_ui',
default=False,
help='Build stock openpilot UI instead of sunnypilot UI')
if external_pythonpath := os.environ.get("PYTHONPATH"):
submodule_python_paths += [p for p in external_pythonpath.split(os.pathsep) if p and p not in submodule_python_paths]
# Detect platform
arch = subprocess.check_output(["uname", "-m"], encoding='utf8').rstrip()
## Architecture name breakdown (arch)
## - larch64: linux tici aarch64
## - aarch64: linux pc aarch64
## - x86_64: linux pc x64
## - Darwin: mac x64 or arm64
real_arch = arch = subprocess.check_output(["uname", "-m"], encoding='utf8').rstrip()
if platform.system() == "Darwin":
arch = "Darwin"
elif arch == "aarch64" and TICI:
brew_prefix = subprocess.check_output(['brew', '--prefix'], encoding='utf8').strip()
elif arch == "aarch64" and AGNOS:
arch = "larch64"
assert arch in [
"larch64", # linux tici arm64
"aarch64", # linux pc arm64
"x86_64", # linux pc x64
"Darwin", # macOS arm64 (x86 not supported)
]
assert arch in ["larch64", "aarch64", "x86_64", "Darwin"]
pkg_names = ['acados', 'bzip2', 'capnproto', 'catch2', 'eigen', 'ffmpeg', 'json11', 'ncurses', 'zeromq', 'zstd']
pkgs = [importlib.import_module(name) for name in pkg_names]
acados = pkgs[pkg_names.index('acados')]
ffmpeg = pkgs[pkg_names.index('ffmpeg')]
# Shared package ships .so/.dylib; older device venvs still have static .a only.
# Keep static link deps (x264/z/va/drm) when the installed package is static so
# TICI CI works without upgrading the device venv yet.
# TODO: drop the static fallback once device venvs have comma-deps-ffmpeg>=7.1.0.post94
_ffmpeg_lib_names = os.listdir(ffmpeg.LIB_DIR) if os.path.isdir(ffmpeg.LIB_DIR) else []
ffmpeg_shared = any(
n.startswith('libavcodec.so') or (n.startswith('libavcodec') and n.endswith('.dylib'))
for n in _ffmpeg_lib_names
)
ffmpeg_libs = ['avformat', 'avcodec', 'swresample', 'avutil']
if not ffmpeg_shared:
ffmpeg_libs += ['x264', 'z']
if arch != "Darwin":
ffmpeg_libs += ['va', 'va-drm', 'drm']
acados_include_dirs = [
acados.INCLUDE_DIR,
os.path.join(acados.INCLUDE_DIR, "blasfeo", "include"),
os.path.join(acados.INCLUDE_DIR, "hpipm", "include"),
]
lenv = {
"PATH": os.environ['PATH'],
"LD_LIBRARY_PATH": [Dir(f"#third_party/acados/{arch}/lib").abspath],
"PYTHONPATH": Dir("#").abspath + ':' + Dir(f"#third_party/acados").abspath,
# ***** enforce a whitelist of system libraries *****
# this prevents silently relying on a 3rd party package,
# e.g. apt-installed libusb. all libraries should either
# be distributed with all Linux distros and macOS, or
# vendored in commaai/dependencies.
allowed_system_libs = {
"EGL", "GLESv2", "GL",
"Qt5Charts", "Qt5Core", "Qt5Gui", "Qt5Widgets",
"dl", "drm", "gbm", "m", "pthread",
"ACADOS_SOURCE_DIR": Dir("#third_party/acados").abspath,
"ACADOS_PYTHON_INTERFACE_PATH": Dir("#third_party/acados/acados_template").abspath,
"TERA_PATH": Dir("#").abspath + f"/third_party/acados/{arch}/t_renderer"
}
def _resolve_lib(env, name):
for d in env.Flatten(env.get('LIBPATH', [])):
p = Dir(str(d)).abspath
for ext in ('.a', '.so', '.dylib'):
f = File(os.path.join(p, f'lib{name}{ext}'))
if f.exists() or f.has_builder():
return name
if name in allowed_system_libs:
return name
raise SCons.Errors.UserError(f"Unexpected non-vendored library '{name}'")
rpath = lenv["LD_LIBRARY_PATH"].copy()
def _libflags(target, source, env, for_signature):
libs = []
lp = env.subst('$LIBLITERALPREFIX')
for lib in env.Flatten(env.get('LIBS', [])):
if isinstance(lib, str):
if os.sep in lib or lib.startswith('#'):
libs.append(File(lib))
elif lib.startswith('-') or (lp and lib.startswith(lp)):
libs.append(lib)
else:
libs.append(_resolve_lib(env, lib))
else:
libs.append(lib)
return _stripixes(env['LIBLINKPREFIX'], libs, env['LIBLINKSUFFIX'],
env['LIBPREFIXES'], env['LIBSUFFIXES'], env, env['LIBLITERALPREFIX'])
if arch == "larch64":
cpppath = [
"#third_party/opencl/include",
]
libpath = [
"/usr/local/lib",
"/system/vendor/lib64",
f"#third_party/acados/{arch}/lib",
]
libpath += [
"#third_party/snpe/larch64",
"#third_party/libyuv/larch64/lib",
"/usr/lib/aarch64-linux-gnu"
]
cflags = ["-DQCOM2", "-mcpu=cortex-a57"]
cxxflags = ["-DQCOM2", "-mcpu=cortex-a57"]
rpath += ["/usr/local/lib"]
else:
cflags = []
cxxflags = []
cpppath = []
rpath += []
# MacOS
if arch == "Darwin":
libpath = [
f"#third_party/libyuv/{arch}/lib",
f"#third_party/acados/{arch}/lib",
f"{brew_prefix}/lib",
f"{brew_prefix}/opt/openssl@3.0/lib",
"/System/Library/Frameworks/OpenGL.framework/Libraries",
]
cflags += ["-DGL_SILENCE_DEPRECATION"]
cxxflags += ["-DGL_SILENCE_DEPRECATION"]
cpppath += [
f"{brew_prefix}/include",
f"{brew_prefix}/opt/openssl@3.0/include",
]
lenv["DYLD_LIBRARY_PATH"] = lenv["LD_LIBRARY_PATH"]
# Linux
else:
libpath = [
f"#third_party/acados/{arch}/lib",
f"#third_party/libyuv/{arch}/lib",
"/usr/lib",
"/usr/local/lib",
]
if arch == "x86_64":
libpath += [
f"#third_party/snpe/{arch}"
]
rpath += [
Dir(f"#third_party/snpe/{arch}").abspath,
]
if GetOption('asan'):
ccflags = ["-fsanitize=address", "-fno-omit-frame-pointer"]
ldflags = ["-fsanitize=address"]
elif GetOption('ubsan'):
ccflags = ["-fsanitize=undefined"]
ldflags = ["-fsanitize=undefined"]
else:
ccflags = []
ldflags = []
# no --as-needed on mac linker
if arch != "Darwin":
ldflags += ["-Wl,--as-needed", "-Wl,--no-undefined"]
if not GetOption('stock_ui'):
cflags += ["-DSUNNYPILOT"]
cxxflags += ["-DSUNNYPILOT"]
ccflags_option = GetOption('ccflags')
if ccflags_option:
ccflags += ccflags_option.split(' ')
env = Environment(
ENV={
"PATH": os.environ['PATH'],
"PYTHONPATH": os.pathsep.join(submodule_python_paths),
"ACADOS_SOURCE_DIR": acados.DIR,
"ACADOS_PYTHON_INTERFACE_PATH": acados.TEMPLATE_DIR,
"TERA_PATH": acados.TERA_PATH
},
ENV=lenv,
CCFLAGS=[
"-g",
"-fPIC",
"-O2",
"-Wunused",
"-Werror",
"-Wshadow" if arch in ("Darwin", "larch64") else "-Wshadow=local",
"-Wshadow",
"-Wno-unknown-warning-option",
"-Wno-inconsistent-missing-override",
"-Wno-c99-designator",
"-Wno-reorder-init-list",
"-Wno-vla-cxx-extension",
],
CFLAGS=["-std=gnu11"],
CXXFLAGS=["-std=c++1z"],
CPPPATH=[
"#openpilot",
"#msgq_repo", # #include "msgq/..."
"#opendbc_repo", # #include "opendbc/..."
"#rednose_repo", # #include "rednose/..."
"#rednose_repo/rednose", # #include "logger/..." (rednose package root)
"#openpilot/cereal/gen/cpp",
acados_include_dirs,
[x.INCLUDE_DIR for x in pkgs],
] + cflags + ccflags,
CPPPATH=cpppath + [
"#",
"#third_party/acados/include",
"#third_party/acados/include/blasfeo/include",
"#third_party/acados/include/hpipm/include",
"#third_party/catch2/include",
"#third_party/libyuv/include",
"#third_party/json11",
"#third_party/linux/include",
"#third_party/snpe/include",
"#third_party",
"#msgq",
],
LIBPATH=[
"#openpilot/common",
CC='clang',
CXX='clang++',
LINKFLAGS=ldflags,
RPATH=rpath,
CFLAGS=["-std=gnu11"] + cflags,
CXXFLAGS=["-std=c++1z"] + cxxflags,
LIBPATH=libpath + [
"#msgq_repo",
"#openpilot/selfdrive/pandad",
"#rednose_repo/rednose/helpers",
[x.LIB_DIR for x in pkgs],
"#third_party",
"#selfdrive/pandad",
"#common",
"#rednose/helpers",
],
RPATH=[ffmpeg.LIB_DIR] if ffmpeg_shared else [],
CYTHONCFILESUFFIX=".cpp",
COMPILATIONDB_USE_ABSPATH=True,
REDNOSE_ROOT="#rednose_repo",
REDNOSE_ROOT="#",
tools=["default", "cython", "compilation_db", "rednose_filter"],
toolpath=["#site_scons/site_tools", "#rednose_repo/site_scons/site_tools"],
)
if arch != "larch64":
env['_LIBFLAGS'] = _libflags
# Arch-specific flags and paths
if arch == "larch64":
env["CC"] = "clang"
env["CXX"] = "clang++"
env.Append(LIBPATH=[
"/usr/lib/aarch64-linux-gnu",
])
arch_flags = ["-D__TICI__", "-mcpu=cortex-a57", "-DQCOM2"]
env.Append(CCFLAGS=arch_flags)
env.Append(CXXFLAGS=arch_flags)
elif arch == "Darwin":
env.Append(LIBPATH=[
"/System/Library/Frameworks/OpenGL.framework/Libraries",
])
env.Append(CCFLAGS=["-DGL_SILENCE_DEPRECATION"])
env.Append(CXXFLAGS=["-DGL_SILENCE_DEPRECATION"])
if arch == "Darwin":
# RPATH is not supported on macOS, instead use the linker flags
darwin_rpath_link_flags = [f"-Wl,-rpath,{path}" for path in env["RPATH"]]
env["LINKFLAGS"] += darwin_rpath_link_flags
_extra_cc = shlex.split(GetOption('ccflags') or '')
if _extra_cc:
env.Append(CCFLAGS=_extra_cc)
if GetOption('compile_db'):
env.CompilationDatabase('compile_commands.json')
# no --as-needed on mac linker
if arch != "Darwin":
env.Append(LINKFLAGS=["-Wl,--as-needed", "-Wl,--no-undefined"])
# Setup cache dir
default_cache_dir = '/data/scons_cache' if AGNOS else '/tmp/scons_cache'
cache_dir = ARGUMENTS.get('cache_dir', default_cache_dir)
CacheDir(cache_dir)
Clean(["."], cache_dir)
# Shorter build output: show brief descriptions instead of full commands.
# Full command lines are still printed on failure by scons.
if not GetOption('verbose'):
for action, short in (
("CC", "CC"),
("CXX", "CXX"),
("LINK", "LINK"),
("SHCC", "CC"),
("SHCXX", "CXX"),
("SHLINK", "LINK"),
("AR", "AR"),
("RANLIB", "RANLIB"),
("AS", "AS"),
):
env[f"{action}COMSTR"] = f" [{short}] $TARGET"
node_interval = 5
node_count = 0
def progress_function(node):
global node_count
node_count += node_interval
sys.stderr.write("progress: %d\n" % node_count)
# ********** Cython build environment **********
if os.environ.get('SCONS_PROGRESS'):
Progress(progress_function, interval=node_interval)
# Cython build environment
py_include = sysconfig.get_paths()['include']
envCython = env.Clone()
envCython["CPPPATH"] += [sysconfig.get_paths()['include'], np.get_include()]
envCython["CCFLAGS"] += ["-Wno-#warnings", "-Wno-cpp", "-Wno-shadow", "-Wno-deprecated-declarations"]
envCython["CPPPATH"] += [py_include, np.get_include()]
envCython["CCFLAGS"] += ["-Wno-#warnings", "-Wno-shadow", "-Wno-deprecated-declarations"]
envCython["CCFLAGS"].remove("-Werror")
envCython["LIBS"] = []
if arch == "Darwin":
envCython["LINKFLAGS"] = env["LINKFLAGS"] + ["-bundle", "-undefined", "dynamic_lookup"]
envCython["LINKFLAGS"] = ["-bundle", "-undefined", "dynamic_lookup"] + darwin_rpath_link_flags
else:
envCython["LINKFLAGS"] = ["-pthread", "-shared"]
np_version = SCons.Script.Value(np.__version__)
Export('envCython', 'np_version')
Export('env', 'arch', 'acados', 'release', 'ffmpeg_libs')
# Qt build environment
qt_env = env.Clone()
qt_modules = ["Widgets", "Gui", "Core", "Network", "Concurrent", "DBus", "Xml"]
# Setup cache dir
default_cache_dir = '/data/scons_cache' if arch == "larch64" else '/tmp/scons_cache'
cache_dir = ARGUMENTS.get('cache_dir', default_cache_dir)
cache_size_limit = 4e9 if "CI" in os.environ else 2e9
CacheDir(cache_dir)
Clean(["."], cache_dir)
qt_libs = []
if arch == "Darwin":
qt_env['QTDIR'] = f"{brew_prefix}/opt/qt@5"
qt_dirs = [
os.path.join(qt_env['QTDIR'], "include"),
]
qt_dirs += [f"{qt_env['QTDIR']}/include/Qt{m}" for m in qt_modules]
qt_env["LINKFLAGS"] += ["-F" + os.path.join(qt_env['QTDIR'], "lib")]
qt_env["FRAMEWORKS"] += [f"Qt{m}" for m in qt_modules] + ["OpenGL"]
qt_env.AppendENVPath('PATH', os.path.join(qt_env['QTDIR'], "bin"))
else:
qt_install_prefix = subprocess.check_output(['qmake', '-query', 'QT_INSTALL_PREFIX'], encoding='utf8').strip()
qt_install_headers = subprocess.check_output(['qmake', '-query', 'QT_INSTALL_HEADERS'], encoding='utf8').strip()
def prune_cache_dir(target=None, source=None, env=None):
cache_files = sorted((os.path.join(root, f) for root, _, files in os.walk(cache_dir) for f in files), key=os.path.getmtime)
cache_size = sum(os.path.getsize(f) for f in cache_files)
for f in cache_files:
if cache_size < cache_size_limit:
break
cache_size -= os.path.getsize(f)
os.unlink(f)
qt_env['QTDIR'] = qt_install_prefix
qt_dirs = [
f"{qt_install_headers}",
]
# ********** start building stuff **********
qt_gui_path = os.path.join(qt_install_headers, "QtGui")
qt_gui_dirs = [d for d in os.listdir(qt_gui_path) if os.path.isdir(os.path.join(qt_gui_path, d))]
qt_dirs += [f"{qt_install_headers}/QtGui/{qt_gui_dirs[0]}/QtGui", ] if qt_gui_dirs else []
qt_dirs += [f"{qt_install_headers}/Qt{m}" for m in qt_modules]
qt_libs = [f"Qt5{m}" for m in qt_modules]
if arch == "larch64":
qt_libs += ["GLESv2", "wayland-client"]
qt_env.PrependENVPath('PATH', Dir("#third_party/qt5/larch64/bin/").abspath)
elif arch != "Darwin":
qt_libs += ["GL"]
qt_env['QT3DIR'] = qt_env['QTDIR']
# compatibility for older SCons versions
try:
qt_env.Tool('qt3')
except SCons.Errors.UserError:
qt_env.Tool('qt')
qt_env['CPPPATH'] += qt_dirs + ["#third_party/qrcode"]
qt_flags = [
"-D_REENTRANT",
"-DQT_NO_DEBUG",
"-DQT_WIDGETS_LIB",
"-DQT_GUI_LIB",
"-DQT_CORE_LIB",
"-DQT_MESSAGELOGCONTEXT",
]
qt_env['CXXFLAGS'] += qt_flags
qt_env['LIBPATH'] += ['#selfdrive/ui', ]
qt_env['LIBS'] = qt_libs
if GetOption("clazy"):
checks = [
"level0",
"level1",
"no-range-loop",
"no-non-pod-global-static",
]
qt_env['CXX'] = 'clazy'
qt_env['ENV']['CLAZY_IGNORE_DIRS'] = qt_dirs[0]
qt_env['ENV']['CLAZY_CHECKS'] = ','.join(checks)
Export('env', 'qt_env', 'arch', 'real_arch')
# Build common module
SConscript(['openpilot/common/SConscript'])
Import('_common')
SConscript(['common/SConscript'])
Import('_common', '_gpucommon')
common = [_common, 'json11', 'zmq']
Export('common')
gpucommon = [_gpucommon]
Export('common', 'gpucommon')
# Build messaging (cereal + msgq + socketmaster + their dependencies)
# Enable swaglog include in submodules
env_swaglog = env.Clone()
env_swaglog['CXXFLAGS'].append('-DSWAGLOG="\\"common/swaglog.h\\""')
SConscript(['msgq_repo/SConscript'], exports={'env': env_swaglog})
SConscript(['opendbc_repo/SConscript'], exports={'env': env_swaglog})
SConscript(['openpilot/cereal/SConscript'])
SConscript(['cereal/SConscript'])
Import('socketmaster', 'msgq')
messaging = [socketmaster, msgq, 'capnp', 'kj',]
messaging = [socketmaster, msgq, 'zmq', 'capnp', 'kj',]
Export('messaging')
@@ -268,79 +369,36 @@ Export('messaging')
SConscript(['panda/SConscript'])
# Build rednose library
SConscript(['rednose_repo/rednose/SConscript'])
SConscript(['rednose/SConscript'])
# Build system services
SConscript([
'openpilot/system/loggerd/SConscript',
'system/ui/SConscript',
'system/proclogd/SConscript',
'system/ubloxd/SConscript',
'system/loggerd/SConscript',
])
if arch == "larch64":
SConscript(['openpilot/system/camerad/SConscript'])
# Build selfdrive
SConscript([
'openpilot/selfdrive/pandad/SConscript',
'openpilot/selfdrive/controls/lib/longitudinal_mpc_lib/SConscript',
'openpilot/selfdrive/locationd/SConscript',
'openpilot/selfdrive/modeld/SConscript',
'openpilot/selfdrive/ui/SConscript',
])
SConscript(['openpilot/sunnypilot/SConscript'])
# Build desktop-only tools
if GetOption('extras') and arch != "larch64":
if arch != "Darwin":
SConscript([
'openpilot/tools/replay/SConscript',
'openpilot/tools/cabana/SConscript',
'openpilot/tools/jotpluggler/SConscript',
'system/sensord/SConscript',
'system/logcatd/SConscript',
])
if arch == "larch64":
SConscript(['system/camerad/SConscript'])
env.CompilationDatabase('compile_commands.json')
# Build openpilot
SConscript(['third_party/SConscript'])
# progress output
def count_scons_nodes(nodes):
seen = set()
stack = list(nodes)
SConscript(['selfdrive/SConscript'])
while stack:
node = stack.pop().disambiguate()
if node in seen:
continue
seen.add(node)
if hasattr(node, 'has_builder') and node.has_builder():
build_product_nodes.add(node)
executor = node.get_executor()
if executor is not None:
stack += executor.get_all_prerequisites() + executor.get_all_children()
SConscript(['sunnypilot/SConscript'])
return len(seen)
if Dir('#tools/cabana/').exists() and GetOption('extras'):
SConscript(['tools/replay/SConscript'])
if arch != "larch64":
SConscript(['tools/cabana/SConscript'])
progress_interval = 5
progress_count = 0
build_product_nodes = set()
progress_total = max(1, count_scons_nodes(env.arg2nodes(BUILD_TARGETS or [Dir('.')], env.fs.Entry)))
def progress_function(node):
global progress_count
if progress_count >= progress_total:
return
progress_count = min(progress_count + progress_interval, progress_total)
progress = round(100. * progress_count / progress_total, 1)
sys.stderr.write("\rBuilding: %5.1f%%" % progress if sys.stderr.isatty() else "progress: %.1f\n" % progress)
if progress == 100. and sys.stderr.isatty():
sys.stderr.write("\n")
sys.stderr.flush()
Progress(progress_function, interval=progress_interval)
AddPostAction(BUILD_TARGETS or [Dir('.')], prune_cache_dir)
def check_build_product_size(target, source, env):
limit = 50 * 1024 * 1024 # GitHub max size
for t in target:
if hasattr(t, 'isfile') and t.isfile() and (size := os.path.getsize(t.abspath)) > limit:
raise SCons.Errors.UserError(f"{t} is {size / (1024 * 1024):.1f} MiB, exceeding the {limit / (1024 * 1024):.1f} MiB limit")
if not GetOption('extras'):
AddPostAction(list(build_product_nodes), Action(check_build_product_size, None))
external_sconscript = GetOption('external_sconscript')
if external_sconscript:
SConscript([external_sconscript])
+51
View File
@@ -0,0 +1,51 @@
# What is cereal?
cereal is the messaging system for openpilot. It uses [msgq](https://github.com/commaai/msgq) as a pub/sub backend, and [Cap'n proto](https://capnproto.org/capnp-tool.html) for serialization of the structs.
## Messaging Spec
You'll find the message types in [log.capnp](log.capnp). It uses [Cap'n proto](https://capnproto.org/capnp-tool.html) and defines one struct called `Event`.
All `Events` have a `logMonoTime` and a `valid`. Then a big union defines the packet type.
### Best Practices
- **All fields must describe quantities in SI units**, unless otherwise specified in the field name.
- In the context of the message they are in, field names should be completely unambiguous.
- All values should be easy to plot and be human-readable with minimal parsing.
### Maintaining backwards-compatibility
When making changes to the messaging spec you want to maintain backwards-compatibility, such that old logs can
be parsed with a new version of cereal. Adding structs and adding members to structs is generally safe, most other
things are not. Read more details [here](https://capnproto.org/language.html).
### Custom forks
Forks of [openpilot](https://github.com/commaai/openpilot) might want to add things to the messaging
spec, however this could conflict with future changes made in mainline cereal/openpilot. Rebasing against mainline openpilot
then means breaking backwards-compatibility with all old logs of your fork. So we added reserved events in
[custom.capnp](custom.capnp) that we will leave empty in mainline cereal/openpilot. **If you only modify those, you can ensure your
fork will remain backwards-compatible with all versions of mainline openpilot and your fork.**
Example
---
```python
import cereal.messaging as messaging
# in subscriber
sm = messaging.SubMaster(['sensorEvents'])
while 1:
sm.update()
print(sm['sensorEvents'])
```
```python
# in publisher
pm = messaging.PubMaster(['sensorEvents'])
dat = messaging.new_message('sensorEvents', size=1)
dat.sensorEvents[0] = {"gyro": {"v": [0.1, -0.1, 0.1]}}
pm.send('sensorEvents', dat)
```
+20
View File
@@ -0,0 +1,20 @@
Import('env', 'common', 'msgq')
cereal_dir = Dir('.')
gen_dir = Dir('gen')
# Build cereal
schema_files = ['log.capnp', 'car.capnp', 'legacy.capnp', 'custom.capnp']
env.Command([f'gen/cpp/{s}.c++' for s in schema_files] + [f'gen/cpp/{s}.h' for s in schema_files],
schema_files,
f"capnpc --src-prefix={cereal_dir.path} $SOURCES -o c++:{gen_dir.path}/cpp/")
cereal = env.Library('cereal', [f'gen/cpp/{s}.c++' for s in schema_files])
# Build messaging
services_h = env.Command(['services.h'], ['services.py'], 'python3 ' + cereal_dir.path + '/services.py > $TARGET')
env.Program('messaging/bridge', ['messaging/bridge.cc', 'messaging/msgq_to_zmq.cc'], LIBS=[msgq, common, 'pthread'])
socketmaster = env.Library('socketmaster', ['messaging/socketmaster.cc'])
Export('cereal', 'socketmaster')
+9
View File
@@ -0,0 +1,9 @@
import os
import capnp
CEREAL_PATH = os.path.dirname(os.path.abspath(__file__))
capnp.remove_import_hook()
log = capnp.load(os.path.join(CEREAL_PATH, "log.capnp"))
car = capnp.load(os.path.join(CEREAL_PATH, "car.capnp"))
custom = capnp.load(os.path.join(CEREAL_PATH, "custom.capnp"))
+1
View File
@@ -0,0 +1 @@
../opendbc_repo/opendbc/car/car.capnp
+107
View File
@@ -0,0 +1,107 @@
using Cxx = import "./include/c++.capnp";
$Cxx.namespace("cereal");
@0xb526ba661d550a59;
# custom.capnp: a home for empty structs reserved for custom forks
# These structs are guaranteed to remain reserved and empty in mainline
# cereal, so use these if you want custom events in your fork.
# you can rename the struct, but don't change the identifier
struct SelfdriveStateSP @0x81c2f05a394cf4af {
mads @0 :ModularAssistiveDrivingSystem;
struct ModularAssistiveDrivingSystem {
state @0 :ModularAssistiveDrivingSystemState;
enabled @1 :Bool;
active @2 :Bool;
available @3 :Bool;
enum ModularAssistiveDrivingSystemState {
disabled @0;
paused @1;
enabled @2;
softDisabling @3;
overriding @4;
}
}
}
struct ModelManagerSP @0xaedffd8f31e7b55d {
activeBundle @0 :ModelBundle;
selectedBundle @1 :ModelBundle;
availableBundles @2 :List(ModelBundle);
struct DownloadUri {
uri @0 :Text;
sha256 @1 :Text;
}
enum Type {
drive @0;
navigation @1;
metadata @2;
}
struct Model {
fullName @0 :Text;
fileName @1 :Text;
downloadUri @2 :DownloadUri;
downloadProgress @3 :DownloadProgress;
type @4 :Type;
}
enum DownloadStatus {
notDownloading @0;
downloading @1;
downloaded @2;
cached @3;
failed @4;
}
struct DownloadProgress {
status @0 :DownloadStatus;
progress @1 :Float32;
eta @2 :UInt32;
}
enum Runner {
snpe @0;
tinygrad @1;
}
struct ModelBundle {
index @0 :UInt32;
internalName @1 :Text;
displayName @2 :Text;
models @3 :List(Model);
status @4 :DownloadStatus;
generation @5 :UInt32;
environment @6 :Text;
runner @7 :Runner;
}
}
struct CustomReserved2 @0xf35cc4560bbf6ec2 {
}
struct CustomReserved3 @0xda96579883444c35 {
}
struct CustomReserved4 @0x80ae746ee2596b11 {
}
struct CustomReserved5 @0xa5cd762cd951a455 {
}
struct CustomReserved6 @0xf98d843bfd7004a3 {
}
struct CustomReserved7 @0xb86e6369214c01c8 {
}
struct CustomReserved8 @0xf416ec09499d9d19 {
}
struct CustomReserved9 @0xa1680744031fdb2d {
}
+574
View File
@@ -0,0 +1,574 @@
using Cxx = import "./include/c++.capnp";
$Cxx.namespace("cereal");
@0x80ef1ec4889c2a63;
# legacy.capnp: a home for deprecated structs
struct LogRotate @0x9811e1f38f62f2d1 {
segmentNum @0 :Int32;
path @1 :Text;
}
struct LiveUI @0xc08240f996aefced {
rearViewCam @0 :Bool;
alertText1 @1 :Text;
alertText2 @2 :Text;
awarenessStatus @3 :Float32;
}
struct UiLayoutState @0x88dcce08ad29dda0 {
activeApp @0 :App;
sidebarCollapsed @1 :Bool;
mapEnabled @2 :Bool;
mockEngaged @3 :Bool;
enum App @0x9917470acf94d285 {
home @0;
music @1;
nav @2;
settings @3;
none @4;
}
}
struct OrbslamCorrection @0x8afd33dc9b35e1aa {
correctionMonoTime @0 :UInt64;
prePositionECEF @1 :List(Float64);
postPositionECEF @2 :List(Float64);
prePoseQuatECEF @3 :List(Float32);
postPoseQuatECEF @4 :List(Float32);
numInliers @5 :UInt32;
}
struct EthernetPacket @0xa99a9d5b33cf5859 {
pkt @0 :Data;
ts @1 :Float32;
}
struct CellInfo @0xcff7566681c277ce {
timestamp @0 :UInt64;
repr @1 :Text; # android toString() for now
}
struct WifiScan @0xd4df5a192382ba0b {
bssid @0 :Text;
ssid @1 :Text;
capabilities @2 :Text;
frequency @3 :Int32;
level @4 :Int32;
timestamp @5 :Int64;
centerFreq0 @6 :Int32;
centerFreq1 @7 :Int32;
channelWidth @8 :ChannelWidth;
operatorFriendlyName @9 :Text;
venueName @10 :Text;
is80211mcResponder @11 :Bool;
passpoint @12 :Bool;
distanceCm @13 :Int32;
distanceSdCm @14 :Int32;
enum ChannelWidth @0xcb6a279f015f6b51 {
w20Mhz @0;
w40Mhz @1;
w80Mhz @2;
w160Mhz @3;
w80Plus80Mhz @4;
}
}
struct LiveEventData @0x94b7baa90c5c321e {
name @0 :Text;
value @1 :Int32;
}
struct ModelData @0xb8aad62cffef28a9 {
frameId @0 :UInt32;
frameAge @12 :UInt32;
frameDropPerc @13 :Float32;
timestampEof @9 :UInt64;
modelExecutionTime @14 :Float32;
gpuExecutionTime @16 :Float32;
rawPred @15 :Data;
path @1 :PathData;
leftLane @2 :PathData;
rightLane @3 :PathData;
lead @4 :LeadData;
freePath @6 :List(Float32);
settings @5 :ModelSettings;
leadFuture @7 :LeadData;
speed @8 :List(Float32);
meta @10 :MetaData;
longitudinal @11 :LongitudinalData;
struct PathData @0x8817eeea389e9f08 {
points @0 :List(Float32);
prob @1 :Float32;
std @2 :Float32;
stds @3 :List(Float32);
poly @4 :List(Float32);
validLen @5 :Float32;
}
struct LeadData @0xd1c9bef96d26fa91 {
dist @0 :Float32;
prob @1 :Float32;
std @2 :Float32;
relVel @3 :Float32;
relVelStd @4 :Float32;
relY @5 :Float32;
relYStd @6 :Float32;
relA @7 :Float32;
relAStd @8 :Float32;
}
struct ModelSettings @0xa26e3710efd3e914 {
bigBoxX @0 :UInt16;
bigBoxY @1 :UInt16;
bigBoxWidth @2 :UInt16;
bigBoxHeight @3 :UInt16;
boxProjection @4 :List(Float32);
yuvCorrection @5 :List(Float32);
inputTransform @6 :List(Float32);
}
struct MetaData @0x9744f25fb60f2bf8 {
engagedProb @0 :Float32;
desirePrediction @1 :List(Float32);
brakeDisengageProb @2 :Float32;
gasDisengageProb @3 :Float32;
steerOverrideProb @4 :Float32;
desireState @5 :List(Float32);
}
struct LongitudinalData @0xf98f999c6a071122 {
distances @2 :List(Float32);
speeds @0 :List(Float32);
accelerations @1 :List(Float32);
}
}
struct ECEFPoint @0xc25bbbd524983447 {
x @0 :Float64;
y @1 :Float64;
z @2 :Float64;
}
struct ECEFPointDEPRECATED @0xe10e21168db0c7f7 {
x @0 :Float32;
y @1 :Float32;
z @2 :Float32;
}
struct GPSPlannerPoints @0xab54c59699f8f9f3 {
curPosDEPRECATED @0 :ECEFPointDEPRECATED;
pointsDEPRECATED @1 :List(ECEFPointDEPRECATED);
curPos @6 :ECEFPoint;
points @7 :List(ECEFPoint);
valid @2 :Bool;
trackName @3 :Text;
speedLimit @4 :Float32;
accelTarget @5 :Float32;
}
struct GPSPlannerPlan @0xf5ad1d90cdc1dd6b {
valid @0 :Bool;
poly @1 :List(Float32);
trackName @2 :Text;
speed @3 :Float32;
acceleration @4 :Float32;
pointsDEPRECATED @5 :List(ECEFPointDEPRECATED);
points @6 :List(ECEFPoint);
xLookahead @7 :Float32;
}
struct UiNavigationEvent @0x90c8426c3eaddd3b {
type @0: Type;
status @1: Status;
distanceTo @2: Float32;
endRoadPointDEPRECATED @3: ECEFPointDEPRECATED;
endRoadPoint @4: ECEFPoint;
enum Type @0xe8db07dcf8fcea05 {
none @0;
laneChangeLeft @1;
laneChangeRight @2;
mergeLeft @3;
mergeRight @4;
turnLeft @5;
turnRight @6;
}
enum Status @0xb9aa88c75ef99a1f {
none @0;
passive @1;
approaching @2;
active @3;
}
}
struct LiveLocationData @0xb99b2bc7a57e8128 {
status @0 :UInt8;
# 3D fix
lat @1 :Float64;
lon @2 :Float64;
alt @3 :Float32; # m
# speed
speed @4 :Float32; # m/s
# NED velocity components
vNED @5 :List(Float32);
# roll, pitch, heading (x,y,z)
roll @6 :Float32; # WRT to center of earth?
pitch @7 :Float32; # WRT to center of earth?
heading @8 :Float32; # WRT to north?
# what are these?
wanderAngle @9 :Float32;
trackAngle @10 :Float32;
# car frame -- https://upload.wikimedia.org/wikipedia/commons/f/f5/RPY_angles_of_cars.png
# gyro, in car frame, deg/s
gyro @11 :List(Float32);
# accel, in car frame, m/s^2
accel @12 :List(Float32);
accuracy @13 :Accuracy;
source @14 :SensorSource;
# if we are fixing a location in the past
fixMonoTime @15 :UInt64;
gpsWeek @16 :Int32;
timeOfWeek @17 :Float64;
positionECEF @18 :List(Float64);
poseQuatECEF @19 :List(Float32);
pitchCalibration @20 :Float32;
yawCalibration @21 :Float32;
imuFrame @22 :List(Float32);
struct Accuracy @0x943dc4625473b03f {
pNEDError @0 :List(Float32);
vNEDError @1 :List(Float32);
rollError @2 :Float32;
pitchError @3 :Float32;
headingError @4 :Float32;
ellipsoidSemiMajorError @5 :Float32;
ellipsoidSemiMinorError @6 :Float32;
ellipsoidOrientationError @7 :Float32;
}
enum SensorSource @0xc871d3cc252af657 {
applanix @0;
kalman @1;
orbslam @2;
timing @3;
dummy @4;
}
}
struct OrbOdometry @0xd7700859ed1f5b76 {
# timing first
startMonoTime @0 :UInt64;
endMonoTime @1 :UInt64;
# fundamental matrix and error
f @2: List(Float64);
err @3: Float64;
# number of inlier points
inliers @4: Int32;
# for debug only
# indexed by endMonoTime features
# value is startMonoTime feature match
# -1 if no match
matches @5: List(Int16);
}
struct OrbFeatures @0xcd60164a8a0159ef {
timestampEof @0 :UInt64;
# transposed arrays of normalized image coordinates
# len(xs) == len(ys) == len(descriptors) * 32
xs @1 :List(Float32);
ys @2 :List(Float32);
descriptors @3 :Data;
octaves @4 :List(Int8);
# match index to last OrbFeatures
# -1 if no match
timestampLastEof @5 :UInt64;
matches @6: List(Int16);
}
struct OrbFeaturesSummary @0xd500d30c5803fa4f {
timestampEof @0 :UInt64;
timestampLastEof @1 :UInt64;
featureCount @2 :UInt16;
matchCount @3 :UInt16;
computeNs @4 :UInt64;
}
struct OrbKeyFrame @0xc8233c0345e27e24 {
# this is a globally unique id for the KeyFrame
id @0: UInt64;
# this is the location of the KeyFrame
pos @1: ECEFPoint;
# these are the features in the world
# len(dpos) == len(descriptors) * 32
dpos @2 :List(ECEFPoint);
descriptors @3 :Data;
}
struct KalmanOdometry @0x92e21bb7ea38793a {
trans @0 :List(Float32); # m/s in device frame
rot @1 :List(Float32); # rad/s in device frame
transStd @2 :List(Float32); # std m/s in device frame
rotStd @3 :List(Float32); # std rad/s in device frame
}
struct OrbObservation @0x9b326d4e436afec7 {
observationMonoTime @0 :UInt64;
normalizedCoordinates @1 :List(Float32);
locationECEF @2 :List(Float64);
matchDistance @3: UInt32;
}
struct CalibrationFeatures @0x8fdfadb254ea867a {
frameId @0 :UInt32;
p0 @1 :List(Float32);
p1 @2 :List(Float32);
status @3 :List(Int8);
}
struct NavStatus @0xbd8822120928120c {
isNavigating @0 :Bool;
currentAddress @1 :Address;
struct Address @0xce7cd672cacc7814 {
title @0 :Text;
lat @1 :Float64;
lng @2 :Float64;
house @3 :Text;
address @4 :Text;
street @5 :Text;
city @6 :Text;
state @7 :Text;
country @8 :Text;
}
}
struct NavUpdate @0xdb98be6565516acb {
isNavigating @0 :Bool;
curSegment @1 :Int32;
segments @2 :List(Segment);
struct LatLng @0x9eaef9187cadbb9b {
lat @0 :Float64;
lng @1 :Float64;
}
struct Segment @0xa5b39b4fc4d7da3f {
from @0 :LatLng;
to @1 :LatLng;
updateTime @2 :Int32;
distance @3 :Int32;
crossTime @4 :Int32;
exitNo @5 :Int32;
instruction @6 :Instruction;
parts @7 :List(LatLng);
enum Instruction @0xc5417a637451246f {
turnLeft @0;
turnRight @1;
keepLeft @2;
keepRight @3;
straight @4;
roundaboutExitNumber @5;
roundaboutExit @6;
roundaboutTurnLeft @7;
unkn8 @8;
roundaboutStraight @9;
unkn10 @10;
roundaboutTurnRight @11;
unkn12 @12;
roundaboutUturn @13;
unkn14 @14;
arrive @15;
exitLeft @16;
exitRight @17;
unkn18 @18;
uturn @19;
# ...
}
}
}
struct TrafficEvent @0xacfa74a094e62626 {
type @0 :Type;
distance @1 :Float32;
action @2 :Action;
resuming @3 :Bool;
enum Type @0xd85d75253435bf4b {
stopSign @0;
lightRed @1;
lightYellow @2;
lightGreen @3;
stopLight @4;
}
enum Action @0xa6f6ce72165ccb49 {
none @0;
yield @1;
stop @2;
resumeReady @3;
}
}
struct AndroidGnss @0xdfdf30d03fc485bd {
union {
measurements @0 :Measurements;
navigationMessage @1 :NavigationMessage;
}
struct Measurements @0xa20710d4f428d6cd {
clock @0 :Clock;
measurements @1 :List(Measurement);
struct Clock @0xa0e27b453a38f450 {
timeNanos @0 :Int64;
hardwareClockDiscontinuityCount @1 :Int32;
hasTimeUncertaintyNanos @2 :Bool;
timeUncertaintyNanos @3 :Float64;
hasLeapSecond @4 :Bool;
leapSecond @5 :Int32;
hasFullBiasNanos @6 :Bool;
fullBiasNanos @7 :Int64;
hasBiasNanos @8 :Bool;
biasNanos @9 :Float64;
hasBiasUncertaintyNanos @10 :Bool;
biasUncertaintyNanos @11 :Float64;
hasDriftNanosPerSecond @12 :Bool;
driftNanosPerSecond @13 :Float64;
hasDriftUncertaintyNanosPerSecond @14 :Bool;
driftUncertaintyNanosPerSecond @15 :Float64;
}
struct Measurement @0xd949bf717d77614d {
svId @0 :Int32;
constellation @1 :Constellation;
timeOffsetNanos @2 :Float64;
state @3 :Int32;
receivedSvTimeNanos @4 :Int64;
receivedSvTimeUncertaintyNanos @5 :Int64;
cn0DbHz @6 :Float64;
pseudorangeRateMetersPerSecond @7 :Float64;
pseudorangeRateUncertaintyMetersPerSecond @8 :Float64;
accumulatedDeltaRangeState @9 :Int32;
accumulatedDeltaRangeMeters @10 :Float64;
accumulatedDeltaRangeUncertaintyMeters @11 :Float64;
hasCarrierFrequencyHz @12 :Bool;
carrierFrequencyHz @13 :Float32;
hasCarrierCycles @14 :Bool;
carrierCycles @15 :Int64;
hasCarrierPhase @16 :Bool;
carrierPhase @17 :Float64;
hasCarrierPhaseUncertainty @18 :Bool;
carrierPhaseUncertainty @19 :Float64;
hasSnrInDb @20 :Bool;
snrInDb @21 :Float64;
multipathIndicator @22 :MultipathIndicator;
enum Constellation @0x9ef1f3ff0deb5ffb {
unknown @0;
gps @1;
sbas @2;
glonass @3;
qzss @4;
beidou @5;
galileo @6;
}
enum State @0xcbb9490adce12d72 {
unknown @0;
codeLock @1;
bitSync @2;
subframeSync @3;
towDecoded @4;
msecAmbiguous @5;
symbolSync @6;
gloStringSync @7;
gloTodDecoded @8;
bdsD2BitSync @9;
bdsD2SubframeSync @10;
galE1bcCodeLock @11;
galE1c2ndCodeLock @12;
galE1bPageSync @13;
sbasSync @14;
}
enum MultipathIndicator @0xc04e7b6231d4caa8 {
unknown @0;
detected @1;
notDetected @2;
}
}
}
struct NavigationMessage @0xe2517b083095fd4e {
type @0 :Int32;
svId @1 :Int32;
messageId @2 :Int32;
submessageId @3 :Int32;
data @4 :Data;
status @5 :Status;
enum Status @0xec1ff7996b35366f {
unknown @0;
parityPassed @1;
parityRebuilt @2;
}
}
}
struct LidarPts @0xe3d6685d4e9d8f7a {
r @0 :List(UInt16); # uint16 m*500.0
theta @1 :List(UInt16); # uint16 deg*100.0
reflect @2 :List(UInt8); # uint8 0-255
# For storing out of file.
idx @3 :UInt64;
# For storing in file
pkt @4 :Data;
}
+2693
View File
File diff suppressed because it is too large Load Diff
+268
View File
@@ -0,0 +1,268 @@
# must be built with scons
from msgq.ipc_pyx import Context, Poller, SubSocket, PubSocket, SocketEventHandle, toggle_fake_events, \
set_fake_prefix, get_fake_prefix, delete_fake_prefix, wait_for_one_event
from msgq.ipc_pyx import MultiplePublishersError, IpcError
from msgq import fake_event_handle, pub_sock, sub_sock, drain_sock_raw
import msgq
import os
import capnp
import time
from typing import Optional, List, Union, Dict, Deque
from collections import deque
from cereal import log
from cereal.services import SERVICE_LIST
NO_TRAVERSAL_LIMIT = 2**64-1
def reset_context():
msgq.context = Context()
def log_from_bytes(dat: bytes, struct: capnp.lib.capnp._StructModule = log.Event) -> capnp.lib.capnp._DynamicStructReader:
with struct.from_bytes(dat, traversal_limit_in_words=NO_TRAVERSAL_LIMIT) as msg:
return msg
def new_message(service: Optional[str], size: Optional[int] = None, **kwargs) -> capnp.lib.capnp._DynamicStructBuilder:
args = {
'valid': False,
'logMonoTime': int(time.monotonic() * 1e9),
**kwargs
}
dat = log.Event.new_message(**args)
if service is not None:
if size is None:
dat.init(service)
else:
dat.init(service, size)
return dat
def drain_sock(sock: SubSocket, wait_for_one: bool = False) -> List[capnp.lib.capnp._DynamicStructReader]:
"""Receive all message currently available on the queue"""
msgs = drain_sock_raw(sock, wait_for_one=wait_for_one)
return [log_from_bytes(m) for m in msgs]
# TODO: print when we drop packets?
def recv_sock(sock: SubSocket, wait: bool = False) -> Optional[capnp.lib.capnp._DynamicStructReader]:
"""Same as drain sock, but only returns latest message. Consider using conflate instead."""
dat = None
while 1:
if wait and dat is None:
recv = sock.receive()
else:
recv = sock.receive(non_blocking=True)
if recv is None: # Timeout hit
break
dat = recv
if dat is not None:
dat = log_from_bytes(dat)
return dat
def recv_one(sock: SubSocket) -> Optional[capnp.lib.capnp._DynamicStructReader]:
dat = sock.receive()
if dat is not None:
dat = log_from_bytes(dat)
return dat
def recv_one_or_none(sock: SubSocket) -> Optional[capnp.lib.capnp._DynamicStructReader]:
dat = sock.receive(non_blocking=True)
if dat is not None:
dat = log_from_bytes(dat)
return dat
def recv_one_retry(sock: SubSocket) -> capnp.lib.capnp._DynamicStructReader:
"""Keep receiving until we get a message"""
while True:
dat = sock.receive()
if dat is not None:
return log_from_bytes(dat)
class FrequencyTracker:
def __init__(self, service_freq: float, update_freq: float, is_poll: bool):
freq = max(min(service_freq, update_freq), 1.)
if is_poll:
min_freq = max_freq = freq
else:
max_freq = min(freq, update_freq)
if service_freq >= 2 * update_freq:
min_freq = update_freq
elif update_freq >= 2* service_freq:
min_freq = freq
else:
min_freq = min(freq, freq / 2.)
self.min_freq = min_freq * 0.8
self.max_freq = max_freq * 1.2
self.recv_dts: Deque[float] = deque(maxlen=int(10 * freq))
self.recv_dts_sum = 0.0
self.recent_recv_dts: Deque[float] = deque(maxlen=int(freq))
self.recent_recv_dts_sum = 0.0
self.prev_time = 0.0
def record_recv_time(self, cur_time: float) -> None:
# TODO: Handle case where cur_time is less than prev_time
if self.prev_time > 1e-5:
dt = cur_time - self.prev_time
if len(self.recv_dts) == self.recv_dts.maxlen:
self.recv_dts_sum -= self.recv_dts[0]
self.recv_dts.append(dt)
self.recv_dts_sum += dt
if len(self.recent_recv_dts) == self.recent_recv_dts.maxlen:
self.recent_recv_dts_sum -= self.recent_recv_dts[0]
self.recent_recv_dts.append(dt)
self.recent_recv_dts_sum += dt
self.prev_time = cur_time
@property
def valid(self) -> bool:
if not self.recv_dts:
return False
avg_freq = len(self.recv_dts) / self.recv_dts_sum
if self.min_freq <= avg_freq <= self.max_freq:
return True
avg_freq_recent = len(self.recent_recv_dts) / self.recent_recv_dts_sum
return self.min_freq <= avg_freq_recent <= self.max_freq
class SubMaster:
def __init__(self, services: List[str], poll: Optional[str] = None,
ignore_alive: Optional[List[str]] = None, ignore_avg_freq: Optional[List[str]] = None,
ignore_valid: Optional[List[str]] = None, addr: str = "127.0.0.1", frequency: Optional[float] = None):
self.frame = -1
self.services = services
self.seen = {s: False for s in services}
self.updated = {s: False for s in services}
self.recv_time = {s: 0. for s in services}
self.recv_frame = {s: 0 for s in services}
self.alive = {s: False for s in services}
self.freq_ok = {s: False for s in services}
self.sock = {}
self.data = {}
self.valid = {}
self.logMonoTime = {}
self.freq_tracker: Dict[str, FrequencyTracker] = {}
self.poller = Poller()
polled_services = set([poll, ] if poll is not None else services)
self.non_polled_services = set(services) - polled_services
self.ignore_average_freq = [] if ignore_avg_freq is None else ignore_avg_freq
self.ignore_alive = [] if ignore_alive is None else ignore_alive
self.ignore_valid = [] if ignore_valid is None else ignore_valid
self.simulation = bool(int(os.getenv("SIMULATION", "0")))
# if freq and poll aren't specified, assume the max to be conservative
assert frequency is None or poll is None, "Do not specify 'frequency' - frequency of the polled service will be used."
self.update_freq = frequency or max([SERVICE_LIST[s].frequency for s in polled_services])
for s in services:
p = self.poller if s not in self.non_polled_services else None
self.sock[s] = sub_sock(s, poller=p, addr=addr, conflate=True)
try:
data = new_message(s)
except capnp.lib.capnp.KjException:
data = new_message(s, 0) # lists
self.data[s] = getattr(data.as_reader(), s)
self.logMonoTime[s] = 0
self.valid[s] = False
self.freq_tracker[s] = FrequencyTracker(SERVICE_LIST[s].frequency, self.update_freq, s == poll)
def __getitem__(self, s: str) -> capnp.lib.capnp._DynamicStructReader:
return self.data[s]
def _check_avg_freq(self, s: str) -> bool:
return SERVICE_LIST[s].frequency > 0.99 and (s not in self.ignore_average_freq) and (s not in self.ignore_alive)
def update(self, timeout: int = 100) -> None:
msgs = []
for sock in self.poller.poll(timeout):
msgs.append(recv_one_or_none(sock))
# non-blocking receive for non-polled sockets
for s in self.non_polled_services:
msgs.append(recv_one_or_none(self.sock[s]))
self.update_msgs(time.monotonic(), msgs)
def update_msgs(self, cur_time: float, msgs: List[capnp.lib.capnp._DynamicStructReader]) -> None:
self.frame += 1
self.updated = dict.fromkeys(self.services, False)
for msg in msgs:
if msg is None:
continue
s = msg.which()
self.seen[s] = True
self.updated[s] = True
self.freq_tracker[s].record_recv_time(cur_time)
self.recv_time[s] = cur_time
self.recv_frame[s] = self.frame
self.data[s] = getattr(msg, s)
self.logMonoTime[s] = msg.logMonoTime
self.valid[s] = msg.valid
for s in self.services:
if SERVICE_LIST[s].frequency > 1e-5 and not self.simulation:
# alive if delay is within 10x the expected frequency
self.alive[s] = (cur_time - self.recv_time[s]) < (10. / SERVICE_LIST[s].frequency)
self.freq_ok[s] = self.freq_tracker[s].valid
else:
self.freq_ok[s] = True
self.alive[s] = self.seen[s] if self.simulation else True
def all_alive(self, service_list: Optional[List[str]] = None) -> bool:
return all(self.alive[s] for s in (service_list or self.services) if s not in self.ignore_alive)
def all_freq_ok(self, service_list: Optional[List[str]] = None) -> bool:
return all(self.freq_ok[s] for s in (service_list or self.services) if self._check_avg_freq(s))
def all_valid(self, service_list: Optional[List[str]] = None) -> bool:
return all(self.valid[s] for s in (service_list or self.services) if s not in self.ignore_valid)
def all_checks(self, service_list: Optional[List[str]] = None) -> bool:
return self.all_alive(service_list) and self.all_freq_ok(service_list) and self.all_valid(service_list)
class PubMaster:
def __init__(self, services: List[str]):
self.sock = {}
for s in services:
self.sock[s] = pub_sock(s)
def send(self, s: str, dat: Union[bytes, capnp.lib.capnp._DynamicStructBuilder]) -> None:
if not isinstance(dat, bytes):
dat = dat.to_bytes()
self.sock[s].send(dat)
def wait_for_readers_to_update(self, s: str, timeout: int, dt: float = 0.05) -> bool:
for _ in range(int(timeout*(1./dt))):
if self.sock[s].all_readers_updated():
return True
time.sleep(dt)
return False
def all_readers_updated(self, s: str) -> bool:
return self.sock[s].all_readers_updated() # type: ignore
+71
View File
@@ -0,0 +1,71 @@
#include <cassert>
#include "cereal/messaging/msgq_to_zmq.h"
#include "cereal/services.h"
#include "common/util.h"
ExitHandler do_exit;
static std::vector<std::string> get_services(std::string whitelist_str, bool zmq_to_msgq) {
std::vector<std::string> service_list;
for (const auto& it : services) {
std::string name = it.second.name;
bool in_whitelist = whitelist_str.find(name) != std::string::npos;
if (name == "plusFrame" || name == "uiLayoutState" || (zmq_to_msgq && !in_whitelist)) {
continue;
}
service_list.push_back(name);
}
return service_list;
}
void msgq_to_zmq(const std::vector<std::string> &endpoints, const std::string &ip) {
MsgqToZmq bridge;
bridge.run(endpoints, ip);
}
void zmq_to_msgq(const std::vector<std::string> &endpoints, const std::string &ip) {
auto poller = std::make_unique<ZMQPoller>();
auto pub_context = std::make_unique<MSGQContext>();
auto sub_context = std::make_unique<ZMQContext>();
std::map<SubSocket *, PubSocket *> sub2pub;
for (auto endpoint : endpoints) {
auto pub_sock = new MSGQPubSocket();
auto sub_sock = new ZMQSubSocket();
pub_sock->connect(pub_context.get(), endpoint);
sub_sock->connect(sub_context.get(), endpoint, ip, false);
poller->registerSocket(sub_sock);
sub2pub[sub_sock] = pub_sock;
}
while (!do_exit) {
for (auto sub_sock : poller->poll(100)) {
std::unique_ptr<Message> msg(sub_sock->receive(true));
if (msg) {
sub2pub[sub_sock]->sendMessage(msg.get());
}
}
}
// Clean up allocated sockets
for (auto &[sub_sock, pub_sock] : sub2pub) {
delete sub_sock;
delete pub_sock;
}
}
int main(int argc, char **argv) {
bool is_zmq_to_msgq = argc > 2;
std::string ip = is_zmq_to_msgq ? argv[1] : "127.0.0.1";
std::string whitelist_str = is_zmq_to_msgq ? std::string(argv[2]) : "";
std::vector<std::string> endpoints = get_services(whitelist_str, is_zmq_to_msgq);
if (is_zmq_to_msgq) {
zmq_to_msgq(endpoints, ip);
} else {
msgq_to_zmq(endpoints, ip);
}
return 0;
}
@@ -8,7 +8,7 @@
#include <capnp/serialize.h>
#include "openpilot/cereal/gen/cpp/log.capnp.h"
#include "cereal/gen/cpp/log.capnp.h"
#include "common/timing.h"
#include "msgq/ipc.h"
@@ -1,8 +1,7 @@
#include "openpilot/cereal/messaging/msgq_to_zmq.h"
#include "cereal/messaging/msgq_to_zmq.h"
#include <cassert>
#include "openpilot/cereal/services.h"
#include "common/util.h"
extern ExitHandler do_exit;
@@ -22,14 +21,14 @@ static std::string recv_zmq_msg(void *sock) {
}
void MsgqToZmq::run(const std::vector<std::string> &endpoints, const std::string &ip) {
zmq_context = std::make_unique<BridgeZmqContext>();
msgq_context = std::make_unique<Context>();
zmq_context = std::make_unique<ZMQContext>();
msgq_context = std::make_unique<MSGQContext>();
// Create ZMQPubSockets for each endpoint
for (const auto &endpoint : endpoints) {
auto &socket_pair = socket_pairs.emplace_back();
socket_pair.endpoint = endpoint;
socket_pair.pub_sock = std::make_unique<BridgeZmqPubSocket>();
socket_pair.pub_sock = std::make_unique<ZMQPubSocket>();
int ret = socket_pair.pub_sock->connect(zmq_context.get(), endpoint);
if (ret != 0) {
printf("Failed to create ZMQ publisher for [%s]: %s\n", endpoint.c_str(), zmq_strerror(zmq_errno()));
@@ -49,7 +48,7 @@ void MsgqToZmq::run(const std::vector<std::string> &endpoints, const std::string
for (auto sub_sock : msgq_poller->poll(100)) {
// Process messages for each socket
BridgeZmqPubSocket *pub_sock = sub2pub.at(sub_sock);
ZMQPubSocket *pub_sock = sub2pub.at(sub_sock);
for (int i = 0; i < MAX_MESSAGES_PER_SOCKET; ++i) {
auto msg = std::unique_ptr<Message>(sub_sock->receive(true));
if (!msg) break;
@@ -72,7 +71,7 @@ void MsgqToZmq::zmqMonitorThread() {
// Set up ZMQ monitor for each pub socket
for (int i = 0; i < socket_pairs.size(); ++i) {
std::string addr = "inproc://op-bridge-monitor-" + std::to_string(i);
zmq_socket_monitor(socket_pairs[i].pub_sock->getRawSocket(), addr.c_str(), ZMQ_EVENT_ACCEPTED | ZMQ_EVENT_DISCONNECTED);
zmq_socket_monitor(socket_pairs[i].pub_sock->sock, addr.c_str(), ZMQ_EVENT_ACCEPTED | ZMQ_EVENT_DISCONNECTED);
void *monitor_socket = zmq_socket(zmq_context->getRawContext(), ZMQ_PAIR);
zmq_connect(monitor_socket, addr.c_str());
@@ -109,8 +108,7 @@ void MsgqToZmq::zmqMonitorThread() {
if (++pair.connected_clients == 1) {
// Create new MSGQ subscriber socket and map to ZMQ publisher
pair.sub_sock = std::make_unique<MSGQSubSocket>();
size_t queue_size = services.at(pair.endpoint).queue_size;
pair.sub_sock->connect(msgq_context.get(), pair.endpoint, "127.0.0.1", false, true, queue_size);
pair.sub_sock->connect(msgq_context.get(), pair.endpoint, "127.0.0.1");
sub2pub[pair.sub_sock.get()] = pair.pub_sock.get();
registerSockets();
}
@@ -130,7 +128,7 @@ void MsgqToZmq::zmqMonitorThread() {
// Clean up monitor sockets
for (int i = 0; i < pollitems.size(); ++i) {
zmq_socket_monitor(socket_pairs[i].pub_sock->getRawSocket(), nullptr, 0);
zmq_socket_monitor(socket_pairs[i].pub_sock->sock, nullptr, 0);
zmq_close(pollitems[i].socket);
}
cv.notify_one();
+37
View File
@@ -0,0 +1,37 @@
#pragma once
#include <condition_variable>
#include <map>
#include <memory>
#include <mutex>
#include <string>
#include <vector>
#define private public
#include "msgq/impl_msgq.h"
#include "msgq/impl_zmq.h"
class MsgqToZmq {
public:
MsgqToZmq() {}
void run(const std::vector<std::string> &endpoints, const std::string &ip);
protected:
void registerSockets();
void zmqMonitorThread();
struct SocketPair {
std::string endpoint;
std::unique_ptr<ZMQPubSocket> pub_sock;
std::unique_ptr<MSGQSubSocket> sub_sock;
int connected_clients = 0;
};
std::unique_ptr<MSGQContext> msgq_context;
std::unique_ptr<ZMQContext> zmq_context;
std::mutex mutex;
std::condition_variable cv;
std::unique_ptr<MSGQPoller> msgq_poller;
std::map<SubSocket *, ZMQPubSocket *> sub2pub;
std::vector<SocketPair> socket_pairs;
};
@@ -3,8 +3,8 @@
#include <string>
#include <mutex>
#include "openpilot/cereal/services.h"
#include "openpilot/cereal/messaging/messaging.h"
#include "cereal/services.h"
#include "cereal/messaging/messaging.h"
const bool SIMULATION = (getenv("SIMULATION") != nullptr) && (std::string(getenv("SIMULATION")) == "1");
@@ -33,7 +33,7 @@ MessageContext message_context;
struct SubMaster::SubMessage {
std::string name;
SubSocket *socket = nullptr;
float freq = 0.0f;
int freq = 0;
bool updated = false, alive = false, valid = false, ignore_alive;
uint64_t rcv_time = 0, rcv_frame = 0;
void *allocated_msg_reader = nullptr;
@@ -50,7 +50,7 @@ SubMaster::SubMaster(const std::vector<const char *> &service_list, const std::v
assert(services.count(std::string(name)) > 0);
service serv = services.at(std::string(name));
SubSocket *socket = SubSocket::create(message_context.context(), name, address ? address : "127.0.0.1", true, true, serv.queue_size);
SubSocket *socket = SubSocket::create(message_context.context(), name, address ? address : "127.0.0.1", true);
assert(socket != 0);
bool is_polled = inList(poll, name) || poll.empty();
if (is_polled) poller_->registerSocket(socket);
@@ -187,8 +187,7 @@ SubMaster::~SubMaster() {
PubMaster::PubMaster(const std::vector<const char *> &service_list) {
for (auto name : service_list) {
assert(services.count(name) > 0);
service serv = services.at(std::string(name));
PubSocket *socket = PubSocket::create(message_context.context(), name, true, serv.queue_size);
PubSocket *socket = PubSocket::create(message_context.context(), name);
assert(socket);
sockets_[name] = socket;
}
@@ -5,13 +5,12 @@ import numbers
import random
import threading
import time
from openpilot.common.parameterized import parameterized
from parameterized import parameterized
import pytest
from openpilot.cereal import log
from opendbc.car.structs import car
import openpilot.cereal.messaging as messaging
from openpilot.cereal.services import SERVICE_LIST
from cereal import log, car
import cereal.messaging as messaging
from cereal.services import SERVICE_LIST
events = [evt for evt in log.Event.schema.union_fields if evt in SERVICE_LIST.keys()]
@@ -31,7 +30,7 @@ def zmq_sleep(t=1):
# TODO: this should take any capnp struct and returrn a msg with random populated data
def random_carstate():
fields = ["vEgo", "aEgo", "steeringTorque", "steeringAngleDeg"]
fields = ["vEgo", "aEgo", "gas", "steeringAngleDeg"]
msg = messaging.new_message("carState")
cs = msg.carState
for f in fields:
@@ -178,8 +177,8 @@ class TestMessaging:
# wait 5 socket timeouts before sending
msg = random_carstate()
start_time = time.monotonic()
delayed_send(sock_timeout*5, pub_sock, msg.to_bytes())
start_time = time.monotonic()
recvd = messaging.recv_one_retry(sub_sock)
assert (time.monotonic() - start_time) >= sock_timeout*5
assert isinstance(recvd, capnp._DynamicStructReader)
@@ -2,11 +2,10 @@ import random
import time
from typing import Sized, cast
import openpilot.cereal.messaging as messaging
from openpilot.cereal.messaging.tests.test_messaging import events, random_sock, random_socks, \
import cereal.messaging as messaging
from cereal.messaging.tests.test_messaging import events, random_sock, random_socks, \
random_bytes, random_carstate, assert_carstate, \
zmq_sleep
from openpilot.cereal.services import SERVICE_LIST
class TestSubMaster:
@@ -27,9 +26,7 @@ class TestSubMaster:
sm = messaging.SubMaster(socks)
assert sm.frame == -1
assert not any(sm.updated.values())
assert not any(sm.seen.values())
on_demand = {s: SERVICE_LIST[s].frequency <= 1e-5 for s in sm.services}
assert all(sm.alive[s] == sm.valid[s] == sm.freq_ok[s] == on_demand[s] for s in sm.services)
assert not any(sm.alive.values())
assert all(t == 0. for t in sm.recv_time.values())
assert all(f == 0 for f in sm.recv_frame.values())
assert all(t == 0 for t in sm.logMonoTime.values())
@@ -86,7 +83,6 @@ class TestSubMaster:
"cameraOdometry": (20, 10),
"liveCalibration": (4, 4),
"carParams": (None, None),
"userBookmark": (None, None),
}
for service, (max_freq, min_freq) in checks.items():
+21
View File
@@ -0,0 +1,21 @@
import os
import tempfile
from typing import Dict
from parameterized import parameterized
import cereal.services as services
from cereal.services import SERVICE_LIST
class TestServices:
@parameterized.expand(SERVICE_LIST.keys())
def test_services(self, s):
service = SERVICE_LIST[s]
assert service.frequency <= 104
assert service.decimation != 0
def test_generated_header(self):
with tempfile.NamedTemporaryFile(suffix=".h") as f:
ret = os.system(f"python3 {services.__file__} > {f.name} && clang++ {f.name} -std=c++11")
assert ret == 0, "generated services header is not valid C"
+126
View File
@@ -0,0 +1,126 @@
#!/usr/bin/env python3
from typing import Optional
class Service:
def __init__(self, should_log: bool, frequency: float, decimation: Optional[int] = None):
self.should_log = should_log
self.frequency = frequency
self.decimation = decimation
_services: dict[str, tuple] = {
# service: (should_log, frequency, qlog decimation (optional))
# note: the "EncodeIdx" packets will still be in the log
"gyroscope": (True, 104., 104),
"gyroscope2": (True, 100., 100),
"accelerometer": (True, 104., 104),
"accelerometer2": (True, 100., 100),
"magnetometer": (True, 25.),
"lightSensor": (True, 100., 100),
"temperatureSensor": (True, 2., 200),
"temperatureSensor2": (True, 2., 200),
"gpsNMEA": (True, 9.),
"deviceState": (True, 2., 1),
"touch": (True, 20., 1),
"can": (True, 100., 2053), # decimation gives ~3 msgs in a full segment
"controlsState": (True, 100., 10),
"selfdriveState": (True, 100., 10),
"pandaStates": (True, 10., 1),
"peripheralState": (True, 2., 1),
"radarState": (True, 20., 5),
"roadEncodeIdx": (False, 20., 1),
"liveTracks": (True, 20.),
"sendcan": (True, 100., 139),
"logMessage": (True, 0.),
"errorLogMessage": (True, 0., 1),
"liveCalibration": (True, 4., 4),
"liveTorqueParameters": (True, 4., 1),
"androidLog": (True, 0.),
"carState": (True, 100., 10),
"carControl": (True, 100., 10),
"carOutput": (True, 100., 10),
"longitudinalPlan": (True, 20., 10),
"driverAssistance": (True, 20., 20),
"procLog": (True, 0.5, 15),
"gpsLocationExternal": (True, 10., 10),
"gpsLocation": (True, 1., 1),
"ubloxGnss": (True, 10.),
"qcomGnss": (True, 2.),
"gnssMeasurements": (True, 10., 10),
"clocks": (True, 0.1, 1),
"ubloxRaw": (True, 20.),
"livePose": (True, 20., 4),
"liveParameters": (True, 20., 5),
"cameraOdometry": (True, 20., 10),
"thumbnail": (True, 0.2, 1),
"onroadEvents": (True, 1., 1),
"carParams": (True, 0.02, 1),
"roadCameraState": (True, 20., 20),
"driverCameraState": (True, 20., 20),
"driverEncodeIdx": (False, 20., 1),
"driverStateV2": (True, 20., 10),
"driverMonitoringState": (True, 20., 10),
"wideRoadEncodeIdx": (False, 20., 1),
"wideRoadCameraState": (True, 20., 20),
"drivingModelData": (True, 20., 10),
"modelV2": (True, 20.),
"managerState": (True, 2., 1),
"uploaderState": (True, 0., 1),
"navInstruction": (True, 1., 10),
"navRoute": (True, 0.),
"navThumbnail": (True, 0.),
"qRoadEncodeIdx": (False, 20.),
"userFlag": (True, 0., 1),
"microphone": (True, 10., 10),
# sunnypilot
"modelManagerSP": (False, 1., 1),
"selfdriveStateSP": (True, 100., 10),
# debug
"uiDebug": (True, 0., 1),
"testJoystick": (True, 0.),
"alertDebug": (True, 20., 5),
"roadEncodeData": (False, 20.),
"driverEncodeData": (False, 20.),
"wideRoadEncodeData": (False, 20.),
"qRoadEncodeData": (False, 20.),
"livestreamWideRoadEncodeIdx": (False, 20.),
"livestreamRoadEncodeIdx": (False, 20.),
"livestreamDriverEncodeIdx": (False, 20.),
"livestreamWideRoadEncodeData": (False, 20.),
"livestreamRoadEncodeData": (False, 20.),
"livestreamDriverEncodeData": (False, 20.),
"customReservedRawData0": (True, 0.),
"customReservedRawData1": (True, 0.),
"customReservedRawData2": (True, 0.),
}
SERVICE_LIST = {name: Service(*vals) for
idx, (name, vals) in enumerate(_services.items())}
def build_header():
h = ""
h += "/* THIS IS AN AUTOGENERATED FILE, PLEASE EDIT services.py */\n"
h += "#ifndef __SERVICES_H\n"
h += "#define __SERVICES_H\n"
h += "#include <map>\n"
h += "#include <string>\n"
h += "struct service { std::string name; bool should_log; int frequency; int decimation; };\n"
h += "static std::map<std::string, service> services = {\n"
for k, v in SERVICE_LIST.items():
should_log = "true" if v.should_log else "false"
decimation = -1 if v.decimation is None else v.decimation
h += ' { "%s", {"%s", %s, %d, %d}},\n' % \
(k, k, should_log, v.frequency, decimation)
h += "};\n"
h += "#endif\n"
return h
if __name__ == "__main__":
print(build_header())
+13
View File
@@ -0,0 +1,13 @@
comment: false
coverage:
status:
project:
default:
informational: true
patch: off
ignore:
- "**/test_*.py"
- "selfdrive/test/**"
- "system/version.py" # codecov changes depending on if we are in a branch or not
- "tools"
+1
View File
@@ -0,0 +1 @@
*.cpp
+39
View File
@@ -0,0 +1,39 @@
Import('env', 'envCython', 'arch')
common_libs = [
'params.cc',
'swaglog.cc',
'util.cc',
'i2c.cc',
'watchdog.cc',
'ratekeeper.cc'
]
if arch != "Darwin":
common_libs.append('gpio.cc')
_common = env.Library('common', common_libs, LIBS="json11")
files = [
'clutil.cc',
]
_gpucommon = env.Library('gpucommon', files)
Export('_common', '_gpucommon')
if GetOption('extras'):
env.Program('tests/test_common',
['tests/test_runner.cc', 'tests/test_params.cc', 'tests/test_util.cc', 'tests/test_swaglog.cc'],
LIBS=[_common, 'json11', 'zmq', 'pthread'])
# Cython bindings
params_python = envCython.Program('params_pyx.so', 'params_pyx.pyx', LIBS=envCython['LIBS'] + [_common, 'zmq', 'json11'])
SConscript([
'transformations/SConscript',
])
Import('transformations_python')
common_python = [params_python, transformations_python]
Export('common_python')
+27
View File
@@ -0,0 +1,27 @@
from openpilot.common.api.comma_connect import CommaConnectApi
from sunnypilot.sunnylink.api import SunnylinkApi
class Api:
def __init__(self, dongle_id, use_sunnylink=False):
if use_sunnylink:
self.service = SunnylinkApi(dongle_id)
else:
self.service = CommaConnectApi(dongle_id)
def request(self, method, endpoint, **params):
return self.service.request(method, endpoint, **params)
def get(self, *args, **kwargs):
return self.service.get(*args, **kwargs)
def post(self, *args, **kwargs):
return self.service.post(*args, **kwargs)
def get_token(self, expiry_hours=1):
return self.service.get_token(expiry_hours)
def api_get(endpoint, method='GET', timeout=None, access_token=None, use_sunnylink=False, **params):
return SunnylinkApi(None).api_get(endpoint, method, timeout, access_token, **params) if use_sunnylink \
else CommaConnectApi(None).api_get(endpoint, method, timeout, access_token, **params)
+56
View File
@@ -0,0 +1,56 @@
import jwt
import requests
import unicodedata
from datetime import datetime, timedelta, UTC
from openpilot.system.hardware.hw import Paths
from openpilot.system.version import get_version
class BaseApi:
def __init__(self, dongle_id, api_host, user_agent="openpilot-"):
self.dongle_id = dongle_id
self.api_host = api_host
self.user_agent = user_agent
with open(f'{Paths.persist_root()}/comma/id_rsa') as f:
self.private_key = f.read()
def get(self, *args, **kwargs):
return self.request('GET', *args, **kwargs)
def post(self, *args, **kwargs):
return self.request('POST', *args, **kwargs)
def request(self, method, endpoint, timeout=None, access_token=None, **params):
return self.api_get(endpoint, method=method, timeout=timeout, access_token=access_token, **params)
def _get_token(self, expiry_hours=1, **extra_payload):
now = datetime.now(UTC).replace(tzinfo=None)
payload = {
'identity': self.dongle_id,
'nbf': now,
'iat': now,
'exp': now + timedelta(hours=expiry_hours),
**extra_payload
}
token = jwt.encode(payload, self.private_key, algorithm='RS256')
if isinstance(token, bytes):
token = token.decode('utf8')
return token
def get_token(self, expiry_hours=1):
return self._get_token(expiry_hours)
def remove_non_ascii_chars(self, text):
normalized_text = unicodedata.normalize('NFD', text)
ascii_encoded_text = normalized_text.encode('ascii', 'ignore')
return ascii_encoded_text.decode()
def api_get(self, endpoint, method='GET', timeout=None, access_token=None, **params):
headers = {}
if access_token is not None:
headers['Authorization'] = "JWT " + access_token
version = self.remove_non_ascii_chars(get_version())
headers['User-Agent'] = self.user_agent + version
return requests.request(method, f"{self.api_host}/{endpoint}", timeout=timeout, headers=headers, params=params)
+4
View File
@@ -0,0 +1,4 @@
import os
BASEDIR = os.path.abspath(os.path.join(os.path.dirname(os.path.realpath(__file__)), "../"))
+204
View File
@@ -0,0 +1,204 @@
#include "common/clutil.h"
#include <cassert>
#include <iostream>
#include <memory>
#include "common/util.h"
#include "common/swaglog.h"
namespace { // helper functions
template <typename Func, typename Id, typename Name>
std::string get_info(Func get_info_func, Id id, Name param_name) {
size_t size = 0;
CL_CHECK(get_info_func(id, param_name, 0, NULL, &size));
std::string info(size, '\0');
CL_CHECK(get_info_func(id, param_name, size, info.data(), NULL));
return info;
}
inline std::string get_platform_info(cl_platform_id id, cl_platform_info name) { return get_info(&clGetPlatformInfo, id, name); }
inline std::string get_device_info(cl_device_id id, cl_device_info name) { return get_info(&clGetDeviceInfo, id, name); }
void cl_print_info(cl_platform_id platform, cl_device_id device) {
size_t work_group_size = 0;
cl_device_type device_type = 0;
clGetDeviceInfo(device, CL_DEVICE_MAX_WORK_GROUP_SIZE, sizeof(work_group_size), &work_group_size, NULL);
clGetDeviceInfo(device, CL_DEVICE_TYPE, sizeof(device_type), &device_type, NULL);
const char *type_str = "Other...";
switch (device_type) {
case CL_DEVICE_TYPE_CPU: type_str ="CL_DEVICE_TYPE_CPU"; break;
case CL_DEVICE_TYPE_GPU: type_str = "CL_DEVICE_TYPE_GPU"; break;
case CL_DEVICE_TYPE_ACCELERATOR: type_str = "CL_DEVICE_TYPE_ACCELERATOR"; break;
}
LOGD("vendor: %s", get_platform_info(platform, CL_PLATFORM_VENDOR).c_str());
LOGD("platform version: %s", get_platform_info(platform, CL_PLATFORM_VERSION).c_str());
LOGD("profile: %s", get_platform_info(platform, CL_PLATFORM_PROFILE).c_str());
LOGD("extensions: %s", get_platform_info(platform, CL_PLATFORM_EXTENSIONS).c_str());
LOGD("name: %s", get_device_info(device, CL_DEVICE_NAME).c_str());
LOGD("device version: %s", get_device_info(device, CL_DEVICE_VERSION).c_str());
LOGD("max work group size: %zu", work_group_size);
LOGD("type = %d, %s", (int)device_type, type_str);
}
void cl_print_build_errors(cl_program program, cl_device_id device) {
cl_build_status status;
clGetProgramBuildInfo(program, device, CL_PROGRAM_BUILD_STATUS, sizeof(status), &status, NULL);
size_t log_size;
clGetProgramBuildInfo(program, device, CL_PROGRAM_BUILD_LOG, 0, NULL, &log_size);
std::string log(log_size, '\0');
clGetProgramBuildInfo(program, device, CL_PROGRAM_BUILD_LOG, log_size, &log[0], NULL);
LOGE("build failed; status=%d, log: %s", status, log.c_str());
}
} // namespace
cl_device_id cl_get_device_id(cl_device_type device_type) {
cl_uint num_platforms = 0;
CL_CHECK(clGetPlatformIDs(0, NULL, &num_platforms));
std::unique_ptr<cl_platform_id[]> platform_ids = std::make_unique<cl_platform_id[]>(num_platforms);
CL_CHECK(clGetPlatformIDs(num_platforms, &platform_ids[0], NULL));
for (size_t i = 0; i < num_platforms; ++i) {
LOGD("platform[%zu] CL_PLATFORM_NAME: %s", i, get_platform_info(platform_ids[i], CL_PLATFORM_NAME).c_str());
// Get first device
if (cl_device_id device_id = NULL; clGetDeviceIDs(platform_ids[i], device_type, 1, &device_id, NULL) == 0 && device_id) {
cl_print_info(platform_ids[i], device_id);
return device_id;
}
}
LOGE("No valid openCL platform found");
assert(0);
return nullptr;
}
cl_context cl_create_context(cl_device_id device_id) {
return CL_CHECK_ERR(clCreateContext(NULL, 1, &device_id, NULL, NULL, &err));
}
void cl_release_context(cl_context context) {
clReleaseContext(context);
}
cl_program cl_program_from_file(cl_context ctx, cl_device_id device_id, const char* path, const char* args) {
return cl_program_from_source(ctx, device_id, util::read_file(path), args);
}
cl_program cl_program_from_source(cl_context ctx, cl_device_id device_id, const std::string& src, const char* args) {
const char *csrc = src.c_str();
cl_program prg = CL_CHECK_ERR(clCreateProgramWithSource(ctx, 1, &csrc, NULL, &err));
if (int err = clBuildProgram(prg, 1, &device_id, args, NULL, NULL); err != 0) {
cl_print_build_errors(prg, device_id);
assert(0);
}
return prg;
}
cl_program cl_program_from_binary(cl_context ctx, cl_device_id device_id, const uint8_t* binary, size_t length, const char* args) {
cl_program prg = CL_CHECK_ERR(clCreateProgramWithBinary(ctx, 1, &device_id, &length, &binary, NULL, &err));
if (int err = clBuildProgram(prg, 1, &device_id, args, NULL, NULL); err != 0) {
cl_print_build_errors(prg, device_id);
assert(0);
}
return prg;
}
// Given a cl code and return a string representation
#define CL_ERR_TO_STR(err) case err: return #err
const char* cl_get_error_string(int err) {
switch (err) {
CL_ERR_TO_STR(CL_SUCCESS);
CL_ERR_TO_STR(CL_DEVICE_NOT_FOUND);
CL_ERR_TO_STR(CL_DEVICE_NOT_AVAILABLE);
CL_ERR_TO_STR(CL_COMPILER_NOT_AVAILABLE);
CL_ERR_TO_STR(CL_MEM_OBJECT_ALLOCATION_FAILURE);
CL_ERR_TO_STR(CL_OUT_OF_RESOURCES);
CL_ERR_TO_STR(CL_OUT_OF_HOST_MEMORY);
CL_ERR_TO_STR(CL_PROFILING_INFO_NOT_AVAILABLE);
CL_ERR_TO_STR(CL_MEM_COPY_OVERLAP);
CL_ERR_TO_STR(CL_IMAGE_FORMAT_MISMATCH);
CL_ERR_TO_STR(CL_IMAGE_FORMAT_NOT_SUPPORTED);
CL_ERR_TO_STR(CL_MAP_FAILURE);
CL_ERR_TO_STR(CL_MISALIGNED_SUB_BUFFER_OFFSET);
CL_ERR_TO_STR(CL_EXEC_STATUS_ERROR_FOR_EVENTS_IN_WAIT_LIST);
CL_ERR_TO_STR(CL_COMPILE_PROGRAM_FAILURE);
CL_ERR_TO_STR(CL_LINKER_NOT_AVAILABLE);
CL_ERR_TO_STR(CL_LINK_PROGRAM_FAILURE);
CL_ERR_TO_STR(CL_DEVICE_PARTITION_FAILED);
CL_ERR_TO_STR(CL_KERNEL_ARG_INFO_NOT_AVAILABLE);
CL_ERR_TO_STR(CL_INVALID_VALUE);
CL_ERR_TO_STR(CL_INVALID_DEVICE_TYPE);
CL_ERR_TO_STR(CL_INVALID_PLATFORM);
CL_ERR_TO_STR(CL_INVALID_DEVICE);
CL_ERR_TO_STR(CL_INVALID_CONTEXT);
CL_ERR_TO_STR(CL_INVALID_QUEUE_PROPERTIES);
CL_ERR_TO_STR(CL_INVALID_COMMAND_QUEUE);
CL_ERR_TO_STR(CL_INVALID_HOST_PTR);
CL_ERR_TO_STR(CL_INVALID_MEM_OBJECT);
CL_ERR_TO_STR(CL_INVALID_IMAGE_FORMAT_DESCRIPTOR);
CL_ERR_TO_STR(CL_INVALID_IMAGE_SIZE);
CL_ERR_TO_STR(CL_INVALID_SAMPLER);
CL_ERR_TO_STR(CL_INVALID_BINARY);
CL_ERR_TO_STR(CL_INVALID_BUILD_OPTIONS);
CL_ERR_TO_STR(CL_INVALID_PROGRAM);
CL_ERR_TO_STR(CL_INVALID_PROGRAM_EXECUTABLE);
CL_ERR_TO_STR(CL_INVALID_KERNEL_NAME);
CL_ERR_TO_STR(CL_INVALID_KERNEL_DEFINITION);
CL_ERR_TO_STR(CL_INVALID_KERNEL);
CL_ERR_TO_STR(CL_INVALID_ARG_INDEX);
CL_ERR_TO_STR(CL_INVALID_ARG_VALUE);
CL_ERR_TO_STR(CL_INVALID_ARG_SIZE);
CL_ERR_TO_STR(CL_INVALID_KERNEL_ARGS);
CL_ERR_TO_STR(CL_INVALID_WORK_DIMENSION);
CL_ERR_TO_STR(CL_INVALID_WORK_GROUP_SIZE);
CL_ERR_TO_STR(CL_INVALID_WORK_ITEM_SIZE);
CL_ERR_TO_STR(CL_INVALID_GLOBAL_OFFSET);
CL_ERR_TO_STR(CL_INVALID_EVENT_WAIT_LIST);
CL_ERR_TO_STR(CL_INVALID_EVENT);
CL_ERR_TO_STR(CL_INVALID_OPERATION);
CL_ERR_TO_STR(CL_INVALID_GL_OBJECT);
CL_ERR_TO_STR(CL_INVALID_BUFFER_SIZE);
CL_ERR_TO_STR(CL_INVALID_MIP_LEVEL);
CL_ERR_TO_STR(CL_INVALID_GLOBAL_WORK_SIZE);
CL_ERR_TO_STR(CL_INVALID_PROPERTY);
CL_ERR_TO_STR(CL_INVALID_IMAGE_DESCRIPTOR);
CL_ERR_TO_STR(CL_INVALID_COMPILER_OPTIONS);
CL_ERR_TO_STR(CL_INVALID_LINKER_OPTIONS);
CL_ERR_TO_STR(CL_INVALID_DEVICE_PARTITION_COUNT);
case -69: return "CL_INVALID_PIPE_SIZE";
case -70: return "CL_INVALID_DEVICE_QUEUE";
case -71: return "CL_INVALID_SPEC_ID";
case -72: return "CL_MAX_SIZE_RESTRICTION_EXCEEDED";
case -1002: return "CL_INVALID_D3D10_DEVICE_KHR";
case -1003: return "CL_INVALID_D3D10_RESOURCE_KHR";
case -1004: return "CL_D3D10_RESOURCE_ALREADY_ACQUIRED_KHR";
case -1005: return "CL_D3D10_RESOURCE_NOT_ACQUIRED_KHR";
case -1006: return "CL_INVALID_D3D11_DEVICE_KHR";
case -1007: return "CL_INVALID_D3D11_RESOURCE_KHR";
case -1008: return "CL_D3D11_RESOURCE_ALREADY_ACQUIRED_KHR";
case -1009: return "CL_D3D11_RESOURCE_NOT_ACQUIRED_KHR";
case -1010: return "CL_INVALID_DX9_MEDIA_ADAPTER_KHR";
case -1011: return "CL_INVALID_DX9_MEDIA_SURFACE_KHR";
case -1012: return "CL_DX9_MEDIA_SURFACE_ALREADY_ACQUIRED_KHR";
case -1013: return "CL_DX9_MEDIA_SURFACE_NOT_ACQUIRED_KHR";
case -1093: return "CL_INVALID_EGL_OBJECT_KHR";
case -1092: return "CL_EGL_RESOURCE_NOT_ACQUIRED_KHR";
case -1001: return "CL_PLATFORM_NOT_FOUND_KHR";
case -1057: return "CL_DEVICE_PARTITION_FAILED_EXT";
case -1058: return "CL_INVALID_PARTITION_COUNT_EXT";
case -1059: return "CL_INVALID_PARTITION_NAME_EXT";
case -1094: return "CL_INVALID_ACCELERATOR_INTEL";
case -1095: return "CL_INVALID_ACCELERATOR_TYPE_INTEL";
case -1096: return "CL_INVALID_ACCELERATOR_DESCRIPTOR_INTEL";
case -1097: return "CL_ACCELERATOR_TYPE_NOT_SUPPORTED_INTEL";
case -1000: return "CL_INVALID_GL_SHAREGROUP_REFERENCE_KHR";
case -1098: return "CL_INVALID_VA_API_MEDIA_ADAPTER_INTEL";
case -1099: return "CL_INVALID_VA_API_MEDIA_SURFACE_INTEL";
case -1100: return "CL_VA_API_MEDIA_SURFACE_ALREADY_ACQUIRED_INTEL";
case -1101: return "CL_VA_API_MEDIA_SURFACE_NOT_ACQUIRED_INTEL";
default: return "CL_UNKNOWN_ERROR";
}
}
+30
View File
@@ -0,0 +1,30 @@
#pragma once
#ifdef __APPLE__
#include <OpenCL/cl.h>
#else
#include <CL/cl.h>
#endif
#include <string>
#define CL_CHECK(_expr) \
do { \
assert(CL_SUCCESS == (_expr)); \
} while (0)
#define CL_CHECK_ERR(_expr) \
({ \
cl_int err = CL_INVALID_VALUE; \
__typeof__(_expr) _ret = _expr; \
assert(_ret&& err == CL_SUCCESS); \
_ret; \
})
cl_device_id cl_get_device_id(cl_device_type device_type);
cl_context cl_create_context(cl_device_id device_id);
void cl_release_context(cl_context context);
cl_program cl_program_from_source(cl_context ctx, cl_device_id device_id, const std::string& src, const char* args = nullptr);
cl_program cl_program_from_binary(cl_context ctx, cl_device_id device_id, const uint8_t* binary, size_t length, const char* args = nullptr);
cl_program cl_program_from_file(cl_context ctx, cl_device_id device_id, const char* path, const char* args);
const char* cl_get_error_string(int err);
+19
View File
@@ -0,0 +1,19 @@
import numpy as np
class Conversions:
# Speed
MPH_TO_KPH = 1.609344
KPH_TO_MPH = 1. / MPH_TO_KPH
MS_TO_KPH = 3.6
KPH_TO_MS = 1. / MS_TO_KPH
MS_TO_MPH = MS_TO_KPH * KPH_TO_MPH
MPH_TO_MS = MPH_TO_KPH * KPH_TO_MS
MS_TO_KNOTS = 1.9438
KNOTS_TO_MS = 1. / MS_TO_KNOTS
# Angle
DEG_TO_RAD = np.pi / 180.
RAD_TO_DEG = 1. / DEG_TO_RAD
# Mass
LB_TO_KG = 0.453592
+9
View File
@@ -0,0 +1,9 @@
# remove all keys that end in DEPRECATED
def strip_deprecated_keys(d):
for k in list(d.keys()):
if isinstance(k, str):
if k.endswith('DEPRECATED'):
d.pop(k)
elif isinstance(d[k], dict):
strip_deprecated_keys(d[k])
return d
+8
View File
@@ -0,0 +1,8 @@
import platform
def suffix():
if platform.system() == "Darwin":
return ".dylib"
else:
return ".so"
+37
View File
@@ -0,0 +1,37 @@
import os
import tempfile
import contextlib
class CallbackReader:
"""Wraps a file, but overrides the read method to also
call a callback function with the number of bytes read so far."""
def __init__(self, f, callback, *args):
self.f = f
self.callback = callback
self.cb_args = args
self.total_read = 0
def __getattr__(self, attr):
return getattr(self.f, attr)
def read(self, *args, **kwargs):
chunk = self.f.read(*args, **kwargs)
self.total_read += len(chunk)
self.callback(*self.cb_args, self.total_read)
return chunk
@contextlib.contextmanager
def atomic_write_in_dir(path: str, mode: str = 'w', buffering: int = -1, encoding: str = None, newline: str = None,
overwrite: bool = False):
"""Write to a file atomically using a temporary file in the same directory as the destination file."""
dir_name = os.path.dirname(path)
if not overwrite and os.path.exists(path):
raise FileExistsError(f"File '{path}' already exists. To overwrite it, set 'overwrite' to True.")
with tempfile.NamedTemporaryFile(mode=mode, buffering=buffering, encoding=encoding, newline=newline, dir=dir_name, delete=False) as tmp_file:
yield tmp_file
tmp_file_name = tmp_file.name
os.replace(tmp_file_name, path)
+18
View File
@@ -0,0 +1,18 @@
class FirstOrderFilter:
# first order filter
def __init__(self, x0, rc, dt, initialized=True):
self.x = x0
self.dt = dt
self.update_alpha(rc)
self.initialized = initialized
def update_alpha(self, rc):
self.alpha = self.dt / (rc + self.dt)
def update(self, x):
if self.initialized:
self.x = (1. - self.alpha) * self.x + self.alpha * x
else:
self.initialized = True
self.x = x
return self.x
+42
View File
@@ -0,0 +1,42 @@
from functools import cache
import subprocess
from openpilot.common.run import run_cmd, run_cmd_default
@cache
def get_commit(cwd: str = None, branch: str = "HEAD") -> str:
return run_cmd_default(["git", "rev-parse", branch], cwd=cwd)
@cache
def get_commit_date(cwd: str = None, commit: str = "HEAD") -> str:
return run_cmd_default(["git", "show", "--no-patch", "--format='%ct %ci'", commit], cwd=cwd)
@cache
def get_short_branch(cwd: str = None) -> str:
return run_cmd_default(["git", "rev-parse", "--abbrev-ref", "HEAD"], cwd=cwd)
@cache
def get_branch(cwd: str = None) -> str:
return run_cmd_default(["git", "rev-parse", "--abbrev-ref", "--symbolic-full-name", "@{u}"], cwd=cwd)
@cache
def get_origin(cwd: str = None) -> str:
try:
local_branch = run_cmd(["git", "name-rev", "--name-only", "HEAD"], cwd=cwd)
tracking_remote = run_cmd(["git", "config", "branch." + local_branch + ".remote"], cwd=cwd)
return run_cmd(["git", "config", "remote." + tracking_remote + ".url"], cwd=cwd)
except subprocess.CalledProcessError: # Not on a branch, fallback
return run_cmd_default(["git", "config", "--get", "remote.origin.url"], cwd=cwd)
@cache
def get_normalized_origin(cwd: str = None) -> str:
return get_origin(cwd) \
.replace("git@", "", 1) \
.replace(".git", "", 1) \
.replace("https://", "", 1) \
.replace(":", "/", 1)
+84
View File
@@ -0,0 +1,84 @@
#include "common/gpio.h"
#include <string>
#ifdef __APPLE__
int gpio_init(int pin_nr, bool output) {
return 0;
}
int gpio_set(int pin_nr, bool high) {
return 0;
}
int gpiochip_get_ro_value_fd(const char* consumer_label, int gpiochiop_id, int pin_nr) {
return 0;
}
#else
#include <fcntl.h>
#include <unistd.h>
#include <cstring>
#include <linux/gpio.h>
#include <sys/ioctl.h>
#include "common/util.h"
#include "common/swaglog.h"
int gpio_init(int pin_nr, bool output) {
char pin_dir_path[50];
int pin_dir_path_len = snprintf(pin_dir_path, sizeof(pin_dir_path),
"/sys/class/gpio/gpio%d/direction", pin_nr);
if (pin_dir_path_len <= 0) {
return -1;
}
const char *value = output ? "out" : "in";
return util::write_file(pin_dir_path, (void*)value, strlen(value));
}
int gpio_set(int pin_nr, bool high) {
char pin_val_path[50];
int pin_val_path_len = snprintf(pin_val_path, sizeof(pin_val_path),
"/sys/class/gpio/gpio%d/value", pin_nr);
if (pin_val_path_len <= 0) {
return -1;
}
return util::write_file(pin_val_path, (void*)(high ? "1" : "0"), 1);
}
int gpiochip_get_ro_value_fd(const char* consumer_label, int gpiochiop_id, int pin_nr) {
// Assumed that all interrupt pins are unexported and rights are given to
// read from gpiochip0.
std::string gpiochip_path = "/dev/gpiochip" + std::to_string(gpiochiop_id);
int fd = open(gpiochip_path.c_str(), O_RDONLY);
if (fd < 0) {
LOGE("Error opening gpiochip0 fd");
return -1;
}
// Setup event
struct gpioevent_request rq;
rq.lineoffset = pin_nr;
rq.handleflags = GPIOHANDLE_REQUEST_INPUT;
/* Requesting both edges as the data ready pulse from the lsm6ds sensor is
very short(75us) and is mostly detected as falling edge instead of rising.
So if it is detected as rising the following falling edge is skipped. */
rq.eventflags = GPIOEVENT_REQUEST_BOTH_EDGES;
strncpy(rq.consumer_label, consumer_label, std::size(rq.consumer_label) - 1);
int ret = util::safe_ioctl(fd, GPIO_GET_LINEEVENT_IOCTL, &rq);
if (ret == -1) {
LOGE("Unable to get line event from ioctl : %s", strerror(errno));
close(fd);
return -1;
}
close(fd);
return rq.fd;
}
#endif
+33
View File
@@ -0,0 +1,33 @@
#pragma once
// Pin definitions
#ifdef QCOM2
#define GPIO_HUB_RST_N 30
#define GPIO_UBLOX_RST_N 32
#define GPIO_UBLOX_SAFEBOOT_N 33
#define GPIO_GNSS_PWR_EN 34 /* SCHEMATIC LABEL: GPIO_UBLOX_PWR_EN */
#define GPIO_STM_RST_N 124
#define GPIO_STM_BOOT0 134
#define GPIO_BMX_ACCEL_INT 21
#define GPIO_BMX_GYRO_INT 23
#define GPIO_BMX_MAGN_INT 87
#define GPIO_LSM_INT 84
#define GPIOCHIP_INT 0
#else
#define GPIO_HUB_RST_N 0
#define GPIO_UBLOX_RST_N 0
#define GPIO_UBLOX_SAFEBOOT_N 0
#define GPIO_GNSS_PWR_EN 0 /* SCHEMATIC LABEL: GPIO_UBLOX_PWR_EN */
#define GPIO_STM_RST_N 0
#define GPIO_STM_BOOT0 0
#define GPIO_BMX_ACCEL_INT 0
#define GPIO_BMX_GYRO_INT 0
#define GPIO_BMX_MAGN_INT 0
#define GPIO_LSM_INT 0
#define GPIOCHIP_INT 0
#endif
int gpio_init(int pin_nr, bool output);
int gpio_set(int pin_nr, bool high);
int gpiochip_get_ro_value_fd(const char* consumer_label, int gpiochiop_id, int pin_nr);
+54
View File
@@ -0,0 +1,54 @@
import os
from functools import cache
def gpio_init(pin: int, output: bool) -> None:
try:
with open(f"/sys/class/gpio/gpio{pin}/direction", 'wb') as f:
f.write(b"out" if output else b"in")
except Exception as e:
print(f"Failed to set gpio {pin} direction: {e}")
def gpio_set(pin: int, high: bool) -> None:
try:
with open(f"/sys/class/gpio/gpio{pin}/value", 'wb') as f:
f.write(b"1" if high else b"0")
except Exception as e:
print(f"Failed to set gpio {pin} value: {e}")
def gpio_read(pin: int) -> bool | None:
val = None
try:
with open(f"/sys/class/gpio/gpio{pin}/value", 'rb') as f:
val = bool(int(f.read().strip()))
except Exception as e:
print(f"Failed to set gpio {pin} value: {e}")
return val
def gpio_export(pin: int) -> None:
if os.path.isdir(f"/sys/class/gpio/gpio{pin}"):
return
try:
with open("/sys/class/gpio/export", 'w') as f:
f.write(str(pin))
except Exception:
print(f"Failed to export gpio {pin}")
@cache
def get_irq_action(irq: int) -> list[str]:
try:
with open(f"/sys/kernel/irq/{irq}/actions") as f:
actions = f.read().strip().split(',')
return actions
except FileNotFoundError:
return []
def get_irqs_for_action(action: str) -> list[str]:
ret = []
with open("/proc/interrupts") as f:
for l in f.readlines():
irq = l.split(':')[0].strip()
if irq.isdigit() and action in get_irq_action(irq):
ret.append(irq)
return ret
+92
View File
@@ -0,0 +1,92 @@
#include "common/i2c.h"
#include <fcntl.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <cassert>
#include <cstdio>
#include <stdexcept>
#include "common/swaglog.h"
#include "common/util.h"
#define UNUSED(x) (void)(x)
#ifdef QCOM2
// TODO: decide if we want to install libi2c-dev everywhere
extern "C" {
#include <linux/i2c-dev.h>
#include <i2c/smbus.h>
}
I2CBus::I2CBus(uint8_t bus_id) {
char bus_name[20];
snprintf(bus_name, 20, "/dev/i2c-%d", bus_id);
i2c_fd = HANDLE_EINTR(open(bus_name, O_RDWR));
if (i2c_fd < 0) {
throw std::runtime_error("Failed to open I2C bus");
}
}
I2CBus::~I2CBus() {
if (i2c_fd >= 0) {
close(i2c_fd);
}
}
int I2CBus::read_register(uint8_t device_address, uint register_address, uint8_t *buffer, uint8_t len) {
std::lock_guard lk(m);
int ret = 0;
ret = HANDLE_EINTR(ioctl(i2c_fd, I2C_SLAVE, device_address));
if (ret < 0) { goto fail; }
ret = i2c_smbus_read_i2c_block_data(i2c_fd, register_address, len, buffer);
if ((ret < 0) || (ret != len)) { goto fail; }
fail:
return ret;
}
int I2CBus::set_register(uint8_t device_address, uint register_address, uint8_t data) {
std::lock_guard lk(m);
int ret = 0;
ret = HANDLE_EINTR(ioctl(i2c_fd, I2C_SLAVE, device_address));
if (ret < 0) { goto fail; }
ret = i2c_smbus_write_byte_data(i2c_fd, register_address, data);
if (ret < 0) { goto fail; }
fail:
return ret;
}
#else
I2CBus::I2CBus(uint8_t bus_id) {
UNUSED(bus_id);
i2c_fd = -1;
}
I2CBus::~I2CBus() {}
int I2CBus::read_register(uint8_t device_address, uint register_address, uint8_t *buffer, uint8_t len) {
UNUSED(device_address);
UNUSED(register_address);
UNUSED(buffer);
UNUSED(len);
return -1;
}
int I2CBus::set_register(uint8_t device_address, uint register_address, uint8_t data) {
UNUSED(device_address);
UNUSED(register_address);
UNUSED(data);
return -1;
}
#endif
+19
View File
@@ -0,0 +1,19 @@
#pragma once
#include <cstdint>
#include <mutex>
#include <sys/types.h>
class I2CBus {
private:
int i2c_fd;
std::mutex m;
public:
I2CBus(uint8_t bus_id);
~I2CBus();
int read_register(uint8_t device_address, uint register_address, uint8_t *buffer, uint8_t len);
int set_register(uint8_t device_address, uint register_address, uint8_t data);
};
@@ -8,7 +8,6 @@ import uuid
import socket
import logging
import traceback
import numpy as np
from threading import local
from collections import OrderedDict
from contextlib import contextmanager
@@ -16,8 +15,6 @@ from contextlib import contextmanager
LOG_TIMESTAMPS = "LOG_TIMESTAMPS" in os.environ
def json_handler(obj):
if isinstance(obj, np.bool_):
return bool(obj)
# if isinstance(obj, (datetime.date, datetime.time)):
# return obj.isoformat()
return repr(obj)
@@ -199,6 +196,7 @@ class SwagLogger(logging.Logger):
co = f.f_code
filename = os.path.normcase(co.co_filename)
# TODO: is this pylint exception correct?
if filename == _srcfile:
f = f.f_back
continue

Some files were not shown because too many files have changed in this diff Show More