Compare commits

..

1 Commits

Author SHA1 Message Date
Jason Wen
7bf90d5372 Tools: Update setup command for macOS native setup 2024-12-10 21:01:31 -05:00
446 changed files with 11294 additions and 10943 deletions

View File

@@ -1,2 +0,0 @@
Wen
REGIST

1
.gitattributes vendored
View File

@@ -9,7 +9,6 @@
*.ttf filter=lfs diff=lfs merge=lfs -text
*.wav filter=lfs diff=lfs merge=lfs -text
selfdrive/test/process_replay/fakedata/*.zst filter=lfs diff=lfs merge=lfs -text
selfdrive/car/tests/test_models_segs.txt filter=lfs diff=lfs merge=lfs -text
system/hardware/tici/updater filter=lfs diff=lfs merge=lfs -text
selfdrive/ui/qt/spinner_larch64 filter=lfs diff=lfs merge=lfs -text

15
.github/PULL_REQUEST_TEMPLATE/bugfix.md vendored Normal file
View File

@@ -0,0 +1,15 @@
---
name: Bug fix
about: For openpilot bug fixes
title: ''
labels: 'bugfix'
assignees: ''
---
**Description**
<!-- A description of the bug and the fix. Also link the issue if it exists. -->
**Verification**
<!-- Explain how you tested this bug fix. -->

View File

@@ -0,0 +1,19 @@
---
name: Car Bug fix
about: For vehicle/brand specific bug fixes
title: ''
labels: 'car bug fix'
assignees: ''
---
**Description**
<!-- A description of the bug and the fix. Also link the issue if it exists. -->
**Verification**
<!-- Explain how you tested this bug fix. -->
**Route**
Route: [a route with the bug fix]

View File

@@ -0,0 +1,15 @@
---
name: Car port
about: For new car ports
title: ''
labels: 'car port'
assignees: ''
---
**Checklist**
- [ ] added entry to CAR in selfdrive/car/*/values.py and ran `selfdrive/car/docs.py` to generate new docs
- [ ] test route added to [routes.py](https://github.com/commaai/openpilot/blob/master/selfdrive/car/tests/routes.py)
- [ ] route with openpilot:
- [ ] route with stock system:
- [ ] car harness used (if comma doesn't sell it, put N/A):

View File

@@ -0,0 +1,13 @@
---
name: Fingerprint
about: For adding fingerprints to existing cars
title: ''
labels: 'fingerprint'
assignees: ''
---
**Car**
Which car (make, model, year) this fingerprint is for
**Route**
A route with the fingerprint

View File

@@ -0,0 +1,15 @@
---
name: Refactor
about: For code refactors
title: ''
labels: 'refactor'
assignees: ''
---
**Description**
<!-- A description of the refactor, including the goals it accomplishes. -->
**Verification**
<!-- Explain how you tested the refactor for regressions. -->

31
.github/PULL_REQUEST_TEMPLATE/tuning.md vendored Normal file
View File

@@ -0,0 +1,31 @@
---
name: Tuning
about: For openpilot tuning changes
title: ''
labels: 'tuning'
assignees: ''
---
**Description**
<!-- A description of what is wrong with the current tuning and how the PR addresses this. -->
**Verification**
<!-- To verify tuning, capture the following scenarios (broadly, not exactly), with current tune and this tune.
Use the PlotJuggler tuning layout to compare planned versus actual behavior.
Run ./juggle.py <route> --layout layouts/tuning.xml , screenshot the full tab of interest, and paste into this PR.
Longitudinal:
* Maintaining speed at 25, 40, 65mph
* Driving up and down hills
* Accelerating from a stop
* Decelerating to a stop
* Following large changes in set speed
* Coming to a stop behind a lead car
Lateral:
* Straight driving at ~25, ~45 and ~65mph
* Turns driving at ~25, ~45 and ~65mph
-->

30
.github/build.py vendored Normal file
View File

@@ -0,0 +1,30 @@
import pathlib
GITHUB_FOLDER = pathlib.Path(__file__).parent
PULL_REQUEST_TEMPLATES = (GITHUB_FOLDER / "PULL_REQUEST_TEMPLATE")
order = ["fingerprint", "car_bugfix", "bugfix", "car_port", "refactor"]
def create_pull_request_template():
with open(GITHUB_FOLDER / "pull_request_template.md", "w") as f:
f.write("<!-- Please copy and paste the relevant template -->\n\n")
for t in order:
template = PULL_REQUEST_TEMPLATES / f"{t}.md"
text = template.read_text()
# Remove metadata for GitHub
start = text.find("---")
end = text.find("---", start+1)
text = text[end + 4:]
# Remove comments
text = text.replace("<!-- ", "").replace("-->", "")
f.write(f"<!--- ***** Template: {template.stem.replace('_', ' ').title()} *****\n")
f.write(text)
f.write("\n\n")
f.write("-->\n\n")
create_pull_request_template()

View File

@@ -24,7 +24,7 @@ jobs:
# Check PR target branch
- name: check branch
uses: Vankka/pr-target-branch-action@def32ec9d93514138d6ac0132ee62e120a72aed5
if: github.repository == 'sunnypilot/sunnypilot'
if: github.repository == 'commaai/openpilot'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
@@ -37,17 +37,17 @@ jobs:
# Welcome comment
- name: "First timers PR"
uses: actions/first-interaction@v1
if: github.event.pull_request.head.repo.full_name != 'sunnypilot/sunnypilot'
if: github.event.pull_request.head.repo.full_name != 'commaai/openpilot'
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
pr-message: |
<!-- _(run_id **${{ github.run_id }}**)_ -->
Thanks for contributing to openpilot! In order for us to review your PR as quickly as possible, check the following:
* Convert your PR to a draft unless it's ready to review
* Read the [contributing docs](https://github.com/sunnypilot/sunnypilot/blob/master/docs/CONTRIBUTING.md)
* Read the [contributing docs](https://github.com/commaai/openpilot/blob/master/docs/CONTRIBUTING.md)
* Before marking as "ready for review", ensure:
* the goal is clearly stated in the description
* all the tests are passing
* the change is [something we merge](https://github.com/sunnypilot/sunnypilot/blob/master/docs/CONTRIBUTING.md#what-gets-merged)
* the change is [something we merge](https://github.com/commaai/openpilot/blob/master/docs/CONTRIBUTING.md#what-gets-merged)
* include a route or your device' dongle ID if relevant

View File

@@ -13,7 +13,7 @@ jobs:
badges:
name: create badges
runs-on: ubuntu-latest
if: github.repository == 'sunnypilot/sunnypilot'
if: github.repository == 'commaai/openpilot'
permissions:
contents: write
steps:
@@ -29,7 +29,7 @@ jobs:
git checkout --orphan badges
git rm -rf --cached .
git config user.email "badge-researcher@sunnypilot.ai"
git config user.email "badge-researcher@comma.ai"
git config user.name "Badge Researcher"
git add translation_badge.svg

View File

@@ -15,7 +15,7 @@ env:
jobs:
setup:
if: github.repository == 'sunnypilot/sunnypilot'
if: github.repository == 'commaai/openpilot'
runs-on: ubuntu-latest
outputs:
ci_runs: ${{ steps.ci_runs_setup.outputs.matrix }}
@@ -31,7 +31,7 @@ jobs:
strategy:
fail-fast: false
matrix: ${{fromJSON(needs.setup.outputs.ci_runs)}}
uses: sunnypilot/sunnypilot/.github/workflows/ci_weekly_run.yaml@master
uses: commaai/openpilot/.github/workflows/ci_weekly_run.yaml@master
with:
run_number: ${{ matrix.run_number }}

View File

@@ -12,6 +12,6 @@ concurrency:
jobs:
selfdrive_tests:
uses: sunnypilot/sunnypilot/.github/workflows/selfdrive_tests.yaml@master
uses: commaai/openpilot/.github/workflows/selfdrive_tests.yaml@master
with:
run_number: ${{ inputs.run_number }}

View File

@@ -15,7 +15,7 @@ runs:
scons -j$(nproc) --cache-populate"
- name: Save scons cache
uses: actions/cache/save@v4
if: (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/master-new')
if: github.ref == 'refs/heads/master'
with:
path: .ci_cache/scons_cache
key: scons-${{ runner.arch }}-${{ env.CACHE_COMMIT_DATE }}-${{ github.sha }}

View File

@@ -18,7 +18,7 @@ concurrency:
jobs:
docs:
name: build docs
runs-on: ubuntu-24.04
runs-on: ubuntu-latest
steps:
- uses: commaai/timeout@v1
@@ -35,13 +35,13 @@ jobs:
# Push to docs.comma.ai
- uses: actions/checkout@v4
if: github.ref == 'refs/heads/master' && github.repository == 'sunnypilot/sunnypilot'
if: github.ref == 'refs/heads/master' && github.repository == 'commaai/openpilot'
with:
path: openpilot-docs
ssh-key: ${{ secrets.OPENPILOT_DOCS_KEY }}
repository: sunnypilot/sunnypilot-docs
repository: commaai/openpilot-docs
- name: Push
if: github.ref == 'refs/heads/master' && github.repository == 'sunnypilot/sunnypilot'
if: github.ref == 'refs/heads/master' && github.repository == 'commaai/openpilot'
run: |
set -x

View File

@@ -1,72 +0,0 @@
name: Sync comma's LFS
env:
LFS_URL: 'https://gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git/info/lfs'
LFS_PUSH_URL: 'ssh://git@gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git'
on:
schedule:
- cron: '0 0 * * *' # Runs at 00:00 UTC every day
push:
branches:
- 'master-new'
pull_request:
branches:
- 'master-new'
workflow_dispatch: # enables manual triggering
inputs:
upstream_branch:
default: 'master'
type: string
jobs:
sync:
runs-on: ubuntu-latest
# Skip if PR is in draft mode
if: github.event_name != 'pull_request' || (github.event_name == 'pull_request' && github.event.pull_request.draft == false)
steps:
- name: Checkout Repository
uses: actions/checkout@v4
with:
repository: 'commaai/openpilot'
ref: ${{ inputs.upstream_branch }}
- name: LFS Fetch
run: |
git lfs fetch
- name: Set up Git
run: |
git config --global user.name 'GitHub Action'
git config --global user.email 'action@github.com'
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
- name: Add GitLab public keys
run: |
ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
- name: Ensure branch
run: |
if git symbolic-ref -q HEAD >/dev/null; then
echo "Already on a branch, proceeding with push"
else
echo "Detached HEAD state detected, creating temporary branch"
git checkout -b temp_branch
fi
- name: Update LFS Config
run: |
echo '[lfs]' > .lfsconfig
echo ' url = ${{ env.LFS_URL }}' >> .lfsconfig
echo ' pushurl = ${{ env.LFS_PUSH_URL }}' >> .lfsconfig
echo ' locksverify = false' >> .lfsconfig
- name: Push LFS
id: sync-and-commit
run: |
git lfs ls-files -l
git lfs push --all origin

View File

@@ -12,7 +12,7 @@ jobs:
build_prebuilt:
name: build prebuilt
runs-on: ubuntu-latest
if: github.repository == 'sunnypilot/sunnypilot'
if: github.repository == 'commaai/openpilot'
env:
PUSH_IMAGE: true
permissions:

View File

@@ -13,7 +13,7 @@ jobs:
container:
image: ghcr.io/commaai/openpilot-base:latest
runs-on: ubuntu-latest
if: github.repository == 'sunnypilot/sunnypilot'
if: github.repository == 'commaai/openpilot'
permissions:
checks: read
contents: write

View File

@@ -11,7 +11,7 @@ jobs:
runs-on: ubuntu-latest
container:
image: ghcr.io/commaai/openpilot-base:latest
if: github.repository == 'sunnypilot/sunnypilot'
if: github.repository == 'commaai/openpilot'
steps:
- uses: actions/checkout@v4
with:

View File

@@ -4,7 +4,6 @@ on:
push:
branches:
- master
- master-new
pull_request:
workflow_dispatch:
workflow_call:
@@ -15,11 +14,10 @@ on:
type: string
concurrency:
group: selfdrive-tests-ci-run-${{ inputs.run_number }}-${{ github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/master-new') && github.run_id || github.head_ref || github.ref }}-${{ github.workflow }}-${{ github.event_name }}
group: selfdrive-tests-ci-run-${{ inputs.run_number }}-${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && github.run_id || github.head_ref || github.ref }}-${{ github.workflow }}-${{ github.event_name }}
cancel-in-progress: true
env:
REPORT_NAME: report-${{ inputs.run_number || '1' }}-${{ github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/master-new') && 'master' || github.event.number }}
PYTHONWARNINGS: error
BASE_IMAGE: openpilot-base
AZURE_TOKEN: ${{ secrets.AZURE_COMMADATACI_OPENPILOTCI_TOKEN }}
@@ -33,7 +31,6 @@ env:
jobs:
build_release:
if: github.repository == 'commaai/openpilot' # build_release blocked for the time being to only comma as we may have a different process.
name: build release
runs-on:
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
@@ -55,7 +52,7 @@ jobs:
run: TARGET_DIR=$STRIPPED_DIR release/build_devel.sh
- uses: ./.github/workflows/setup-with-retry
- name: Check submodules
if: github.repository == 'sunnypilot/sunnypilot'
if: github.repository == 'commaai/openpilot'
timeout-minutes: 3
run: release/check-submodules.sh
- name: Build openpilot and run checks
@@ -96,9 +93,7 @@ jobs:
build_mac:
name: build macOS
runs-on: ${{ ((github.repository == 'commaai/openpilot') &&
((github.event_name != 'pull_request') ||
(github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-macos-8x14' || 'macos-latest' }}
runs-on: ${{ github.repository == 'commaai/openpilot' && 'namespace-profile-macos-8x14' || 'macos-latest' }}
steps:
- uses: actions/checkout@v4
with:
@@ -107,7 +102,6 @@ jobs:
uses: ./.github/workflows/auto-cache
with:
path: ~/Library/Caches/Homebrew
key: build_macos_${{ hashFiles('.github/workflows/selfdrive_tests.yaml') }}
- name: Install dependencies
run: ./tools/mac_setup.sh
env:
@@ -119,7 +113,6 @@ jobs:
uses: ./.github/workflows/auto-cache
with:
path: /tmp/scons_cache
key: build_macos_${{ hashFiles('.github/workflows/selfdrive_tests.yaml') }}
- name: Building openpilot
run: . .venv/bin/activate && scons -j$(nproc)
@@ -238,7 +231,7 @@ jobs:
uses: actions/cache@v4
with:
path: .ci_cache/comma_download_cache
key: car_models-${{ hashFiles('selfdrive/car/tests/test_models.py', 'opendbc/car/tests/routes.py') }}-${{ matrix.job }}
key: car_models-${{ hashFiles('selfdrive/car/tests/test_models.py', 'selfdrive/car/tests/routes.py') }}-${{ matrix.job }}
- name: Build openpilot
run: ${{ env.RUN }} "scons -j$(nproc)"
- name: Test car models
@@ -316,7 +309,6 @@ jobs:
runs-on:
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
- ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
if: (github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))
steps:
- uses: actions/checkout@v4
with:
@@ -361,5 +353,5 @@ jobs:
- name: Upload Test Report
uses: actions/upload-artifact@v4
with:
name: ${{ env.REPORT_NAME }}
name: report-${{ inputs.run_number || '1' }}-${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && 'master' || github.event.number }}
path: selfdrive/ui/tests/test_ui/report_1/screenshots

View File

@@ -20,7 +20,7 @@ jobs:
stale-pr-message: 'This PR has had no activity for ${{ env.DAYS_BEFORE_PR_STALE }} days. It will be automatically closed in ${{ env.DAYS_BEFORE_PR_CLOSE }} days if there is no activity.'
close-pr-message: 'This PR has been automatically closed due to inactivity. Feel free to re-open once activity resumes.'
stale-pr-label: stale
delete-branch: ${{ github.event.pull_request.head.repo.full_name == 'sunnypilot/sunnypilot' }} # only delete branches on the main repo
delete-branch: ${{ github.event.pull_request.head.repo.full_name == 'commaai/openpilot' }} # only delete branches on the main repo
exempt-pr-labels: "ignore stale,needs testing" # if wip or it needs testing from the community, don't mark as stale
days-before-pr-stale: ${{ env.DAYS_BEFORE_PR_STALE }}
days-before-pr-close: ${{ env.DAYS_BEFORE_PR_CLOSE }}

View File

@@ -1,86 +0,0 @@
name: Build Model from Upstream
env:
BUILD_DIR: "/data/openpilot"
OUTPUT_DIR: ${{ github.workspace }}/output
SCONS_CACHE_DIR: ${{ github.workspace }}/release/ci/scons_cache
UPSTREAM_REPO: "commaai/openpilot"
on:
workflow_dispatch:
inputs:
upstream_branch:
description: 'Upstream branch to build from'
required: true
default: 'master'
type: string
custom_name:
description: 'Custom name for the model'
required: false
type: string
jobs:
build_model:
runs-on: self-hosted
steps:
- uses: actions/checkout@v4
with:
repository: ${{ env.UPSTREAM_REPO }}
ref: ${{ github.event.inputs.upstream_branch }}
submodules: recursive
- run: git lfs pull
- name: Cache SCons
uses: actions/cache@v4
with:
path: ${{env.SCONS_CACHE_DIR}}
key: scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-model-${{ github.sha }}
restore-keys: |
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-model-${{ github.sha }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-model
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}
scons-${{ runner.os }}-${{ runner.arch }}-master-new
scons-${{ runner.os }}-${{ runner.arch }}-master
scons-${{ runner.os }}-${{ runner.arch }}
- name: Setup build environment
run: |
mkdir -p "${BUILD_DIR}/"
sudo find $BUILD_DIR/ -mindepth 1 -delete
echo "Starting build stage..."
echo "Building from: ${{ env.UPSTREAM_REPO }} branch: ${{ github.event.inputs.upstream_branch }}"
- name: Patch SConstruct to pass arbitrary cache
run: |
sed -i.bak 's#cache_dir =#default_cache_dir =#' ${{ github.workspace }}/SConstruct
printf '/default_cache_dir/a\\\ncache_dir = ARGUMENTS.get("cache_dir", default_cache_dir)\n' | sed -i.bak -f - ${{ github.workspace }}/SConstruct
cat ${{ github.workspace }}/SConstruct
- name: Build Model
run: |
source /etc/profile
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
scons -j$(nproc) cache_dir=${{ env.SCONS_CACHE_DIR }} ${{ github.workspace }}/selfdrive/modeld
- name: Prepare Output
run: |
sudo rm -rf ${OUTPUT_DIR}
mkdir -p ${OUTPUT_DIR}
rsync -avm \
--include='*.dlc' \
--include='*.thneed' \
--include='*.pkl' \
--include='*.onnx' \
--exclude='*' \
--delete-excluded \
--chown=comma:comma \
./selfdrive/modeld/models/ ${OUTPUT_DIR}/
- name: Upload Build Artifacts
uses: actions/upload-artifact@v4
with:
name: model-${{ github.event.inputs.custom_name || github.event.inputs.upstream_branch }}-${{ github.run_number }}
path: ${{ env.OUTPUT_DIR }}

View File

@@ -1,268 +0,0 @@
name: sunnypilot prebuilt action
env:
BUILD_DIR: "/data/openpilot"
OUTPUT_DIR: ${{ github.workspace }}/output
CI_DIR: ${{ github.workspace }}/release/ci
SCONS_CACHE_DIR: ${{ github.workspace }}/release/ci/scons_cache
PUBLIC_REPO_URL: "https://github.com/sunnypilot/sunnypilot"
# Branch configurations
MASTER_BRANCH: "master"
MASTER_NEW_BRANCH: "master-new"
DEV_C3_SOURCE_BRANCH: "master-dev-c3-new"
# Target branch configurations
STAGING_TARGET_BRANCH: "staging-c3-new"
DEV_TARGET_BRANCH: "dev-c3-new"
RELEASE_TARGET_BRANCH: "release-c3-new"
on:
push:
branches: [ master, master-new, master-dev-c3-new ]
tags: [ '*' ]
pull_request:
branches: [ master, master-new ]
workflow_dispatch:
inputs:
extra_version:
description: 'Extra version identifier'
required: false
default: ''
jobs:
build:
concurrency:
group: build-${{ github.head_ref || github.ref_name }}
cancel-in-progress: false
runs-on: self-hosted
outputs:
new_branch: ${{ steps.set-env.outputs.new_branch }}
version: ${{ steps.set-env.outputs.version }}
extra_version_identifier: ${{ steps.set-env.outputs.extra_version_identifier }}
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- run: git lfs pull
- name: Cache SCons
uses: actions/cache@v4
with:
path: ${{env.SCONS_CACHE_DIR}}
key: scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref || github.ref_name }}-${{ github.sha }}
restore-keys: |
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.head_ref }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ github.ref_name }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_NEW_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}-${{ env.MASTER_BRANCH }}
scons-${{ runner.os }}-${{ runner.arch }}
- name: Set Configuration
run: |
if [[ "${{ github.ref_name }}" == "${{ env.DEV_C3_SOURCE_BRANCH }}" ]]; then
# Dev configuration
echo "BRANCH_TYPE=dev" >> $GITHUB_ENV
echo "NEW_BRANCH=${{ env.DEV_TARGET_BRANCH }}" >> $GITHUB_ENV
echo "VERSION=$(date '+%Y.%m.%d')-${{ github.run_number }}" >> $GITHUB_ENV
echo "EXTRA_VERSION_IDENTIFIER=${{ github.run_number }}" >> $GITHUB_ENV
elif [[ "${{ github.ref_name }}" == "${{ env.MASTER_BRANCH }}" || "${{ github.ref_name }}" == "${{ env.MASTER_NEW_BRANCH }}" ]]; then
# Master configuration
echo "BRANCH_TYPE=master" >> $GITHUB_ENV
echo "NEW_BRANCH=${{ env.STAGING_TARGET_BRANCH }}" >> $GITHUB_ENV
echo "EXTRA_VERSION_IDENTIFIER=staging" >> $GITHUB_ENV
echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')-staging" >> $GITHUB_ENV
elif [[ "${{ github.ref }}" == refs/tags/* ]]; then
# Tag configuration
echo "BRANCH_TYPE=tag" >> $GITHUB_ENV
echo "NEW_BRANCH=${{ env.RELEASE_TARGET_BRANCH }}" >> $GITHUB_ENV
echo "EXTRA_VERSION_IDENTIFIER=release" >> $GITHUB_ENV
echo "VERSION=$(cat common/version.h | grep COMMA_VERSION | sed -e 's/[^0-9|.]//g')-release" >> $GITHUB_ENV
else
# Feature branch configuration
echo "BRANCH_TYPE=dispatch" >> $GITHUB_ENV
echo "NEW_BRANCH=${{ github.head_ref || github.ref_name }}-prebuilt" >> $GITHUB_ENV
echo "VERSION=$(date '+%Y.%m.%d')-${{ github.run_number }}" >> $GITHUB_ENV
fi
- name: Set environment variables
id: set-env
run: |
# Write to GITHUB_OUTPUT from environment variables
echo "new_branch=$NEW_BRANCH" >> $GITHUB_OUTPUT
[[ ! -z "$EXTRA_VERSION_IDENTIFIER" ]] && echo "extra_version_identifier=$EXTRA_VERSION_IDENTIFIER" >> $GITHUB_OUTPUT
[[ ! -z "$VERSION" ]] && echo "version=$VERSION" >> $GITHUB_OUTPUT
# Set up common environment
source /etc/profile;
export UV_PROJECT_ENVIRONMENT=${HOME}/venv
export VIRTUAL_ENV=$UV_PROJECT_ENVIRONMENT
printenv >> $GITHUB_ENV
if [[ "${{ runner.debug }}" == "1" ]]; then
cat $GITHUB_OUTPUT
fi
- name: Setup build environment
run: |
mkdir -p "${BUILD_DIR}/"
sudo find $BUILD_DIR/ -mindepth 1 -delete
echo "Starting build stage..."
echo "BUILD_DIR: ${BUILD_DIR}"
echo "CI_DIR: ${CI_DIR}"
echo "VERSION: ${{ steps.set-env.outputs.version }}"
echo "UV_PROJECT_ENVIRONMENT: ${UV_PROJECT_ENVIRONMENT}"
echo "VIRTUAL_ENV: ${VIRTUAL_ENV}"
echo "-------"
if [[ "${{ runner.debug }}" == "1" ]]; then
printenv
fi
- name: Build Panda
run: |
scons -j$(nproc) cache_dir=${{env.SCONS_CACHE_DIR}} ${{ github.workspace }}/panda
- name: Build Main Project
run: |
export PYTHONPATH="$BUILD_DIR"
./release/release_files.py | sort | uniq | rsync -rRl${RUNNER_DEBUG:+v} --files-from=- . $BUILD_DIR/
cd $BUILD_DIR
sed -i '/from .board.jungle import PandaJungle, PandaJungleDFU/s/^/#/' panda/__init__.py
scons -j$(nproc) cache_dir=${{env.SCONS_CACHE_DIR}} --minimal
touch ${BUILD_DIR}/prebuilt
if [[ "${{ runner.debug }}" == "1" ]]; then
ls -la ${BUILD_DIR}
fi
- name: Prepare Output
run: |
sudo rm -rf ${OUTPUT_DIR}
mkdir -p ${OUTPUT_DIR}
rsync -am${RUNNER_DEBUG:+v} \
--include='**/panda/board/' \
--include='**/panda/board/obj' \
--include='**/panda/board/obj/panda.bin.signed' \
--include='**/panda/board/obj/panda_h7.bin.signed' \
--include='**/panda/board/obj/bootstub.panda.bin' \
--include='**/panda/board/obj/bootstub.panda_h7.bin' \
--exclude='.sconsign.dblite' \
--exclude='*.a' \
--exclude='*.o' \
--exclude='*.os' \
--exclude='*.pyc' \
--exclude='moc_*' \
--exclude='*.cc' \
--exclude='Jenkinsfile' \
--exclude='supercombo.onnx' \
--exclude='**/panda/board/*' \
--exclude='**/panda/board/obj/**' \
--exclude='**/panda/certs/' \
--exclude='**/panda/crypto/' \
--exclude='**/release/' \
--exclude='**/.github/' \
--exclude='**/selfdrive/ui/replay/' \
--exclude='**/__pycache__/' \
--exclude='**/selfdrive/ui/*.h' \
--exclude='**/selfdrive/ui/**/*.h' \
--exclude='**/selfdrive/ui/qt/offroad/sunnypilot/' \
--exclude='${{env.SCONS_CACHE_DIR}}' \
--exclude='**/.git/' \
--exclude='**/SConstruct' \
--exclude='**/SConscript' \
--exclude='**/.venv/' \
--delete-excluded \
--chown=comma:comma \
${BUILD_DIR}/ ${OUTPUT_DIR}/
- name: 'Tar.gz files'
run: |
tar czf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }} .
ls -la prebuilt.tar.gz
- name: 'Upload Artifact'
uses: actions/upload-artifact@v4
with:
name: prebuilt
path: prebuilt.tar.gz
publish:
concurrency:
group: publish-${{ github.head_ref || github.ref_name }}
cancel-in-progress: true
if: ${{ github.event_name != 'pull_request' || github.event_name == 'pull_request' && github.event.pull_request.draft }}
needs: build
runs-on: ubuntu-24.04
environment: ${{ contains(fromJSON(vars.AUTO_DEPLOY_PREBUILT_BRANCHES), github.head_ref || github.ref_name) && 'auto-deploy' || 'feature-branch' }}
steps:
- uses: actions/checkout@v4
- name: Download build artifacts
uses: actions/download-artifact@v4
with:
name: prebuilt
- name: Untar prebuilt
run: |
mkdir -p ${{ env.OUTPUT_DIR }}
tar xzf prebuilt.tar.gz -C ${{ env.OUTPUT_DIR }}
- name: Configure Git
run: |
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"
- name: Publish to Public Repository
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
echo '${{ toJSON(needs.build.outputs) }}'
ls -la ${{ env.OUTPUT_DIR }}
${{ env.CI_DIR }}/publish.sh \
"${{ github.workspace }}" \
"${{ env.OUTPUT_DIR }}" \
"${{ needs.build.outputs.new_branch }}" \
"${{ needs.build.outputs.version }}" \
"https://x-access-token:${{github.token}}@github.com/sunnypilot/sunnypilot.git" \
"-${{ needs.build.outputs.extra_version_identifier }}"
echo ""
echo "---- To update the list of branches that auto deploy prebuilts -----"
echo ""
echo "1. Go to: ${{ github.server_url }}/${{ github.repository }}/settings/variables/actions/AUTO_DEPLOY_PREBUILT_BRANCHES"
echo "2. Current value: ${{ vars.AUTO_DEPLOY_PREBUILT_BRANCHES }}"
echo "3. Update as needed (JSON array with no spaces)"
notify:
needs: [ build, publish ]
runs-on: ubuntu-24.04
if: success()
steps:
- uses: actions/checkout@v4
- name: Setup Alpine Linux environment
uses: jirutka/setup-alpine@v1.2.0
with:
packages: 'jq gettext curl'
- name: Send Discord Notification
env:
DISCORD_WEBHOOK: ${{ contains(fromJSON(vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES), github.head_ref || github.ref_name) && secrets.DISCORD_DEV_FEEDBACK_CHANNEL_WEBHOOK || secrets.DISCORD_DEV_PRIVATE_CHANNEL_WEBHOOK }}
run: |
TEMPLATE='${{ vars.DISCORD_GENERAL_UPDATE_NOTICE }}'
export EXTRA_VERSION_IDENTIFIER="${{ needs.build.outputs.extra_version_identifier }}"
export VERSION="${{ needs.build.outputs.version }}"
export branch_name=${{ github.head_ref || github.ref_name }}
export new_branch=${{ needs.build.outputs.new_branch }}
export extra_version_identifier=${{ needs.build.outputs.extra_version_identifier || github.run_number}}
echo ${TEMPLATE} | envsubst | jq -c '.' | tee payload.json
curl -X POST -H "Content-Type: application/json" -d @payload.json $DISCORD_WEBHOOK
echo ""
echo "---- To update the list of branches that notify to dev-feedback -----"
echo ""
echo "1. Go to: ${{ github.server_url }}/${{ github.repository }}/settings/variables/actions/DEV_FEEDBACK_NOTIFICATION_BRANCHES"
echo "2. Current value: ${{ vars.DEV_FEEDBACK_NOTIFICATION_BRANCHES }}"
echo "3. Update as needed (JSON array with no spaces)"
shell: alpine.sh {0}

View File

@@ -3,25 +3,23 @@ on:
push:
branches:
- master
- master-new
pull_request_target:
types: [assigned, opened, synchronize, reopened, edited]
branches:
- 'master'
- 'master-new'
paths:
- 'selfdrive/ui/**'
workflow_dispatch:
env:
UI_JOB_NAME: "Create UI Report"
REPORT_NAME: ${{ github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/master-new') && 'master' || github.event.number }}
SHA: ${{ github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/master-new') && github.sha || github.event.pull_request.head.sha }}
REPORT_NAME: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && 'master' || github.event.number }}
SHA: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && github.sha || github.event.pull_request.head.sha }}
BRANCH_NAME: "openpilot/pr-${{ github.event.number }}"
jobs:
preview:
if: github.repository == 'sunnypilot/sunnypilot'
if: github.repository == 'commaai/openpilot'
name: preview
runs-on: ubuntu-latest
timeout-minutes: 20
@@ -60,13 +58,13 @@ jobs:
- name: Getting master ui
uses: actions/checkout@v4
with:
repository: sunnypilot/ci-artifacts
repository: commaai/ci-artifacts
ssh-key: ${{ secrets.CI_ARTIFACTS_DEPLOY_KEY }}
path: ${{ github.workspace }}/master_ui
ref: openpilot_master_ui
- name: Saving new master ui
if: (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/master-new') && github.event_name == 'push'
if: github.ref == 'refs/heads/master' && github.event_name == 'push'
working-directory: ${{ github.workspace }}/master_ui
run: |
git checkout --orphan=new_master_ui
@@ -86,7 +84,7 @@ jobs:
run: >-
sudo apt-get install -y imagemagick
scenes="homescreen settings_device settings_software settings_sunnylink settings_toggles settings_sunnypilot settings_sunnypilot_mads settings_developer offroad_alert update_available prime onroad onroad_disengaged onroad_override onroad_sidebar onroad_wide onroad_wide_sidebar onroad_alert_small onroad_alert_mid onroad_alert_full driver_camera body keyboard"
scenes="homescreen settings_device settings_software settings_toggles settings_developer offroad_alert update_available prime onroad onroad_disengaged onroad_override onroad_sidebar onroad_wide onroad_wide_sidebar onroad_alert_small onroad_alert_mid onroad_alert_full driver_camera body keyboard"
A=($scenes)
DIFF=""
@@ -108,13 +106,13 @@ jobs:
DIFF="${DIFF}<table>"
DIFF="${DIFF}<tr>"
DIFF="${DIFF} <td> master <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}_master_ref.png\"> </td>"
DIFF="${DIFF} <td> proposed <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}.png\"> </td>"
DIFF="${DIFF} <td> master <img src=\"https://raw.githubusercontent.com/commaai/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}_master_ref.png\"> </td>"
DIFF="${DIFF} <td> proposed <img src=\"https://raw.githubusercontent.com/commaai/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}.png\"> </td>"
DIFF="${DIFF}</tr>"
DIFF="${DIFF}<tr>"
DIFF="${DIFF} <td> diff <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}_diff.png\"> </td>"
DIFF="${DIFF} <td> composite diff <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}_diff.gif\"> </td>"
DIFF="${DIFF} <td> diff <img src=\"https://raw.githubusercontent.com/commaai/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}_diff.png\"> </td>"
DIFF="${DIFF} <td> composite diff <img src=\"https://raw.githubusercontent.com/commaai/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}_diff.gif\"> </td>"
DIFF="${DIFF}</tr>"
DIFF="${DIFF}</table>"
@@ -127,7 +125,7 @@ jobs:
if [[ $INDEX -eq 0 ]]; then
TABLE="${TABLE}<tr>"
fi
TABLE="${TABLE} <td> <img src=\"https://raw.githubusercontent.com/sunnypilot/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}.png\"> </td>"
TABLE="${TABLE} <td> <img src=\"https://raw.githubusercontent.com/commaai/ci-artifacts/${{ env.BRANCH_NAME }}/${A[$i]}.png\"> </td>"
if [[ $INDEX -eq 1 || $(($i + 1)) -eq ${#A[*]} ]]; then
TABLE="${TABLE}</tr>"
fi

5
.gitignore vendored
View File

@@ -103,8 +103,3 @@ Pipfile
# Ignore all local history of files
.history
.ionide
### JetBrains ###
!.idea/customTargets.xml
!.idea/tools/*
!.run/*

12
.gitmodules vendored
View File

@@ -1,18 +1,18 @@
[submodule "panda"]
path = panda
url = https://github.com/sunnyhaibin/panda.git
url = ../../commaai/panda.git
[submodule "opendbc"]
path = opendbc_repo
url = https://github.com/sunnypilot/opendbc.git
url = ../../commaai/opendbc.git
[submodule "msgq"]
path = msgq_repo
url = https://github.com/sunnypilot/msgq.git
url = ../../commaai/msgq.git
[submodule "rednose_repo"]
path = rednose_repo
url = https://github.com/commaai/rednose.git
url = ../../commaai/rednose.git
[submodule "teleoprtc_repo"]
path = teleoprtc_repo
url = https://github.com/commaai/teleoprtc
url = ../../commaai/teleoprtc
[submodule "tinygrad"]
path = tinygrad_repo
url = https://github.com/tinygrad/tinygrad.git
url = https://github.com/commaai/tinygrad.git

View File

@@ -1,25 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="CLionExternalBuildManager">
<target id="a62f99e8-5ec4-434c-8122-49efed5af108" name="uv Scons Build Debug" defaultType="TOOL">
<configuration id="b93ec964-16e5-4962-a12e-3ed360ce8f02" name="uv Scons Build Debug">
<build type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Build Debug" />
</build>
<clean type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Clean" />
</clean>
</configuration>
</target>
<target id="edd8ad9d-183b-467c-a355-0d9a0ecab026" name="uv Scons Build Release" defaultType="TOOL">
<configuration id="09523339-5ce3-4223-ab9e-904f38ad7752" name="uv Scons Build Release">
<build type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Build Release" />
</build>
<clean type="TOOL">
<tool actionId="Tool_External Tools_uv Scons Clean" />
</clean>
</configuration>
</target>
</component>
</project>

View File

@@ -1,23 +0,0 @@
<toolSet name="External Tools">
<tool name="uv Scons Build Debug" showInMainMenu="false" showInEditor="false" showInProject="false" showInSearchPopup="false" disabled="false" useConsole="true" showConsoleOnStdOut="false" showConsoleOnStdErr="false" synchronizeAfterRun="true">
<exec>
<option name="COMMAND" value="bash" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -u -j$(nproc) --compile_db --ccflags=\&quot;-fno-inline\&quot;&quot;" />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
<tool name="uv Scons Clean" showInMainMenu="false" showInEditor="false" showInProject="false" showInSearchPopup="false" disabled="false" useConsole="true" showConsoleOnStdOut="false" showConsoleOnStdErr="false" synchronizeAfterRun="true">
<exec>
<option name="COMMAND" value="bash" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -c&quot; " />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
<tool name="uv Scons Build Release" showInMainMenu="false" showInEditor="false" showInProject="false" showInSearchPopup="false" disabled="false" useConsole="true" showConsoleOnStdOut="false" showConsoleOnStdErr="false" synchronizeAfterRun="true">
<exec>
<option name="COMMAND" value="bash" />
<option name="PARAMETERS" value="-c &quot;source .venv/bin/activate &amp;&amp; scons -u -j$(nproc) --compile_db&quot; " />
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$" />
</exec>
</tool>
</toolSet>

View File

@@ -1,4 +1,4 @@
[lfs]
url = https://gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git/info/lfs
pushurl = ssh://git@gitlab.com/sunnypilot/public/sunnypilot-new-lfs.git
url = https://gitlab.com/commaai/openpilot-lfs.git/info/lfs
pushurl = ssh://git@gitlab.com/commaai/openpilot-lfs.git
locksverify = false

View File

@@ -1,4 +0,0 @@
[lfs]
url = https://gitlab.com/commaai/openpilot-lfs.git/info/lfs
pushurl = ssh://git@gitlab.com/commaai/openpilot-lfs.git
locksverify = false

View File

@@ -1,10 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Build Debug" type="CLionExternalRunConfiguration" factoryName="Application" REDIRECT_INPUT="false" ELEVATE="false" USE_EXTERNAL_CONSOLE="false" EMULATE_TERMINAL="false" WORKING_DIR="file://$ProjectFileDir$/selfdrive/ui" PASS_PARENT_ENVS_2="true" PROJECT_NAME="sunnypilot" TARGET_NAME="uv Scons Build Debug" CONFIG_NAME="uv Scons Build Debug" RUN_PATH="ui">
<envs>
<env name="QT_DBL_CLICK_DIST" value="150" />
</envs>
<method v="2">
<option name="CLION.EXTERNAL.BUILD" enabled="true" />
</method>
</configuration>
</component>

View File

@@ -1,10 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="Build Release" type="CLionExternalRunConfiguration" factoryName="Application" REDIRECT_INPUT="false" ELEVATE="false" USE_EXTERNAL_CONSOLE="false" EMULATE_TERMINAL="false" WORKING_DIR="file://$ProjectFileDir$/selfdrive/ui" PASS_PARENT_ENVS_2="true" PROJECT_NAME="sunnypilot" TARGET_NAME="uv Scons Build Release" CONFIG_NAME="uv Scons Build Release" RUN_PATH="ui">
<envs>
<env name="QT_DBL_CLICK_DIST" value="150" />
</envs>
<method v="2">
<option name="CLION.EXTERNAL.BUILD" enabled="true" />
</method>
</configuration>
</component>

View File

@@ -1,9 +1,9 @@
FROM ghcr.io/commaai/openpilot-base:latest
ENV PYTHONUNBUFFERED=1
ENV PYTHONUNBUFFERED 1
ENV OPENPILOT_PATH=/home/batman/openpilot
ENV PYTHONPATH=${OPENPILOT_PATH}:${PYTHONPATH}
ENV OPENPILOT_PATH /home/batman/openpilot
ENV PYTHONPATH ${OPENPILOT_PATH}:${PYTHONPATH}
RUN mkdir -p ${OPENPILOT_PATH}
WORKDIR ${OPENPILOT_PATH}

View File

@@ -1,16 +1,16 @@
FROM ubuntu:24.04
ENV PYTHONUNBUFFERED=1
ENV PYTHONUNBUFFERED 1
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -y --no-install-recommends sudo tzdata locales ssh pulseaudio xvfb x11-xserver-utils gnome-screenshot python3-tk python3-dev && \
apt-get install -y --no-install-recommends sudo tzdata locales ssh pulseaudio xvfb x11-xserver-utils gnome-screenshot && \
rm -rf /var/lib/apt/lists/*
RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && locale-gen
ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US:en
ENV LC_ALL=en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
COPY tools/install_ubuntu_dependencies.sh /tmp/tools/
RUN /tmp/tools/install_ubuntu_dependencies.sh && \
@@ -55,9 +55,9 @@ RUN mkdir -p /tmp/opencl-driver-intel && \
cd / && \
rm -rf /tmp/opencl-driver-intel
ENV NVIDIA_VISIBLE_DEVICES=all
ENV NVIDIA_DRIVER_CAPABILITIES=graphics,utility,compute
ENV QTWEBENGINE_DISABLE_SANDBOX=1
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES graphics,utility,compute
ENV QTWEBENGINE_DISABLE_SANDBOX 1
RUN dbus-uuidgen > /etc/machine-id

View File

@@ -1,21 +0,0 @@
# Custom MIT License
Copyright (c) 2024, Haibin Wen, SUNNYPILOT LLC
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to view and modify the Software, subject to the following conditions:
1. **Permission Required**: Permission Required for Commercial, For-Profit, or Closed Source Use: Use of the Software, in whole or in part, for any commercial purposes, for-profit projects, or in closed source projects requires explicit written permission from the original author(s).
2. **Redistribution**: Any redistribution of the Software, modified or unmodified, must retain this license notice and the following acknowledgment:
"This software is licensed under a custom license requiring permission for use."
3. **Visibility**: Any project that uses the Software must visibly mention the following acknowledgment:
"This project uses software from Haibin Wen and SUNNYPILOT LLC and is licensed under a custom license requiring permission for use."
4. **No Warranty**: THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Contact sunnypilot Support <support@sunnypilot.ai> for permission requests.
---
Haibin Wen, SUNNYPILOT LLC

View File

@@ -38,8 +38,7 @@ Quick start: `bash <(curl -fsSL openpilot.comma.ai)`
</tr>
</table>
Using openpilot in a car
To start using openpilot in a car
------
To use openpilot in a car, you need four things:
@@ -50,14 +49,6 @@ To use openpilot in a car, you need four things:
We have detailed instructions for [how to install the harness and device in a car](https://comma.ai/setup). Note that it's possible to run openpilot on [other hardware](https://blog.comma.ai/self-driving-car-for-free/), although it's not plug-and-play.
### Branches
| branch | URL | description |
|------------------|----------------------------------------|-------------------------------------------------------------------------------------|
| `release3` | openpilot.comma.ai | This is openpilot's release branch. |
| `release3-staging` | openpilot-test.comma.ai | This is the staging branch for releases. Use it to get new releases slightly early. |
| `nightly` | openpilot-nightly.comma.ai | This is the bleeding edge development branch. Do not expect this to be stable. |
| `nightly-dev` | installer.comma.ai/commaai/nightly-dev | Same as nightly, but includes experimental development features for some cars. |
To start developing openpilot
------

View File

@@ -49,6 +49,10 @@ AddOption('--ccflags',
default='',
help='pass arbitrary flags over the command line')
AddOption('--snpe',
action='store_true',
help='use SNPE on PC')
AddOption('--external-sconscript',
action='store',
metavar='FILE',
@@ -70,12 +74,6 @@ AddOption('--minimal',
default=os.path.exists(File('#.lfsconfig').abspath), # minimal by default on release branch (where there's no LFS)
help='the minimum build to run openpilot. no tests, tools, etc.')
AddOption('--stock-ui',
action='store_true',
dest='stock_ui',
default=False,
help='Build stock openpilot UI instead of sunnypilot UI')
## Architecture name breakdown (arch)
## - larch64: linux tici aarch64
## - aarch64: linux pc aarch64
@@ -174,10 +172,6 @@ else:
if arch != "Darwin":
ldflags += ["-Wl,--as-needed", "-Wl,--no-undefined"]
if not GetOption('stock_ui'):
cflags += ["-DSUNNYPILOT"]
cxxflags += ["-DSUNNYPILOT"]
ccflags_option = GetOption('ccflags')
if ccflags_option:
ccflags += ccflags_option.split(' ')
@@ -243,8 +237,7 @@ if GetOption('compile_db'):
env.CompilationDatabase('compile_commands.json')
# Setup cache dir
default_cache_dir = '/data/scons_cache' if AGNOS else '/tmp/scons_cache'
cache_dir = ARGUMENTS.get('cache_dir', default_cache_dir)
cache_dir = '/data/scons_cache' if AGNOS else '/tmp/scons_cache'
CacheDir(cache_dir)
Clean(["."], cache_dir)

View File

@@ -8,73 +8,10 @@ $Cxx.namespace("cereal");
# cereal, so use these if you want custom events in your fork.
# you can rename the struct, but don't change the identifier
struct SelfdriveStateSP @0x81c2f05a394cf4af {
mads @0 :ModularAssistiveDrivingSystem;
struct ModularAssistiveDrivingSystem {
state @0 :ModularAssistiveDrivingSystemState;
enabled @1 :Bool;
active @2 :Bool;
available @3 :Bool;
enum ModularAssistiveDrivingSystemState {
disabled @0;
paused @1;
enabled @2;
softDisabling @3;
overriding @4;
}
}
struct CustomReserved0 @0x81c2f05a394cf4af {
}
struct ModelManagerSP @0xaedffd8f31e7b55d {
activeBundle @0 :ModelBundle;
selectedBundle @1 :ModelBundle;
availableBundles @2 :List(ModelBundle);
struct DownloadUri {
uri @0 :Text;
sha256 @1 :Text;
}
enum Type {
drive @0;
navigation @1;
metadata @2;
}
struct Model {
fullName @0 :Text;
fileName @1 :Text;
downloadUri @2 :DownloadUri;
downloadProgress @3 :DownloadProgress;
type @4 :Type;
}
enum DownloadStatus {
notDownloading @0;
downloading @1;
downloaded @2;
cached @3;
failed @4;
}
struct DownloadProgress {
status @0 :DownloadStatus;
progress @1 :Float32;
eta @2 :UInt32;
}
struct ModelBundle {
index @0 :UInt32;
internalName @1 :Text;
displayName @2 :Text;
models @3 :List(Model);
status @4 :DownloadStatus;
generation @5 :UInt32;
environment @6 :Text;
is20hz @7 :Bool;
}
struct CustomReserved1 @0xaedffd8f31e7b55d {
}
struct CustomReserved2 @0xf35cc4560bbf6ec2 {

View File

@@ -125,79 +125,6 @@ struct OnroadEvent @0xc4fa6047f024e718 {
espActive @90;
personalityChanged @91;
aeb @92;
eventReserved93 @93;
eventReserved94 @94;
eventReserved95 @95;
eventReserved96 @96;
eventReserved97 @97;
eventReserved98 @98;
eventReserved99 @99;
eventReserved100 @100;
eventReserved101 @101;
eventReserved102 @102;
eventReserved103 @103;
eventReserved104 @104;
eventReserved105 @105;
eventReserved106 @106;
eventReserved107 @107;
eventReserved108 @108;
eventReserved109 @109;
eventReserved110 @110;
eventReserved111 @111;
eventReserved112 @112;
eventReserved113 @113;
eventReserved114 @114;
eventReserved115 @115;
eventReserved116 @116;
eventReserved117 @117;
eventReserved118 @118;
eventReserved119 @119;
eventReserved120 @120;
eventReserved121 @121;
eventReserved122 @122;
eventReserved123 @123;
eventReserved124 @124;
eventReserved125 @125;
eventReserved126 @126;
eventReserved127 @127;
eventReserved128 @128;
eventReserved129 @129;
eventReserved130 @130;
eventReserved131 @131;
eventReserved132 @132;
eventReserved133 @133;
eventReserved134 @134;
eventReserved135 @135;
eventReserved136 @136;
eventReserved137 @137;
eventReserved138 @138;
eventReserved139 @139;
eventReserved140 @140;
eventReserved141 @141;
eventReserved142 @142;
eventReserved143 @143;
eventReserved144 @144;
eventReserved145 @145;
eventReserved146 @146;
eventReserved147 @147;
eventReserved148 @148;
eventReserved149 @149;
eventReserved150 @150;
# sunnypilot
lkasEnable @151;
lkasDisable @152;
manualSteeringRequired @153;
manualLongitudinalRequired @154;
silentLkasEnable @155;
silentLkasDisable @156;
silentBrakeHold @157;
silentWrongGear @158;
silentReverseGear @159;
silentDoorOpen @160;
silentSeatbeltNotLatched @161;
silentParkBrake @162;
controlsMismatchLateral @163;
soundsUnavailableDEPRECATED @47;
}
@@ -662,7 +589,6 @@ struct PandaState @0xa7649e2575e4591e {
# safety stuff
controlsAllowed @3 :Bool;
controlsAllowedLat @5 :Bool;
safetyRxInvalid @19 :UInt32;
safetyTxBlocked @24 :UInt32;
safetyModel @14 :Car.CarParams.SafetyModel;
@@ -770,6 +696,7 @@ struct PandaState @0xa7649e2575e4591e {
}
gasInterceptorDetectedDEPRECATED @4 :Bool;
startedSignalDetectedDEPRECATED @5 :Bool;
hasGpsDEPRECATED @6 :Bool;
gmlanSendErrsDEPRECATED @9 :UInt32;
fanSpeedRpmDEPRECATED @11 :UInt16;
@@ -2513,14 +2440,6 @@ struct Microphone {
filteredSoundPressureWeightedDb @2 :Float32;
}
struct Touch {
sec @0 :Int64;
usec @1 :Int64;
type @2 :UInt8;
code @3 :Int32;
value @4 :Int32;
}
struct Event {
logMonoTime @0 :UInt64; # nanoseconds
valid @67 :Bool = true;
@@ -2601,9 +2520,6 @@ struct Event {
logMessage @18 :Text;
errorLogMessage @85 :Text;
# touch frame
touch @135 :List(Touch);
# navigation
navInstruction @82 :NavInstruction;
navRoute @83 :NavRoute;
@@ -2631,8 +2547,8 @@ struct Event {
customReservedRawData2 @126 :Data;
# *********** Custom: reserved for forks ***********
selfdriveStateSP @107 :Custom.SelfdriveStateSP;
modelManagerSP @108 :Custom.ModelManagerSP;
customReserved0 @107 :Custom.CustomReserved0;
customReserved1 @108 :Custom.CustomReserved1;
customReserved2 @109 :Custom.CustomReserved2;
customReserved3 @110 :Custom.CustomReserved3;
customReserved4 @111 :Custom.CustomReserved4;

49
cereal/maptile.capnp Normal file
View File

@@ -0,0 +1,49 @@
using Cxx = import "./include/c++.capnp";
$Cxx.namespace("cereal");
@0xa086df597ef5d7a0;
# Geometry
struct Point {
x @0: Float64;
y @1: Float64;
z @2: Float64;
}
struct PolyLine {
points @0: List(Point);
}
# Map features
struct Lane {
id @0 :Text;
leftBoundary @1 :LaneBoundary;
rightBoundary @2 :LaneBoundary;
leftAdjacentId @3 :Text;
rightAdjacentId @4 :Text;
inboundIds @5 :List(Text);
outboundIds @6 :List(Text);
struct LaneBoundary {
polyLine @0 :PolyLine;
startHeading @1 :Float32; # WRT north
}
}
# Map tiles
struct TileSummary {
version @0 :Text;
updatedAt @1 :UInt64; # Millis since epoch
level @2 :UInt8;
x @3 :UInt16;
y @4 :UInt16;
}
struct MapTile {
summary @0 :TileSummary;
lanes @1 :List(Lane);
}

View File

@@ -22,7 +22,6 @@ _services: dict[str, tuple] = {
"temperatureSensor2": (True, 2., 200),
"gpsNMEA": (True, 9.),
"deviceState": (True, 2., 1),
"touch": (True, 20., 1),
"can": (True, 100., 2053), # decimation gives ~3 msgs in a full segment
"controlsState": (True, 100., 10),
"selfdriveState": (True, 100., 10),
@@ -74,10 +73,6 @@ _services: dict[str, tuple] = {
"userFlag": (True, 0., 1),
"microphone": (True, 10., 10),
# sunnypilot
"modelManagerSP": (False, 1., 1),
"selfdriveStateSP": (True, 100., 10),
# debug
"uiDebug": (True, 0., 1),
"testJoystick": (True, 0.),

View File

@@ -1,27 +1,46 @@
from openpilot.common.api.comma_connect import CommaConnectApi
from sunnypilot.sunnylink.api import SunnylinkApi
import jwt
import os
import requests
from datetime import datetime, timedelta, UTC
from openpilot.system.hardware.hw import Paths
from openpilot.system.version import get_version
API_HOST = os.getenv('API_HOST', 'https://api.commadotai.com')
class Api:
def __init__(self, dongle_id, use_sunnylink=False):
if use_sunnylink:
self.service = SunnylinkApi(dongle_id)
else:
self.service = CommaConnectApi(dongle_id)
def request(self, method, endpoint, **params):
return self.service.request(method, endpoint, **params)
def __init__(self, dongle_id):
self.dongle_id = dongle_id
with open(Paths.persist_root()+'/comma/id_rsa') as f:
self.private_key = f.read()
def get(self, *args, **kwargs):
return self.service.get(*args, **kwargs)
return self.request('GET', *args, **kwargs)
def post(self, *args, **kwargs):
return self.service.post(*args, **kwargs)
return self.request('POST', *args, **kwargs)
def request(self, method, endpoint, timeout=None, access_token=None, **params):
return api_get(endpoint, method=method, timeout=timeout, access_token=access_token, **params)
def get_token(self, expiry_hours=1):
return self.service.get_token(expiry_hours)
now = datetime.now(UTC).replace(tzinfo=None)
payload = {
'identity': self.dongle_id,
'nbf': now,
'iat': now,
'exp': now + timedelta(hours=expiry_hours)
}
token = jwt.encode(payload, self.private_key, algorithm='RS256')
if isinstance(token, bytes):
token = token.decode('utf8')
return token
def api_get(endpoint, method='GET', timeout=None, access_token=None, use_sunnylink=False, **params):
return SunnylinkApi(None).api_get(endpoint, method, timeout, access_token, **params) if use_sunnylink \
else CommaConnectApi(None).api_get(endpoint, method, timeout, access_token, **params)
def api_get(endpoint, method='GET', timeout=None, access_token=None, **params):
headers = {}
if access_token is not None:
headers['Authorization'] = "JWT " + access_token
headers['User-Agent'] = "openpilot-" + get_version()
return requests.request(method, API_HOST + "/" + endpoint, timeout=timeout, headers=headers, params=params)

View File

@@ -1,56 +0,0 @@
import jwt
import requests
import unicodedata
from datetime import datetime, timedelta, UTC
from openpilot.system.hardware.hw import Paths
from openpilot.system.version import get_version
class BaseApi:
def __init__(self, dongle_id, api_host, user_agent="openpilot-"):
self.dongle_id = dongle_id
self.api_host = api_host
self.user_agent = user_agent
with open(f'{Paths.persist_root()}/comma/id_rsa') as f:
self.private_key = f.read()
def get(self, *args, **kwargs):
return self.request('GET', *args, **kwargs)
def post(self, *args, **kwargs):
return self.request('POST', *args, **kwargs)
def request(self, method, endpoint, timeout=None, access_token=None, **params):
return self.api_get(endpoint, method=method, timeout=timeout, access_token=access_token, **params)
def _get_token(self, expiry_hours=1, **extra_payload):
now = datetime.now(UTC).replace(tzinfo=None)
payload = {
'identity': self.dongle_id,
'nbf': now,
'iat': now,
'exp': now + timedelta(hours=expiry_hours),
**extra_payload
}
token = jwt.encode(payload, self.private_key, algorithm='RS256')
if isinstance(token, bytes):
token = token.decode('utf8')
return token
def get_token(self, expiry_hours=1):
return self._get_token(expiry_hours)
def remove_non_ascii_chars(self, text):
normalized_text = unicodedata.normalize('NFD', text)
ascii_encoded_text = normalized_text.encode('ascii', 'ignore')
return ascii_encoded_text.decode()
def api_get(self, endpoint, method='GET', timeout=None, access_token=None, **params):
headers = {}
if access_token is not None:
headers['Authorization'] = "JWT " + access_token
version = self.remove_non_ascii_chars(get_version())
headers['User-Agent'] = self.user_agent + version
return requests.request(method, f"{self.api_host}/{endpoint}", timeout=timeout, headers=headers, params=params)

View File

@@ -1,11 +0,0 @@
import os
from openpilot.common.api.base import BaseApi
API_HOST = os.getenv('API_HOST', 'https://api.commadotai.com')
class CommaConnectApi(BaseApi):
def __init__(self, dongle_id):
super().__init__(dongle_id, API_HOST)
self.user_agent = "openpilot-"

View File

@@ -109,35 +109,36 @@ std::unordered_map<std::string, uint32_t> keys = {
{"CurrentBootlog", PERSISTENT},
{"CurrentRoute", CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION},
{"DisableLogging", CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION},
{"DisablePowerDown", PERSISTENT | BACKUP},
{"DisableUpdates", PERSISTENT | BACKUP},
{"DisengageOnAccelerator", PERSISTENT | BACKUP},
{"DisablePowerDown", PERSISTENT},
{"DisableUpdates", PERSISTENT},
{"DisengageOnAccelerator", PERSISTENT},
{"DongleId", PERSISTENT},
{"DoReboot", CLEAR_ON_MANAGER_START},
{"DoShutdown", CLEAR_ON_MANAGER_START},
{"DoUninstall", CLEAR_ON_MANAGER_START},
{"ExperimentalLongitudinalEnabled", PERSISTENT | DEVELOPMENT_ONLY | BACKUP},
{"ExperimentalMode", PERSISTENT | BACKUP},
{"ExperimentalModeConfirmed", PERSISTENT | BACKUP},
{"ExperimentalLongitudinalEnabled", PERSISTENT | DEVELOPMENT_ONLY},
{"ExperimentalMode", PERSISTENT},
{"ExperimentalModeConfirmed", PERSISTENT},
{"FirmwareQueryDone", CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION},
{"ForcePowerDown", PERSISTENT},
{"GitBranch", PERSISTENT},
{"GitCommit", PERSISTENT},
{"GitCommitDate", PERSISTENT},
{"GitDiff", PERSISTENT},
{"GithubSshKeys", PERSISTENT | BACKUP},
{"GithubUsername", PERSISTENT | BACKUP},
{"GithubSshKeys", PERSISTENT},
{"GithubUsername", PERSISTENT},
{"GitRemote", PERSISTENT},
{"GsmApn", PERSISTENT | BACKUP},
{"GsmMetered", PERSISTENT | BACKUP},
{"GsmRoaming", PERSISTENT | BACKUP},
{"GsmApn", PERSISTENT},
{"GsmMetered", PERSISTENT},
{"GsmRoaming", PERSISTENT},
{"HardwareSerial", PERSISTENT},
{"HasAcceptedTerms", PERSISTENT},
{"IMEI", PERSISTENT},
{"InstallDate", PERSISTENT},
{"IsDriverViewEnabled", CLEAR_ON_MANAGER_START},
{"IsEngaged", PERSISTENT},
{"IsLdwEnabled", PERSISTENT | BACKUP},
{"IsMetric", PERSISTENT | BACKUP},
{"IsLdwEnabled", PERSISTENT},
{"IsMetric", PERSISTENT},
{"IsOffroad", CLEAR_ON_MANAGER_START},
{"IsOnroad", PERSISTENT},
{"IsRhdDetected", PERSISTENT},
@@ -145,7 +146,7 @@ std::unordered_map<std::string, uint32_t> keys = {
{"IsTakingSnapshot", CLEAR_ON_MANAGER_START},
{"IsTestedBranch", CLEAR_ON_MANAGER_START},
{"JoystickDebugMode", CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION},
{"LanguageSetting", PERSISTENT | BACKUP},
{"LanguageSetting", PERSISTENT},
{"LastAthenaPingTime", CLEAR_ON_MANAGER_START},
{"LastGPSPosition", PERSISTENT},
{"LastManagerExitReason", CLEAR_ON_MANAGER_START},
@@ -157,7 +158,7 @@ std::unordered_map<std::string, uint32_t> keys = {
{"LiveTorqueParameters", PERSISTENT | DONT_LOG},
{"LocationFilterInitialState", PERSISTENT},
{"LongitudinalManeuverMode", CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION},
{"LongitudinalPersonality", PERSISTENT | BACKUP},
{"LongitudinalPersonality", PERSISTENT},
{"NetworkMetered", PERSISTENT},
{"ObdMultiplexingChanged", CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION},
{"ObdMultiplexingEnabled", CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION},
@@ -173,17 +174,17 @@ std::unordered_map<std::string, uint32_t> keys = {
{"Offroad_TemperatureTooHigh", CLEAR_ON_MANAGER_START},
{"Offroad_UnofficialHardware", CLEAR_ON_MANAGER_START},
{"Offroad_UpdateFailed", CLEAR_ON_MANAGER_START},
{"OpenpilotEnabledToggle", PERSISTENT | BACKUP},
{"OpenpilotEnabledToggle", PERSISTENT},
{"PandaHeartbeatLost", CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION},
{"PandaSomResetTriggered", CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION},
{"PandaSignatures", CLEAR_ON_MANAGER_START},
{"PrimeType", PERSISTENT},
{"RecordFront", PERSISTENT | BACKUP},
{"RecordFront", PERSISTENT},
{"RecordFrontLock", PERSISTENT}, // for the internal fleet
{"SecOCKey", PERSISTENT | DONT_LOG}, // Candidate for | BACKUP
{"SecOCKey", PERSISTENT | DONT_LOG},
{"RouteCount", PERSISTENT},
{"SnoozeUpdate", CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION},
{"SshEnabled", PERSISTENT | BACKUP},
{"SshEnabled", PERSISTENT},
{"TermsVersion", PERSISTENT},
{"TrainingVersion", PERSISTENT},
{"UbloxAvailable", PERSISTENT},
@@ -199,28 +200,6 @@ std::unordered_map<std::string, uint32_t> keys = {
{"UpdaterTargetBranch", CLEAR_ON_MANAGER_START},
{"UpdaterLastFetchTime", PERSISTENT},
{"Version", PERSISTENT},
// --- sunnypilot params --- //
{"EnableGithubRunner", PERSISTENT | BACKUP},
// MADS params
{"Mads", PERSISTENT | BACKUP},
{"MadsMainCruiseAllowed", PERSISTENT | BACKUP},
{"MadsPauseLateralOnBrake", PERSISTENT | BACKUP},
{"MadsUnifiedEngagementMode", PERSISTENT | BACKUP},
// Model Manager params
{"ModelManager_ActiveBundle", PERSISTENT},
{"ModelManager_DownloadIndex", CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION},
{"ModelManager_LastSyncTime", CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION},
{"ModelManager_ModelsCache", PERSISTENT | BACKUP},
// sunnylink params
{"EnableSunnylinkUploader", PERSISTENT | BACKUP},
{"LastSunnylinkPingTime", CLEAR_ON_MANAGER_START},
{"SunnylinkDongleId", PERSISTENT},
{"SunnylinkdPid", PERSISTENT},
{"SunnylinkEnabled", PERSISTENT},
};
} // namespace

View File

@@ -16,7 +16,6 @@ enum ParamKeyType {
CLEAR_ON_OFFROAD_TRANSITION = 0x10,
DONT_LOG = 0x20,
DEVELOPMENT_ONLY = 0x40,
BACKUP = 0x80,
ALL = 0xFFFFFFFF
};

View File

@@ -1,7 +1,7 @@
import numpy as np
from openpilot.common.transformations.orientation import rot_from_euler
from openpilot.common.transformations.camera import get_view_frame_from_calib_frame, view_frame_from_device_frame, _ar_ox_fisheye
from openpilot.common.transformations.camera import get_view_frame_from_calib_frame, view_frame_from_device_frame
# segnet
SEGNET_SIZE = (512, 384)
@@ -39,13 +39,6 @@ sbigmodel_intrinsics = np.array([
[0.0, sbigmodel_fl, 0.5 * (256 + MEDMODEL_CY)],
[0.0, 0.0, 1.0]])
DM_INPUT_SIZE = (1440, 960)
dmonitoringmodel_fl = _ar_ox_fisheye.focal_length
dmonitoringmodel_intrinsics = np.array([
[dmonitoringmodel_fl, 0.0, DM_INPUT_SIZE[0]/2],
[0.0, dmonitoringmodel_fl, DM_INPUT_SIZE[1]/2 - (_ar_ox_fisheye.height - DM_INPUT_SIZE[1])/2],
[0.0, 0.0, 1.0]])
bigmodel_frame_from_calib_frame = np.dot(bigmodel_intrinsics,
get_view_frame_from_calib_frame(0, 0, 0, 0))

View File

@@ -255,29 +255,6 @@ bool ends_with(const std::string& s, const std::string& suffix) {
strcmp(s.c_str() + (s.size() - suffix.size()), suffix.c_str()) == 0;
}
std::string strip(const std::string &str) {
auto should_trim = [](unsigned char ch) {
// trim whitespace or a null character
return std::isspace(ch) || ch == '\0';
};
size_t start = 0;
while (start < str.size() && should_trim(static_cast<unsigned char>(str[start]))) {
start++;
}
if (start == str.size()) {
return "";
}
size_t end = str.size() - 1;
while (end > 0 && should_trim(static_cast<unsigned char>(str[end]))) {
end--;
}
return str.substr(start, end - start + 1);
}
std::string check_output(const std::string& command) {
char buffer[128];
std::string result;

View File

@@ -74,7 +74,6 @@ float getenv(const char* key, float default_val);
std::string hexdump(const uint8_t* in, const size_t size);
bool starts_with(const std::string &s1, const std::string &s2);
bool ends_with(const std::string &s, const std::string &suffix);
std::string strip(const std::string &str);
// ***** random helpers *****
int random_int(int min, int max);

View File

@@ -4,7 +4,7 @@
A supported vehicle is one that just works when you install a comma device. All supported cars provide a better experience than any stock system. Supported vehicles reference the US market unless otherwise specified.
# 290 Supported Cars
# 289 Supported Cars
|Make|Model|Supported Package|ACC|No ACC accel below|No ALC below|Steering Torque|Resume from stop|<a href="##"><img width=2000></a>Hardware Needed<br>&nbsp;|Video|
|---|---|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
@@ -56,7 +56,6 @@ A supported vehicle is one that just works when you install a comma device. All
|Genesis|GV60 (Performance Trim) 2022-23[<sup>5</sup>](#footnotes)|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai K connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Genesis&model=GV60 (Performance Trim) 2022-23">Buy Here</a></sub></details>||
|Genesis|GV70 (2.5T Trim, without HDA II) 2022-23[<sup>5</sup>](#footnotes)|All|Stock|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai L connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Genesis&model=GV70 (2.5T Trim, without HDA II) 2022-23">Buy Here</a></sub></details>||
|Genesis|GV70 (3.5T Trim, without HDA II) 2022-23[<sup>5</sup>](#footnotes)|All|Stock|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai M connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Genesis&model=GV70 (3.5T Trim, without HDA II) 2022-23">Buy Here</a></sub></details>||
|Genesis|GV70 Electrified (Australia Only) 2022[<sup>5</sup>](#footnotes)|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai Q connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Genesis&model=GV70 Electrified (Australia Only) 2022">Buy Here</a></sub></details>||
|Genesis|GV70 Electrified (with HDA II) 2023[<sup>5</sup>](#footnotes)|Highway Driving Assist II|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai Q connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Genesis&model=GV70 Electrified (with HDA II) 2023">Buy Here</a></sub></details>||
|Genesis|GV80 2023[<sup>5</sup>](#footnotes)|All|Stock|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai M connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Genesis&model=GV80 2023">Buy Here</a></sub></details>||
|GMC|Sierra 1500 2020-21|Driver Alert Package II|openpilot available[<sup>1</sup>](#footnotes)|0 mph|6 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 GM connector<br>- 1 comma 3X<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=GMC&model=Sierra 1500 2020-21">Buy Here</a></sub></details>|<a href="https://youtu.be/5HbNoBLzRwE" target="_blank"><img height="18px" src="assets/icon-youtube.svg"></img></a>|
@@ -80,7 +79,7 @@ A supported vehicle is one that just works when you install a comma device. All
|Honda|Odyssey 2018-20|Honda Sensing|openpilot|26 mph|0 mph|[![star](assets/icon-star-empty.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 Honda Nidec connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Honda&model=Odyssey 2018-20">Buy Here</a></sub></details>||
|Honda|Passport 2019-23|All|openpilot|26 mph|12 mph|[![star](assets/icon-star-empty.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 Honda Nidec connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Honda&model=Passport 2019-23">Buy Here</a></sub></details>||
|Honda|Pilot 2016-22|Honda Sensing|openpilot|26 mph|12 mph|[![star](assets/icon-star-empty.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 Honda Nidec connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Honda&model=Pilot 2016-22">Buy Here</a></sub></details>||
|Honda|Ridgeline 2017-25|Honda Sensing|openpilot|26 mph|12 mph|[![star](assets/icon-star-empty.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 Honda Nidec connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Honda&model=Ridgeline 2017-25">Buy Here</a></sub></details>||
|Honda|Ridgeline 2017-24|Honda Sensing|openpilot|26 mph|12 mph|[![star](assets/icon-star-empty.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 Honda Nidec connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Honda&model=Ridgeline 2017-24">Buy Here</a></sub></details>||
|Hyundai|Azera 2022|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai K connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Azera 2022">Buy Here</a></sub></details>||
|Hyundai|Azera Hybrid 2019|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai C connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Azera Hybrid 2019">Buy Here</a></sub></details>||
|Hyundai|Azera Hybrid 2020|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai K connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Azera Hybrid 2020">Buy Here</a></sub></details>||
@@ -92,7 +91,7 @@ A supported vehicle is one that just works when you install a comma device. All
|Hyundai|Elantra Hybrid 2021-23|Smart Cruise Control (SCC)|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai K connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Elantra Hybrid 2021-23">Buy Here</a></sub></details>|<a href="https://youtu.be/_EdYQtV52-c" target="_blank"><img height="18px" src="assets/icon-youtube.svg"></img></a>|
|Hyundai|Genesis 2015-16|Smart Cruise Control (SCC)|Stock|19 mph|37 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai J connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Genesis 2015-16">Buy Here</a></sub></details>||
|Hyundai|i30 2017-19|Smart Cruise Control (SCC)|Stock|0 mph|32 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai E connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=i30 2017-19">Buy Here</a></sub></details>||
|Hyundai|Ioniq 5 (Southeast Asia and Europe only) 2022-24[<sup>5</sup>](#footnotes)|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai Q connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Ioniq 5 (Southeast Asia and Europe only) 2022-24">Buy Here</a></sub></details>||
|Hyundai|Ioniq 5 (Non-US only) 2022-24[<sup>5</sup>](#footnotes)|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai Q connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Ioniq 5 (Non-US only) 2022-24">Buy Here</a></sub></details>||
|Hyundai|Ioniq 5 (with HDA II) 2022-24[<sup>5</sup>](#footnotes)|Highway Driving Assist II|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai Q connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Ioniq 5 (with HDA II) 2022-24">Buy Here</a></sub></details>||
|Hyundai|Ioniq 5 (without HDA II) 2022-24[<sup>5</sup>](#footnotes)|Highway Driving Assist|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai K connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Ioniq 5 (without HDA II) 2022-24">Buy Here</a></sub></details>||
|Hyundai|Ioniq 6 (with HDA II) 2023-24[<sup>5</sup>](#footnotes)|Highway Driving Assist II|Stock|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai P connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Ioniq 6 (with HDA II) 2023-24">Buy Here</a></sub></details>||
@@ -104,7 +103,7 @@ A supported vehicle is one that just works when you install a comma device. All
|Hyundai|Ioniq Plug-in Hybrid 2020-22|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai H connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Ioniq Plug-in Hybrid 2020-22">Buy Here</a></sub></details>||
|Hyundai|Kona 2020|Smart Cruise Control (SCC)|openpilot available[<sup>1</sup>](#footnotes)|6 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai B connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Kona 2020">Buy Here</a></sub></details>||
|Hyundai|Kona Electric 2018-21|Smart Cruise Control (SCC)|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai G connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Kona Electric 2018-21">Buy Here</a></sub></details>||
|Hyundai|Kona Electric 2022-23|Smart Cruise Control (SCC)|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai O connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Kona Electric 2022-23">Buy Here</a></sub></details>||
|Hyundai|Kona Electric 2022-23|Smart Cruise Control (SCC)|Stock|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai O connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Kona Electric 2022-23">Buy Here</a></sub></details>||
|Hyundai|Kona Electric (with HDA II, Korea only) 2023[<sup>5</sup>](#footnotes)|Smart Cruise Control (SCC)|Stock|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai R connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Kona Electric (with HDA II, Korea only) 2023">Buy Here</a></sub></details>|<a href="https://www.youtube.com/watch?v=U2fOCmcQ8hw" target="_blank"><img height="18px" src="assets/icon-youtube.svg"></img></a>|
|Hyundai|Kona Hybrid 2020|Smart Cruise Control (SCC)|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai I connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Kona Hybrid 2020">Buy Here</a></sub></details>||
|Hyundai|Palisade 2020-22|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai H connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Hyundai&model=Palisade 2020-22">Buy Here</a></sub></details>|<a href="https://youtu.be/TAnDqjF4fDY?t=456" target="_blank"><img height="18px" src="assets/icon-youtube.svg"></img></a>|
@@ -295,7 +294,7 @@ A supported vehicle is one that just works when you install a comma device. All
|Volkswagen|Teramont 2018-22|Adaptive Cruise Control (ACC) & Lane Assist|openpilot available[<sup>1,12</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 USB-C coupler<br>- 1 VW J533 connector<br>- 1 comma 3X<br>- 1 harness box<br>- 1 long OBD-C cable<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Volkswagen&model=Teramont 2018-22">Buy Here</a></sub></details>||
|Volkswagen|Teramont Cross Sport 2021-22|Adaptive Cruise Control (ACC) & Lane Assist|openpilot available[<sup>1,12</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 USB-C coupler<br>- 1 VW J533 connector<br>- 1 comma 3X<br>- 1 harness box<br>- 1 long OBD-C cable<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Volkswagen&model=Teramont Cross Sport 2021-22">Buy Here</a></sub></details>||
|Volkswagen|Teramont X 2021-22|Adaptive Cruise Control (ACC) & Lane Assist|openpilot available[<sup>1,12</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 USB-C coupler<br>- 1 VW J533 connector<br>- 1 comma 3X<br>- 1 harness box<br>- 1 long OBD-C cable<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Volkswagen&model=Teramont X 2021-22">Buy Here</a></sub></details>||
|Volkswagen|Tiguan 2018-23|Adaptive Cruise Control (ACC) & Lane Assist|openpilot available[<sup>1,12</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 USB-C coupler<br>- 1 VW J533 connector<br>- 1 comma 3X<br>- 1 harness box<br>- 1 long OBD-C cable<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Volkswagen&model=Tiguan 2018-23">Buy Here</a></sub></details>||
|Volkswagen|Tiguan 2018-24|Adaptive Cruise Control (ACC) & Lane Assist|openpilot available[<sup>1,12</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 USB-C coupler<br>- 1 VW J533 connector<br>- 1 comma 3X<br>- 1 harness box<br>- 1 long OBD-C cable<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Volkswagen&model=Tiguan 2018-24">Buy Here</a></sub></details>||
|Volkswagen|Tiguan eHybrid 2021-23|Adaptive Cruise Control (ACC) & Lane Assist|openpilot available[<sup>1,12</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 USB-C coupler<br>- 1 VW J533 connector<br>- 1 comma 3X<br>- 1 harness box<br>- 1 long OBD-C cable<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Volkswagen&model=Tiguan eHybrid 2021-23">Buy Here</a></sub></details>||
|Volkswagen|Touran 2016-23|Adaptive Cruise Control (ACC) & Lane Assist|openpilot available[<sup>1,12</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 USB-C coupler<br>- 1 VW J533 connector<br>- 1 comma 3X<br>- 1 harness box<br>- 1 long OBD-C cable<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Volkswagen&model=Touran 2016-23">Buy Here</a></sub></details>||

View File

@@ -1,44 +0,0 @@
[data-tooltip] {
position: relative;
display: inline-block;
border-bottom: 1px dotted black;
}
[data-tooltip] .tooltip-content {
width: max-content;
max-width: 25em;
position: absolute;
top: 100%;
left: 50%;
transform: translateX(-50%);
background-color: white;
color: #404040;
box-shadow: 0 4px 14px 0 rgba(0,0,0,.2), 0 0 0 1px rgba(0,0,0,.05);
padding: 10px;
font: 14px/1.5 Lato, proxima-nova, Helvetica Neue, Arial, sans-serif;
text-decoration: none;
opacity: 0;
visibility: hidden;
transition: opacity 0.1s, visibility 0s;
z-index: 1000;
pointer-events: none; /* Prevent accidental interaction */
}
[data-tooltip]:hover .tooltip-content {
opacity: 1;
visibility: visible;
pointer-events: auto; /* Allow interaction when visible */
}
.tooltip-content .tooltip-glossary-link {
display: inline-block;
margin-top: 8px;
font-size: 12px;
color: #007bff;
text-decoration: none;
}
.tooltip-content .tooltip-glossary-link:hover {
color: #0056b3;
text-decoration: underline;
}

View File

View File

@@ -1,68 +0,0 @@
import re
import tomllib
def load_glossary(file_path="docs/glossary.toml"):
with open(file_path, "rb") as f:
glossary_data = tomllib.load(f)
return glossary_data.get("glossary", {})
def generate_anchor_id(name):
return name.replace(" ", "-").replace("_", "-").lower()
def format_markdown_term(name, definition):
anchor_id = generate_anchor_id(name)
markdown = f"* [**{name.replace('_', ' ').title()}**](#{anchor_id})"
if definition.get("abbreviation"):
markdown += f" *({definition['abbreviation']})*"
if definition.get("description"):
markdown += f": {definition['description']}\n"
return markdown
def glossary_markdown(vocabulary):
markdown = ""
for category, terms in vocabulary.items():
markdown += f"## {category.replace('_', ' ').title()}\n\n"
for name, definition in terms.items():
markdown += format_markdown_term(name, definition)
return markdown
def format_tooltip_html(term_key, definition, html):
display_term = term_key.replace("_", " ").title()
clean_description = re.sub(r"\[(.+)]\(.+\)", r"\1", definition["description"])
glossary_link = (
f"<a href='/concepts/glossary#{term_key}' class='tooltip-glossary-link' title='View in glossary'>Glossary🔗</a>"
)
return re.sub(
re.escape(display_term),
lambda
match: f"<span data-tooltip>{match.group(0)}<span class='tooltip-content'>{clean_description} {glossary_link}</span></span>",
html,
flags=re.IGNORECASE,
)
def apply_tooltip(_term_key, _definition, pattern, html):
return re.sub(
pattern,
lambda match: format_tooltip_html(_term_key, _definition, match.group(0)),
html,
flags=re.IGNORECASE,
)
def tooltip_html(vocabulary, html):
for _category, terms in vocabulary.items():
for term_key, definition in terms.items():
if definition.get("description"):
pattern = rf"(?<!\w){re.escape(term_key.replace('_', ' ').title())}(?![^<]*<\/a>)(?!\([^)]*\))"
html = apply_tooltip(term_key, definition, pattern, html)
return html
# Page Hooks
def on_page_markdown(markdown, **kwargs):
glossary = load_glossary()
return markdown.replace("{{GLOSSARY_DEFINITIONS}}", glossary_markdown(glossary))
def on_page_content(html, **kwargs):
if kwargs.get("page").title == "Glossary":
return html
glossary = load_glossary()
return tooltip_html(glossary, html)

View File

@@ -7,7 +7,7 @@ export OPENBLAS_NUM_THREADS=1
export VECLIB_MAXIMUM_THREADS=1
if [ -z "$AGNOS_VERSION" ]; then
export AGNOS_VERSION="11.4"
export AGNOS_VERSION="11.3"
fi
export STAGING_ROOT="/data/safe_staging"

View File

@@ -8,10 +8,6 @@ strict: true
docs_dir: docs
site_dir: docs_site/
hooks:
- docs/hooks/glossary.py
extra_css:
- css/tooltip.css
theme:
name: readthedocs
navigation_depth: 3

View File

@@ -1 +0,0 @@
../sunnypilot

2
panda

Submodule panda updated: 781af8b4f1...c7cc2deaf0

View File

@@ -1,6 +1,6 @@
[project]
name = "openpilot"
requires-python = ">= 3.11, < 3.13"
requires-python = ">= 3.11, <= 3.12"
license = {text = "MIT License"}
version = "0.1.0"
description = "an open source driver assistance system"
@@ -42,7 +42,8 @@ dependencies = [
# modeld
"onnx >= 1.14.0",
"onnxruntime >=1.16.3",
"onnxruntime >=1.16.3; platform_system == 'Linux' and platform_machine == 'aarch64'",
"onnxruntime-gpu >=1.16.3; platform_system == 'Linux' and platform_machine == 'x86_64'",
# logging
"pyzmq",
@@ -103,6 +104,7 @@ dev = [
"lru-dict",
"matplotlib",
"parameterized >=0.8, <0.9",
#"pprofile",
"pyautogui",
"pyopencl; platform_machine != 'aarch64'", # broken on arm64
"pytools < 2024.1.11; platform_machine != 'aarch64'", # pyopencl use a broken version
@@ -111,10 +113,13 @@ dev = [
"tabulate",
"types-requests",
"types-tabulate",
# this is only pinned since 5.15.11 is broken
"pyqt5 ==5.15.2; platform_machine == 'x86_64'", # no aarch64 wheels for macOS/linux
]
tools = [
"metadrive-simulator @ https://github.com/commaai/metadrive/releases/download/MetaDrive-minimal-0.4.2.4/metadrive_simulator-0.4.2.4-py3-none-any.whl ; (platform_machine != 'aarch64')",
"metadrive-simulator @ https://github.com/commaai/metadrive/releases/download/MetaDrive-minimal/metadrive_simulator-0.4.2.3-py3-none-any.whl ; (platform_machine != 'aarch64')",
"rerun-sdk >= 0.18",
]
@@ -159,7 +164,6 @@ testpaths = [
"tools/replay",
"tools/cabana",
"cereal/messaging/tests",
"sunnypilot",
]
[tool.codespell]

View File

@@ -1,147 +0,0 @@
#!/usr/bin/env bash
set -e
# Default values
DEFAULT_REPO_URL="https://github.com/sunnypilot"
START_AT_BOOT=false
# Parse command line arguments
while [[ $# -gt 0 ]]; do
case $1 in
--start-at-boot)
START_AT_BOOT=true
shift
;;
--token)
GITHUB_TOKEN="$2"
shift 2
;;
--repo)
REPO_URL="$2"
shift 2
;;
*)
if [ -z "$GITHUB_TOKEN" ]; then
GITHUB_TOKEN="$1"
elif [ -z "$REPO_URL" ]; then
REPO_URL="$1"
fi
shift
;;
esac
done
# Check required arguments
if [ -z "$GITHUB_TOKEN" ]; then
echo "Usage: $0 [--start-at-boot] [--token <github_token>] [--repo <repository_url>]"
echo "Required argument: github_token"
echo "Optional arguments:"
echo " --start-at-boot Enable auto-start at boot (default: false)"
echo " --repo Repository URL (default: ${DEFAULT_REPO_URL})"
exit 1
fi
# Set repository URL if not provided
REPO_URL="${REPO_URL:-$DEFAULT_REPO_URL}"
# Determine BASE_DIR based on mount point
if mountpoint -q /data/media; then
BASE_DIR="/data/media/0/github"
else
BASE_DIR="/data/github"
fi
# Constants
RUNNER_USER="github-runner"
USER_GROUPS="comma,gpu,gpio,sudo"
RUNNER_DIR="${BASE_DIR}/runner"
BUILDS_DIR="${BASE_DIR}/builds"
LOGS_DIR="${BASE_DIR}/logs"
CACHE_DIR="${BASE_DIR}/cache"
OPENPILOT_DIR="${BASE_DIR}/openpilot"
create_directories() {
sudo mkdir -p "$RUNNER_DIR" "$BUILDS_DIR" "$LOGS_DIR" "$CACHE_DIR" "$OPENPILOT_DIR"
mkdir -p "/data/openpilot"
sudo chown -R comma:comma "/data/openpilot"
}
download_and_setup_runner() {
cd "$RUNNER_DIR"
curl -o actions-runner-linux-arm64-2.321.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.321.0/actions-runner-linux-arm64-2.321.0.tar.gz
tar xzf ./actions-runner-linux-arm64-2.321.0.tar.gz
rm ./actions-runner-linux-arm64-2.321.0.tar.gz
chmod +x ./config.sh
}
setup_runner_user() {
sudo useradd --comment 'GitHub Runner' --create-home --home-dir ${BASE_DIR} ${RUNNER_USER} --shell /bin/bash -G ${USER_GROUPS} || sudo usermod -aG ${USER_GROUPS} ${RUNNER_USER}
export BASE_DIR
sudo -u ${RUNNER_USER} bash -c "truncate -s 0 '${BASE_DIR}/.bash_logout'"
}
create_sudoers_entry() {
sudo grep -qxF "${RUNNER_USER} ALL=(ALL) NOPASSWD: ALL" /etc/sudoers || echo "${RUNNER_USER} ALL=(ALL) NOPASSWD: ALL" | sudo tee -a /etc/sudoers
}
configure_runner() {
cd "$RUNNER_DIR"
sudo -u ${RUNNER_USER} ./config.sh --url "$REPO_URL" --token "$GITHUB_TOKEN" --name $(hostname) --runnergroup "tici-tizi" --labels "tici" --work "$BUILDS_DIR" --unattended
}
set_directory_permissions() {
sudo chown -R ${RUNNER_USER}:comma "$BASE_DIR"
sudo chmod g+rwx "$BASE_DIR"
sudo chmod g+s "$BASE_DIR"
}
modify_service_template() {
cat <<EOL > "$RUNNER_DIR/bin/actions.runner.service.template"
[Unit]
Description={{Description}}
After=network-online.target nss-lookup.target time-sync.target
Wants=network-online.target nss-lookup.target time-sync.target
StartLimitInterval=5
StartLimitBurst=10
[Service]
Type=simple
User=root
ExecStart=/usr/bin/unshare -m -- /bin/bash -c 'mount --bind ${OPENPILOT_DIR} /data/openpilot && setpriv --reuid={{User}} --regid={{User}} --init-groups env HOME=${BASE_DIR} USER={{User}} LOGNAME={{User}} MAIL=/var/mail/{{User}} {{RunnerRoot}}/runsvc.sh'
WorkingDirectory={{RunnerRoot}}
KillMode=process
KillSignal=SIGTERM
TimeoutStopSec=5min
Restart=always
RestartSec=120
[Install]
WantedBy=multi-user.target
EOL
}
# Make filesystem writable
sudo mount -o remount,rw /
# Ensure filesystem is remounted as read-only on script exit
trap "sudo mount -o remount,ro /" EXIT
# Execute installation steps
setup_runner_user
create_sudoers_entry
create_directories
download_and_setup_runner
modify_service_template
configure_runner
set_directory_permissions
# Install and start service using built-in installer
cd "$RUNNER_DIR"
sudo ./svc.sh install $RUNNER_USER
# Handle auto-start configuration
if [ "$START_AT_BOOT" = false ]; then
sudo systemctl disable actions.runner.sunnypilot.$(uname -n)
fi
sudo ./svc.sh start

View File

@@ -1,78 +0,0 @@
#!/usr/bin/env bash
set -e
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
cd $DIR
# Take parameters as arguments
SOURCE_DIR=$1
OUTPUT_DIR=$2
DEV_BRANCH=$3
VERSION=$4
GIT_ORIGIN=$5
EXTRA_VERSION_IDENTIFIER=$6
# Check parameters
if [ -z "$SOURCE_DIR" ] || [ -z "$OUTPUT_DIR" ]; then
echo "Error: No source or output directory provided."
exit 1
fi
if [ -z "$DEV_BRANCH" ] || [ -z "$VERSION" ]; then
echo "Error: No dev branch or version provided."
exit 1
fi
if [ -z "$GIT_ORIGIN" ]; then
echo "Error: No GIT_ORIGIN provided"
exit 1
fi
# "Tagging"
echo "#define COMMA_VERSION \"$VERSION\"" > ${OUTPUT_DIR}/common/version.h
## set git identity
#source $DIR/identity.sh
#export GIT_SSH_COMMAND="ssh -i /data/gitkey"
echo "[-] Setting up repo T=$SECONDS"
cd $OUTPUT_DIR
git init
# set git username/password
#source /data/identity.sh
git rm -rf $OUTPUT_DIR/.git || true # Doing cleanup, but it might fail if the .git doesn't exist or not allowed to delete
git remote remove origin || true # ensure cleanup
git remote add origin $GIT_ORIGIN
#git push origin -d $DEV_BRANCH || true # Ensuring we delete the remote branch if it exists as we are wiping it out
git fetch origin $DEV_BRANCH || (git checkout -b $DEV_BRANCH && git commit --allow-empty -m "sunnypilot v$VERSION release" && git push -u origin $DEV_BRANCH)
echo "[-] committing version $VERSION T=$SECONDS"
git add -f .
git commit -a -m "sunnypilot v$VERSION release"
git branch --set-upstream-to=origin/$DEV_BRANCH
# include source commit hash and build date in commit
GIT_HASH=$(git --git-dir=$SOURCE_DIR/.git rev-parse HEAD)
DATETIME=$(date '+%Y-%m-%dT%H:%M:%S')
SP_VERSION=$(cat $SOURCE_DIR/common/version.h | awk -F\" '{print $2}')
# Add built files to git
git add -f .
if [ "$EXTRA_VERSION_IDENTIFIER" = "-release" ] || [ "$EXTRA_VERSION_IDENTIFIER" = "-staging" ]; then
export VERSION=${VERSION%"$EXTRA_VERSION_IDENTIFIER"}
git commit --amend -m "sunnypilot v$VERSION"
else
git commit --amend -m "sunnypilot v$VERSION
version: sunnypilot v$SP_VERSION release
date: $DATETIME
master commit: $GIT_HASH
"
fi
git branch -m $DEV_BRANCH
# Push!
echo "[-] pushing T=$SECONDS"
git push -f origin $DEV_BRANCH

View File

@@ -1,361 +0,0 @@
#!/usr/bin/env python3
import argparse
import subprocess
import sys
import shutil
import signal
import contextlib
import tempfile
import os
def run_command(command: str) -> tuple[int, str, str]:
"""Run a shell command and return exit code, stdout, and stderr."""
process = subprocess.Popen(
command,
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True
)
stdout, stderr = process.communicate()
return process.returncode, stdout.strip(), stderr.strip()
def is_gh_available() -> bool:
"""Check if GitHub CLI is available."""
return shutil.which('gh') is not None
def get_current_branch() -> str | None:
"""Get the name of the current git branch."""
code, output, error = run_command("git rev-parse --abbrev-ref HEAD")
if code != 0:
print(f"Error getting current branch: {error}")
return None
return output
def backup_branch(branch_name: str) -> bool:
"""Create a backup of the current branch."""
backup_name = f"{branch_name}-backup-$(date +%Y%m%d_%H%M%S)"
code, _, error = run_command(f"git branch {backup_name}")
if code != 0:
print(f"Error creating backup branch: {error}")
return False
print(f"Created backup branch: {backup_name}")
return True
def get_commit_messages(source_branch: str, target_branch: str) -> list[str] | None:
"""Get all commit messages between source and target branches."""
code, output, error = run_command(f"git log {target_branch}..{source_branch} --format=%B")
if code != 0:
print(f"Error getting commit messages: {error}")
return None
return [msg.strip() for msg in output.splitlines() if msg and not msg.startswith('Merge')]
def get_pr_info(branch_name: str) -> str | None:
"""Get PR title using GitHub CLI."""
if not is_gh_available():
print("Warning: GitHub CLI not found. Install it to auto-fetch PR titles:")
print(" https://cli.github.com/")
return None
# Try to get PR info using gh cli
code, output, error = run_command(f"gh pr view --json title --jq .title {branch_name}")
if code != 0:
print(f"No open PR found for branch '{branch_name}'")
return None
return output
def create_squash_message(pr_title: str | None, commit_messages: list[str], source_branch: str) -> str:
"""Create a squash commit message from PR title and commit messages."""
parts = []
# Add PR title if provided
if pr_title:
parts.append(pr_title)
else:
parts.append(f"Squashed changes from {source_branch}")
parts.append("") # Empty line after title
# Add original commits section
if commit_messages:
parts.append("Original commits:")
parts.append("") # Empty line before list
parts.extend(f"* {msg}" for msg in commit_messages)
return '\n'.join(parts)
def prompt_for_title() -> str:
"""Prompt user for a commit title."""
return input("Enter commit title (or press Enter to use default): ").strip()
@contextlib.contextmanager
def workspace_manager(original_branch: str):
"""Context manager to handle workspace state and cleanup."""
stash_created = False
stash_restored = False
temp_branch: str | None = None
def cleanup_handler(signum=None, frame=None):
"""Clean up workspace state."""
nonlocal temp_branch, stash_created, stash_restored
try:
if signum and stash_restored:
# If we're handling Ctrl+C but stash was already restored,
# just clean up branches and exit
current = get_current_branch()
if current and current != original_branch:
run_command(f"git checkout {original_branch}")
if temp_branch:
run_command(f"git branch -D {temp_branch}")
print("\nOperation interrupted, but changes were already restored.")
sys.exit(1)
# First, switch back to original branch
current = get_current_branch()
if current and current != original_branch:
run_command(f"git checkout {original_branch}")
# Then clean up temp branch
if temp_branch:
run_command(f"git branch -D {temp_branch}")
# Finally, restore stash if needed - AFTER switching branches
if stash_created and not stash_restored:
print("Restoring your uncommitted changes...")
code, stash_list, _ = run_command("git stash list")
if code == 0 and "Automatic stash by squash script" in stash_list:
run_command("git stash pop")
stash_restored = True
stash_created = False
if signum:
print("\nOperation interrupted. Cleaned up and restored original state.")
sys.exit(1)
except Exception as e:
print(f"Error during cleanup: {e}")
if signum:
sys.exit(1)
try:
# Set up signal handlers
signal.signal(signal.SIGINT, cleanup_handler)
signal.signal(signal.SIGTERM, cleanup_handler)
# Check for changes (including untracked files)
code, output, _ = run_command("git status --porcelain")
if output:
print("Stashing uncommitted changes...")
run_command("git stash push -u -m 'Automatic stash by squash script'")
stash_created = True
yield lambda x: setattr(x, 'temp_branch', temp_branch)
except Exception as e:
print(f"\nError occurred: {str(e)}")
cleanup_handler()
raise
finally:
cleanup_handler()
def create_commit_with_message(message: str) -> bool:
"""Create a commit with the given message using a temporary file."""
try:
with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
f.write(message)
temp_path = f.name
# Use the temporary file for the commit message
code, _, error = run_command(f"git commit -F {temp_path}")
os.unlink(temp_path) # Clean up the temp file
if code != 0:
print(f"Error creating commit: {error}")
return False
return True
except Exception as e:
print(f"Error handling commit message: {e}")
if os.path.exists(temp_path):
os.unlink(temp_path)
return False
def squash_and_merge(source_branch: str, target_branch: str, manual_title: str | None, backup: bool = False, push: bool = False) -> bool:
"""
Squash the source branch and merge into target branch.
"""
# Get original branch right away
original_branch = get_current_branch()
if not original_branch:
return False
class State:
temp_branch: str | None = None
state = State()
with workspace_manager(original_branch) as set_temp_branch:
# Validate source branch exists
code, _, error = run_command(f"git rev-parse --verify {source_branch}")
if code != 0:
print(f"Error: Source branch {source_branch} not found")
return False
if source_branch == target_branch:
print(f"Error: Source and target branches cannot be the same ({source_branch})")
return False
# Ensure target branch exists
code, _, error = run_command(f"git rev-parse --verify {target_branch}")
if code != 0:
print(f"Error: Target branch {target_branch} not found")
return False
# Find merge base
code, merge_base, error = run_command(f"git merge-base {target_branch} {source_branch}")
if code != 0:
print(f"Error finding merge base: {error}")
return False
# Create backup unless explicitly skipped
if backup and not backup_branch(source_branch):
return False
# Get commit messages
commit_messages = get_commit_messages(source_branch, target_branch)
if commit_messages is None:
return False
# Get title (priority: manual title > PR title > prompt user)
title = manual_title
if not title:
title = get_pr_info(source_branch)
if not title:
title = prompt_for_title()
try:
# Create and switch to temporary branch
temp_branch = f"temp-squash-{source_branch}"
state.temp_branch = temp_branch
set_temp_branch(state)
print(f"\nCreating temporary branch {temp_branch}...")
code, _, error = run_command(f"git checkout -b {temp_branch} {source_branch}")
if code != 0:
print(f"Error creating temp branch: {error}")
return False
print("Preparing squash by resetting temporary branch to merge base...")
code, _, error = run_command(f"git reset --soft {merge_base}")
if code != 0:
print(f"Error resetting for squash: {error}")
return False
# Create commit with message
print("Creating squash commit...")
squash_message = create_squash_message(title, commit_messages, source_branch)
if not create_commit_with_message(squash_message):
return False
# Switch to target and try merge
print(f"\nSwitching to target branch {target_branch}...")
code, _, error = run_command(f"git checkout {target_branch}")
if code != 0:
print(f"Error checking out target branch: {error}")
return False
print(f"Attempting to merge changes from {temp_branch}...")
code, _, error = run_command(f"git merge {temp_branch}")
if code != 0:
print(f"\nMerge failed with error: {error}")
print("\nThe squash was successful, and your changes are preserved in the temporary branch.")
print("To complete the merge manually, follow these steps:")
print(f"\n1. Your squashed changes are in branch: '{temp_branch}'")
print(f"2. The target branch is: '{target_branch}'")
print("\nTo resolve the conflicts:")
print(f" git checkout {target_branch}")
print(f" git merge {temp_branch}")
print(" # resolve conflicts in your editor")
print(" git add <resolved-files>")
print(" git commit")
print(f" git push origin {target_branch} # when ready to push")
print("\nTo clean up after successful merge:")
print(f" git branch -D {temp_branch}")
# Make sure to abort the merge
print("\nAborting current merge attempt...")
run_command("git merge --abort")
# Return to original branch, but keep temp branch
print(f"Returning to {original_branch}...")
run_command(f"git checkout {original_branch}")
return False
# Clean up temp branch on success
run_command(f"git branch -D {temp_branch}")
# Push if requested
if push:
code, _, error = run_command(f"git push origin {target_branch}")
if code != 0:
print(f"Error pushing to {target_branch}: {error}")
return False
print(f"Successfully pushed to {target_branch}")
else:
print(f"Changes squashed and merged into {target_branch} locally")
print(f"To push the changes: git push origin {target_branch}")
# Return to original branch
code, _, error = run_command(f"git checkout {original_branch}")
if code != 0:
print(f"Warning: Failed to return to original branch: {error}")
return False
return True
except Exception as e:
print(f"Error during squash process: {e}")
return False
def main():
parser = argparse.ArgumentParser(
description='Squash branch and merge into target branch'
)
parser.add_argument('--target', '-t', required=True,
help='Target branch to merge changes into')
parser.add_argument('--source', '-s',
help='Source branch to squash (default: current branch)')
parser.add_argument('--title', '-m',
help='Optional manual title (overrides PR title)')
parser.add_argument('--backup', action='store_true',
help='Creates a backup branch for the source branch')
parser.add_argument('--push', action='store_true',
help='Push changes to remote after squashing')
args = parser.parse_args()
# Determine source branch early
source_branch = args.source
if not source_branch:
source_branch = get_current_branch()
if not source_branch:
sys.exit(1)
if not squash_and_merge(source_branch, args.target, args.title, args.backup, args.push):
sys.exit(1)
if __name__ == "__main__":
main()

View File

@@ -1,66 +0,0 @@
#!/usr/bin/env bash
# Determine BASE_DIR based on mount point
if mountpoint -q /data/media; then
GITHUB_BASE_DIR="/data/media/0/github"
else
GITHUB_BASE_DIR="/data/github"
fi
# Define directories and user
BIN_DIR="$GITHUB_BASE_DIR/bin"
BUILDS_DIR="$GITHUB_BASE_DIR/builds"
OPENPILOT_DIR="$GITHUB_BASE_DIR/openpilot"
LOGS_DIR="$GITHUB_BASE_DIR/logs"
CACHE_DIR="$GITHUB_BASE_DIR/cache"
RUNNER_USERNAME="github-runner"
# Define the systemd service name
SERVICE_NAME="github-runner"
USER_GROUPS="comma,gpu,gpio,sudo"
# Function to stop and disable the systemd service
stop_and_uninstall_service() {
cd $GITHUB_BASE_DIR/runner
sudo ./svc.sh stop
sudo ./svc.sh uninstall
}
# Function to remove the systemd service file
remove_runner() {
cd $GITHUB_BASE_DIR/runner
sudo rm .runner
sudo su -c './config.sh remove' github-runner
}
# Function to delete the Github Runner directories
delete_directories() {
sudo rm -rf "$BIN_DIR/github-runner"
sudo rm -rf "$GITHUB_BASE_DIR" "$BIN_DIR" "$BUILDS_DIR" "$LOGS_DIR" "$CACHE_DIR" "$OPENPILOT_DIR"
}
# Function to remove the Github Runner user
delete_user() {
for group in ${USER_GROUPS//,/ }
do
sudo gpasswd -d ${RUNNER_USERNAME} ${group}
done
sudo userdel -r ${RUNNER_USERNAME}
}
# Function to remove sudoers entry
remove_sudoers_entry() {
sudo sed -i.bak "/${RUNNER_USERNAME} ALL=(ALL) NOPASSWD: ALL/d" /etc/sudoers
}
# Make filesystem writable
sudo mount -o remount rw /
# Ensure filesystem is remounted as read-only on script exit
trap "sudo mount -o remount ro /" EXIT
# Call functions
stop_and_uninstall_service
remove_runner
delete_directories
delete_user
remove_sudoers_entry
# End of uninstall script

View File

@@ -32,7 +32,6 @@ blacklist = [
".git/",
".github/",
".devcontainer/",
"Darwin/",
".vscode",
@@ -48,42 +47,6 @@ blacklist = [
".gitmodules",
]
# Sunnypilot blacklist
sunnypilot_blacklist = [
"system/loggerd/sunnylink_uploader.py", # Temporarily, until we are ready to roll it out widely
".idea/",
".run/",
".*__pycache__/.*",
".*\\.pyc",
"teleoprtc/*",
"third_party/snpe/x86_64/*",
"body/board/canloader.py",
"body/board/flash_base.sh",
"body/board/flash_knee.sh",
"body/board/recover.sh",
".*/test/",
".*/tests/",
".*tinygrad_repo/tinygrad/renderer/",
"README.md",
".*internal/",
"docs/.*",
".sconsign.dblite",
"release/ci/scons_cache/",
".gitlab-ci.yml",
".clang-tidy",
".dockerignore",
".editorconfig",
".python-version",
"SECURITY.md",
"codecov.yml",
"conftest.py",
"poetry.lock",
".venv/",
]
# Merge the blacklists
blacklist += sunnypilot_blacklist
# gets you through the blacklist
whitelist = [
"tools/lib/",
@@ -91,7 +54,7 @@ whitelist = [
"tools/joystick/",
"tools/longitudinal_maneuvers/",
"tinygrad_repo/examples/openpilot/compile3.py",
"tinygrad_repo/openpilot/compile2.py",
"tinygrad_repo/extra/onnx.py",
"tinygrad_repo/extra/onnx_ops.py",
"tinygrad_repo/extra/thneed.py",
@@ -156,45 +119,8 @@ whitelist = [
"opendbc_repo/dbc/toyota_tss2_adas.dbc",
"opendbc_repo/dbc/vw_golf_mk4.dbc",
"opendbc_repo/dbc/vw_mqb_2010.dbc",
"opendbc_repo/dbc/tesla_can.dbc",
"opendbc_repo/dbc/tesla_radar_bosch_generated.dbc",
"opendbc_repo/dbc/tesla_radar_continental_generated.dbc",
"opendbc_repo/dbc/tesla_powertrain.dbc",
]
# Sunnypilot whitelist
sunnypilot_whitelist = [
"^README.md",
".*selfdrive/test/fuzzy_generation.py",
".*selfdrive/test/helpers.py",
".*selfdrive/test/__init__.py",
".*selfdrive/test/setup_device_ci.sh",
".*selfdrive/test/test_time_to_onroad.py",
".*selfdrive/test/test_onroad.py",
".*system/manager/test/test_manager.py",
".*system/manager/test/__init__.py",
".*system/qcomgpsd/tests/test_qcomgpsd.py",
".*system/updated/casync/tests/test_casync.py",
".*system/updated/tests/test_git.py",
".*system/updated/tests/test_base.py",
".*selfdrive/ui/tests/test_translations.py",
".*selfdrive/car/tests/__init__.py",
".*selfdrive/car/tests/test_car_interfaces.py",
".*selfdrive/navd/tests/test_navd.py",
".*selfdrive/navd/tests/test_map_renderer.py",
".*selfdrive/boardd/tests/test_boardd_loopback.py",
".*INTEGRATION.md",
".*HOW-TOS.md",
".*CARS.md",
".*LIMITATIONS.md",
".*CONTRIBUTING.md",
".*sunnyhaibin0850_qrcode_paypal.me.png",
"opendbc/.*.dbc",
]
# Merge the whitelists
whitelist += sunnypilot_whitelist
if __name__ == "__main__":
for f in Path(ROOT).rglob("**/*"):

39
scripts/build_small.sh Executable file
View File

@@ -0,0 +1,39 @@
#!/usr/bin/env bash
set -ex
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
cd $DIR
# git clone --mirror
SRC=/tmp/openpilot.git/
OUT=/tmp/smallpilot/
echo "starting size $(du -hs .git/)"
rm -rf $OUT
cd $SRC
git remote update
# copy contents
#rsync -a --exclude='.git/' $DIR $OUT
cp -r $SRC $OUT
cd $OUT
# remove all tags
git tag -l | xargs git tag -d
# remove non-master branches
BRANCHES="release2 release3 devel master-ci nightly"
for branch in $BRANCHES; do
git branch -D $branch
git branch -D ${branch}-staging || true
done
#git gc
git reflog expire --expire=now --all
git gc --prune=now
git gc --aggressive --prune=now
echo "new one is $(du -hs .)"

54
scripts/code_stats.py Executable file
View File

@@ -0,0 +1,54 @@
#!/usr/bin/env python3
import os
import ast
import stat
import subprocess
fouts = {x.decode('utf-8') for x in subprocess.check_output(['git', 'ls-files']).strip().split()}
pyf = []
for d in ["cereal", "common", "scripts", "selfdrive", "tools"]:
for root, _, files in os.walk(d):
for f in files:
if f.endswith(".py"):
pyf.append(os.path.join(root, f))
imps: set[str] = set()
class Analyzer(ast.NodeVisitor):
def visit_Import(self, node):
for alias in node.names:
imps.add(alias.name)
self.generic_visit(node)
def visit_ImportFrom(self, node):
imps.add(node.module)
self.generic_visit(node)
tlns = 0
carlns = 0
scriptlns = 0
testlns = 0
for f in sorted(pyf):
if f not in fouts:
continue
xbit = bool(os.stat(f)[stat.ST_MODE] & stat.S_IXUSR)
src = open(f).read()
lns = len(src.split("\n"))
tree = ast.parse(src)
Analyzer().visit(tree)
print(f"{lns:5d} {f} {xbit}")
if 'test' in f:
testlns += lns
elif f.startswith(('tools/', 'scripts/', 'selfdrive/debug')):
scriptlns += lns
elif f.startswith('selfdrive/car'):
carlns += lns
else:
tlns += lns
print(f"{tlns} lines of openpilot python")
print(f"{carlns} lines of car ports")
print(f"{scriptlns} lines of tools/scripts/debug")
print(f"{testlns} lines of tests")
#print(sorted(list(imps)))

11
scripts/count_cars.py Executable file
View File

@@ -0,0 +1,11 @@
#!/usr/bin/env python3
from collections import Counter
from pprint import pprint
from opendbc.car.docs import get_all_car_docs
if __name__ == "__main__":
cars = get_all_car_docs()
make_count = Counter(l.make for l in cars)
print("\n", "*" * 20, len(cars), "total", "*" * 20, "\n")
pprint(make_count)

View File

@@ -0,0 +1,391 @@
#!/usr/bin/env bash
set -e
SRC=/tmp/openpilot/
SRC_CLONE=/tmp/openpilot-clone/
OUT=/tmp/openpilot-tiny/
REWRITE_IGNORE_BRANCHES=(
dashcam3
devel
master-ci
nightly
release2
release3
release3-staging
)
VALIDATE_IGNORE_FILES=(
".github/ISSUE_TEMPLATE/bug_report.md"
".github/pull_request_template.md"
)
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
cd $DIR
LOGS_DIR=$DIR/git-rewrite-$(date +"%Y-%m-%dT%H:%M:%S%z")
mkdir -p $LOGS_DIR
GIT_REWRITE_LOG=$LOGS_DIR/git-rewrite-log.txt
BRANCH_DIFF_LOG=$LOGS_DIR/branch-diff-log.txt
COMMIT_DIFF_LOG=$LOGS_DIR/commit-diff-log.txt
START_TIME=$(date +%s)
exec > >(while IFS= read -r line; do
CURRENT_TIME=$(date +%s)
ELAPSED_TIME=$((CURRENT_TIME - START_TIME))
echo "[${ELAPSED_TIME}s] $line"
done | tee -a "$GIT_REWRITE_LOG") 2>&1
# INSTALL git-filter-repo
if [ ! -f /tmp/git-filter-repo ]; then
echo "Installing git-filter-repo..."
curl -sSo /tmp/git-filter-repo https://raw.githubusercontent.com/newren/git-filter-repo/main/git-filter-repo
chmod +x /tmp/git-filter-repo
fi
# MIRROR openpilot
if [ ! -d $SRC ]; then
echo "Mirroring openpilot..."
git clone --mirror https://github.com/commaai/openpilot.git $SRC # 4.18 GiB (488034 objects)
cd $SRC
echo "Starting size $(du -sh .)"
git remote update
# the git-filter-repo analysis is bliss - can be found in the repo root/filter-repo/analysis
echo "Analyzing with git-filter-repo..."
/tmp/git-filter-repo --force --analyze
echo "Pushing to openpilot-archive..."
# push to archive repo - in smaller parts because the 2 GB push limit - https://docs.github.com/en/get-started/using-git/troubleshooting-the-2-gb-push-limit
ARCHIVE_REPO=git@github.com:commaai/openpilot-archive.git
git push --prune $ARCHIVE_REPO +refs/heads/master:refs/heads/master # push master first so it's the default branch (when openpilot-archive is an empty repo)
git push --prune $ARCHIVE_REPO +refs/heads/*:refs/heads/* # 956.39 MiB (110725 objects)
git push --prune $ARCHIVE_REPO +refs/tags/*:refs/tags/* # 1.75 GiB (21694 objects)
# git push --mirror $ARCHIVE_REPO || true # fails to push refs/pull/* (deny updating a hidden ref) for pull requests
# we fail and continue - more reading: https://stackoverflow.com/a/34266401/639708 and https://blog.plataformatec.com.br/2013/05/how-to-properly-mirror-a-git-repository/
fi
# REWRITE master and tags
if [ ! -d $SRC_CLONE ]; then
echo "Cloning $SRC..."
GIT_LFS_SKIP_SMUDGE=1 git clone $SRC $SRC_CLONE
cd $SRC_CLONE
echo "Checking out old history..."
git checkout tags/v0.7.1 > /dev/null 2>&1
# checkout as main, since we need master ref later
git checkout -b main
echo "Creating setup commits..."
# rm these so we don't get conflicts later
git rm -r cereal opendbc panda selfdrive/ui/ui > /dev/null
git commit -m "removed conflicting files" > /dev/null
# skip-smudge to get rid of some lfs errors that it can't find the reference of some lfs files
# we don't care about fetching/pushing lfs right now
git lfs install --skip-smudge --local
# squash initial setup commits
git cherry-pick -n -X theirs 6c33a5c..59b3d06 > /dev/null
git commit -m "switching to master" > /dev/null
# squash the two commits
git reset --soft HEAD~2
git commit -m "switching to master" -m "$(git log --reverse --format=%B 6c33a5c..59b3d06)" -m "removed conflicting files" > /dev/null
# get commits we want to cherry-pick
# will start with the next commit after #59b3d06 tools is local now
COMMITS=$(git rev-list --reverse 59b3d06..master)
# we need this for logging
TOTAL_COMMITS=$(echo $COMMITS | wc -w | xargs)
CURRENT_COMMIT_NUMBER=0
# empty this file
> commit-map.txt
echo "Rewriting master commits..."
for COMMIT in $COMMITS; do
CURRENT_COMMIT_NUMBER=$((CURRENT_COMMIT_NUMBER + 1))
# echo -ne "[$CURRENT_COMMIT_NUMBER/$TOTAL_COMMITS] Cherry-picking commit: $COMMIT"\\r
echo "[$CURRENT_COMMIT_NUMBER/$TOTAL_COMMITS] Cherry-picking commit: $COMMIT"
# set environment variables to preserve author/committer and dates
export GIT_AUTHOR_NAME=$(git show -s --format='%an' $COMMIT)
export GIT_AUTHOR_EMAIL=$(git show -s --format='%ae' $COMMIT)
export GIT_COMMITTER_NAME=$(git show -s --format='%cn' $COMMIT)
export GIT_COMMITTER_EMAIL=$(git show -s --format='%ce' $COMMIT)
export GIT_AUTHOR_DATE=$(git show -s --format='%ad' $COMMIT)
export GIT_COMMITTER_DATE=$(git show -s --format='%cd' $COMMIT)
# cherry-pick the commit
if ! GIT_OUTPUT=$(git cherry-pick -m 1 -X theirs $COMMIT 2>&1); then
# check if the failure is because of an empty commit
if [[ "$GIT_OUTPUT" == *"The previous cherry-pick is now empty"* ]]; then
echo "Empty commit detected. Skipping commit $COMMIT"
git cherry-pick --skip
# log it was empty to the mapping file
echo "$COMMIT EMPTY" >> commit-map.txt
else
# handle other errors or conflicts
echo "Cherry-pick failed. Handling error..."
echo "$GIT_OUTPUT"
exit 1
fi
else
# capture the new commit hash
NEW_COMMIT=$(git rev-parse HEAD)
# save the old and new commit hashes to the mapping file
echo "$COMMIT $NEW_COMMIT" >> commit-map.txt
# append the old commit ID to the commit message
git commit --amend -m "$(git log -1 --pretty=%B)" -m "Former-commit-id: $COMMIT" > /dev/null
fi
# prune every 3000 commits to avoid gc errors
if [ $((CURRENT_COMMIT_NUMBER % 3000)) -eq 0 ]; then
echo "Pruning repo..."
git gc
fi
done
echo "Rewriting tags..."
# remove all old tags
git tag -l | xargs git tag -d
# read each line from the tag-commit-map.txt
while IFS=' ' read -r TAG OLD_COMMIT; do
# search for the new commit in commit-map.txt corresponding to the old commit
NEW_COMMIT=$(grep "^$OLD_COMMIT " "commit-map.txt" | awk '{print $2}')
# check if this is a rebased commit
if [ -z "$NEW_COMMIT" ]; then
# if not, then just use old commit hash
NEW_COMMIT=$OLD_COMMIT
fi
echo "Rewriting tag $TAG from commit $NEW_COMMIT"
git tag -f "$TAG" "$NEW_COMMIT"
done < "$DIR/tag-commit-map.txt"
# uninstall lfs since we don't want to touch (push to) lfs right now
# git push will also push lfs, if we don't uninstall (--local so just for this repo)
git lfs uninstall --local
# force push new master
git push --force origin main:master
# force push new tags
git push --force --tags
fi
# REWRITE branches based on master
if [ ! -f "$SRC_CLONE/rewrite-branches-done" ]; then
cd $SRC_CLONE
> rewrite-branches-done
# empty file
> $BRANCH_DIFF_LOG
echo "Rewriting branches based on master..."
# will store raw diffs here, if exist
mkdir -p differences
# get a list of all branches except master and REWRITE_IGNORE_BRANCHES
BRANCHES=$(git branch -r | grep -v ' -> ' | sed 's/.*origin\///' | grep -v '^master$' | grep -v -f <(echo "${REWRITE_IGNORE_BRANCHES[*]}" | tr ' ' '\n'))
for BRANCH in $BRANCHES; do
# check if the branch is based on master history
MERGE_BASE=$(git merge-base master origin/$BRANCH) || true
if [ -n "$MERGE_BASE" ]; then
echo "Rewriting branch: $BRANCH"
# create a new branch based on the new master
NEW_MERGE_BASE=$(grep "^$MERGE_BASE " "commit-map.txt" | awk '{print $2}')
if [ -z "$NEW_MERGE_BASE" ]; then
echo "Error: could not find new merge base for branch $BRANCH" >> $BRANCH_DIFF_LOG
continue
fi
git checkout -b ${BRANCH}_new $NEW_MERGE_BASE
# get the range of commits unique to this branch
COMMITS=$(git rev-list --reverse $MERGE_BASE..origin/${BRANCH})
HAS_ERROR=0
# simple delimiter
echo "BRANCH ${BRANCH}" >> commit-map.txt
for COMMIT in $COMMITS; do
# set environment variables to preserve author/committer and dates
export GIT_AUTHOR_NAME=$(git show -s --format='%an' $COMMIT)
export GIT_AUTHOR_EMAIL=$(git show -s --format='%ae' $COMMIT)
export GIT_COMMITTER_NAME=$(git show -s --format='%cn' $COMMIT)
export GIT_COMMITTER_EMAIL=$(git show -s --format='%ce' $COMMIT)
export GIT_AUTHOR_DATE=$(git show -s --format='%ad' $COMMIT)
export GIT_COMMITTER_DATE=$(git show -s --format='%cd' $COMMIT)
# cherry-pick the commit
if ! GIT_OUTPUT=$(git cherry-pick -m 1 -X theirs $COMMIT 2>&1); then
# check if the failure is because of an empty commit
if [[ "$GIT_OUTPUT" == *"The previous cherry-pick is now empty"* ]]; then
echo "Empty commit detected. Skipping commit $COMMIT"
git cherry-pick --skip
# log it was empty to the mapping file
echo "$COMMIT EMPTY" >> commit-map.txt
else
# handle other errors or conflicts
echo "Cherry-pick of ${BRANCH} branch failed. Removing branch upstream..." >> $BRANCH_DIFF_LOG
echo "$GIT_OUTPUT" > "$LOGS_DIR/branch-${BRANCH}"
git cherry-pick --abort
git push --delete origin ${BRANCH}
HAS_ERROR=1
break
fi
else
# capture the new commit hash
NEW_COMMIT=$(git rev-parse HEAD)
# save the old and new commit hashes to the mapping file
echo "$COMMIT $NEW_COMMIT" >> commit-map.txt
# append the old commit ID to the commit message
git commit --amend -m "$(git log -1 --pretty=%B)" -m "Former-commit-id: $COMMIT" > /dev/null
fi
done
# force push the new branch
if [ $HAS_ERROR -eq 0 ]; then
# git lfs goes haywire here, so we need to install and uninstall
# git lfs install --skip-smudge --local
git lfs uninstall --local > /dev/null
git push -f origin ${BRANCH}_new:${BRANCH}
fi
# clean up local branch
git checkout master > /dev/null
git branch -D ${BRANCH}_new > /dev/null
else
echo "Deleting branch $BRANCH as it's not based on master history" >> $BRANCH_DIFF_LOG
git push --delete origin ${BRANCH}
fi
done
fi
# VALIDATE cherry-pick
if [ ! -f "$SRC_CLONE/validation-done" ]; then
cd $SRC_CLONE
> validation-done
TOTAL_COMMITS=$(grep -cve '^\s*$' commit-map.txt)
CURRENT_COMMIT_NUMBER=0
COUNT_SAME=0
COUNT_DIFF=0
# empty file
> $COMMIT_DIFF_LOG
echo "Validating commits..."
# will store raw diffs here, if exist
mkdir -p differences
# read each line from commit-map.txt
while IFS=' ' read -r OLD_COMMIT NEW_COMMIT; do
if [ "$NEW_COMMIT" == "EMPTY" ]; then
continue
fi
if [ "$OLD_COMMIT" == "BRANCH" ]; then
echo "Branch ${NEW_COMMIT} below:" >> $COMMIT_DIFF_LOG
continue
fi
CURRENT_COMMIT_NUMBER=$((CURRENT_COMMIT_NUMBER + 1))
# retrieve short hashes and dates for the old and new commits
OLD_COMMIT_SHORT=$(git rev-parse --short $OLD_COMMIT)
NEW_COMMIT_SHORT=$(git rev-parse --short $NEW_COMMIT)
OLD_DATE=$(git show -s --format='%cd' $OLD_COMMIT)
NEW_DATE=$(git show -s --format='%cd' $NEW_COMMIT)
# echo -ne "[$CURRENT_COMMIT_NUMBER/$TOTAL_COMMITS] Comparing old commit $OLD_COMMIT_SHORT ($OLD_DATE) with new commit $NEW_COMMIT_SHORT ($NEW_DATE)"\\r
echo "[$CURRENT_COMMIT_NUMBER/$TOTAL_COMMITS] Comparing old commit $OLD_COMMIT_SHORT ($OLD_DATE) with new commit $NEW_COMMIT_SHORT ($NEW_DATE)"
# generate lists of files and their hashes for the old and new commits, excluding ignored files
OLD_FILES=$(git ls-tree -r $OLD_COMMIT | grep -vE "$(IFS='|'; echo "${VALIDATE_IGNORE_FILES[*]}")")
NEW_FILES=$(git ls-tree -r $NEW_COMMIT | grep -vE "$(IFS='|'; echo "${VALIDATE_IGNORE_FILES[*]}")")
# Compare the diffs
if diff <(echo "$OLD_FILES") <(echo "$NEW_FILES") > /dev/null; then
# echo "Old commit $OLD_COMMIT_SHORT and new commit $NEW_COMMIT_SHORT are equivalent."
COUNT_SAME=$((COUNT_SAME + 1))
else
echo "[$CURRENT_COMMIT_NUMBER/$TOTAL_COMMITS] Difference found between old commit $OLD_COMMIT_SHORT and new commit $NEW_COMMIT_SHORT" >> $COMMIT_DIFF_LOG
COUNT_DIFF=$((COUNT_DIFF + 1))
set +e
diff -u <(echo "$OLD_FILES") <(echo "$NEW_FILES") > "$LOGS_DIR/commit-$CURRENT_COMMIT_NUMBER-$OLD_COMMIT_SHORT-$NEW_COMMIT_SHORT"
set -e
fi
done < "commit-map.txt"
echo "Summary:" >> $COMMIT_DIFF_LOG
echo "Equivalent commits: $COUNT_SAME" >> $COMMIT_DIFF_LOG
echo "Different commits: $COUNT_DIFF" >> $COMMIT_DIFF_LOG
fi
if [ ! -d $OUT ]; then
cp -r $SRC $OUT
cd $OUT
# remove all non-master branches
# git branch | grep -v "^ master$" | grep -v "\*" | xargs git branch -D
# echo "cleaning up refs"
# delete pull request refs since we can't alter them anyway (https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally#error-failed-to-push-some-refs)
# git for-each-ref --format='%(refname)' | grep '^refs/pull/' | xargs -I {} git update-ref -d {}
echo "importing new lfs files"
# import "almost" everything to lfs
BRANCHES=$(git for-each-ref --format='%(refname)' refs/heads/ | sed 's%refs/heads/%%g' | grep -v -f <(echo "${REWRITE_IGNORE_BRANCHES[*]}" | tr ' ' '\n') | tr '\n' ' ')
git lfs migrate import --include="*.dlc,*.onnx,*.svg,*.png,*.gif,*.ttf,*.wav,selfdrive/car/tests/test_models_segs.txt,system/hardware/tici/updater,selfdrive/ui/qt/spinner_larch64,selfdrive/ui/qt/text_larch64,third_party/**/*.a,third_party/**/*.so,third_party/**/*.so.*,third_party/**/*.dylib,third_party/acados/*/t_renderer,third_party/qt5/larch64/bin/lrelease,third_party/qt5/larch64/bin/lupdate,third_party/catch2/include/catch2/catch.hpp,*.apk,*.apkpatch,*.jar,*.pdf,*.jpg,*.mp3,*.thneed,*.tar.gz,*.npy,*.csv,*.a,*.so*,*.dylib,*.o,*.b64,selfdrive/hardware/tici/updater,selfdrive/boardd/tests/test_boardd,selfdrive/ui/qt/spinner_aarch64,installer/updater/updater,selfdrive/debug/profiling/simpleperf/**/*,selfdrive/hardware/eon/updater,selfdrive/ui/qt/text_aarch64,selfdrive/debug/profiling/pyflame/**/*,installer/installers/installer_openpilot,installer/installers/installer_dashcam,selfdrive/ui/text/text,selfdrive/ui/android/text/text,selfdrive/ui/spinner/spinner,selfdrive/visiond/visiond,selfdrive/loggerd/loggerd,selfdrive/sensord/sensord,selfdrive/sensord/gpsd,selfdrive/ui/android/spinner/spinner,selfdrive/ui/qt/spinner,selfdrive/ui/qt/text,_stringdefs.py,dfu-util-aarch64-linux,dfu-util-aarch64,dfu-util-x86_64-linux,dfu-util-x86_64,stb_image.h,clpeak3,clwaste,apk/**/*,external/**/*,phonelibs/**/*,third_party/boringssl/**/*,flask/**/*,panda/**/*,board/**/*,messaging/**/*,opendbc/**/*,tools/cabana/chartswidget.cc,third_party/nanovg/**/*,selfdrive/controls/lib/lateral_mpc/lib_mpc_export/**/*,selfdrive/ui/paint.cc,werkzeug/**/*,pyextra/**/*,third_party/android_hardware_libhardware/**/*,selfdrive/controls/lib/lead_mpc_lib/lib_mpc_export/**/*,selfdrive/locationd/laikad.py,selfdrive/locationd/test/test_laikad.py,tools/gpstest/test_laikad.py,selfdrive/locationd/laikad_helpers.py,tools/nui/**/*,jsonrpc/**/*,selfdrive/controls/lib/longitudinal_mpc/lib_mpc_export/**/*,selfdrive/controls/lib/lateral_mpc/mpc_export/**/*,selfdrive/camerad/cameras/camera_qcom.cc,selfdrive/manager.py,selfdrive/modeld/models/driving.cc,third_party/curl/**/*,selfdrive/modeld/thneed/debug/**/*,selfdrive/modeld/thneed/include/**/*,third_party/openmax/**/*,selfdrive/controls/lib/longitudinal_mpc/mpc_export/**/*,selfdrive/controls/lib/longitudinal_mpc_model/lib_mpc_export/**/*,Pipfile,Pipfile.lock,gunicorn/**/*,*.qm,jinja2/**/*,click/**/*,dbcs/**/*,websocket/**/*" $BRANCHES
echo "reflog and gc"
# this is needed after lfs import
git reflog expire --expire=now --all
git gc --prune=now --aggressive
# check the git-filter-repo analysis again - can be found in the repo root/filter-repo/analysis
echo "Analyzing with git-filter-repo..."
/tmp/git-filter-repo --force --analyze
echo "New size is $(du -sh .)"
fi
cd $OUT
# fetch all lfs files from https://github.com/commaai/openpilot.git
# some lfs files are missing on gitlab, but they can be found on github
git config lfs.url https://github.com/commaai/openpilot.git/info/lfs
git config lfs.pushurl ssh://git@github.com/commaai/openpilot.git
git lfs fetch --all || true
# also fetch all lfs files from https://gitlab.com/commaai/openpilot-lfs.git
git config lfs.url https://gitlab.com/commaai/openpilot-lfs.git/info/lfs
git config lfs.pushurl ssh://git@gitlab.com/commaai/openpilot-lfs.git
git lfs fetch --all || true
# final push - will also push lfs
# TODO: switch to git@github.com:commaai/openpilot.git when ready
# git push --mirror git@github.com:commaai/openpilot-tiny.git
# using this instead to ignore refs/pull/* - since this is also what --mirror does - https://blog.plataformatec.com.br/2013/05/how-to-properly-mirror-a-git-repository/
git push --prune git@github.com:commaai/openpilot-tiny.git +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/*

59
scripts/git_rewrite/rewrite.sh Executable file
View File

@@ -0,0 +1,59 @@
#!/usr/bin/env bash
set -e
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
cd $DIR
git clone --bare https://github.com/commaai/openpilot
cp -r openpilot.git openpilot_backup
cd openpilot.git
# backup old repo
git push git@github.com:commaai/openpilot-archive.git +refs/heads/master:refs/heads/master
git push git@github.com:commaai/openpilot-archive.git +refs/heads/*:refs/heads/*
git push git@github.com:commaai/openpilot-archive.git +refs/tags/*:refs/tags/*
git push --mirror git@github.com:commaai/openpilot-archive.git
# ignore all release branches
git for-each-ref --format='delete %(refname)' | grep 'dashcam3\|devel\|master-ci\|nightly\|release2\|release3\|release3-staging' | git update-ref --stdin
# re-tag old releases on master
declare -A TAGS=( ["f8cb04e4a8b032b72a909f68b808a50936184bee"]="v0.9.7" ["0b4d08fab8e35a264bc7383e878538f8083c33e5"]="v0.9.6" ["3b1e9017c560499786d8a0e46aaaeea65037acac"]="v0.9.5" ["fa310d9e2542cf497d92f007baec8fd751ffa99c"]="v0.9.4" ["8704c1ff952b5c85a44f50143bbd1a4f7b4887e2"]="v0.9.3" ["c7d3b28b93faa6c955fb24bc64031512ee985ee9"]="v0.9.2" ["89f68bf0cbf53a81b0553d3816fdbe522f941fa1"]="v0.9.1" ["58b84fb401a804967aa0dd5ee66fafa90194fd30"]="v0.9.0" ["f41dc62a12cc0f3cb8c5453c0caa0ba21e1bd01e"]="v0.8.16" ["5a7c2f90361e72e9c35e88abd2e11acdc4aba354"]="v0.8.15" ["71901c94dbbaa2f9f156a80c14cc7ea65219fc7c"]="v0.8.14" ["95da47079510afc91665263619e5939126da637c"]="v0.8.13" ["472177e2a8a1d002e56f9096326fd2dff62e54f9"]="v0.8.12" ["08078acbd0b4f7da469c7dff6159000e358974a9"]="v0.8.11" ["687925c775c375495f9827946138a724bde00b9d"]="v0.8.10" ["204e5a090735a059d69c29145a4cee49450da07e"]="v0.8.9" ["4be956f8861ecbb521ef9503a3c87b07c9d36721"]="v0.8.8" ["589f82c76627d634761a31a34b2488403556eb0b"]="v0.8.7" ["507cfc8910f74ddb8810039d68b880b426ff9ff9"]="v0.8.6" ["d47b00b45a866bef088f51d1ff31de5885ab04e9"]="v0.8.5" ["553e7d1cce314e7eb0587186b1764c3ff43bed62"]="v0.8.4" ["9896438d1511602a1ff87f7c4eb3c7172b30104a"]="v0.8.3" ["280192ed1443f112463417c2d815ea8ee2762fbd"]="v0.8.2" ["8039361567e4659eae2a084e6f39f34acadf4cac"]="v0.8.1" ["d56e04c0d960c8d3d4ab88b578dc508a2b4e07dc"]="v0.8" ["3d456e5d0fbf0c9887d0499dee812f2b029edf6d"]="v0.7.10" ["81763a18b5d0e379b749e090ecce36a91fca7c43"]="v0.7.9" ["9bc0b350fd273bbb2deb3dcaef0312944e4f6cfd"]="v0.7.8" ["ede5b632b58c55e4ff003f948efae07fe03c2280"]="v0.7.7" ["775acd11ba2e0a8c2f5a5655338718d796491b36"]="v0.7.6.1" ["302417b4cf0dcf00d45e4995b5410e543ad121d1"]="v0.7.5" ["12ff088b42221dd17d9d97decb1fc61a7cb0a861"]="v0.7.4" ["9563f7730252451fdcba9bc3d9fe36dab9c86a26"]="v0.7.3" ["8321cf283abbc2ca3fda7e0c7a069a77a492fe0c"]="v0.7.2" ["1e1de64a1e59476b7b3d3558b92149246d5c3292"]="v0.7.1" ["a2ae18d1dbd1e59c38ce22fa25ddffbd1d3084e3"]="v0.7" ["d4eb5a6eafdd4803d09e6f3963918216cca5a81f"]="v0.6.6" ["70d17cd69b80e7627dcad8fd5b6438f2309ac307"]="v0.6.5" ["58f376002e0c654fbc2de127765fa297cf694a33"]="v0.6.4" ["d5f9caa82d80cdcc7f1b7748f2cf3ccbf94f82a3"]="v0.6.3" ["095ef5f9f60fca1b269aabcc3cfd322b17b9e674"]="v0.6.2" ["cf5c4aeacb1703d0ffd35bdb5297d3494fee9a22"]="v0.6.1" ["60a20537c5f3fcc7f11946d81aebc8f90c08c117"]="v0.6" ["dd34ccfe288ebda8e2568cf550994ae890379f45"]="v0.5.13" ["3f9059fea886f1fa3b0c19a62a981d891dcc84eb"]="v0.5.12" ["2f92d577f995ff6ae1945ef6b89df3cb69b92999"]="v0.5.11" ["5a9d89ed42ddcd209d001a10d7eb828ef0e6d9de"]="v0.5.10" ["0207a970400ee28d3e366f2e8f5c551281accf02"]="v0.5.9" ["b967da5fc1f7a07e3561db072dd714d325e857b0"]="v0.5.8" ["210db686bb89f8696aa040e6e16de65424b808c9"]="v0.5.7" ["860a48765d1016ba226fb2c64aea35a45fe40e4a"]="v0.5.6" ["8f3539a27b28851153454eb737da9624cccaed2d"]="v0.5.5" ["a422246dc30bce11e970514f13f7c110f4470cc3"]="v0.5.4" ["285c52eb693265a0a530543e9ca0aeb593a2a55e"]="v0.5.3" ["0129a8a4ff8da5314e8e4d4d3336e89667ff6d54"]="v0.5.2" ["6f3d10a4c475c4c4509f0b370805419acd13912d"]="v0.5.1" ["de33bc46452b1046387ee2b3a03191b2c71135fb"]="v0.5" ["ae5cb7a0dab8b1bed9d52292f9b4e8e66a0f8ec9"]="v0.4.7" ["c6df34f55ba8c5a911b60d3f9eb20e3fa45f68c1"]="v0.4.6" ["37285038d3f91fa1b49159c4a35a8383168e644f"]="v0.4.5" ["9a9ff839a9b70cb2601d7696af743f5652395389"]="v0.4.4" ["28c0797d30175043bbfa31307b63aab4197cf996"]="v0.4.2" ["4474b9b3718653aeb0aee26422caefb90460cc0e"]="v0.4.1" ["da52d065a4c4f52d6017a537f3a80326f5af8bdc"]="v0.4.0.2" ["9d3963559ae7b15193057937ff3e72481899f40d"]="v0.3.5" ["1b8c44b5067525a5d266b6e99799d8097da76a29"]="v0.3.4" ["5cf91d0496688fed4f2a6c7021349b1fc0e057a2"]="v0.3.3" ["7fe46f1e1df5dec08a940451ba0feefd5c039165"]="v0.3.2" ["41e3a0f699f5c39cb61a15c0eb7a4aa816d47c24"]="v0.3.1" ["c5d8aec28b5230d34ae4b677c2091cc3dec7e3e8"]="v0.3.0" ["693bcb0f83478f2651db6bac9be5ca5ad60d03f3"]="v0.2.9" ["95a349abcc050712c50d4d85a1c8a804eee7f6c2"]="v0.2.8" ["c6ba5dc5391d3ca6cda479bf1923b88ce45509a0"]="v0.2.7" ["6c3afeec0fb439070b2912978b8dbb659033b1d9"]="v0.2.6" ["29c58b45882ac79595356caf98580c1d2a626011"]="v0.2.5" ["ecc565aa3fdc4c7e719aadc000e1fdc4d80d4fe0"]="v0.2.4" ["adaa4ed350acda4067fc0b455ad15b54cdf4c768"]="v0.2.3" ["a64b9aa9b8cb5863c917b6926516291a63c02fe5"]="v0.2.2" ["17d9becd3c673091b22f09aa02559a9ed9230f50"]="v0.2.1" ["449b482cc3236ccf31829830b4f6a44b2dcc06c2"]="v0.2" ["e94a30bec07e719c5a7b037ca1f4db8312702cce"]="v0.1" )
for tag in "${!TAGS[@]}"; do git tag -f "${TAGS[$tag]}" "$tag" ; done
# get master root commit
ROOT_COMMIT=$(git rev-list --max-parents=0 HEAD | tail -n 1)
# link master and devel
git replace --graft $ROOT_COMMIT v0.7.1
git-filter-repo --prune-empty never --force --commit-callback 'h=commit.original_id.decode("utf-8");m=commit.message.decode("utf-8");commit.message=str.encode(m + "\n" + "old-commit-hash: " + h)'
# delete replace refs
git for-each-ref --format='delete %(refname)' refs/replace | git update-ref --stdin
# machine validation
tail -n +2 "filter-repo/commit-map" | tr ' ' '\n' | xargs -P $(nproc) -n 2 bash -c 'H1=$(cd ../openpilot_backup && git ls-tree -r $0 | sha1sum) && H2=$(git ls-tree -r $1 | sha1sum) && echo "$H1 $H2" >> /tmp/GIT_HASHES && diff <(echo $H1) <(echo $H2) || exit 255'
# human validation
less /tmp/GIT_HASH
# cleanup
git reflog expire --expire=now --all
git gc --prune=now --aggressive
# get all lfs files
set +e
git config lfs.url https://github.com/commaai/openpilot.git/info/lfs
git lfs fetch --all
git config lfs.url https://gitlab.com/commaai/openpilot-lfs.git/info/lfs
git lfs fetch --all
set -e
# add new files to lfs
git lfs migrate import --everything --include="*.ico,*.dlc,*.onnx,*.svg,*.png,*.gif,*.ttf,*.wav,system/hardware/tici/updater,selfdrive/ui/qt/spinner_larch64,selfdrive/ui/qt/text_larch64,third_party/**/*.a,third_party/**/*.so,third_party/**/*.so.*,third_party/**/*.dylib,third_party/acados/*/t_renderer,third_party/qt5/larch64/bin/lrelease,third_party/qt5/larch64/bin/lupdate,third_party/catch2/include/catch2/catch.hpp,*.apk,*.apkpatch,*.jar,*.pdf,*.jpg,*.mp3,*.thneed,*.tar.gz,*.npy,*.csv,*.a,*.so*,*.dylib,*.o,*.b64,selfdrive/hardware/tici/updater,selfdrive/boardd/tests/test_boardd,selfdrive/ui/qt/spinner_aarch64,installer/updater/updater,selfdrive/debug/profiling/simpleperf/**/*,selfdrive/hardware/eon/updater,selfdrive/ui/qt/text_aarch64,selfdrive/debug/profiling/pyflame/**/*,installer/installers/installer_openpilot,installer/installers/installer_dashcam,selfdrive/ui/text/text,selfdrive/ui/android/text/text,selfdrive/ui/spinner/spinner,selfdrive/visiond/visiond,selfdrive/loggerd/loggerd,selfdrive/sensord/sensord,selfdrive/sensord/gpsd,selfdrive/ui/android/spinner/spinner,selfdrive/ui/qt/spinner,selfdrive/ui/qt/text,_stringdefs.py,dfu-util-aarch64-linux,dfu-util-aarch64,dfu-util-x86_64-linux,dfu-util-x86_64,stb_image.h,clpeak3,clwaste,apk/**/*,external/**/*,phonelibs/**/*,third_party/boringssl/**/*,pyextra/**/*,panda/board/**/inc/*.h,panda/board/obj/*.elf,board/inc/*.h,third_party/nanovg/**/*,selfdrive/controls/lib/lateral_mpc/lib_mpc_export/**/*,pyextra/**/*,third_party/android_hardware_libhardware/**/*,selfdrive/controls/lib/lead_mpc_lib/lib_mpc_export/**/*,*.pro,selfdrive/controls/lib/longitudinal_mpc/lib_mpc_export/**/*,selfdrive/controls/lib/lateral_mpc/mpc_export/**/*,third_party/curl/**/*,selfdrive/modeld/thneed/debug/**/*,selfdrive/modeld/thneed/include/**/*,third_party/openmax/**/*,selfdrive/controls/lib/longitudinal_mpc/mpc_export/**/*,selfdrive/controls/lib/longitudinal_mpc_model/lib_mpc_export/**/*,Pipfile,Pipfile.lock,poetry.lock,*.qm"
# set new lfs endpoint
git config lfs.url https://gitlab.com/commaai/openpilot-lfs.git/info/lfs
git config lfs.pushurl ssh://git@gitlab.com/commaai/openpilot-lfs.git
# push all branch+tag (scary stuff...)
git push -f --set-upstream git@github.com:commaai/openpilot.git +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/*

View File

@@ -0,0 +1,82 @@
v0.1 e94a30bec07e719c5a7b037ca1f4db8312702cce
v0.2 449b482cc3236ccf31829830b4f6a44b2dcc06c2
v0.2.1 17d9becd3c673091b22f09aa02559a9ed9230f50
v0.2.2 a64b9aa9b8cb5863c917b6926516291a63c02fe5
v0.2.3 adaa4ed350acda4067fc0b455ad15b54cdf4c768
v0.2.4 ecc565aa3fdc4c7e719aadc000e1fdc4d80d4fe0
v0.2.5 29c58b45882ac79595356caf98580c1d2a626011
v0.2.6 6c3afeec0fb439070b2912978b8dbb659033b1d9
v0.2.7 c6ba5dc5391d3ca6cda479bf1923b88ce45509a0
v0.2.8 95a349abcc050712c50d4d85a1c8a804eee7f6c2
v0.2.9 693bcb0f83478f2651db6bac9be5ca5ad60d03f3
v0.3.0 c5d8aec28b5230d34ae4b677c2091cc3dec7e3e8
v0.3.1 41e3a0f699f5c39cb61a15c0eb7a4aa816d47c24
v0.3.2 7fe46f1e1df5dec08a940451ba0feefd5c039165
v0.3.3 5cf91d0496688fed4f2a6c7021349b1fc0e057a2
v0.3.4 1b8c44b5067525a5d266b6e99799d8097da76a29
v0.3.5 b111277f464cf66fa34b67819a83ea683e0f64df
v0.4.0.2 da52d065a4c4f52d6017a537f3a80326f5af8bdc
v0.4.1 4474b9b3718653aeb0aee26422caefb90460cc0e
v0.4.2 28c0797d30175043bbfa31307b63aab4197cf996
v0.4.4 9a9ff839a9b70cb2601d7696af743f5652395389
v0.4.5 37285038d3f91fa1b49159c4a35a8383168e644f
v0.4.6 c6df34f55ba8c5a911b60d3f9eb20e3fa45f68c1
v0.4.7 ae5cb7a0dab8b1bed9d52292f9b4e8e66a0f8ec9
v0.5 de33bc46452b1046387ee2b3a03191b2c71135fb
v0.5.1 8f22f52235c48eada586795ac57edb22688e4d08
v0.5.2 0129a8a4ff8da5314e8e4d4d3336e89667ff6d54
v0.5.3 285c52eb693265a0a530543e9ca0aeb593a2a55e
v0.5.4 a422246dc30bce11e970514f13f7c110f4470cc3
v0.5.5 8f3539a27b28851153454eb737da9624cccaed2d
v0.5.6 860a48765d1016ba226fb2c64aea35a45fe40e4a
v0.5.7 9ce3045f139ee29bf0eea5ec59dfe7df9c3d2c51
v0.5.8 2cee2e05ba0f3824fdbb8b957958800fa99071a1
v0.5.9 ad145da3bcded0fe75306df02061d07a633963c3
v0.5.10 ff4c1557d8358f158f4358788ff18ef93d2470ef
v0.5.11 d1866845df423c6855e2b365ff230cf7d89a420b
v0.5.12 f6e8ef27546e9a406724841e75f8df71cc4c2c97
v0.5.13 dd34ccfe288ebda8e2568cf550994ae890379f45
v0.6 60a20537c5f3fcc7f11946d81aebc8f90c08c117
v0.6.1 cf5c4aeacb1703d0ffd35bdb5297d3494fee9a22
v0.6.2 095ef5f9f60fca1b269aabcc3cfd322b17b9e674
v0.6.3 d5f9caa82d80cdcc7f1b7748f2cf3ccbf94f82a3
v0.6.4 58f376002e0c654fbc2de127765fa297cf694a33
v0.6.5 70d17cd69b80e7627dcad8fd5b6438f2309ac307
v0.6.6 d4eb5a6eafdd4803d09e6f3963918216cca5a81f
v0.7 a2ae18d1dbd1e59c38ce22fa25ddffbd1d3084e3
v0.7.1 1e1de64a1e59476b7b3d3558b92149246d5c3292
v0.7.2 59bd58c940673b4c4a6a86f299022614bcf42b22
v0.7.3 d7acd8b68f8131e0e714400cf124a3e228638643
v0.7.4 e93649882c5e914eec4a8b8b593dc0587e497033
v0.7.5 8abc0afe464626a461d2c7e192c912eeebeccc65
v0.7.6 69aacd9d179fe6dd3110253a099c38b34cff7899
v0.7.7 f1caed7299cdba5e45635d8377da6cc1e5fd7072
v0.7.8 2189fe8741b635d8394d55dee28959425cfd5ad0
v0.7.9 86dc54b836a973f132ed26db9f5a60b29f9b25b2
v0.7.10 47a42ff432db8a2494e922ca5e767e58020f0446
v0.7.11 f46ed718ba8d6bb4d42cd7b0f0150c406017c373
v0.8 d56e04c0d960c8d3d4ab88b578dc508a2b4e07dc
v0.8.1 cd6f26664cb8d32a13847d6648567c47c580e248
v0.8.2 7cc0999aebfe63b6bb6dd83c1dff62c3915c4820
v0.8.3 986500fe2f10870018f1fba1e5465476b8915977
v0.8.4 f0d0b82b8d6f5f450952113e234d0a5a49e80c48
v0.8.5 f5d9ddc6c2a2802a61e5ce590c6b6688bf736a69
v0.8.6 75904ed7452c6cbfb2a70cd379a899d8a75b97c2
v0.8.7 4f9e568019492126e236da85b5ca0a059f292900
v0.8.8 a949a49d5efaaf2d881143d23e9fb5ff9e28e88c
v0.8.9 a034926264cd1025c69d6ceb3fe444965f960b75
v0.8.10 59accdd814398b884167c0f41dbf46dcccf0c29c
v0.8.11 d630ec9092f039cb5e51c5dd6d92fc47b91407e4
v0.8.12 57871c99031cf597ffa0d819057ac1401e129f32
v0.8.13 e43e6e876513450d235124fcb711f1724ed9814c
v0.8.14 71901c94dbbaa2f9f156a80c14cc7ea65219fc7c
v0.8.15 5a7c2f90361e72e9c35e88abd2e11acdc4aba354
v0.8.16 f41dc62a12cc0f3cb8c5453c0caa0ba21e1bd01e
v0.9.0 58b84fb401a804967aa0dd5ee66fafa90194fd30
v0.9.1 89f68bf0cbf53a81b0553d3816fdbe522f941fa1
v0.9.2 c7d3b28b93faa6c955fb24bc64031512ee985ee9
v0.9.3 8704c1ff952b5c85a44f50143bbd1a4f7b4887e2
v0.9.4 fa310d9e2542cf497d92f007baec8fd751ffa99c
v0.9.5 3b1e9017c560499786d8a0e46aaaeea65037acac
v0.9.6 0b4d08fab8e35a264bc7383e878538f8083c33e5
v0.9.7 f8cb04e4a8b032b72a909f68b808a50936184bee

View File

@@ -13,7 +13,7 @@ cd $ROOT
FAILED=0
IGNORED_FILES="uv\.lock|docs\/CARS.md|LICENSE\.md|.*\.zst"
IGNORED_FILES="uv\.lock|docs\/CARS.md"
IGNORED_DIRS="^third_party.*|^msgq.*|^msgq_repo.*|^opendbc.*|^opendbc_repo.*|^cereal.*|^panda.*|^rednose.*|^rednose_repo.*|^tinygrad.*|^tinygrad_repo.*|^teleoprtc.*|^teleoprtc_repo.*"
function run() {
@@ -57,7 +57,7 @@ function run_tests() {
if [[ -z "$FAST" ]]; then
run "mypy" mypy $PYTHON_FILES
run "codespell" codespell $ALL_FILES --ignore-words=$ROOT/.codespellignore
run "codespell" codespell $ALL_FILES
fi
return $FAILED

14
scripts/pyqt_demo.py Executable file
View File

@@ -0,0 +1,14 @@
#!/usr/bin/env python3
from PyQt5.QtWidgets import QApplication, QLabel
from openpilot.selfdrive.ui.qt.python_helpers import set_main_window
if __name__ == "__main__":
app = QApplication([])
label = QLabel('Hello World!')
# Set full screen and rotate
set_main_window(label)
app.exec_()

Binary file not shown.

View File

@@ -148,8 +148,7 @@ class CarSpecificEvents:
# To avoid re-engaging when openpilot cancels, check user engagement intention via buttons
# Main button also can trigger an engagement on these cars
self.cruise_buttons.append(any(ev.type in HYUNDAI_ENABLE_BUTTONS for ev in CS.buttonEvents))
events = self.create_common_events(CS, CS_prev, extra_gears=(GearShifter.sport, GearShifter.manumatic),
pcm_enable=self.CP.pcmCruise, allow_enable=any(self.cruise_buttons))
events = self.create_common_events(CS, CS_prev, pcm_enable=self.CP.pcmCruise, allow_enable=any(self.cruise_buttons))
# low speed steer alert hysteresis logic (only for cars with steer cut off above 10 m/s)
if CS.vEgo < (self.CP.minSteerSpeed + 2.) and self.CP.minSteerSpeed > 10.:

View File

@@ -22,8 +22,6 @@ from openpilot.selfdrive.pandad import can_capnp_to_list, can_list_to_can_capnp
from openpilot.selfdrive.car.cruise import VCruiseHelper
from openpilot.selfdrive.car.car_specific import MockCarState
from openpilot.sunnypilot.mads.mads import MadsParams
REPLAY = "REPLAY" in os.environ
EventName = log.OnroadEvent.EventName
@@ -115,10 +113,6 @@ class Car:
if not disengage_on_accelerator:
self.CP.alternativeExperience |= ALTERNATIVE_EXPERIENCE.DISABLE_DISENGAGE_ON_GAS
# mads
MadsParams().set_alternative_experience(self.CP)
MadsParams().set_car_specific_params(self.CP)
openpilot_enabled_toggle = self.params.get_bool("OpenpilotEnabledToggle")
controller_available = self.CI.CC is not None and openpilot_enabled_toggle and not self.CP.dashcamOnly

View File

@@ -19,7 +19,6 @@ from openpilot.selfdrive.controls.lib.longcontrol import LongControl
from openpilot.selfdrive.controls.lib.vehicle_model import VehicleModel
from openpilot.selfdrive.locationd.helpers import PoseCalibrator, Pose
from opendbc.sunnypilot import SunnypilotParamFlags
State = log.SelfdriveState.OpenpilotState
LaneChangeState = log.LaneChangeState
@@ -57,9 +56,6 @@ class Controls:
elif self.CP.lateralTuning.which() == 'torque':
self.LaC = LatControlTorque(self.CP, self.CI)
data_services = list(self.sm.data.keys()) + ['selfdriveStateSP']
self.sm = messaging.SubMaster(data_services, poll='selfdriveState')
def update(self):
self.sm.update(15)
if self.sm.updated["liveCalibration"]:
@@ -92,16 +88,7 @@ class Controls:
# Check which actuators can be enabled
standstill = abs(CS.vEgo) <= max(self.CP.minSteerSpeed, MIN_LATERAL_CONTROL_SPEED) or CS.standstill
ss_sp = self.sm['selfdriveStateSP']
CC.madsEnabled = ss_sp.mads.enabled
if ss_sp.mads.available:
CC.sunnypilotParams |= SunnypilotParamFlags.ENABLE_MADS.value
_lat_active = ss_sp.mads.active
else:
_lat_active = self.sm['selfdriveState'].active
CC.latActive = _lat_active and not CS.steerFaultTemporary and not CS.steerFaultPermanent and not standstill
CC.latActive = self.sm['selfdriveState'].active and not CS.steerFaultTemporary and not CS.steerFaultPermanent and not standstill
CC.longActive = CC.enabled and not any(e.overrideLongitudinal for e in self.sm['onroadEvents']) and self.CP.openpilotLongitudinalControl
actuators = CC.actuators

View File

@@ -5,15 +5,12 @@ from openpilot.common.realtime import DT_CTRL
MIN_SPEED = 1.0
CONTROL_N = 17
CAR_ROTATION_RADIUS = 0.0
# This is a turn radius smaller than most cars can achieve
MAX_CURVATURE = 0.2
# EU guidelines
MAX_LATERAL_JERK = 5.0
MAX_VEL_ERR = 5.0
def clip_curvature(v_ego, prev_curvature, new_curvature):
new_curvature = clip(new_curvature, -MAX_CURVATURE, MAX_CURVATURE)
v_ego = max(MIN_SPEED, v_ego)
max_curvature_rate = MAX_LATERAL_JERK / (v_ego**2) # inexact calculation, check https://github.com/commaai/openpilot/pull/24755
safe_desired_curvature = clip(new_curvature,

View File

@@ -50,20 +50,24 @@ def limit_accel_in_turns(v_ego, angle_steers, a_target, CP):
return [a_target[0], min(a_target[1], a_x_allowed)]
def get_accel_from_plan(speeds, accels, action_t=DT_MDL, vEgoStopping=0.05):
def get_accel_from_plan(CP, speeds, accels):
if len(speeds) == CONTROL_N:
v_now = speeds[0]
a_now = accels[0]
v_target_now = interp(DT_MDL, CONTROL_N_T_IDX, speeds)
a_target_now = interp(DT_MDL, CONTROL_N_T_IDX, accels)
v_target = interp(action_t, CONTROL_N_T_IDX, speeds)
a_target = 2 * (v_target - v_now) / (action_t) - a_now
v_target_1sec = interp(action_t + 1.0, CONTROL_N_T_IDX, speeds)
v_target = interp(CP.longitudinalActuatorDelay + DT_MDL, CONTROL_N_T_IDX, speeds)
if v_target != v_target_now:
a_target = 2 * (v_target - v_target_now) / CP.longitudinalActuatorDelay - a_target_now
else:
a_target = a_target_now
v_target_1sec = interp(CP.longitudinalActuatorDelay + DT_MDL + 1.0, CONTROL_N_T_IDX, speeds)
else:
v_target = 0.0
v_target_1sec = 0.0
a_target = 0.0
should_stop = (v_target < vEgoStopping and
v_target_1sec < vEgoStopping)
should_stop = (v_target < CP.vEgoStopping and
v_target_1sec < CP.vEgoStopping)
return a_target, should_stop
@@ -197,9 +201,7 @@ class LongitudinalPlanner:
longitudinalPlan.longitudinalPlanSource = self.mpc.source
longitudinalPlan.fcw = self.fcw
action_t = self.CP.longitudinalActuatorDelay + DT_MDL
a_target, should_stop = get_accel_from_plan(longitudinalPlan.speeds, longitudinalPlan.accels,
action_t=action_t, vEgoStopping=self.CP.vEgoStopping)
a_target, should_stop = get_accel_from_plan(self.CP, longitudinalPlan.speeds, longitudinalPlan.accels)
longitudinalPlan.aTarget = a_target
longitudinalPlan.shouldStop = should_stop
longitudinalPlan.allowBrake = True

View File

@@ -1,54 +0,0 @@
#!/usr/bin/env python3
import argparse
import numpy as np
import matplotlib.pyplot as plt
from openpilot.tools.lib.logreader import LogReader
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--width', default=2160, type=int)
parser.add_argument('--height', default=1080, type=int)
parser.add_argument('--route', default='rlog', type=str)
args = parser.parse_args()
w = args.width
h = args.height
route = args.route
fingers = [[-1, -1]] * 5
touch_points = []
current_slot = 0
lr = list(LogReader(route))
for msg in lr:
if msg.which() == 'touch':
for event in msg.touch:
if event.type == 3 and event.code == 47:
current_slot = event.value
elif event.type == 3 and event.code == 57 and event.value == -1:
fingers[current_slot] = [-1, -1]
elif event.type == 3 and event.code == 53:
fingers[current_slot][1] = event.value
if fingers[current_slot][0] != -1:
touch_points.append(fingers[current_slot].copy())
elif event.type == 3 and event.code == 54:
fingers[current_slot][0] = w - event.value
if fingers[current_slot][1] != -1:
touch_points.append(fingers[current_slot].copy())
if not touch_points:
print(f'No touch events found for {route}')
quit()
unique_points, counts = np.unique(touch_points, axis=0, return_counts=True)
plt.figure(figsize=(10, 3))
plt.scatter(unique_points[:, 0], unique_points[:, 1], c=counts, s=counts * 20, edgecolors='red')
plt.colorbar()
plt.title(f'Touches for {route}')
plt.xlim(0, w)
plt.ylim(0, h)
plt.grid(True)
plt.show()

View File

@@ -13,6 +13,20 @@ common_src = [
"transforms/transform.cc",
]
thneed_src_common = [
"thneed/thneed_common.cc",
"thneed/serialize.cc",
]
thneed_src_qcom = thneed_src_common + ["thneed/thneed_qcom2.cc"]
thneed_src_pc = thneed_src_common + ["thneed/thneed_pc.cc"]
thneed_src = thneed_src_qcom if arch == "larch64" else thneed_src_pc
# SNPE except on Mac and ARM Linux
snpe_lib = []
if arch != "Darwin" and arch != "aarch64":
common_src += ['runners/snpemodel.cc']
snpe_lib += ['SNPE']
# OpenCL is a framework on Mac
if arch == "Darwin":
@@ -26,25 +40,39 @@ for pathdef, fn in {'TRANSFORM': 'transforms/transform.cl', 'LOADYUV': 'transfor
xenv['CXXFLAGS'].append(f'-D{pathdef}_PATH=\\"{File(fn).abspath}\\"')
# Compile cython
snpe_rpath_qcom = "/data/pythonpath/third_party/snpe/larch64"
snpe_rpath_pc = f"{Dir('#').abspath}/third_party/snpe/x86_64-linux-clang"
snpe_rpath = lenvCython['RPATH'] + [snpe_rpath_qcom if arch == "larch64" else snpe_rpath_pc]
cython_libs = envCython["LIBS"] + libs
snpemodel_lib = lenv.Library('snpemodel', ['runners/snpemodel.cc'])
commonmodel_lib = lenv.Library('commonmodel', common_src)
lenvCython.Program('runners/runmodel_pyx.so', 'runners/runmodel_pyx.pyx', LIBS=cython_libs, FRAMEWORKS=frameworks)
lenvCython.Program('runners/snpemodel_pyx.so', 'runners/snpemodel_pyx.pyx', LIBS=[snpemodel_lib, snpe_lib, *cython_libs], FRAMEWORKS=frameworks, RPATH=snpe_rpath)
lenvCython.Program('models/commonmodel_pyx.so', 'models/commonmodel_pyx.pyx', LIBS=[commonmodel_lib, *cython_libs], FRAMEWORKS=frameworks)
tinygrad_files = ["#"+x for x in glob.glob(env.Dir("#tinygrad_repo").relpath + "/**", recursive=True, root_dir=env.Dir("#").abspath) if 'pycache' not in x]
tinygrad_files = ["#"+x for x in glob.glob(env.Dir("#tinygrad_repo").relpath + "/**", recursive=True, root_dir=env.Dir("#").abspath)]
# Get model metadata
fn = File("models/supercombo").abspath
cmd = f'python3 {Dir("#selfdrive/modeld").abspath}/get_model_metadata.py {fn}.onnx'
lenv.Command(fn + "_metadata.pkl", [fn + ".onnx"] + tinygrad_files, cmd)
# Compile tinygrad model
pythonpath_string = 'PYTHONPATH="${PYTHONPATH}:' + env.Dir("#tinygrad_repo").abspath + '"'
if arch == 'larch64':
device_string = 'QCOM=1'
else:
device_string = 'CLANG=1 IMAGE=0'
# Build thneed model
if arch == "larch64" or GetOption('pc_thneed'):
tinygrad_opts = []
if not GetOption('pc_thneed'):
# use FLOAT16 on device for speed + don't cache the CL kernels for space
tinygrad_opts += ["FLOAT16=1", "PYOPENCL_NO_CACHE=1"]
cmd = f"cd {Dir('#').abspath}/tinygrad_repo && " + ' '.join(tinygrad_opts) + f" python3 openpilot/compile2.py {fn}.onnx {fn}.thneed"
for model_name in ['supercombo', 'dmonitoring_model']:
fn = File(f"models/{model_name}").abspath
cmd = f'{pythonpath_string} {device_string} python3 {Dir("#tinygrad_repo").abspath}/examples/openpilot/compile3.py {fn}.onnx {fn}_tinygrad.pkl'
lenv.Command(fn + "_tinygrad.pkl", [fn + ".onnx"] + tinygrad_files, cmd)
lenv.Command(fn + ".thneed", [fn + ".onnx"] + tinygrad_files, cmd)
fn_dm = File("models/dmonitoring_model").abspath
cmd = f"cd {Dir('#').abspath}/tinygrad_repo && " + ' '.join(tinygrad_opts) + f" python3 openpilot/compile2.py {fn_dm}.onnx {fn_dm}.thneed"
lenv.Command(fn_dm + ".thneed", [fn_dm + ".onnx"] + tinygrad_files, cmd)
thneed_lib = env.SharedLibrary('thneed', thneed_src, LIBS=[gpucommon, common, 'OpenCL', 'dl'])
thneedmodel_lib = env.Library('thneedmodel', ['runners/thneedmodel.cc'])
lenvCython.Program('runners/thneedmodel_pyx.so', 'runners/thneedmodel_pyx.pyx', LIBS=envCython["LIBS"]+[thneedmodel_lib, thneed_lib, gpucommon, common, 'dl', 'OpenCL'])

View File

@@ -16,6 +16,7 @@ class ModelConstants:
MODEL_FREQ = 20
FEATURE_LEN = 512
FULL_HISTORY_BUFFER_LEN = 99
HISTORY_BUFFER_LEN = 24
DESIRE_LEN = 8
TRAFFIC_CONVENTION_LEN = 2
LAT_PLANNER_STATE_LEN = 4
@@ -72,13 +73,14 @@ class Plan:
class Meta:
ENGAGED = slice(0, 1)
# next 2, 4, 6, 8, 10 seconds
GAS_DISENGAGE = slice(1, 36, 7)
BRAKE_DISENGAGE = slice(2, 36, 7)
STEER_OVERRIDE = slice(3, 36, 7)
HARD_BRAKE_3 = slice(4, 36, 7)
HARD_BRAKE_4 = slice(5, 36, 7)
HARD_BRAKE_5 = slice(6, 36, 7)
GAS_PRESS = slice(7, 36, 7)
GAS_DISENGAGE = slice(1, 31, 6)
BRAKE_DISENGAGE = slice(2, 31, 6)
STEER_OVERRIDE = slice(3, 31, 6)
HARD_BRAKE_3 = slice(4, 31, 6)
HARD_BRAKE_4 = slice(5, 31, 6)
HARD_BRAKE_5 = slice(6, 31, 6)
# next 0, 2, 4, 6, 8, 10 seconds
LEFT_BLINKER = slice(36, 48, 2)
RIGHT_BLINKER = slice(37, 48, 2)
GAS_PRESS = slice(31, 55, 4)
BRAKE_PRESS = slice(32, 55, 4)
LEFT_BLINKER = slice(33, 55, 4)
RIGHT_BLINKER = slice(34, 55, 4)

View File

@@ -1,4 +1,10 @@
#!/usr/bin/env bash
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
cd "$DIR/../../"
if [ -f "$DIR/libthneed.so" ]; then
export LD_PRELOAD="$DIR/libthneed.so"
fi
exec "$DIR/dmonitoringmodeld.py" "$@"

View File

@@ -1,17 +1,8 @@
#!/usr/bin/env python3
import os
from openpilot.system.hardware import TICI
if TICI:
from tinygrad.tensor import Tensor
from tinygrad.dtype import dtypes
from openpilot.selfdrive.modeld.runners.tinygrad_helpers import qcom_tensor_from_opencl_address
os.environ['QCOM'] = '1'
else:
from openpilot.selfdrive.modeld.runners.ort_helpers import make_onnx_cpu_runner
import gc
import math
import time
import pickle
import ctypes
import numpy as np
from pathlib import Path
@@ -22,20 +13,21 @@ from cereal.messaging import PubMaster, SubMaster
from msgq.visionipc import VisionIpcClient, VisionStreamType, VisionBuf
from openpilot.common.swaglog import cloudlog
from openpilot.common.realtime import set_realtime_priority
from openpilot.common.transformations.model import dmonitoringmodel_intrinsics, DM_INPUT_SIZE
from openpilot.common.transformations.camera import _ar_ox_fisheye, _os_fisheye
from openpilot.selfdrive.modeld.models.commonmodel_pyx import CLContext, MonitoringModelFrame
from openpilot.selfdrive.modeld.runners import ModelRunner, Runtime
from openpilot.selfdrive.modeld.models.commonmodel_pyx import CLContext
from openpilot.selfdrive.modeld.parse_model_outputs import sigmoid
MODEL_WIDTH, MODEL_HEIGHT = DM_INPUT_SIZE
CALIB_LEN = 3
MODEL_WIDTH = 1440
MODEL_HEIGHT = 960
FEATURE_LEN = 512
OUTPUT_SIZE = 84 + FEATURE_LEN
PROCESS_NAME = "selfdrive.modeld.dmonitoringmodeld"
SEND_RAW_PRED = os.getenv('SEND_RAW_PRED')
MODEL_PATH = Path(__file__).parent / 'models/dmonitoring_model.onnx'
MODEL_PKL_PATH = Path(__file__).parent / 'models/dmonitoring_model_tinygrad.pkl'
MODEL_PATHS = {
ModelRunner.THNEED: Path(__file__).parent / 'models/dmonitoring_model.thneed',
ModelRunner.ONNX: Path(__file__).parent / 'models/dmonitoring_model.onnx'}
class DriverStateResult(ctypes.Structure):
_fields_ = [
@@ -66,42 +58,33 @@ class DMonitoringModelResult(ctypes.Structure):
class ModelState:
inputs: dict[str, np.ndarray]
output: np.ndarray
model: ModelRunner
def __init__(self, cl_ctx):
assert ctypes.sizeof(DMonitoringModelResult) == OUTPUT_SIZE * ctypes.sizeof(ctypes.c_float)
self.output = np.zeros(OUTPUT_SIZE, dtype=np.float32)
self.inputs = {
'input_img': np.zeros(MODEL_HEIGHT * MODEL_WIDTH, dtype=np.uint8),
'calib': np.zeros(CALIB_LEN, dtype=np.float32)}
self.frame = MonitoringModelFrame(cl_ctx)
self.numpy_inputs = {
'calib': np.zeros((1, CALIB_LEN), dtype=np.float32),
}
self.model = ModelRunner(MODEL_PATHS, self.output, Runtime.GPU, False, cl_ctx)
self.model.addInput("input_img", None)
self.model.addInput("calib", self.inputs['calib'])
if TICI:
self.tensor_inputs = {k: Tensor(v, device='NPY').realize() for k,v in self.numpy_inputs.items()}
with open(MODEL_PKL_PATH, "rb") as f:
self.model_run = pickle.load(f)
else:
self.onnx_cpu_runner = make_onnx_cpu_runner(MODEL_PATH)
def run(self, buf:VisionBuf, calib:np.ndarray) -> tuple[np.ndarray, float]:
self.inputs['calib'][:] = calib
def run(self, buf:VisionBuf, calib:np.ndarray, transform:np.ndarray) -> tuple[np.ndarray, float]:
self.numpy_inputs['calib'][0,:] = calib
v_offset = buf.height - MODEL_HEIGHT
h_offset = (buf.width - MODEL_WIDTH) // 2
buf_data = buf.data.reshape(-1, buf.stride)
input_data = self.inputs['input_img'].reshape(MODEL_HEIGHT, MODEL_WIDTH)
input_data[:] = buf_data[v_offset:v_offset+MODEL_HEIGHT, h_offset:h_offset+MODEL_WIDTH]
self.model.setInputBuffer("input_img", self.inputs['input_img'].view(np.float32))
t1 = time.perf_counter()
input_img_cl = self.frame.prepare(buf, transform.flatten())
if TICI:
# The imgs tensors are backed by opencl memory, only need init once
if 'input_img' not in self.tensor_inputs:
self.tensor_inputs['input_img'] = qcom_tensor_from_opencl_address(input_img_cl.mem_address, (1, MODEL_WIDTH*MODEL_HEIGHT), dtype=dtypes.uint8)
else:
self.numpy_inputs['input_img'] = self.frame.buffer_from_cl(input_img_cl).reshape((1, MODEL_WIDTH*MODEL_HEIGHT))
if TICI:
output = self.model_run(**self.tensor_inputs).numpy().flatten()
else:
output = self.onnx_cpu_runner.run(None, self.numpy_inputs)[0].flatten()
self.model.execute()
t2 = time.perf_counter()
return output, t2 - t1
return self.output, t2 - t1
def fill_driver_state(msg, ds_result: DriverStateResult):
@@ -154,23 +137,18 @@ def main():
pm = PubMaster(["driverStateV2"])
calib = np.zeros(CALIB_LEN, dtype=np.float32)
model_transform = None
while True:
buf = vipc_client.recv()
if buf is None:
continue
if model_transform is None:
cam = _os_fisheye if buf.width == _os_fisheye.width else _ar_ox_fisheye
model_transform = np.linalg.inv(np.dot(dmonitoringmodel_intrinsics, np.linalg.inv(cam.intrinsics))).astype(np.float32)
sm.update(0)
if sm.updated["liveCalibration"]:
calib[:] = np.array(sm["liveCalibration"].rpyCalib)
t1 = time.perf_counter()
model_output, gpu_execution_time = model.run(buf, calib, model_transform)
model_output, gpu_execution_time = model.run(buf, calib)
t2 = time.perf_counter()
pm.send("driverStateV2", get_driverstate_packet(model_output, vipc_client.frame_id, vipc_client.timestamp_sof, t2 - t1, gpu_execution_time))

View File

@@ -8,7 +8,6 @@ SEND_RAW_PRED = os.getenv('SEND_RAW_PRED')
ConfidenceClass = log.ModelDataV2.ConfidenceClass
class PublishState:
def __init__(self):
self.disengage_buffer = np.zeros(ModelConstants.CONFIDENCE_BUFFER_LEN*ModelConstants.DISENGAGE_WIDTH, dtype=np.float32)
@@ -56,10 +55,9 @@ def fill_lane_line_meta(builder, lane_lines, lane_line_probs):
builder.rightProb = lane_line_probs[2]
def fill_model_msg(base_msg: capnp._DynamicStructBuilder, extended_msg: capnp._DynamicStructBuilder,
net_output_data: dict[str, np.ndarray], v_ego: float, delay: float,
publish_state: PublishState, vipc_frame_id: int, vipc_frame_id_extra: int,
frame_id: int, frame_drop: float, timestamp_eof: int, model_execution_time: float,
valid: bool) -> None:
net_output_data: dict[str, np.ndarray], publish_state: PublishState,
vipc_frame_id: int, vipc_frame_id_extra: int, frame_id: int, frame_drop: float,
timestamp_eof: int, model_execution_time: float, valid: bool) -> None:
frame_age = frame_id - vipc_frame_id if frame_id > vipc_frame_id else 0
frame_drop_perc = frame_drop * 100
extended_msg.valid = valid
@@ -169,8 +167,8 @@ def fill_model_msg(base_msg: capnp._DynamicStructBuilder, extended_msg: capnp._D
disengage_predictions.brake3MetersPerSecondSquaredProbs = net_output_data['meta'][0,Meta.HARD_BRAKE_3].tolist()
disengage_predictions.brake4MetersPerSecondSquaredProbs = net_output_data['meta'][0,Meta.HARD_BRAKE_4].tolist()
disengage_predictions.brake5MetersPerSecondSquaredProbs = net_output_data['meta'][0,Meta.HARD_BRAKE_5].tolist()
#disengage_predictions.gasPressProbs = net_output_data['meta'][0,Meta.GAS_PRESS].tolist()
#disengage_predictions.brakePressProbs = net_output_data['meta'][0,Meta.BRAKE_PRESS].tolist()
disengage_predictions.gasPressProbs = net_output_data['meta'][0,Meta.GAS_PRESS].tolist()
disengage_predictions.brakePressProbs = net_output_data['meta'][0,Meta.BRAKE_PRESS].tolist()
publish_state.prev_brake_5ms2_probs[:-1] = publish_state.prev_brake_5ms2_probs[1:]
publish_state.prev_brake_5ms2_probs[-1] = net_output_data['meta'][0,Meta.HARD_BRAKE_5][0]

View File

@@ -1,4 +1,10 @@
#!/usr/bin/env bash
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
cd "$DIR/../../"
if [ -f "$DIR/libthneed.so" ]; then
export LD_PRELOAD="$DIR/libthneed.so"
fi
exec "$DIR/modeld.py" "$@"

View File

@@ -1,15 +1,11 @@
#!/usr/bin/env python3
from openpilot.system.hardware import TICI
from openpilot.selfdrive.modeld.runners.model_runner import ONNXRunner, TinygradRunner
from openpilot.sunnypilot.models.helpers import is_active_model_20hz
#
import os
import time
import pickle
import numpy as np
import cereal.messaging as messaging
from cereal import car, log
from pathlib import Path
from setproctitle import setproctitle
from cereal.messaging import PubMaster, SubMaster
from msgq.visionipc import VisionIpcClient, VisionStreamType, VisionBuf
@@ -21,16 +17,22 @@ from openpilot.common.realtime import config_realtime_process
from openpilot.common.transformations.camera import DEVICE_CAMERAS
from openpilot.common.transformations.model import get_warp_matrix
from openpilot.system import sentry
from openpilot.system.hardware import PC
from openpilot.selfdrive.controls.lib.desire_helper import DesireHelper
from openpilot.selfdrive.modeld.runners import ModelRunner, Runtime
from openpilot.selfdrive.modeld.parse_model_outputs import Parser
from openpilot.selfdrive.modeld.fill_model_msg import fill_model_msg, fill_pose_msg, PublishState
from openpilot.selfdrive.modeld.constants import ModelConstants
from openpilot.selfdrive.modeld.models.commonmodel_pyx import DrivingModelFrame, CLContext
from openpilot.selfdrive.modeld.models.commonmodel_pyx import ModelFrame, CLContext
PROCESS_NAME = "selfdrive.modeld.modeld"
USE_ONNX = bool(os.getenv('USE_ONNX', PC))
IS_20HZ_MODEL_DEFAULT = False
SEND_RAW_PRED = os.getenv('SEND_RAW_PRED')
MODEL_PATHS = {
ModelRunner.THNEED: Path(__file__).parent / 'models/supercombo.thneed',
ModelRunner.ONNX: Path(__file__).parent / 'models/supercombo.onnx'}
METADATA_PATH = Path(__file__).parent / 'models/supercombo_metadata.pkl'
class FrameMeta:
frame_id: int = 0
@@ -42,92 +44,83 @@ class FrameMeta:
self.frame_id, self.timestamp_sof, self.timestamp_eof = vipc.frame_id, vipc.timestamp_sof, vipc.timestamp_eof
class ModelState:
frames: dict[str, DrivingModelFrame]
frame: ModelFrame
wide_frame: ModelFrame
inputs: dict[str, np.ndarray]
output: np.ndarray
prev_desire: np.ndarray # for tracking the rising edge of the pulse
model: ModelRunner
def __init__(self, context: CLContext):
self.is_20hz = IS_20HZ_MODEL_DEFAULT
if (active_20hz := is_active_model_20hz(None)) is not None:
self.is_20hz = active_20hz
self.frames = {'input_imgs': DrivingModelFrame(context, self.is_20hz), 'big_input_imgs': DrivingModelFrame(context, self.is_20hz)}
self.frame = ModelFrame(context)
self.wide_frame = ModelFrame(context)
self.prev_desire = np.zeros(ModelConstants.DESIRE_LEN, dtype=np.float32)
self.full_features_20Hz = np.zeros((ModelConstants.FULL_HISTORY_BUFFER_LEN, ModelConstants.FEATURE_LEN), dtype=np.float32)
self.desire_20Hz = np.zeros((ModelConstants.FULL_HISTORY_BUFFER_LEN + 1, ModelConstants.DESIRE_LEN), dtype=np.float32)
# Initialize model runner
self.model_runner = ONNXRunner(self.frames) if (not TICI) and USE_ONNX else TinygradRunner(self.frames)
self.prev_desired_curv_20hz = np.zeros((ModelConstants.FULL_HISTORY_BUFFER_LEN + 1, ModelConstants.PREV_DESIRED_CURV_LEN), dtype=np.float32)
# img buffers are managed in openCL transform code
self.numpy_inputs = {}
self.inputs = {
'desire': np.zeros(ModelConstants.DESIRE_LEN * (ModelConstants.HISTORY_BUFFER_LEN+1), dtype=np.float32),
'traffic_convention': np.zeros(ModelConstants.TRAFFIC_CONVENTION_LEN, dtype=np.float32),
'lateral_control_params': np.zeros(ModelConstants.LATERAL_CONTROL_PARAMS_LEN, dtype=np.float32),
'prev_desired_curv': np.zeros(ModelConstants.PREV_DESIRED_CURV_LEN * (ModelConstants.HISTORY_BUFFER_LEN+1), dtype=np.float32),
'features_buffer': np.zeros(ModelConstants.HISTORY_BUFFER_LEN * ModelConstants.FEATURE_LEN, dtype=np.float32),
}
for key, shape in self.model_runner.input_shapes.items():
if key not in self.frames: # Managed by opencl
self.numpy_inputs[key] = np.zeros(shape, dtype=np.float32)
with open(METADATA_PATH, 'rb') as f:
model_metadata = pickle.load(f)
self.output_slices = model_metadata['output_slices']
net_output_size = model_metadata['output_shapes']['outputs'][1]
self.output = np.zeros(net_output_size, dtype=np.float32)
self.parser = Parser()
net_output_size = self.model_runner.model_metadata['output_shapes']['outputs'][1]
self.output = np.zeros(net_output_size, dtype=np.float32)
self.model = ModelRunner(MODEL_PATHS, self.output, Runtime.GPU, False, context)
self.model.addInput("input_imgs", None)
self.model.addInput("big_input_imgs", None)
for k,v in self.inputs.items():
self.model.addInput(k, v)
num_elements = self.numpy_inputs['features_buffer'].shape[1]
step_size = int(-100 / num_elements)
self.full_features_20Hz_idxs = np.arange(step_size, step_size * (num_elements + 1), step_size)[::-1]
self.desire_reshape_dims = (self.numpy_inputs['desire'].shape[0], self.numpy_inputs['desire'].shape[1], -1, self.numpy_inputs['desire'].shape[2])
def slice_outputs(self, model_outputs: np.ndarray) -> dict[str, np.ndarray]:
parsed_model_outputs = {k: model_outputs[np.newaxis, v] for k,v in self.output_slices.items()}
if SEND_RAW_PRED:
parsed_model_outputs['raw_pred'] = model_outputs.copy()
return parsed_model_outputs
def run(self, buf: VisionBuf, wbuf: VisionBuf, transform: np.ndarray, transform_wide: np.ndarray,
inputs: dict[str, np.ndarray], prepare_only: bool) -> dict[str, np.ndarray] | None:
inputs: dict[str, np.ndarray], prepare_only: bool) -> dict[str, np.ndarray] | None:
# Model decides when action is completed, so desire input is just a pulse triggered on rising edge
inputs['desire'][0] = 0
new_desire = np.where(inputs['desire'] - self.prev_desire > .99, inputs['desire'], 0)
self.prev_desire[:] = inputs['desire']
if self.is_20hz:
self.desire_20Hz[:-1] = self.desire_20Hz[1:]
self.desire_20Hz[-1] = new_desire
self.numpy_inputs['desire'][:] = self.desire_20Hz.reshape(self.desire_reshape_dims).max(axis=2)
else:
self.numpy_inputs['desire'][0,:-1] = self.numpy_inputs['desire'][0,1:]
self.numpy_inputs['desire'][0,-1] = new_desire
self.desire_20Hz[:-1] = self.desire_20Hz[1:]
self.desire_20Hz[-1] = new_desire
self.inputs['desire'][:] = self.desire_20Hz.reshape((25,4,-1)).max(axis=1).flatten()
for key in self.numpy_inputs:
if key in inputs and key not in ['desire']:
self.numpy_inputs[key][:] = inputs[key]
self.inputs['traffic_convention'][:] = inputs['traffic_convention']
self.inputs['lateral_control_params'][:] = inputs['lateral_control_params']
imgs_cl = {'input_imgs': self.frames['input_imgs'].prepare(buf, transform.flatten()),
'big_input_imgs': self.frames['big_input_imgs'].prepare(wbuf, transform_wide.flatten())}
# Prepare inputs using the model runner
self.model_runner.prepare_inputs(imgs_cl, self.numpy_inputs)
self.model.setInputBuffer("input_imgs", self.frame.prepare(buf, transform.flatten(), self.model.getCLBuffer("input_imgs")))
self.model.setInputBuffer("big_input_imgs", self.wide_frame.prepare(wbuf, transform_wide.flatten(), self.model.getCLBuffer("big_input_imgs")))
if prepare_only:
return None
# Run model inference
self.output = self.model_runner.run_model()
outputs = self.parser.parse_outputs(self.model_runner.slice_outputs(self.output))
self.model.execute()
outputs = self.parser.parse_outputs(self.slice_outputs(self.output))
if self.is_20hz:
self.full_features_20Hz[:-1] = self.full_features_20Hz[1:]
self.full_features_20Hz[-1] = outputs['hidden_state'][0, :]
self.numpy_inputs['features_buffer'][:] = self.full_features_20Hz[self.full_features_20Hz_idxs]
else:
self.numpy_inputs['features_buffer'][0,:-1] = self.numpy_inputs['features_buffer'][0,1:]
self.numpy_inputs['features_buffer'][0,-1] = outputs['hidden_state'][0, :]
self.full_features_20Hz[:-1] = self.full_features_20Hz[1:]
self.full_features_20Hz[-1] = outputs['hidden_state'][0, :]
if "desired_curvature" in outputs:
input_name_prev = None
self.prev_desired_curv_20hz[:-1] = self.prev_desired_curv_20hz[1:]
self.prev_desired_curv_20hz[-1] = outputs['desired_curvature'][0, :]
if "prev_desired_curvs" in self.numpy_inputs.keys():
input_name_prev = 'prev_desired_curvs'
elif "prev_desired_curv" in self.numpy_inputs.keys():
input_name_prev = 'prev_desired_curv'
if input_name_prev is not None:
len = outputs['desired_curvature'][0].size
self.numpy_inputs['prev_desired_curv'][0,:-len] = self.numpy_inputs['prev_desired_curv'][0,len:]
self.numpy_inputs['prev_desired_curv'][0,-len,:] = outputs['desired_curvature'][0, :]
idxs = np.arange(-4,-100,-4)[::-1]
self.inputs['features_buffer'][:] = self.full_features_20Hz[idxs].flatten()
# TODO model only uses last value now, once that changes we need to input strided action history buffer
self.inputs['prev_desired_curv'][-ModelConstants.PREV_DESIRED_CURV_LEN:] = 0. * self.prev_desired_curv_20hz[-4, :]
return outputs
@@ -238,6 +231,7 @@ def main(demo=False):
is_rhd = sm["driverMonitoringState"].isRHD
frame_id = sm["roadCameraState"].frameId
v_ego = max(sm["carState"].vEgo, 0.)
lateral_control_params = np.array([v_ego, steer_delay], dtype=np.float32)
if sm.updated["liveCalibration"] and sm.seen['roadCameraState'] and sm.seen['deviceState']:
device_from_calib_euler = np.array(sm["liveCalibration"].rpyCalib, dtype=np.float32)
dc = DEVICE_CAMERAS[(str(sm['deviceState'].deviceType), str(sm['roadCameraState'].sensor))]
@@ -268,10 +262,8 @@ def main(demo=False):
inputs:dict[str, np.ndarray] = {
'desire': vec_desire,
'traffic_convention': traffic_convention,
}
if "lateral_control_params" in model.numpy_inputs.keys():
inputs['lateral_control_params'] = np.array([v_ego, steer_delay], dtype=np.float32)
'lateral_control_params': lateral_control_params,
}
mt1 = time.perf_counter()
model_output = model.run(buf_main, buf_extra, model_transform_main, model_transform_extra, inputs, prepare_only)
@@ -282,8 +274,7 @@ def main(demo=False):
modelv2_send = messaging.new_message('modelV2')
drivingdata_send = messaging.new_message('drivingModelData')
posenet_send = messaging.new_message('cameraOdometry')
fill_model_msg(drivingdata_send, modelv2_send, model_output, v_ego, steer_delay,
publish_state, meta_main.frame_id, meta_extra.frame_id, frame_id,
fill_model_msg(drivingdata_send, modelv2_send, model_output, publish_state, meta_main.frame_id, meta_extra.frame_id, frame_id,
frame_drop_ratio, meta_main.timestamp_eof, model_execution_time, live_calib_seen)
desire_state = modelv2_send.modelV2.meta.desireState
@@ -300,6 +291,7 @@ def main(demo=False):
pm.send('modelV2', modelv2_send)
pm.send('drivingModelData', drivingdata_send)
pm.send('cameraOdometry', posenet_send)
last_vipc_frame_id = meta_main.frame_id

View File

@@ -1,61 +1,58 @@
#include "selfdrive/modeld/models/commonmodel.h"
#include <cassert>
#include <cmath>
#include <cstring>
#include "common/clutil.h"
DrivingModelFrame::DrivingModelFrame(cl_device_id device_id, cl_context context, bool is_20hz) : ModelFrame(device_id, context), is_20hz(is_20hz) {
ModelFrame::ModelFrame(cl_device_id device_id, cl_context context) {
input_frames = std::make_unique<uint8_t[]>(buf_size);
input_frames_cl = CL_CHECK_ERR(clCreateBuffer(context, CL_MEM_READ_WRITE, buf_size, NULL, &err));
img_buffer_20hz_cl = CL_CHECK_ERR(clCreateBuffer(context, CL_MEM_READ_WRITE, buf_len*frame_size_bytes, NULL, &err));
region.origin = (buf_len - 1) * frame_size_bytes;
q = CL_CHECK_ERR(clCreateCommandQueue(context, device_id, 0, &err));
y_cl = CL_CHECK_ERR(clCreateBuffer(context, CL_MEM_READ_WRITE, MODEL_WIDTH * MODEL_HEIGHT, NULL, &err));
u_cl = CL_CHECK_ERR(clCreateBuffer(context, CL_MEM_READ_WRITE, (MODEL_WIDTH / 2) * (MODEL_HEIGHT / 2), NULL, &err));
v_cl = CL_CHECK_ERR(clCreateBuffer(context, CL_MEM_READ_WRITE, (MODEL_WIDTH / 2) * (MODEL_HEIGHT / 2), NULL, &err));
img_buffer_20hz_cl = CL_CHECK_ERR(clCreateBuffer(context, CL_MEM_READ_WRITE, 5*frame_size_bytes, NULL, &err));
region.origin = 4 * frame_size_bytes;
region.size = frame_size_bytes;
last_img_cl = CL_CHECK_ERR(clCreateSubBuffer(img_buffer_20hz_cl, CL_MEM_READ_WRITE, CL_BUFFER_CREATE_TYPE_REGION, &region, &err));
transform_init(&transform, context, device_id);
loadyuv_init(&loadyuv, context, device_id, MODEL_WIDTH, MODEL_HEIGHT);
init_transform(device_id, context, MODEL_WIDTH, MODEL_HEIGHT);
}
cl_mem* DrivingModelFrame::prepare(cl_mem yuv_cl, int frame_width, int frame_height, int frame_stride, int frame_uv_offset, const mat3& projection) {
run_transform(yuv_cl, MODEL_WIDTH, MODEL_HEIGHT, frame_width, frame_height, frame_stride, frame_uv_offset, projection);
uint8_t* ModelFrame::prepare(cl_mem yuv_cl, int frame_width, int frame_height, int frame_stride, int frame_uv_offset, const mat3 &projection, cl_mem *output) {
transform_queue(&this->transform, q,
yuv_cl, frame_width, frame_height, frame_stride, frame_uv_offset,
y_cl, u_cl, v_cl, MODEL_WIDTH, MODEL_HEIGHT, projection);
for (int i = 0; i < (buf_len - 1); i++) {
for (int i = 0; i < 4; i++) {
CL_CHECK(clEnqueueCopyBuffer(q, img_buffer_20hz_cl, img_buffer_20hz_cl, (i+1)*frame_size_bytes, i*frame_size_bytes, frame_size_bytes, 0, nullptr, nullptr));
}
loadyuv_queue(&loadyuv, q, y_cl, u_cl, v_cl, last_img_cl);
if (output == NULL) {
CL_CHECK(clEnqueueReadBuffer(q, img_buffer_20hz_cl, CL_TRUE, 0, frame_size_bytes, &input_frames[0], 0, nullptr, nullptr));
CL_CHECK(clEnqueueReadBuffer(q, last_img_cl, CL_TRUE, 0, frame_size_bytes, &input_frames[MODEL_FRAME_SIZE], 0, nullptr, nullptr));
clFinish(q);
return &input_frames[0];
} else {
copy_queue(&loadyuv, q, img_buffer_20hz_cl, *output, 0, 0, frame_size_bytes);
copy_queue(&loadyuv, q, last_img_cl, *output, 0, frame_size_bytes, frame_size_bytes);
copy_queue(&loadyuv, q, img_buffer_20hz_cl, input_frames_cl, 0, 0, frame_size_bytes);
copy_queue(&loadyuv, q, last_img_cl, input_frames_cl, 0, frame_size_bytes, frame_size_bytes);
// NOTE: Since thneed is using a different command queue, this clFinish is needed to ensure the image is ready.
clFinish(q);
return &input_frames_cl;
// NOTE: Since thneed is using a different command queue, this clFinish is needed to ensure the image is ready.
clFinish(q);
return NULL;
}
}
DrivingModelFrame::~DrivingModelFrame() {
deinit_transform();
ModelFrame::~ModelFrame() {
transform_destroy(&transform);
loadyuv_destroy(&loadyuv);
CL_CHECK(clReleaseMemObject(img_buffer_20hz_cl));
CL_CHECK(clReleaseMemObject(last_img_cl));
CL_CHECK(clReleaseMemObject(v_cl));
CL_CHECK(clReleaseMemObject(u_cl));
CL_CHECK(clReleaseMemObject(y_cl));
CL_CHECK(clReleaseCommandQueue(q));
}
MonitoringModelFrame::MonitoringModelFrame(cl_device_id device_id, cl_context context) : ModelFrame(device_id, context) {
input_frames = std::make_unique<uint8_t[]>(buf_size);
input_frame_cl = CL_CHECK_ERR(clCreateBuffer(context, CL_MEM_READ_WRITE, buf_size, NULL, &err));
init_transform(device_id, context, MODEL_WIDTH, MODEL_HEIGHT);
}
cl_mem* MonitoringModelFrame::prepare(cl_mem yuv_cl, int frame_width, int frame_height, int frame_stride, int frame_uv_offset, const mat3& projection) {
run_transform(yuv_cl, MODEL_WIDTH, MODEL_HEIGHT, frame_width, frame_height, frame_stride, frame_uv_offset, projection);
clFinish(q);
return &y_cl;
}
MonitoringModelFrame::~MonitoringModelFrame() {
deinit_transform();
CL_CHECK(clReleaseCommandQueue(q));
}
}

View File

@@ -2,7 +2,6 @@
#include <cfloat>
#include <cstdlib>
#include <cassert>
#include <memory>
@@ -19,54 +18,9 @@
class ModelFrame {
public:
ModelFrame(cl_device_id device_id, cl_context context) {
q = CL_CHECK_ERR(clCreateCommandQueue(context, device_id, 0, &err));
}
virtual ~ModelFrame() {}
virtual cl_mem* prepare(cl_mem yuv_cl, int frame_width, int frame_height, int frame_stride, int frame_uv_offset, const mat3& projection) { return NULL; }
uint8_t* buffer_from_cl(cl_mem *in_frames, int buffer_size) {
CL_CHECK(clEnqueueReadBuffer(q, *in_frames, CL_TRUE, 0, buffer_size, input_frames.get(), 0, nullptr, nullptr));
clFinish(q);
return &input_frames[0];
}
int MODEL_WIDTH;
int MODEL_HEIGHT;
int MODEL_FRAME_SIZE;
int buf_size;
protected:
cl_mem y_cl, u_cl, v_cl;
Transform transform;
cl_command_queue q;
std::unique_ptr<uint8_t[]> input_frames;
void init_transform(cl_device_id device_id, cl_context context, int model_width, int model_height) {
y_cl = CL_CHECK_ERR(clCreateBuffer(context, CL_MEM_READ_WRITE, model_width * model_height, NULL, &err));
u_cl = CL_CHECK_ERR(clCreateBuffer(context, CL_MEM_READ_WRITE, (model_width / 2) * (model_height / 2), NULL, &err));
v_cl = CL_CHECK_ERR(clCreateBuffer(context, CL_MEM_READ_WRITE, (model_width / 2) * (model_height / 2), NULL, &err));
transform_init(&transform, context, device_id);
}
void deinit_transform() {
transform_destroy(&transform);
CL_CHECK(clReleaseMemObject(v_cl));
CL_CHECK(clReleaseMemObject(u_cl));
CL_CHECK(clReleaseMemObject(y_cl));
}
void run_transform(cl_mem yuv_cl, int model_width, int model_height, int frame_width, int frame_height, int frame_stride, int frame_uv_offset, const mat3& projection) {
transform_queue(&transform, q,
yuv_cl, frame_width, frame_height, frame_stride, frame_uv_offset,
y_cl, u_cl, v_cl, model_width, model_height, projection);
}
};
class DrivingModelFrame : public ModelFrame {
public:
DrivingModelFrame(cl_device_id device_id, cl_context context, bool is_20hz = false);
~DrivingModelFrame();
cl_mem* prepare(cl_mem yuv_cl, int frame_width, int frame_height, int frame_stride, int frame_uv_offset, const mat3& projection);
ModelFrame(cl_device_id device_id, cl_context context);
~ModelFrame();
uint8_t* prepare(cl_mem yuv_cl, int width, int height, int frame_stride, int frame_uv_offset, const mat3& transform, cl_mem *output);
const int MODEL_WIDTH = 512;
const int MODEL_HEIGHT = 256;
@@ -74,26 +28,11 @@ public:
const int buf_size = MODEL_FRAME_SIZE * 2;
const size_t frame_size_bytes = MODEL_FRAME_SIZE * sizeof(uint8_t);
const bool is_20hz;
const int buf_len = is_20hz ? 5 : 2;
private:
Transform transform;
LoadYUVState loadyuv;
cl_mem img_buffer_20hz_cl, last_img_cl, input_frames_cl;
cl_command_queue q;
cl_mem y_cl, u_cl, v_cl, img_buffer_20hz_cl, last_img_cl;
cl_buffer_region region;
};
class MonitoringModelFrame : public ModelFrame {
public:
MonitoringModelFrame(cl_device_id device_id, cl_context context);
~MonitoringModelFrame();
cl_mem* prepare(cl_mem yuv_cl, int frame_width, int frame_height, int frame_stride, int frame_uv_offset, const mat3& projection);
const int MODEL_WIDTH = 1440;
const int MODEL_HEIGHT = 960;
const int MODEL_FRAME_SIZE = MODEL_WIDTH * MODEL_HEIGHT;
const int buf_size = MODEL_FRAME_SIZE;
private:
cl_mem input_frame_cl;
};
std::unique_ptr<uint8_t[]> input_frames;
};

View File

@@ -14,13 +14,5 @@ cdef extern from "common/clutil.h":
cdef extern from "selfdrive/modeld/models/commonmodel.h":
cppclass ModelFrame:
int buf_size
unsigned char * buffer_from_cl(cl_mem*, int);
cl_mem * prepare(cl_mem, int, int, int, int, mat3)
cppclass DrivingModelFrame:
int buf_size
DrivingModelFrame(cl_device_id, cl_context, bint)
cppclass MonitoringModelFrame:
int buf_size
MonitoringModelFrame(cl_device_id, cl_context)
ModelFrame(cl_device_id, cl_context)
unsigned char * prepare(cl_mem, int, int, int, int, mat3, cl_mem*)

View File

@@ -4,12 +4,11 @@
import numpy as np
cimport numpy as cnp
from libc.string cimport memcpy
from libc.stdint cimport uintptr_t
from msgq.visionipc.visionipc cimport cl_mem
from msgq.visionipc.visionipc_pyx cimport VisionBuf, CLContext as BaseCLContext
from .commonmodel cimport CL_DEVICE_TYPE_DEFAULT, cl_get_device_id, cl_create_context
from .commonmodel cimport mat3, ModelFrame as cppModelFrame, DrivingModelFrame as cppDrivingModelFrame, MonitoringModelFrame as cppMonitoringModelFrame
from .commonmodel cimport mat3, ModelFrame as cppModelFrame
cdef class CLContext(BaseCLContext):
@@ -24,47 +23,23 @@ cdef class CLMem:
mem.mem = <cl_mem*> cmem
return mem
@property
def mem_address(self):
return <uintptr_t>(self.mem)
def cl_from_visionbuf(VisionBuf buf):
return CLMem.create(<void*>&buf.buf.buf_cl)
cdef class ModelFrame:
cdef cppModelFrame * frame
cdef int buf_size
def __cinit__(self, CLContext context):
self.frame = new cppModelFrame(context.device_id, context.context)
def __dealloc__(self):
del self.frame
def prepare(self, VisionBuf buf, float[:] projection):
def prepare(self, VisionBuf buf, float[:] projection, CLMem output):
cdef mat3 cprojection
memcpy(cprojection.v, &projection[0], 9*sizeof(float))
cdef cl_mem * data
data = self.frame.prepare(buf.buf.buf_cl, buf.width, buf.height, buf.stride, buf.uv_offset, cprojection)
return CLMem.create(data)
def buffer_from_cl(self, CLMem in_frames):
cdef unsigned char * data2
data2 = self.frame.buffer_from_cl(in_frames.mem, self.buf_size)
return np.asarray(<cnp.uint8_t[:self.buf_size]> data2)
cdef class DrivingModelFrame(ModelFrame):
cdef cppDrivingModelFrame * _frame
def __cinit__(self, CLContext context, bint is_20hz=False):
self._frame = new cppDrivingModelFrame(context.device_id, context.context, is_20hz)
self.frame = <cppModelFrame*>(self._frame)
self.buf_size = self._frame.buf_size
cdef class MonitoringModelFrame(ModelFrame):
cdef cppMonitoringModelFrame * _frame
def __cinit__(self, CLContext context):
self._frame = new cppMonitoringModelFrame(context.device_id, context.context)
self.frame = <cppModelFrame*>(self._frame)
self.buf_size = self._frame.buf_size
cdef unsigned char * data
if output is None:
data = self.frame.prepare(buf.buf.buf_cl, buf.width, buf.height, buf.stride, buf.uv_offset, cprojection, NULL)
else:
data = self.frame.prepare(buf.buf.buf_cl, buf.width, buf.height, buf.stride, buf.uv_offset, cprojection, output.mem)
if not data:
return None
return np.asarray(<cnp.uint8_t[:self.frame.buf_size]> data)

Binary file not shown.

View File

@@ -85,7 +85,6 @@ class Parser:
outs[name + '_stds'] = pred_std_final.reshape(final_shape)
def parse_outputs(self, outs: dict[str, np.ndarray]) -> dict[str, np.ndarray]:
""" Parse the model outputs into a dictionary of numpy arrays. The input_keys are used to determine how the output should be parsed. """
self.parse_mdn('plan', outs, in_N=ModelConstants.PLAN_MHP_N, out_N=ModelConstants.PLAN_MHP_SELECTION,
out_shape=(ModelConstants.IDX_N,ModelConstants.PLAN_WIDTH))
self.parse_mdn('lane_lines', outs, in_N=0, out_N=0, out_shape=(ModelConstants.NUM_LANE_LINES,ModelConstants.IDX_N,ModelConstants.LANE_LINES_WIDTH))

View File

@@ -0,0 +1,27 @@
import os
from openpilot.system.hardware import TICI
from openpilot.selfdrive.modeld.runners.runmodel_pyx import RunModel, Runtime
assert Runtime
USE_THNEED = int(os.getenv('USE_THNEED', str(int(TICI))))
USE_SNPE = int(os.getenv('USE_SNPE', str(int(TICI))))
class ModelRunner(RunModel):
THNEED = 'THNEED'
SNPE = 'SNPE'
ONNX = 'ONNX'
def __new__(cls, paths, *args, **kwargs):
if ModelRunner.THNEED in paths and USE_THNEED:
from openpilot.selfdrive.modeld.runners.thneedmodel_pyx import ThneedModel as Runner
runner_type = ModelRunner.THNEED
elif ModelRunner.SNPE in paths and USE_SNPE:
from openpilot.selfdrive.modeld.runners.snpemodel_pyx import SNPEModel as Runner
runner_type = ModelRunner.SNPE
elif ModelRunner.ONNX in paths:
from openpilot.selfdrive.modeld.runners.onnxmodel import ONNXModel as Runner
runner_type = ModelRunner.ONNX
else:
raise Exception("Couldn't select a model runner, make sure to pass at least one valid model path")
return Runner(str(paths[runner_type]), *args, **kwargs)

View File

@@ -1,120 +0,0 @@
import os
from openpilot.system.hardware import TICI
from openpilot.sunnypilot.modeld.run_helpers import get_custom_model_paths
#
from tinygrad.tensor import Tensor, dtypes
from openpilot.selfdrive.modeld.runners.tinygrad_helpers import qcom_tensor_from_opencl_address
from openpilot.selfdrive.modeld.runners.ort_helpers import make_onnx_cpu_runner, ORT_TYPES_TO_NP_TYPES
import pickle
import numpy as np
from pathlib import Path
from abc import ABC, abstractmethod
from openpilot.selfdrive.modeld.models.commonmodel_pyx import DrivingModelFrame, CLMem
from openpilot.system.hardware import PC
if TICI:
os.environ['QCOM'] = '1'
SEND_RAW_PRED = os.getenv('SEND_RAW_PRED')
MODEL_PATH = Path(__file__).parent / '../models/supercombo.onnx'
MODEL_PKL_PATH = Path(__file__).parent / '../models/supercombo_tinygrad.pkl'
METADATA_PATH = Path(__file__).parent / '../models/supercombo_metadata.pkl'
USE_ONNX = os.getenv('USE_ONNX', PC)
class ModelRunner(ABC):
"""Abstract base class for model runners that defines the interface for running ML models."""
def __init__(self):
"""Initialize the model runner with paths to model and metadata files."""
self.model_paths = ({"model": MODEL_PATH, "metadata": METADATA_PATH} if USE_ONNX else
get_custom_model_paths() or {"model": MODEL_PKL_PATH, "metadata": METADATA_PATH})
with open(self.model_paths["metadata"], 'rb') as f:
self.model_metadata = pickle.load(f)
self.input_shapes = self.model_metadata['input_shapes']
self.output_slices = self.model_metadata['output_slices']
self.inputs: dict = {}
@abstractmethod
def prepare_inputs(self, imgs_cl: dict[str, CLMem], numpy_inputs: dict[str, np.ndarray]) -> dict:
"""Prepare inputs for model inference."""
@abstractmethod
def run_model(self):
"""Run model inference with prepared inputs."""
def slice_outputs(self, model_outputs: np.ndarray) -> dict:
"""Slice model outputs according to metadata configuration."""
parsed_outputs = {k: model_outputs[np.newaxis, v] for k, v in self.output_slices.items()}
if SEND_RAW_PRED:
parsed_outputs['raw_pred'] = model_outputs.copy()
return parsed_outputs
class TinygradRunner(ModelRunner):
"""Tinygrad implementation of model runner for TICI hardware."""
def __init__(self, frames: dict[str, DrivingModelFrame] | None = None):
super().__init__()
if not str(self.model_paths["model"]).endswith("_tinygrad.pkl"):
raise ValueError(f"Tinygrad model must be a _tinygrad.pkl file, we got {self.model_paths['model']}")
# Load Tinygrad model
with open(self.model_paths["model"], "rb") as f:
self.model_run = pickle.load(f)
self.input_to_dtype = {}
self.input_to_device = {}
for idx, name in enumerate(self.model_run.captured.expected_names):
self.input_to_dtype[name] = self.model_run.captured.expected_st_vars_dtype_device[idx][2] # 2 is the dtype
self.input_to_device[name] = self.model_run.captured.expected_st_vars_dtype_device[idx][3] # 3 is the device
assert TICI or frames is not None, "TinygradRunner requires frames for non-TICI hardware"
self.frames = frames
self.is_memory_model = None # Use None to indicate that it hasn't been determined yet
def prepare_inputs(self, imgs_cl: dict[str, CLMem], numpy_inputs: dict[str, np.ndarray]) -> dict:
# Initialize image tensors if not already done
for key in imgs_cl:
if TICI and key not in self.inputs:
self.inputs[key] = qcom_tensor_from_opencl_address(imgs_cl[key].mem_address, self.input_shapes[key], dtype=dtypes.uint8)
elif not TICI:
shape = self.frames[key].buffer_from_cl(imgs_cl[key]).reshape(self.input_shapes[key])
self.inputs[key] = Tensor(shape, device=self.input_to_device[key], dtype=self.input_to_dtype[key]).realize()
# Update numpy inputs
for key, value in numpy_inputs.items():
if key not in imgs_cl:
self.inputs[key] = Tensor(value, device=self.input_to_device[key], dtype=self.input_to_dtype[key]).realize()
return self.inputs
def run_model(self):
return self.model_run(**self.inputs).numpy().flatten()
class ONNXRunner(ModelRunner):
"""ONNX implementation of model runner for non-TICI hardware."""
def __init__(self, frames: dict[str, DrivingModelFrame]):
super().__init__()
self.runner = make_onnx_cpu_runner(self.model_paths["model"])
self.frames = frames
self.input_to_nptype = {
model_input.name: ORT_TYPES_TO_NP_TYPES[model_input.type]
for model_input in self.runner.get_inputs()
}
def prepare_inputs(self, imgs_cl: dict[str, CLMem], numpy_inputs: dict[str, np.ndarray]) -> dict:
self.inputs = numpy_inputs.copy()
for key in imgs_cl:
self.inputs[key] = self.frames[key].buffer_from_cl(imgs_cl[key]).reshape(self.input_shapes[key]).astype(dtype=np.float32)
return self.inputs
def run_model(self):
return self.runner.run(None, self.inputs)[0].flatten()

Some files were not shown because too many files have changed in this diff Show More