mawei/onepilot
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

sunnypilot v2026.002.000 release

Browse Source 
date: 2026-06-02T19:04:03
master commit: dfc3c98b226da57a653daf57131a8a3d66166fcb
This commit is contained in:
github-actions[bot]
2026-06-02 19:04:12 +08:00
commit 7c44d10fed
4111 changed files with 1168255 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
.codespellignore Normal file
View File

@@ -0,0 +1,6 @@
Wen
REGIST
PullRequest
cancelled
FOF
NoO

18
.dockerignore Normal file
View File

@@ -0,0 +1,18 @@
**/.git
.DS_Store
*.dylib
*.DSYM
*.d
*.pyc
*.pyo
.*.swp
.*.swo
.*.un~
*.tmp
*.o
*.o-*
*.os
*.os-*
venv/
.venv/

11
.editorconfig Normal file
View File

@@ -0,0 +1,11 @@
root = true
[*]
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
[*.{py,pyx,pxd}]
charset = utf-8
indent_style = space
indent_size = 2

11
.github/CODEOWNERS vendored Normal file
View File

@@ -0,0 +1,11 @@
* @sunnypilot/dev-internal
/.github/ @devtekve @sunnyhaibin
/release/ci/ @devtekve @sunnyhaibin
/tinygrad_repo @devtekve @Discountchubbs
/tinygrad/ @devtekve @Discountchubbs
/selfdrive/controls/lib/longitudinal_planner.py @devtekve @Discountchubbs
/selfdrive/controls/lib/longitudinal_mpc_lib/long_mpc.py @devtekve @Discountchubbs
/selfdrive/modeld/ @devtekve @Discountchubbs
/sunnypilot/model* @devtekve @Discountchubbs
/sunnypilot/sunnylink/ @devtekve
/system/athena/ @devtekve

47
.github/ISSUE_TEMPLATE/bug_report.yml vendored Normal file
View File

@@ -0,0 +1,47 @@
name: Bug report
description: For issues with running openpilot on your comma device
labels: ["bug"]
body:
- type: markdown
attributes:
value: >
Before creating a **bug report**, please check the following:
* If the issue likely only affects your car model or make, go back and open a **car bug report** instead.
* If the issue is related to the driving or driver monitoring models, you should open a [discussion](https://github.com/commaai/openpilot/discussions/categories/model-feedback) instead.
* Ensure you're running the latest openpilot release.
* Ensure you're using officially supported hardware. Issues running on PCs have a different issue template.
* Ensure there isn't an existing issue for your bug. If there is, leave a comment on the existing issue.
* Ensure you're running stock openpilot. We cannot look into bug reports from forks.
If you're unsure whether you've hit a bug, check out the #installation-help channel in the [community Discord server](https://discord.comma.ai).
- type: textarea
attributes:
label: Describe the bug
description: Also include a description of how to reproduce the bug
validations:
required: true
- type: input
id: route
attributes:
label: Provide a route where the issue occurs
description: Ensure the route is fully uploaded at https://useradmin.comma.ai. We cannot look into issues without routes, or at least a Dongle ID.
placeholder: 77611a1fac303767|2020-05-11--16-37-07
validations:
required: true
- type: input
id: version
attributes:
label: openpilot version
description: If you're not on release, provide the commit hash
placeholder: 0.8.10
validations:
required: true
- type: textarea
attributes:
label: Additional info

14
.github/ISSUE_TEMPLATE/config.yml vendored Normal file
View File

@@ -0,0 +1,14 @@
blank_issues_enabled: false
contact_links:
- name: Car bug report
url: https://github.com/commaai/opendbc/issues/new
about: For issues with a particular car make or model
- name: Join the Discord
url: https://discord.comma.ai
about: The community Discord is for both openpilot development and experience discussion
- name: Report driving behavior feedback
url: https://discord.com/channels/469524606043160576/1254834193066623017
about: Feedback for the driving and driver monitoring models goes in the #driving-feedback in Discord
- name: Community Wiki
url: https://github.com/commaai/openpilot/wiki
about: Check out our community wiki

8
.github/ISSUE_TEMPLATE/enhancement.md vendored Normal file
View File

@@ -0,0 +1,8 @@
---
name: Enhancement
about: For openpilot enhancement suggestions
title: ''
labels: 'enhancement'
assignees: ''
---

42
.github/ISSUE_TEMPLATE/pc_bug_report.yml vendored Normal file
View File

@@ -0,0 +1,42 @@
name: PC bug report
description: For issues with running openpilot on PC
labels: ["PC"]
body:
- type: markdown
attributes:
value: >
Before creating a **bug report**, please check the following:
* Ensure you're running the latest openpilot release.
* Ensure there isn't an existing issue for your bug. If there is, leave a comment on the existing issue.
* Ensure you're running stock openpilot. We cannot look into bug reports from forks.
If you're unsure whether you've hit a bug, check out the #installation-help channel in the [community Discord server](https://discord.comma.ai).
- type: textarea
attributes:
label: Describe the bug
description: Also include a description of how to reproduce the bug
validations:
required: true
- type: input
id: os-version
attributes:
label: OS Version
placeholder: Ubuntu 24.04
validations:
required: true
- type: input
id: version
attributes:
label: openpilot version or commit
placeholder: bd36f2ec8d3559909678eff2690c10a520938367
validations:
required: false
- type: textarea
attributes:
label: Additional info

31
.github/labeler.yaml vendored Normal file
View File

@@ -0,0 +1,31 @@
ci:
- changed-files:
- any-glob-to-all-files: "{.github/**,**/test_*,**/test/**,Jenkinsfile}"
chore:
- changed-files:
- any-glob-to-all-files: "{.github/**}"
car:
- changed-files:
- any-glob-to-all-files: '{selfdrive/car/**,opendbc_repo}'
simulation:
- changed-files:
- any-glob-to-all-files: 'tools/sim/**'
ui:
- changed-files:
- any-glob-to-all-files: '{selfdrive/assets/**,selfdrive/ui/**,system/ui/**}'
tools:
- changed-files:
- any-glob-to-all-files: 'tools/**'
multilanguage:
- changed-files:
- any-glob-to-all-files: 'selfdrive/ui/translations/**'
autonomy:
- changed-files:
- any-glob-to-all-files: "{selfdrive/modeld/models/**,selfdrive/test/process_replay/model_replay_ref_commit,sunnypilot/modeld*/models/**}"

68
.github/pull_request_template.md vendored Normal file
View File

@@ -0,0 +1,68 @@
<!-- Please copy and paste the relevant template -->
<!--- ***** Template: Fingerprint *****
**Car**
Which car (make, model, year) this fingerprint is for
**Route**
A route with the fingerprint
-->
<!--- ***** Template: Car Bugfix *****
**Description**
A description of the bug and the fix. Also link the issue if it exists.
**Verification**
Explain how you tested this bug fix.
**Route**
Route: [a route with the bug fix]
-->
<!--- ***** Template: Bugfix *****
**Description**
A description of the bug and the fix. Also link the issue if it exists.
**Verification**
Explain how you tested this bug fix.
-->
<!--- ***** Template: Car Port *****
**Checklist**
- [ ] added entry to CAR in selfdrive/car/*/values.py and ran `selfdrive/car/docs.py` to generate new docs
- [ ] test route added to [routes.py](https://github.com/commaai/openpilot/blob/master/selfdrive/car/tests/routes.py)
- [ ] route with openpilot:
- [ ] route with stock system:
- [ ] car harness used (if comma doesn't sell it, put N/A):
-->
<!--- ***** Template: Refactor *****
**Description**
A description of the refactor, including the goals it accomplishes.
**Verification**
Explain how you tested the refactor for regressions.
-->

43
.github/release-drafter.yml vendored Normal file
View File

@@ -0,0 +1,43 @@
exclude-labels:
- 'no-changelog'
categories:
- title: '🚀 Features'
labels:
- 'feature'
- 'enhancement'
- title: '🐛 Bug Fixes'
collapse-after: 5
labels:
- 'fix'
- 'bugfix'
- 'bug'
- title: '🧰 Maintenance'
collapse-after: 5
label: 'chore'
change-template: '- $TITLE @$AUTHOR (#$NUMBER)'
change-title-escapes: '\<*_&'
replacers:
- search: '/[Ss][Uu][Nn][Nn][Yy][Pp][Ii][Ll][Oo][Tt]/g'
replace: 'sunnypilot'
- search: '/\b[Ss][Pp]\b/g'
replace: 'SP'
version-resolver:
major:
labels:
- 'major'
minor:
labels:
- 'minor'
patch:
labels:
- 'patch'
default: patch
name-template: 'v$RESOLVED_VERSION 🚀'
tag-template: 'v$RESOLVED_VERSION'
version-template: "0.$MAJOR.$MINOR.$PATCH" # The day OP becomes v1, we need to bump this
tag-prefix: "v0." # The day OP becomes v1, we need to bump this
prerelease-identifier: "staging"
template: |
## Changes
$CHANGES

97
.gitignore vendored Normal file
View File

@@ -0,0 +1,97 @@
venv/
.venv/
.ci_cache
.env
.clang-format
.DS_Store
.tags
.ipynb_checkpoints
.idea
.overlay_init
.overlay_consistent
.sconsign.dblite
a.out
.hypothesis
.cache/
bin/
*.mp4
*.dylib
*.DSYM
*.d
*.pem
*.pyc
*.pyo
.*.swp
.*.swo
.*.un~
*.tmp
*.o
*.o-*
*.os
*.os-*
*.so
*.a
*.clb
*.class
*.pyxbldc
*.vcd
*.mo
*_pyx.cpp
*.stats
*.pkl
*.pkl*
config.json
compile_commands.json
compare_runtime*.html
selfdrive/modeld/models/tg_compiled_flags.json
# build artifacts
docs_site/
selfdrive/pandad/pandad
cereal/services.h
cereal/gen
cereal/messaging/bridge
selfdrive/ui/translations/tmp
selfdrive/car/tests/cars_dump
system/camerad/camerad
system/camerad/test/ae_gray_test
.coverage*
coverage.xml
htmlcov
# openpilot log files
*.bz2
*.zst
*.rlog
build/
!**/.gitkeep
### VisualStudioCode ###
*.vsix
.history
.ionide
.vscode/*
.history/
!.vscode/settings.json
!.vscode/tasks.json
!.vscode/launch.json
!.vscode/extensions.json
!.vscode/*.code-snippets
# agents
.claude/
.context/
PLAN.md
TASK.md
CLAUDE.md
SKILL.md
### JetBrains ###
!.idea/customTargets.xml
!.idea/tools/*
!.run/*

1
.python-version Normal file
View File

@@ -0,0 +1 @@
3.12.13

8
.vscode/extensions.json vendored Normal file
View File

@@ -0,0 +1,8 @@
{
"recommendations": [
"ms-python.python",
"ms-vscode.cpptools",
"elagil.pre-commit-helper",
"charliermarsh.ruff",
]
}

88
.vscode/launch.json vendored Normal file
View File

@@ -0,0 +1,88 @@
{
"version": "0.2.0",
"inputs": [
{
"id": "python_process",
"type": "pickString",
"description": "Select the process to debug",
"options": [
"selfdrive/controls/controlsd.py",
"system/timed/timed.py",
"tools/sim/run_bridge.py"
]
},
{
"id": "cpp_process",
"type": "pickString",
"description": "Select the process to debug",
"options": [
"selfdrive/ui/ui"
]
},
{
"id": "args",
"description": "Arguments to pass to the process",
"type": "promptString"
},
{
"id": "replayArg",
"type": "promptString",
"description": "Enter route or segment to replay."
}
],
"configurations": [
{
"name": "Python: openpilot Process",
"type": "debugpy",
"request": "launch",
"program": "${input:python_process}",
"console": "integratedTerminal",
"justMyCode": true,
"args": "${input:args}"
},
{
"name": "C++: openpilot Process",
"type": "cppdbg",
"request": "launch",
"program": "${workspaceFolder}/${input:cpp_process}",
"cwd": "${workspaceFolder}"
},
{
"name": "Attach LLDB to Replay drive",
"type": "lldb",
"request": "attach",
"pid": "${command:pickMyProcess}",
"sourceMap": {
".": "${workspaceFolder}/opendbc/safety"
},
"initCommands": [
"script import time; time.sleep(3)"
]
},
{
"name": "Replay drive",
"type": "debugpy",
"request": "launch",
"program": "${workspaceFolder}/opendbc/safety/tests/safety_replay/replay_drive.py",
"args": [
"${input:replayArg}"
],
"console": "integratedTerminal",
"justMyCode": false,
"env": {
"PYTHONPATH": "${workspaceFolder}"
},
"subProcess": true,
"stopOnEntry": false
}
],
"compounds": [
{
"name": "Replay drive + Safety LLDB",
"configurations": [
"Replay drive",
"Attach LLDB to Replay drive"
]
}
]
}

27
.vscode/settings.json vendored Normal file
View File

@@ -0,0 +1,27 @@
{
"editor.tabSize": 2,
"editor.insertSpaces": true,
"editor.renderWhitespace": "trailing",
"files.trimTrailingWhitespace": true,
"search.exclude": {
"**/.git": true,
"**/.venv": true,
"**/__pycache__": true
},
"files.exclude": {
"**/.git": true,
"**/.venv": true,
"**/__pycache__": true
},
"python.analysis.exclude": [
"**/.git",
"**/.venv",
"**/__pycache__",
// exclude directories that should be using the symlinked version
"common/**",
"selfdrive/**",
"system/**",
"third_party/**",
"tools/**",
]
}

1291
CHANGELOG.md Normal file
View File

File diff suppressed because it is too large Load Diff

38
Dockerfile.openpilot Normal file
View File

@@ -0,0 +1,38 @@
FROM ubuntu:24.04
ENV PYTHONUNBUFFERED=1
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -y --no-install-recommends sudo tzdata locales && \
rm -rf /var/lib/apt/lists/*
RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && locale-gen
ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US:en
ENV LC_ALL=en_US.UTF-8
ENV NVIDIA_VISIBLE_DEVICES=all
ENV NVIDIA_DRIVER_CAPABILITIES=graphics,utility,compute
ARG USER=batman
ARG USER_UID=1001
RUN useradd -m -s /bin/bash -u $USER_UID $USER
RUN usermod -aG sudo $USER
RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
USER $USER
ENV OPENPILOT_PATH=/home/$USER/openpilot
RUN mkdir -p ${OPENPILOT_PATH}
WORKDIR ${OPENPILOT_PATH}
COPY --chown=$USER . ${OPENPILOT_PATH}/
ENV UV_BIN="/home/$USER/.local/bin/"
ENV VIRTUAL_ENV=${OPENPILOT_PATH}/.venv
ENV PATH="$UV_BIN:$VIRTUAL_ENV/bin:$PATH"
RUN tools/setup_dependencies.sh && \
sudo rm -rf /var/lib/apt/lists/*
USER root
RUN git config --global --add safe.directory '*'

260
Jenkinsfile vendored Normal file
View File

@@ -0,0 +1,260 @@
def retryWithDelay(int maxRetries, int delay, Closure body) {
for (int i = 0; i < maxRetries; i++) {
try {
return body()
} catch (Exception e) {
sleep(delay)
}
}
throw Exception("Failed after ${maxRetries} retries")
}
def device(String ip, String step_label, String cmd) {
withCredentials([file(credentialsId: 'id_rsa', variable: 'key_file')]) {
def ssh_cmd = """
ssh -o ConnectTimeout=5 -o ServerAliveInterval=5 -o ServerAliveCountMax=2 -o BatchMode=yes -o StrictHostKeyChecking=no -i ${key_file} 'comma@${ip}' exec /usr/bin/bash <<'END'
set -e
export TERM=xterm-256color
shopt -s huponexit # kill all child processes when the shell exits
export CI=1
export PYTHONWARNINGS=error
#export LOGPRINT=debug # this has gotten too spammy...
export TEST_DIR=${env.TEST_DIR}
export SOURCE_DIR=${env.SOURCE_DIR}
export GIT_BRANCH=${env.GIT_BRANCH}
export GIT_COMMIT=${env.GIT_COMMIT}
export CI_ARTIFACTS_TOKEN=${env.CI_ARTIFACTS_TOKEN}
export GITHUB_COMMENTS_TOKEN=${env.GITHUB_COMMENTS_TOKEN}
export AZURE_TOKEN='${env.AZURE_TOKEN}'
# only use 1 thread for tici tests since most require HIL
export PYTEST_ADDOPTS="-n0 -s"
export GIT_SSH_COMMAND="ssh -i /data/gitkey"
source ~/.bash_profile
if [ -f /TICI ]; then
source /etc/profile
rm -rf /tmp/tmp*
rm -rf ~/.commacache
rm -rf /dev/shm/*
rm -rf /dev/tmp/tmp*
if ! systemctl is-active --quiet systemd-resolved; then
echo "restarting resolved"
sudo systemctl start systemd-resolved
sleep 3
fi
# restart aux USB
if [ -e /sys/bus/usb/drivers/hub/3-0:1.0 ]; then
echo "restarting aux usb"
echo "3-0:1.0" | sudo tee /sys/bus/usb/drivers/hub/unbind
sleep 0.5
echo "3-0:1.0" | sudo tee /sys/bus/usb/drivers/hub/bind
fi
fi
if [ -f /data/openpilot/launch_env.sh ]; then
source /data/openpilot/launch_env.sh
fi
ln -snf ${env.TEST_DIR} /data/pythonpath
cd ${env.TEST_DIR} || true
time ${cmd}
END"""
sh script: ssh_cmd, label: step_label
}
}
def deviceStage(String stageName, String deviceType, List extra_env, def steps) {
stage(stageName) {
if (currentBuild.result != null) {
return
}
if (isReplay()) {
error("REPLAYING TESTS IS NOT ALLOWED. FIX THEM INSTEAD.")
}
def extra = extra_env.collect { "export ${it}" }.join('\n');
def branch = env.BRANCH_NAME ?: 'master';
def gitDiff = sh returnStdout: true, script: 'curl -s -H "Authorization: Bearer ${GITHUB_COMMENTS_TOKEN}" https://api.github.com/repos/commaai/openpilot/compare/master...${GIT_BRANCH} | jq .files[].filename || echo "/"', label: 'Getting changes'
lock(resource: "", label: deviceType, inversePrecedence: true, variable: 'device_ip', quantity: 1, resourceSelectStrategy: 'random') {
docker.image('ghcr.io/commaai/alpine-ssh').inside('--user=root') {
timeout(time: 35, unit: 'MINUTES') {
retry (3) {
def date = sh(script: 'date', returnStdout: true).trim();
device(device_ip, "set time", "date -s '" + date + "'")
device(device_ip, "git checkout", extra + "\n" + readFile("selfdrive/test/setup_device_ci.sh"))
}
steps.each { item ->
def name = item[0]
def cmd = item[1]
def args = item[2]
def diffPaths = args.diffPaths ?: []
def cmdTimeout = args.timeout ?: 9999
if (branch != "master" && !branch.contains("__jenkins_loop_") && diffPaths && !hasPathChanged(gitDiff, diffPaths)) {
println "Skipping ${name}: no changes in ${diffPaths}."
return
} else {
timeout(time: cmdTimeout, unit: 'SECONDS') {
device(device_ip, name, cmd)
}
}
}
}
}
}
}
}
def hasPathChanged(String gitDiff, List<String> paths) {
for (path in paths) {
if (gitDiff.contains(path)) {
return true
}
}
return false
}
def isReplay() {
def replayClass = "org.jenkinsci.plugins.workflow.cps.replay.ReplayCause"
return currentBuild.rawBuild.getCauses().any{ cause -> cause.toString().contains(replayClass) }
}
def setupCredentials() {
withCredentials([
string(credentialsId: 'azure_token', variable: 'AZURE_TOKEN'),
]) {
env.AZURE_TOKEN = "${AZURE_TOKEN}"
}
withCredentials([
string(credentialsId: 'ci_artifacts_pat', variable: 'CI_ARTIFACTS_TOKEN'),
]) {
env.CI_ARTIFACTS_TOKEN = "${CI_ARTIFACTS_TOKEN}"
}
withCredentials([
string(credentialsId: 'post_comments_github_pat', variable: 'GITHUB_COMMENTS_TOKEN'),
]) {
env.GITHUB_COMMENTS_TOKEN = "${GITHUB_COMMENTS_TOKEN}"
}
}
def step(String name, String cmd, Map args = [:]) {
return [name, cmd, args]
}
node {
env.CI = "1"
env.PYTHONWARNINGS = "error"
env.TEST_DIR = "/data/openpilot"
env.SOURCE_DIR = "/data/openpilot_source/"
setupCredentials()
env.GIT_BRANCH = checkout(scm).GIT_BRANCH
env.GIT_COMMIT = checkout(scm).GIT_COMMIT
def excludeBranches = ['__nightly', 'devel', 'devel-staging', 'release3', 'release3-staging',
'release-tici', 'release-tizi', 'release-tizi-staging', 'release-mici-staging', 'testing-closet*', 'hotfix-*']
def excludeRegex = excludeBranches.join('|').replaceAll('\\*', '.*')
if (env.BRANCH_NAME != 'master' && !env.BRANCH_NAME.contains('__jenkins_loop_')) {
properties([
disableConcurrentBuilds(abortPrevious: true)
])
}
try {
if (env.BRANCH_NAME == 'devel-staging') {
deviceStage("build release-tizi-staging", "tizi-needs-can", [], [
step("build release-tizi-staging", "RELEASE_BRANCH=release-tizi-staging $SOURCE_DIR/release/build_release.sh && git push -f origin release-tizi-staging:release-mici-staging"),
])
}
if (env.BRANCH_NAME == '__nightly') {
parallel (
'nightly': {
deviceStage("build nightly", "tizi-needs-can", [], [
step("build nightly", "RELEASE_BRANCH=nightly $SOURCE_DIR/release/build_release.sh"),
])
},
'nightly-dev': {
deviceStage("build nightly-dev", "tizi-needs-can", [], [
step("build nightly-dev", "PANDA_DEBUG_BUILD=1 RELEASE_BRANCH=nightly-dev $SOURCE_DIR/release/build_release.sh"),
])
},
)
}
if (!env.BRANCH_NAME.matches(excludeRegex)) {
parallel (
'onroad tests': {
deviceStage("onroad", "tizi-needs-can", ["UNSAFE=1"], [
step("build openpilot", "cd system/manager && ./build.py"),
step("check dirty", "release/check-dirty.sh"),
step("onroad tests", "pytest selfdrive/test/test_onroad.py -s", [timeout: 60]),
])
},
'HW + Unit Tests': {
deviceStage("tizi-hardware", "tizi-common", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test power draw", "pytest -s system/hardware/tici/tests/test_power_draw.py"),
step("test encoder", "LD_LIBRARY_PATH=/usr/local/lib pytest system/loggerd/tests/test_encoder.py", [diffPaths: ["system/loggerd/"]]),
step("test manager", "pytest system/manager/test/test_manager.py"),
])
},
'camerad OX03C10': {
deviceStage("OX03C10", "tizi-ox03c10", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test pandad", "pytest selfdrive/pandad/tests/test_pandad.py"),
step("test camerad", "pytest system/camerad/test/test_camerad.py", [timeout: 90]),
])
},
'camerad OS04C10': {
deviceStage("OS04C10", "tici-os04c10", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test pandad", "pytest selfdrive/pandad/tests/test_pandad.py"),
step("test camerad", "pytest system/camerad/test/test_camerad.py", [timeout: 90]),
])
},
'sensord': {
deviceStage("LSM + MMC", "tizi-lsmc", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py"),
step("test sensord", "pytest system/sensord/tests/test_sensord.py"),
])
},
'replay': {
deviceStage("model-replay", "tizi-replay", ["UNSAFE=1"], [
step("build", "cd system/manager && ./build.py", [diffPaths: ["selfdrive/modeld/", "tinygrad_repo", "selfdrive/test/process_replay/model_replay.py"]]),
step("model replay", "selfdrive/test/process_replay/model_replay.py", [diffPaths: ["selfdrive/modeld/", "tinygrad_repo", "selfdrive/test/process_replay/model_replay.py"]]),
])
},
'tizi': {
deviceStage("tizi", "tizi", ["UNSAFE=1"], [
step("build openpilot", "cd system/manager && ./build.py"),
step("test pandad loopback", "pytest selfdrive/pandad/tests/test_pandad_loopback.py"),
step("test pandad spi", "pytest selfdrive/pandad/tests/test_pandad_spi.py"),
step("test amp", "pytest system/hardware/tici/tests/test_amplifier.py"),
step("test qcomgpsd", "pytest system/qcomgpsd/tests/test_qcomgpsd.py", [diffPaths: ["system/qcomgpsd/"]]),
])
},
)
}
} catch (Exception e) {
currentBuild.result = 'FAILED'
throw e
}
}

7
LICENSE Normal file
View File

@@ -0,0 +1,7 @@
Copyright (c) 2018, Comma.ai, Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

21
LICENSE.md Normal file
View File

@@ -0,0 +1,21 @@
# Custom MIT License
Copyright (c) 2024, Haibin Wen, SUNNYPILOT LLC
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to view and modify the Software, subject to the following conditions:
1. **Permission Required**: Permission Required for Commercial, For-Profit, or Closed Source Use: Use of the Software, in whole or in part, for any commercial purposes, for-profit projects, or in closed source projects requires explicit written permission from the original author(s).
2. **Redistribution**: Any redistribution of the Software, modified or unmodified, must retain this license notice and the following acknowledgment:
"This software is licensed under a custom license requiring permission for use."
3. **Visibility**: Any project that uses the Software must visibly mention the following acknowledgment:
"This project uses software from Haibin Wen and SUNNYPILOT LLC and is licensed under a custom license requiring permission for use."
4. **No Warranty**: THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Contact sunnypilot Support <support@sunnypilot.ai> for permission requests.
---
Haibin Wen, SUNNYPILOT LLC

74
README.md Normal file
View File

@@ -0,0 +1,74 @@
![](https://user-images.githubusercontent.com/47793918/233812617-beab2e71-57b9-479e-8bff-c3931347ca40.png)
## 🌞 What is sunnypilot?
[sunnypilot](https://github.com/sunnyhaibin/sunnypilot) is a fork of comma.ai's openpilot, an open source driver assistance system. sunnypilot offers the user a unique driving experience for over 300+ supported car makes and models with modified behaviors of driving assist engagements. sunnypilot complies with comma.ai's safety rules as accurately as possible.
## 💭 Join our Community Forum
Join the official sunnypilot community forum to stay up to date with all the latest features and be a part of shaping the future of sunnypilot!
* https://community.sunnypilot.ai/
## Documentation
https://docs.sunnypilot.ai/ is your one stop shop for everything from features to installation to FAQ about the sunnypilot
## 🚘 Running on a dedicated device in a car
First, check out this list of items you'll need to [get started](https://community.sunnypilot.ai/t/getting-started-using-sunnypilot-in-your-supported-car/251).
## Installation
Next, refer to the sunnypilot community forum for [installation instructions](https://community.sunnypilot.ai/t/read-before-installing-sunnypilot/254), as well as a complete list of [Recommended Branch Installations](https://community.sunnypilot.ai/t/recommended-branch-installations/235).
## 🎆 Pull Requests
We welcome both pull requests and issues on GitHub. Bug fixes are encouraged.
Pull requests should be against the most current `master` branch.
## 📊 User Data
By default, sunnypilot uploads the driving data to comma servers. You can also access your data through [comma connect](https://connect.comma.ai/).
sunnypilot is open source software. The user is free to disable data collection if they wish to do so.
sunnypilot logs the road-facing camera, CAN, GPS, IMU, magnetometer, thermal sensors, crashes, and operating system logs.
The driver-facing camera and microphone are only logged if you explicitly opt-in in settings.
By using this software, you understand that use of this software or its related services will generate certain types of user data, which may be logged and stored at the sole discretion of comma. By accepting this agreement, you grant an irrevocable, perpetual, worldwide right to comma for the use of this data.
## Licensing
sunnypilot is released under the [MIT License](LICENSE). This repository includes original work as well as significant portions of code derived from [openpilot by comma.ai](https://github.com/commaai/openpilot), which is also released under the MIT license with additional disclaimers.
The original openpilot license notice, including comma.ai’s indemnification and alpha software disclaimer, is reproduced below as required:
> openpilot is released under the MIT license. Some parts of the software are released under other licenses as specified.
>
> Any user of this software shall indemnify and hold harmless Comma.ai, Inc. and its directors, officers, employees, agents, stockholders, affiliates, subcontractors and customers from and against all allegations, claims, actions, suits, demands, damages, liabilities, obligations, losses, settlements, judgments, costs and expenses (including without limitation attorneys’ fees and costs) which arise out of, relate to or result from any use of this software by user.
>
> **THIS IS ALPHA QUALITY SOFTWARE FOR RESEARCH PURPOSES ONLY. THIS IS NOT A PRODUCT.
> YOU ARE RESPONSIBLE FOR COMPLYING WITH LOCAL LAWS AND REGULATIONS.
> NO WARRANTY EXPRESSED OR IMPLIED.**
For full license terms, please see the [`LICENSE`](LICENSE) file.
## 💰 Support sunnypilot
If you find any of the features useful, consider becoming a [sponsor on GitHub](https://github.com/sponsors/sunnyhaibin) to support future feature development and improvements.
By becoming a sponsor, you will gain access to exclusive content, early access to new features, and the opportunity to directly influence the project's development.
<h3>GitHub Sponsor</h3>
<a href="https://github.com/sponsors/sunnyhaibin">
<img src="https://user-images.githubusercontent.com/47793918/244135584-9800acbd-69fd-4b2b-bec9-e5fa2d85c817.png" alt="Become a Sponsor" width="300" style="max-width: 100%; height: auto;">
</a>
<br>
<h3>PayPal</h3>
<a href="https://paypal.me/sunnyhaibin0850" target="_blank">
<img src="https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif" alt="PayPal this" title="PayPal - The safer, easier way to pay online!" border="0" />
</a>
<br></br>
Your continuous love and support are greatly appreciated! Enjoy 🥰
<span>-</span> Jason, Founder of sunnypilot

1116
RELEASES.md Normal file
View File

File diff suppressed because it is too large Load Diff

269
SConstruct Normal file
View File

@@ -0,0 +1,269 @@
import os
import subprocess
import sys
import sysconfig
import platform
import shlex
import importlib
import numpy as np
import SCons.Errors
from SCons.Defaults import _stripixes
SCons.Warnings.warningAsException(True)
Decider('MD5-timestamp')
SetOption('num_jobs', max(1, int(os.cpu_count()/2)))
AddOption('--ccflags', action='store', type='string', default='', help='pass arbitrary flags over the command line')
AddOption('--verbose', action='store_true', default=False, help='show full build commands')
AddOption('--minimal',
action='store_false',
dest='extras',
default=os.path.exists(File('#.gitattributes').abspath), # minimal by default on release branch (where there's no LFS)
help='the minimum build to run openpilot. no tests, tools, etc.')
# Detect platform
arch = subprocess.check_output(["uname", "-m"], encoding='utf8').rstrip()
if platform.system() == "Darwin":
arch = "Darwin"
elif arch == "aarch64" and os.path.isfile('/TICI'):
arch = "larch64"
assert arch in [
"larch64", # linux tici arm64
"aarch64", # linux pc arm64
"x86_64", # linux pc x64
"Darwin", # macOS arm64 (x86 not supported)
]
pkg_names = ['bzip2', 'capnproto', 'eigen', 'ffmpeg', 'libjpeg', 'libyuv', 'ncurses', 'zeromq', 'zstd']
pkgs = [importlib.import_module(name) for name in pkg_names]
# ***** enforce a whitelist of system libraries *****
# this prevents silently relying on a 3rd party package,
# e.g. apt-installed libusb. all libraries should either
# be distributed with all Linux distros and macOS, or
# vendored in commaai/dependencies.
allowed_system_libs = {
"EGL", "GLESv2", "GL",
"Qt5Charts", "Qt5Core", "Qt5Gui", "Qt5Widgets",
"dl", "drm", "gbm", "m", "pthread",
}
def _resolve_lib(env, name):
for d in env.Flatten(env.get('LIBPATH', [])):
p = Dir(str(d)).abspath
for ext in ('.a', '.so', '.dylib'):
f = File(os.path.join(p, f'lib{name}{ext}'))
if f.exists() or f.has_builder():
return name
if name in allowed_system_libs:
return name
raise SCons.Errors.UserError(f"Unexpected non-vendored library '{name}'")
def _libflags(target, source, env, for_signature):
libs = []
lp = env.subst('$LIBLITERALPREFIX')
for lib in env.Flatten(env.get('LIBS', [])):
if isinstance(lib, str):
if os.sep in lib or lib.startswith('#'):
libs.append(File(lib))
elif lib.startswith('-') or (lp and lib.startswith(lp)):
libs.append(lib)
else:
libs.append(_resolve_lib(env, lib))
else:
libs.append(lib)
return _stripixes(env['LIBLINKPREFIX'], libs, env['LIBLINKSUFFIX'],
env['LIBPREFIXES'], env['LIBSUFFIXES'], env, env['LIBLITERALPREFIX'])
env = Environment(
ENV={
"PATH": os.environ['PATH'],
"PYTHONPATH": Dir("#").abspath + ':' + Dir(f"#third_party/acados").abspath,
"ACADOS_SOURCE_DIR": Dir("#third_party/acados").abspath,
"ACADOS_PYTHON_INTERFACE_PATH": Dir("#third_party/acados/acados_template").abspath,
"TERA_PATH": Dir("#").abspath + f"/third_party/acados/{arch}/t_renderer"
},
CCFLAGS=[
"-g",
"-fPIC",
"-O2",
"-Wunused",
"-Werror",
"-Wshadow" if arch in ("Darwin", "larch64") else "-Wshadow=local",
"-Wno-unknown-warning-option",
"-Wno-inconsistent-missing-override",
"-Wno-c99-designator",
"-Wno-reorder-init-list",
"-Wno-vla-cxx-extension",
],
CFLAGS=["-std=gnu11"],
CXXFLAGS=["-std=c++1z"],
CPPPATH=[
"#",
"#msgq",
"#third_party",
"#third_party/json11",
"#third_party/linux/include",
"#third_party/acados/include",
"#third_party/acados/include/blasfeo/include",
"#third_party/acados/include/hpipm/include",
"#third_party/catch2/include",
[x.INCLUDE_DIR for x in pkgs],
],
LIBPATH=[
"#common",
"#msgq_repo",
"#third_party",
"#selfdrive/pandad",
"#rednose/helpers",
f"#third_party/acados/{arch}/lib",
[x.LIB_DIR for x in pkgs],
],
RPATH=[],
CYTHONCFILESUFFIX=".cpp",
COMPILATIONDB_USE_ABSPATH=True,
REDNOSE_ROOT="#",
tools=["default", "cython", "compilation_db", "rednose_filter"],
toolpath=["#site_scons/site_tools", "#rednose_repo/site_scons/site_tools"],
)
if arch != "larch64":
env['_LIBFLAGS'] = _libflags
# Arch-specific flags and paths
if arch == "larch64":
env["CC"] = "clang"
env["CXX"] = "clang++"
env.Append(LIBPATH=[
"/usr/lib/aarch64-linux-gnu",
])
arch_flags = ["-D__TICI__", "-mcpu=cortex-a57", "-DQCOM2"]
env.Append(CCFLAGS=arch_flags)
env.Append(CXXFLAGS=arch_flags)
elif arch == "Darwin":
env.Append(LIBPATH=[
"/System/Library/Frameworks/OpenGL.framework/Libraries",
])
env.Append(CCFLAGS=["-DGL_SILENCE_DEPRECATION"])
env.Append(CXXFLAGS=["-DGL_SILENCE_DEPRECATION"])
_extra_cc = shlex.split(GetOption('ccflags') or '')
if _extra_cc:
env.Append(CCFLAGS=_extra_cc)
# no --as-needed on mac linker
if arch != "Darwin":
env.Append(LINKFLAGS=["-Wl,--as-needed", "-Wl,--no-undefined"])
# Shorter build output: show brief descriptions instead of full commands.
# Full command lines are still printed on failure by scons.
if not GetOption('verbose'):
for action, short in (
("CC", "CC"),
("CXX", "CXX"),
("LINK", "LINK"),
("SHCC", "CC"),
("SHCXX", "CXX"),
("SHLINK", "LINK"),
("AR", "AR"),
("RANLIB", "RANLIB"),
("AS", "AS"),
):
env[f"{action}COMSTR"] = f" [{short}] $TARGET"
# progress output
node_interval = 5
node_count = 0
def progress_function(node):
global node_count
node_count += node_interval
sys.stderr.write("progress: %d\n" % node_count)
if os.environ.get('SCONS_PROGRESS'):
Progress(progress_function, interval=node_interval)
# ********** Cython build environment **********
envCython = env.Clone()
envCython["CPPPATH"] += [sysconfig.get_paths()['include'], np.get_include()]
envCython["CCFLAGS"] += ["-Wno-#warnings", "-Wno-cpp", "-Wno-shadow", "-Wno-deprecated-declarations"]
envCython["CCFLAGS"].remove("-Werror")
envCython["LIBS"] = []
if arch == "Darwin":
envCython["LINKFLAGS"] = env["LINKFLAGS"] + ["-bundle", "-undefined", "dynamic_lookup"]
else:
envCython["LINKFLAGS"] = ["-pthread", "-shared"]
np_version = SCons.Script.Value(np.__version__)
Export('envCython', 'np_version')
Export('env', 'arch')
# Setup cache dir
default_cache_dir = '/data/scons_cache' if arch == "larch64" else '/tmp/scons_cache'
cache_dir = ARGUMENTS.get('cache_dir', default_cache_dir)
CacheDir(cache_dir)
Clean(["."], cache_dir)
# ********** start building stuff **********
# Build common module
SConscript(['common/SConscript'])
Import('_common')
common = [_common, 'json11', 'zmq']
Export('common')
# Build messaging (cereal + msgq + socketmaster + their dependencies)
# Enable swaglog include in submodules
env_swaglog = env.Clone()
env_swaglog['CXXFLAGS'].append('-DSWAGLOG="\\"common/swaglog.h\\""')
SConscript(['msgq_repo/SConscript'], exports={'env': env_swaglog})
SConscript(['cereal/SConscript'])
Import('socketmaster', 'msgq')
messaging = [socketmaster, msgq, 'capnp', 'kj',]
Export('messaging')
# Build other submodules
SConscript(['panda/SConscript'])
# Build rednose library
SConscript(['rednose/SConscript'])
# Build system services
SConscript([
'system/loggerd/SConscript',
])
if arch == "larch64":
SConscript(['system/camerad/SConscript'])
# Build openpilot
SConscript(['third_party/SConscript'])
# Build selfdrive
SConscript([
'selfdrive/pandad/SConscript',
'selfdrive/controls/lib/lateral_mpc_lib/SConscript',
'selfdrive/controls/lib/longitudinal_mpc_lib/SConscript',
'selfdrive/locationd/SConscript',
'selfdrive/modeld/SConscript',
'selfdrive/ui/SConscript',
])
SConscript(['sunnypilot/SConscript'])
# Build tools
if arch != "larch64":
SConscript([
'tools/replay/SConscript',
'tools/cabana/SConscript',
'tools/jotpluggler/SConscript',
])
env.CompilationDatabase('compile_commands.json')

5
SECURITY.md Normal file
View File

@@ -0,0 +1,5 @@
# Security Policy
## Reporting a Vulnerability
Suspected vulnerabilities can be reported to both `adeeb@comma.ai` and `security@comma.ai`.

95
cereal/README.md Normal file
View File

@@ -0,0 +1,95 @@
# What is cereal?
cereal is the messaging system for openpilot. It uses [msgq](https://github.com/commaai/msgq) as a pub/sub backend, and [Cap'n proto](https://capnproto.org/capnp-tool.html) for serialization of the structs.
## Messaging Spec
You'll find the message types in [log.capnp](log.capnp). It uses [Cap'n proto](https://capnproto.org/capnp-tool.html) and defines one struct called `Event`.
All `Events` have a `logMonoTime` and a `valid`. Then a big union defines the packet type.
### Best Practices
- **All fields must describe quantities in SI units**, unless otherwise specified in the field name.
- In the context of the message they are in, field names should be completely unambiguous.
- All values should be easy to plot and be human-readable with minimal parsing.
### Maintaining backwards-compatibility
When making changes to the messaging spec you want to maintain backwards-compatibility, such that old logs can
be parsed with a new version of cereal. Adding structs and adding members to structs is generally safe, most other
things are not. Read more details [here](https://capnproto.org/language.html).
### Custom forks
Forks of [openpilot](https://github.com/commaai/openpilot) might want to add things to the messaging
spec, however this could conflict with future changes made in mainline cereal/openpilot. Rebasing against mainline openpilot
then means breaking backwards-compatibility with all old logs of your fork. So we added reserved events in
[custom.capnp](custom.capnp) that we will leave empty in mainline cereal/openpilot. **If you only modify those, you can ensure your
fork will remain backwards-compatible with all versions of mainline openpilot and your fork.**
An example of compatible changes:
```diff
diff --git a/cereal/custom.capnp b/cereal/custom.capnp
index 3348e859e..3365c7b98 100644
--- a/cereal/custom.capnp
+++ b/cereal/custom.capnp
@@ -10,7 +10,11 @@ $Cxx.namespace("cereal");
# DO rename the structs
# DON'T change the identifier (e.g. @0x81c2f05a394cf4af)
-struct CustomReserved0 @0x81c2f05a394cf4af {
+struct SteeringInfo @0x81c2f05a394cf4af {
+ active @0 :Bool;
+ steeringAngleDeg @1 :Float32;
+ steeringRateDeg @2 :Float32;
+ steeringAccelDeg @3 :Float32;
}
struct CustomReserved1 @0xaedffd8f31e7b55d {
diff --git a/cereal/log.capnp b/cereal/log.capnp
index 1209f3fd9..b189f58b6 100644
--- a/cereal/log.capnp
+++ b/cereal/log.capnp
@@ -2558,14 +2558,14 @@ struct Event {
# DO change the name of the field
# DON'T change anything after the "@"
- customReservedRawData0 @124 :Data;
+ rawCanData @124 :Data;
customReservedRawData1 @125 :Data;
customReservedRawData2 @126 :Data;
# DO change the name of the field and struct
# DON'T change the ID (e.g. @107)
# DON'T change which struct it points to
- customReserved0 @107 :Custom.CustomReserved0;
+ steeringInfo @107 :Custom.SteeringInfo;
customReserved1 @108 :Custom.CustomReserved1;
customReserved2 @109 :Custom.CustomReserved2;
customReserved3 @110 :Custom.CustomReserved3;
```
---
Example
---
```python
import cereal.messaging as messaging
# in subscriber
sm = messaging.SubMaster(['sensorEvents'])
while 1:
sm.update()
print(sm['sensorEvents'])
```
```python
# in publisher
pm = messaging.PubMaster(['sensorEvents'])
dat = messaging.new_message('sensorEvents', size=1)
dat.sensorEvents[0] = {"gyro": {"v": [0.1, -0.1, 0.1]}}
pm.send('sensorEvents', dat)
```

20
cereal/SConscript Normal file
View File

@@ -0,0 +1,20 @@
Import('env', 'common', 'msgq')
cereal_dir = Dir('.')
gen_dir = Dir('gen')
# Build cereal
schema_files = ['log.capnp', 'car.capnp', 'deprecated.capnp', 'custom.capnp']
env.Command([f'gen/cpp/{s}.c++' for s in schema_files] + [f'gen/cpp/{s}.h' for s in schema_files],
schema_files,
f"capnpc --src-prefix={cereal_dir.path} $SOURCES -o c++:{gen_dir.path}/cpp/")
cereal = env.Library('cereal', [f'gen/cpp/{s}.c++' for s in schema_files])
# Build messaging
services_h = env.Command(['services.h'], ['services.py'], 'python3 ' + cereal_dir.path + '/services.py > $TARGET')
env.Program('messaging/bridge', ['messaging/bridge.cc', 'messaging/msgq_to_zmq.cc', 'messaging/bridge_zmq.cc'], LIBS=[msgq, common, 'pthread'])
socketmaster = env.Library('socketmaster', ['messaging/socketmaster.cc'])
Export('cereal', 'socketmaster')

11
cereal/__init__.py Normal file
View File

@@ -0,0 +1,11 @@
import os
import capnp
from importlib.resources import as_file, files
capnp.remove_import_hook()
with as_file(files("cereal")) as fspath:
CEREAL_PATH = fspath.as_posix()
log = capnp.load(os.path.join(CEREAL_PATH, "log.capnp"))
car = capnp.load(os.path.join(CEREAL_PATH, "car.capnp"))
custom = capnp.load(os.path.join(CEREAL_PATH, "custom.capnp"))

1
cereal/car.capnp Symbolic link
View File

@@ -0,0 +1 @@
../opendbc_repo/opendbc/car/car.capnp

487
cereal/custom.capnp Normal file
View File

@@ -0,0 +1,487 @@
using Cxx = import "./include/c++.capnp";
$Cxx.namespace("cereal");
@0xb526ba661d550a59;
# custom.capnp: a home for empty structs reserved for custom forks
# These structs are guaranteed to remain reserved and empty in mainline
# cereal, so use these if you want custom events in your fork.
# DO rename the structs
# DON'T change the identifier (e.g. @0x81c2f05a394cf4af)
struct ModularAssistiveDrivingSystem {
state @0 :ModularAssistiveDrivingSystemState;
enabled @1 :Bool;
active @2 :Bool;
available @3 :Bool;
enum ModularAssistiveDrivingSystemState {
disabled @0;
paused @1;
enabled @2;
softDisabling @3;
overriding @4;
}
}
struct IntelligentCruiseButtonManagement {
state @0 :IntelligentCruiseButtonManagementState;
sendButton @1 :SendButtonState;
vTarget @2 :Float32;
enum IntelligentCruiseButtonManagementState {
inactive @0; # No button press or default state
preActive @1; # Pre-active state before transitioning to increasing or decreasing
increasing @2; # Increasing speed
decreasing @3; # Decreasing speed
holding @4; # Holding steady speed
}
enum SendButtonState {
none @0;
increase @1;
decrease @2;
}
}
# Same struct as Log.RadarState.LeadData
struct LeadData {
dRel @0 :Float32;
yRel @1 :Float32;
vRel @2 :Float32;
aRel @3 :Float32;
vLead @4 :Float32;
dPath @6 :Float32;
vLat @7 :Float32;
vLeadK @8 :Float32;
aLeadK @9 :Float32;
fcw @10 :Bool;
status @11 :Bool;
aLeadTau @12 :Float32;
modelProb @13 :Float32;
radar @14 :Bool;
radarTrackId @15 :Int32 = -1;
aLeadDEPRECATED @5 :Float32;
}
struct SelfdriveStateSP @0x81c2f05a394cf4af {
mads @0 :ModularAssistiveDrivingSystem;
intelligentCruiseButtonManagement @1 :IntelligentCruiseButtonManagement;
enum AudibleAlert {
none @0;
engage @1;
disengage @2;
refuse @3;
warningSoft @4;
warningImmediate @5;
prompt @6;
promptRepeat @7;
promptDistracted @8;
# unused, these are reserved for upstream events so we don't collide
reserved9 @9;
reserved10 @10;
reserved11 @11;
reserved12 @12;
reserved13 @13;
reserved14 @14;
reserved15 @15;
reserved16 @16;
reserved17 @17;
reserved18 @18;
reserved19 @19;
reserved20 @20;
reserved21 @21;
reserved22 @22;
reserved23 @23;
reserved24 @24;
reserved25 @25;
reserved26 @26;
reserved27 @27;
reserved28 @28;
reserved29 @29;
reserved30 @30;
promptSingleLow @31;
promptSingleHigh @32;
}
}
struct ModelManagerSP @0xaedffd8f31e7b55d {
activeBundle @0 :ModelBundle;
selectedBundle @1 :ModelBundle;
availableBundles @2 :List(ModelBundle);
struct DownloadUri {
uri @0 :Text;
sha256 @1 :Text;
}
enum DownloadStatus {
notDownloading @0;
downloading @1;
downloaded @2;
cached @3;
failed @4;
}
struct DownloadProgress {
status @0 :DownloadStatus;
progress @1 :Float32;
eta @2 :UInt32;
}
struct Artifact {
fileName @0 :Text;
downloadUri @1 :DownloadUri;
downloadProgress @2 :DownloadProgress;
}
struct Model {
type @0 :Type;
artifact @1 :Artifact; # Main artifact
metadata @2 :Artifact; # Metadata artifact
enum Type {
supercombo @0;
navigation @1;
vision @2;
policy @3;
offPolicy @4;
onPolicy @5;
}
}
enum Runner {
snpe @0;
tinygrad @1;
stock @2;
}
struct Override {
key @0 :Text;
value @1 :Text;
}
struct ModelBundle {
index @0 :UInt32;
internalName @1 :Text;
displayName @2 :Text;
models @3 :List(Model);
status @4 :DownloadStatus;
generation @5 :UInt32;
environment @6 :Text;
runner @7 :Runner;
is20hz @8 :Bool;
ref @9 :Text;
minimumSelectorVersion @10 :UInt32;
overrides @11 :List(Override);
}
}
struct LongitudinalPlanSP @0xf35cc4560bbf6ec2 {
dec @0 :DynamicExperimentalControl;
longitudinalPlanSource @1 :LongitudinalPlanSource;
smartCruiseControl @2 :SmartCruiseControl;
speedLimit @3 :SpeedLimit;
vTarget @4 :Float32;
aTarget @5 :Float32;
events @6 :List(OnroadEventSP.Event);
e2eAlerts @7 :E2eAlerts;
struct DynamicExperimentalControl {
state @0 :DynamicExperimentalControlState;
enabled @1 :Bool;
active @2 :Bool;
enum DynamicExperimentalControlState {
acc @0;
blended @1;
}
}
struct SmartCruiseControl {
vision @0 :Vision;
map @1 :Map;
struct Vision {
state @0 :VisionState;
vTarget @1 :Float32;
aTarget @2 :Float32;
currentLateralAccel @3 :Float32;
maxPredictedLateralAccel @4 :Float32;
enabled @5 :Bool;
active @6 :Bool;
}
struct Map {
state @0 :MapState;
vTarget @1 :Float32;
aTarget @2 :Float32;
enabled @3 :Bool;
active @4 :Bool;
}
enum VisionState {
disabled @0; # System disabled or inactive.
enabled @1; # No predicted substantial turn on vision range.
entering @2; # A substantial turn is predicted ahead, adapting speed to turn comfort levels.
turning @3; # Actively turning. Managing acceleration to provide a roll on turn feeling.
leaving @4; # Road ahead straightens. Start to allow positive acceleration.
overriding @5; # System overriding with manual control.
}
enum MapState {
disabled @0; # System disabled or inactive.
enabled @1; # No predicted substantial turn on map range.
turning @2; # Actively turning. Managing acceleration to provide a roll on turn feeling.
overriding @3; # System overriding with manual control.
}
}
struct SpeedLimit {
resolver @0 :Resolver;
assist @1 :Assist;
struct Resolver {
speedLimit @0 :Float32;
distToSpeedLimit @1 :Float32;
source @2 :Source;
speedLimitOffset @3 :Float32;
speedLimitLast @4 :Float32;
speedLimitFinal @5 :Float32;
speedLimitFinalLast @6 :Float32;
speedLimitValid @7 :Bool;
speedLimitLastValid @8 :Bool;
}
struct Assist {
state @0 :AssistState;
enabled @1 :Bool;
active @2 :Bool;
vTarget @3 :Float32;
aTarget @4 :Float32;
}
enum Source {
none @0;
car @1;
map @2;
}
enum AssistState {
disabled @0;
inactive @1; # No speed limit set or not enabled by parameter.
preActive @2;
pending @3; # Awaiting new speed limit.
adapting @4; # Reducing speed to match new speed limit.
active @5; # Cruising at speed limit.
}
}
enum LongitudinalPlanSource {
cruise @0;
sccVision @1;
sccMap @2;
speedLimitAssist @3;
}
struct E2eAlerts {
greenLightAlert @0 :Bool;
leadDepartAlert @1 :Bool;
}
}
struct OnroadEventSP @0xda96579883444c35 {
events @0 :List(Event);
struct Event {
name @0 :EventName;
# event types
enable @1 :Bool;
noEntry @2 :Bool;
warning @3 :Bool; # alerts presented only when enabled or soft disabling
userDisable @4 :Bool;
softDisable @5 :Bool;
immediateDisable @6 :Bool;
preEnable @7 :Bool;
permanent @8 :Bool; # alerts presented regardless of openpilot state
overrideLateral @10 :Bool;
overrideLongitudinal @9 :Bool;
}
enum EventName {
lkasEnable @0;
lkasDisable @1;
manualSteeringRequired @2;
manualLongitudinalRequired @3;
silentLkasEnable @4;
silentLkasDisable @5;
silentBrakeHold @6;
silentWrongGear @7;
silentReverseGear @8;
silentDoorOpen @9;
silentSeatbeltNotLatched @10;
silentParkBrake @11;
controlsMismatchLateral @12;
hyundaiRadarTracksConfirmed @13;
experimentalModeSwitched @14;
wrongCarModeAlertOnly @15;
pedalPressedAlertOnly @16;
laneTurnLeft @17;
laneTurnRight @18;
speedLimitPreActive @19;
speedLimitActive @20;
speedLimitChanged @21;
speedLimitPending @22;
e2eChime @23;
}
}
struct CarParamsSP @0x80ae746ee2596b11 {
flags @0 :UInt32; # flags for car specific quirks in sunnypilot
safetyParam @1 : Int16; # flags for sunnypilot's custom safety flags
pcmCruiseSpeed @3 :Bool;
intelligentCruiseButtonManagementAvailable @4 :Bool;
enableGasInterceptor @5 :Bool;
neuralNetworkLateralControl @2 :NeuralNetworkLateralControl;
struct NeuralNetworkLateralControl {
model @0 :Model;
fuzzyFingerprint @1 :Bool;
struct Model {
path @0 :Text;
name @1 :Text;
}
}
}
struct CarControlSP @0xa5cd762cd951a455 {
mads @0 :ModularAssistiveDrivingSystem;
params @1 :List(Param);
leadOne @2 :LeadData;
leadTwo @3 :LeadData;
intelligentCruiseButtonManagement @4 :IntelligentCruiseButtonManagement;
struct Param {
key @0 :Text;
type @2 :ParamType;
value @3 :Data;
valueDEPRECATED @1 :Text; # The data type change may cause issues with backwards compatibility.
}
enum ParamType {
string @0;
bool @1;
int @2;
float @3;
time @4;
json @5;
bytes @6;
}
}
struct BackupManagerSP @0xf98d843bfd7004a3 {
backupStatus @0 :Status;
restoreStatus @1 :Status;
backupProgress @2 :Float32;
restoreProgress @3 :Float32;
lastError @4 :Text;
currentBackup @5 :BackupInfo;
backupHistory @6 :List(BackupInfo);
enum Status {
idle @0;
inProgress @1;
completed @2;
failed @3;
}
struct Version {
major @0 :UInt16;
minor @1 :UInt16;
patch @2 :UInt16;
build @3 :UInt16;
branch @4 :Text;
}
struct MetadataEntry {
key @0 :Text;
value @1 :Text;
tags @2 :List(Text);
}
struct BackupInfo {
deviceId @0 :Text;
version @1 :UInt32;
config @2 :Text;
isEncrypted @3 :Bool;
createdAt @4 :Text; # ISO timestamp
updatedAt @5 :Text; # ISO timestamp
sunnypilotVersion @6 :Version;
backupMetadata @7 :List(MetadataEntry);
}
}
struct CarStateSP @0xb86e6369214c01c8 {
speedLimit @0 :Float32;
}
struct LiveMapDataSP @0xf416ec09499d9d19 {
speedLimitValid @0 :Bool;
speedLimit @1 :Float32;
speedLimitAheadValid @2 :Bool;
speedLimitAhead @3 :Float32;
speedLimitAheadDistance @4 :Float32;
roadName @5 :Text;
}
struct ModelDataV2SP @0xa1680744031fdb2d {
laneTurnDirection @0 :TurnDirection;
enum TurnDirection {
none @0;
turnLeft @1;
turnRight @2;
}
}
struct CustomReserved10 @0xcb9fd56c7057593a {
}
struct CustomReserved11 @0xc2243c65e0340384 {
}
struct CustomReserved12 @0x9ccdc8676701b412 {
}
struct CustomReserved13 @0xcd96dafb67a082d0 {
}
struct CustomReserved14 @0xb057204d7deadf3f {
}
struct CustomReserved15 @0xbd443b539493bc68 {
}
struct CustomReserved16 @0xfc6241ed8877b611 {
}
struct CustomReserved17 @0xa30662f84033036c {
}
struct CustomReserved18 @0xc86a3d38d13eb3ef {
}
struct CustomReserved19 @0xa4f1eb3323f5f582 {
}

789
cereal/deprecated.capnp Normal file
View File

@@ -0,0 +1,789 @@
using Cxx = import "./include/c++.capnp";
$Cxx.namespace("cereal");
@0x80ef1ec4889c2a63;
# deprecated.capnp: a home for deprecated structs
struct LogRotate @0x9811e1f38f62f2d1 {
segmentNum @0 :Int32;
path @1 :Text;
}
struct LiveUI @0xc08240f996aefced {
rearViewCam @0 :Bool;
alertText1 @1 :Text;
alertText2 @2 :Text;
awarenessStatus @3 :Float32;
}
struct UiLayoutState @0x88dcce08ad29dda0 {
activeApp @0 :App;
sidebarCollapsed @1 :Bool;
mapEnabled @2 :Bool;
mockEngaged @3 :Bool;
enum App @0x9917470acf94d285 {
home @0;
music @1;
nav @2;
settings @3;
none @4;
}
}
struct OrbslamCorrection @0x8afd33dc9b35e1aa {
correctionMonoTime @0 :UInt64;
prePositionECEF @1 :List(Float64);
postPositionECEF @2 :List(Float64);
prePoseQuatECEF @3 :List(Float32);
postPoseQuatECEF @4 :List(Float32);
numInliers @5 :UInt32;
}
struct EthernetPacket @0xa99a9d5b33cf5859 {
pkt @0 :Data;
ts @1 :Float32;
}
struct CellInfo @0xcff7566681c277ce {
timestamp @0 :UInt64;
repr @1 :Text; # android toString() for now
}
struct WifiScan @0xd4df5a192382ba0b {
bssid @0 :Text;
ssid @1 :Text;
capabilities @2 :Text;
frequency @3 :Int32;
level @4 :Int32;
timestamp @5 :Int64;
centerFreq0 @6 :Int32;
centerFreq1 @7 :Int32;
channelWidth @8 :ChannelWidth;
operatorFriendlyName @9 :Text;
venueName @10 :Text;
is80211mcResponder @11 :Bool;
passpoint @12 :Bool;
distanceCm @13 :Int32;
distanceSdCm @14 :Int32;
enum ChannelWidth @0xcb6a279f015f6b51 {
w20Mhz @0;
w40Mhz @1;
w80Mhz @2;
w160Mhz @3;
w80Plus80Mhz @4;
}
}
struct LiveEventData @0x94b7baa90c5c321e {
name @0 :Text;
value @1 :Int32;
}
struct ModelData @0xb8aad62cffef28a9 {
frameId @0 :UInt32;
frameAge @12 :UInt32;
frameDropPerc @13 :Float32;
timestampEof @9 :UInt64;
modelExecutionTime @14 :Float32;
gpuExecutionTime @16 :Float32;
rawPred @15 :Data;
path @1 :PathData;
leftLane @2 :PathData;
rightLane @3 :PathData;
lead @4 :LeadData;
freePath @6 :List(Float32);
settings @5 :ModelSettings;
leadFuture @7 :LeadData;
speed @8 :List(Float32);
meta @10 :MetaData;
longitudinal @11 :LongitudinalData;
struct PathData @0x8817eeea389e9f08 {
points @0 :List(Float32);
prob @1 :Float32;
std @2 :Float32;
stds @3 :List(Float32);
poly @4 :List(Float32);
validLen @5 :Float32;
}
struct LeadData @0xd1c9bef96d26fa91 {
dist @0 :Float32;
prob @1 :Float32;
std @2 :Float32;
relVel @3 :Float32;
relVelStd @4 :Float32;
relY @5 :Float32;
relYStd @6 :Float32;
relA @7 :Float32;
relAStd @8 :Float32;
}
struct ModelSettings @0xa26e3710efd3e914 {
bigBoxX @0 :UInt16;
bigBoxY @1 :UInt16;
bigBoxWidth @2 :UInt16;
bigBoxHeight @3 :UInt16;
boxProjection @4 :List(Float32);
yuvCorrection @5 :List(Float32);
inputTransform @6 :List(Float32);
}
struct MetaData @0x9744f25fb60f2bf8 {
engagedProb @0 :Float32;
desirePrediction @1 :List(Float32);
brakeDisengageProb @2 :Float32;
gasDisengageProb @3 :Float32;
steerOverrideProb @4 :Float32;
desireState @5 :List(Float32);
}
struct LongitudinalData @0xf98f999c6a071122 {
distances @2 :List(Float32);
speeds @0 :List(Float32);
accelerations @1 :List(Float32);
}
}
struct ECEFPoint @0xc25bbbd524983447 {
x @0 :Float64;
y @1 :Float64;
z @2 :Float64;
}
struct ECEFPointDEPRECATED @0xe10e21168db0c7f7 {
x @0 :Float32;
y @1 :Float32;
z @2 :Float32;
}
struct GPSPlannerPoints @0xab54c59699f8f9f3 {
curPosDEPRECATED @0 :ECEFPointDEPRECATED;
pointsDEPRECATED @1 :List(ECEFPointDEPRECATED);
curPos @6 :ECEFPoint;
points @7 :List(ECEFPoint);
valid @2 :Bool;
trackName @3 :Text;
speedLimit @4 :Float32;
accelTarget @5 :Float32;
}
struct GPSPlannerPlan @0xf5ad1d90cdc1dd6b {
valid @0 :Bool;
poly @1 :List(Float32);
trackName @2 :Text;
speed @3 :Float32;
acceleration @4 :Float32;
pointsDEPRECATED @5 :List(ECEFPointDEPRECATED);
points @6 :List(ECEFPoint);
xLookahead @7 :Float32;
}
struct UiNavigationEvent @0x90c8426c3eaddd3b {
type @0: Type;
status @1: Status;
distanceTo @2: Float32;
endRoadPointDEPRECATED @3: ECEFPointDEPRECATED;
endRoadPoint @4: ECEFPoint;
enum Type @0xe8db07dcf8fcea05 {
none @0;
laneChangeLeft @1;
laneChangeRight @2;
mergeLeft @3;
mergeRight @4;
turnLeft @5;
turnRight @6;
}
enum Status @0xb9aa88c75ef99a1f {
none @0;
passive @1;
approaching @2;
active @3;
}
}
struct LiveLocationData @0xb99b2bc7a57e8128 {
status @0 :UInt8;
# 3D fix
lat @1 :Float64;
lon @2 :Float64;
alt @3 :Float32; # m
# speed
speed @4 :Float32; # m/s
# NED velocity components
vNED @5 :List(Float32);
# roll, pitch, heading (x,y,z)
roll @6 :Float32; # WRT to center of earth?
pitch @7 :Float32; # WRT to center of earth?
heading @8 :Float32; # WRT to north?
# what are these?
wanderAngle @9 :Float32;
trackAngle @10 :Float32;
# car frame -- https://upload.wikimedia.org/wikipedia/commons/f/f5/RPY_angles_of_cars.png
# gyro, in car frame, deg/s
gyro @11 :List(Float32);
# accel, in car frame, m/s^2
accel @12 :List(Float32);
accuracy @13 :Accuracy;
source @14 :SensorSource;
# if we are fixing a location in the past
fixMonoTime @15 :UInt64;
gpsWeek @16 :Int32;
timeOfWeek @17 :Float64;
positionECEF @18 :List(Float64);
poseQuatECEF @19 :List(Float32);
pitchCalibration @20 :Float32;
yawCalibration @21 :Float32;
imuFrame @22 :List(Float32);
struct Accuracy @0x943dc4625473b03f {
pNEDError @0 :List(Float32);
vNEDError @1 :List(Float32);
rollError @2 :Float32;
pitchError @3 :Float32;
headingError @4 :Float32;
ellipsoidSemiMajorError @5 :Float32;
ellipsoidSemiMinorError @6 :Float32;
ellipsoidOrientationError @7 :Float32;
}
enum SensorSource @0xc871d3cc252af657 {
applanix @0;
kalman @1;
orbslam @2;
timing @3;
dummy @4;
}
}
struct OrbOdometry @0xd7700859ed1f5b76 {
# timing first
startMonoTime @0 :UInt64;
endMonoTime @1 :UInt64;
# fundamental matrix and error
f @2: List(Float64);
err @3: Float64;
# number of inlier points
inliers @4: Int32;
# for debug only
# indexed by endMonoTime features
# value is startMonoTime feature match
# -1 if no match
matches @5: List(Int16);
}
struct OrbFeatures @0xcd60164a8a0159ef {
timestampEof @0 :UInt64;
# transposed arrays of normalized image coordinates
# len(xs) == len(ys) == len(descriptors) * 32
xs @1 :List(Float32);
ys @2 :List(Float32);
descriptors @3 :Data;
octaves @4 :List(Int8);
# match index to last OrbFeatures
# -1 if no match
timestampLastEof @5 :UInt64;
matches @6: List(Int16);
}
struct OrbFeaturesSummary @0xd500d30c5803fa4f {
timestampEof @0 :UInt64;
timestampLastEof @1 :UInt64;
featureCount @2 :UInt16;
matchCount @3 :UInt16;
computeNs @4 :UInt64;
}
struct OrbKeyFrame @0xc8233c0345e27e24 {
# this is a globally unique id for the KeyFrame
id @0: UInt64;
# this is the location of the KeyFrame
pos @1: ECEFPoint;
# these are the features in the world
# len(dpos) == len(descriptors) * 32
dpos @2 :List(ECEFPoint);
descriptors @3 :Data;
}
struct KalmanOdometry @0x92e21bb7ea38793a {
trans @0 :List(Float32); # m/s in device frame
rot @1 :List(Float32); # rad/s in device frame
transStd @2 :List(Float32); # std m/s in device frame
rotStd @3 :List(Float32); # std rad/s in device frame
}
struct OrbObservation @0x9b326d4e436afec7 {
observationMonoTime @0 :UInt64;
normalizedCoordinates @1 :List(Float32);
locationECEF @2 :List(Float64);
matchDistance @3: UInt32;
}
struct CalibrationFeatures @0x8fdfadb254ea867a {
frameId @0 :UInt32;
p0 @1 :List(Float32);
p1 @2 :List(Float32);
status @3 :List(Int8);
}
struct NavStatus @0xbd8822120928120c {
isNavigating @0 :Bool;
currentAddress @1 :Address;
struct Address @0xce7cd672cacc7814 {
title @0 :Text;
lat @1 :Float64;
lng @2 :Float64;
house @3 :Text;
address @4 :Text;
street @5 :Text;
city @6 :Text;
state @7 :Text;
country @8 :Text;
}
}
struct NavUpdate @0xdb98be6565516acb {
isNavigating @0 :Bool;
curSegment @1 :Int32;
segments @2 :List(Segment);
struct LatLng @0x9eaef9187cadbb9b {
lat @0 :Float64;
lng @1 :Float64;
}
struct Segment @0xa5b39b4fc4d7da3f {
from @0 :LatLng;
to @1 :LatLng;
updateTime @2 :Int32;
distance @3 :Int32;
crossTime @4 :Int32;
exitNo @5 :Int32;
instruction @6 :Instruction;
parts @7 :List(LatLng);
enum Instruction @0xc5417a637451246f {
turnLeft @0;
turnRight @1;
keepLeft @2;
keepRight @3;
straight @4;
roundaboutExitNumber @5;
roundaboutExit @6;
roundaboutTurnLeft @7;
unkn8 @8;
roundaboutStraight @9;
unkn10 @10;
roundaboutTurnRight @11;
unkn12 @12;
roundaboutUturn @13;
unkn14 @14;
arrive @15;
exitLeft @16;
exitRight @17;
unkn18 @18;
uturn @19;
# ...
}
}
}
struct TrafficEvent @0xacfa74a094e62626 {
type @0 :Type;
distance @1 :Float32;
action @2 :Action;
resuming @3 :Bool;
enum Type @0xd85d75253435bf4b {
stopSign @0;
lightRed @1;
lightYellow @2;
lightGreen @3;
stopLight @4;
}
enum Action @0xa6f6ce72165ccb49 {
none @0;
yield @1;
stop @2;
resumeReady @3;
}
}
struct AndroidGnss @0xdfdf30d03fc485bd {
union {
measurements @0 :Measurements;
navigationMessage @1 :NavigationMessage;
}
struct Measurements @0xa20710d4f428d6cd {
clock @0 :Clock;
measurements @1 :List(Measurement);
struct Clock @0xa0e27b453a38f450 {
timeNanos @0 :Int64;
hardwareClockDiscontinuityCount @1 :Int32;
hasTimeUncertaintyNanos @2 :Bool;
timeUncertaintyNanos @3 :Float64;
hasLeapSecond @4 :Bool;
leapSecond @5 :Int32;
hasFullBiasNanos @6 :Bool;
fullBiasNanos @7 :Int64;
hasBiasNanos @8 :Bool;
biasNanos @9 :Float64;
hasBiasUncertaintyNanos @10 :Bool;
biasUncertaintyNanos @11 :Float64;
hasDriftNanosPerSecond @12 :Bool;
driftNanosPerSecond @13 :Float64;
hasDriftUncertaintyNanosPerSecond @14 :Bool;
driftUncertaintyNanosPerSecond @15 :Float64;
}
struct Measurement @0xd949bf717d77614d {
svId @0 :Int32;
constellation @1 :Constellation;
timeOffsetNanos @2 :Float64;
state @3 :Int32;
receivedSvTimeNanos @4 :Int64;
receivedSvTimeUncertaintyNanos @5 :Int64;
cn0DbHz @6 :Float64;
pseudorangeRateMetersPerSecond @7 :Float64;
pseudorangeRateUncertaintyMetersPerSecond @8 :Float64;
accumulatedDeltaRangeState @9 :Int32;
accumulatedDeltaRangeMeters @10 :Float64;
accumulatedDeltaRangeUncertaintyMeters @11 :Float64;
hasCarrierFrequencyHz @12 :Bool;
carrierFrequencyHz @13 :Float32;
hasCarrierCycles @14 :Bool;
carrierCycles @15 :Int64;
hasCarrierPhase @16 :Bool;
carrierPhase @17 :Float64;
hasCarrierPhaseUncertainty @18 :Bool;
carrierPhaseUncertainty @19 :Float64;
hasSnrInDb @20 :Bool;
snrInDb @21 :Float64;
multipathIndicator @22 :MultipathIndicator;
enum Constellation @0x9ef1f3ff0deb5ffb {
unknown @0;
gps @1;
sbas @2;
glonass @3;
qzss @4;
beidou @5;
galileo @6;
}
enum State @0xcbb9490adce12d72 {
unknown @0;
codeLock @1;
bitSync @2;
subframeSync @3;
towDecoded @4;
msecAmbiguous @5;
symbolSync @6;
gloStringSync @7;
gloTodDecoded @8;
bdsD2BitSync @9;
bdsD2SubframeSync @10;
galE1bcCodeLock @11;
galE1c2ndCodeLock @12;
galE1bPageSync @13;
sbasSync @14;
}
enum MultipathIndicator @0xc04e7b6231d4caa8 {
unknown @0;
detected @1;
notDetected @2;
}
}
}
struct NavigationMessage @0xe2517b083095fd4e {
type @0 :Int32;
svId @1 :Int32;
messageId @2 :Int32;
submessageId @3 :Int32;
data @4 :Data;
status @5 :Status;
enum Status @0xec1ff7996b35366f {
unknown @0;
parityPassed @1;
parityRebuilt @2;
}
}
}
struct LidarPts @0xe3d6685d4e9d8f7a {
r @0 :List(UInt16); # uint16 m*500.0
theta @1 :List(UInt16); # uint16 deg*100.0
reflect @2 :List(UInt8); # uint8 0-255
# For storing out of file.
idx @3 :UInt64;
# For storing in file
pkt @4 :Data;
}
struct LiveTracksDEPRECATED @0xb16f60103159415a {
trackId @0 :Int32;
dRel @1 :Float32;
yRel @2 :Float32;
vRel @3 :Float32;
aRel @4 :Float32;
timeStamp @5 :Float32;
status @6 :Float32;
currentTime @7 :Float32;
stationary @8 :Bool;
oncoming @9 :Bool;
}
struct LiveMpcData @0x92a5e332a85f32a0 {
x @0 :List(Float32);
y @1 :List(Float32);
psi @2 :List(Float32);
curvature @3 :List(Float32);
qpIterations @4 :UInt32;
calculationTime @5 :UInt64;
cost @6 :Float64;
}
struct LiveLongitudinalMpcData @0xe7e17c434f865ae2 {
xEgo @0 :List(Float32);
vEgo @1 :List(Float32);
aEgo @2 :List(Float32);
xLead @3 :List(Float32);
vLead @4 :List(Float32);
aLead @5 :List(Float32);
aLeadTau @6 :Float32; # lead accel time constant
qpIterations @7 :UInt32;
mpcId @8 :UInt32;
calculationTime @9 :UInt64;
cost @10 :Float64;
}
struct DriverStateDEPRECATED @0xb83c6cc593ed0a00 {
frameId @0 :UInt32;
modelExecutionTime @14 :Float32;
dspExecutionTime @16 :Float32;
rawPredictions @15 :Data;
faceOrientation @3 :List(Float32);
facePosition @4 :List(Float32);
faceProb @5 :Float32;
leftEyeProb @6 :Float32;
rightEyeProb @7 :Float32;
leftBlinkProb @8 :Float32;
rightBlinkProb @9 :Float32;
faceOrientationStd @11 :List(Float32);
facePositionStd @12 :List(Float32);
sunglassesProb @13 :Float32;
poorVision @17 :Float32;
partialFace @18 :Float32;
distractedPose @19 :Float32;
distractedEyes @20 :Float32;
eyesOnRoad @21 :Float32;
phoneUse @22 :Float32;
occludedProb @23 :Float32;
readyProb @24 :List(Float32);
notReadyProb @25 :List(Float32);
irPwrDEPRECATED @10 :Float32;
descriptorDEPRECATED @1 :List(Float32);
stdDEPRECATED @2 :Float32;
}
struct NavModelData @0xac3de5c437be057a {
frameId @0 :UInt32;
locationMonoTime @6 :UInt64;
modelExecutionTime @1 :Float32;
dspExecutionTime @2 :Float32;
features @3 :List(Float32);
# predicted future position
position @4 :XYData;
desirePrediction @5 :List(Float32);
# All SI units and in device frame
struct XYData @0xbe09e615b2507e26 {
x @0 :List(Float32);
y @1 :List(Float32);
xStd @2 :List(Float32);
yStd @3 :List(Float32);
}
}
struct AndroidBuildInfo @0xfe2919d5c21f426c {
board @0 :Text;
bootloader @1 :Text;
brand @2 :Text;
device @3 :Text;
display @4 :Text;
fingerprint @5 :Text;
hardware @6 :Text;
host @7 :Text;
id @8 :Text;
manufacturer @9 :Text;
model @10 :Text;
product @11 :Text;
radioVersion @12 :Text;
serial @13 :Text;
supportedAbis @14 :List(Text);
tags @15 :Text;
time @16 :Int64;
type @17 :Text;
user @18 :Text;
versionCodename @19 :Text;
versionRelease @20 :Text;
versionSdk @21 :Int32;
versionSecurityPatch @22 :Text;
}
struct AndroidSensor @0x9b513b93a887dbcd {
id @0 :Int32;
name @1 :Text;
vendor @2 :Text;
version @3 :Int32;
handle @4 :Int32;
type @5 :Int32;
maxRange @6 :Float32;
resolution @7 :Float32;
power @8 :Float32;
minDelay @9 :Int32;
fifoReservedEventCount @10 :UInt32;
fifoMaxEventCount @11 :UInt32;
stringType @12 :Text;
maxDelay @13 :Int32;
}
struct IosBuildInfo @0xd97e3b28239f5580 {
appVersion @0 :Text;
appBuild @1 :UInt32;
osVersion @2 :Text;
deviceModel @3 :Text;
}
enum FrameTypeDEPRECATED @0xa37f0d8558e193fd {
unknown @0;
neo @1;
chffrAndroid @2;
front @3;
}
struct AndroidCaptureResult @0xbcc3efbac41d2048 {
sensitivity @0 :Int32;
frameDuration @1 :Int64;
exposureTime @2 :Int64;
rollingShutterSkew @3 :UInt64;
colorCorrectionTransform @4 :List(Int32);
colorCorrectionGains @5 :List(Float32);
displayRotation @6 :Int8;
}
enum UsbPowerModeDEPRECATED @0xa8883583b32c9877 {
none @0;
client @1;
cdp @2;
dcp @3;
}
struct LateralINDIState @0x939463348632375e {
active @0 :Bool;
steeringAngleDeg @1 :Float32;
steeringRateDeg @2 :Float32;
steeringAccelDeg @3 :Float32;
rateSetPoint @4 :Float32;
accelSetPoint @5 :Float32;
accelError @6 :Float32;
delayedOutput @7 :Float32;
delta @8 :Float32;
output @9 :Float32;
saturated @10 :Bool;
steeringAngleDesiredDeg @11 :Float32;
steeringRateDesiredDeg @12 :Float32;
}
struct LateralLQRState @0x9024e2d790c82ade {
active @0 :Bool;
steeringAngleDeg @1 :Float32;
i @2 :Float32;
output @3 :Float32;
lqrOutput @4 :Float32;
saturated @5 :Bool;
steeringAngleDesiredDeg @6 :Float32;
}
struct LateralCurvatureState @0xad9d8095c06f7c61 {
active @0 :Bool;
actualCurvature @1 :Float32;
desiredCurvature @2 :Float32;
error @3 :Float32;
p @4 :Float32;
i @5 :Float32;
f @6 :Float32;
output @7 :Float32;
saturated @8 :Bool;
}
struct LateralPlannerSolution @0x84caeca5a6b4acfe {
x @0 :List(Float32);
y @1 :List(Float32);
yaw @2 :List(Float32);
yawRate @3 :List(Float32);
xStd @4 :List(Float32);
yStd @5 :List(Float32);
yawStd @6 :List(Float32);
yawRateStd @7 :List(Float32);
}
struct GpsTrajectory @0x8cfeb072f5301000 {
x @0 :List(Float32);
y @1 :List(Float32);
}

26
cereal/include/c++.capnp Normal file
View File

@@ -0,0 +1,26 @@
# Copyright (c) 2013-2014 Sandstorm Development Group, Inc. and contributors
# Licensed under the MIT License:
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
@0xbdf87d7bb8304e81;
$namespace("capnp::annotations");
annotation namespace(file): Text;
annotation name(field, enumerant, struct, enum, interface, method, param, group, union): Text;

2555
cereal/log.capnp Normal file
View File

File diff suppressed because it is too large Load Diff

269
cereal/messaging/__init__.py Normal file
View File

@@ -0,0 +1,269 @@
# must be built with scons
from msgq import fake_event_handle, drain_sock_raw, MultiplePublishersError, IpcError, \
Context, Poller, SubSocket, PubSocket, SocketEventHandle, toggle_fake_events, \
set_fake_prefix, get_fake_prefix, delete_fake_prefix, wait_for_one_event
import msgq
import os
import capnp
import time
from typing import Optional, List, Union, Dict
from cereal import log
from cereal.services import SERVICE_LIST
from openpilot.common.utils import MovingAverage
NO_TRAVERSAL_LIMIT = 2**64-1
def pub_sock(endpoint: str) -> PubSocket:
service = SERVICE_LIST.get(endpoint)
segment_size = service.queue_size if service else 0
return msgq.pub_sock(endpoint, segment_size)
def sub_sock(endpoint: str, poller: Optional[Poller] = None, addr: str = "127.0.0.1",
conflate: bool = False, timeout: Optional[int] = None) -> SubSocket:
service = SERVICE_LIST.get(endpoint)
segment_size = service.queue_size if service else 0
return msgq.sub_sock(endpoint, poller=poller, addr=addr, conflate=conflate,
timeout=timeout, segment_size=segment_size)
def reset_context():
msgq.context = Context()
def log_from_bytes(dat: bytes, struct: capnp.lib.capnp._StructModule = log.Event) -> capnp.lib.capnp._DynamicStructReader:
with struct.from_bytes(dat, traversal_limit_in_words=NO_TRAVERSAL_LIMIT) as msg:
return msg
def new_message(service: Optional[str], size: Optional[int] = None, **kwargs) -> capnp.lib.capnp._DynamicStructBuilder:
args = {
'valid': False,
'logMonoTime': int(time.monotonic() * 1e9),
**kwargs
}
dat = log.Event.new_message(**args)
if service is not None:
if size is None:
dat.init(service)
else:
dat.init(service, size)
return dat
def drain_sock(sock: SubSocket, wait_for_one: bool = False) -> List[capnp.lib.capnp._DynamicStructReader]:
"""Receive all message currently available on the queue"""
msgs = drain_sock_raw(sock, wait_for_one=wait_for_one)
return [log_from_bytes(m) for m in msgs]
# TODO: print when we drop packets?
def recv_sock(sock: SubSocket, wait: bool = False) -> Optional[capnp.lib.capnp._DynamicStructReader]:
"""Same as drain sock, but only returns latest message. Consider using conflate instead."""
dat = None
while 1:
if wait and dat is None:
recv = sock.receive()
else:
recv = sock.receive(non_blocking=True)
if recv is None: # Timeout hit
break
dat = recv
if dat is not None:
dat = log_from_bytes(dat)
return dat
def recv_one(sock: SubSocket) -> Optional[capnp.lib.capnp._DynamicStructReader]:
dat = sock.receive()
if dat is not None:
dat = log_from_bytes(dat)
return dat
def recv_one_or_none(sock: SubSocket) -> Optional[capnp.lib.capnp._DynamicStructReader]:
dat = sock.receive(non_blocking=True)
if dat is not None:
dat = log_from_bytes(dat)
return dat
def recv_one_retry(sock: SubSocket) -> capnp.lib.capnp._DynamicStructReader:
"""Keep receiving until we get a message"""
while True:
dat = sock.receive()
if dat is not None:
return log_from_bytes(dat)
class FrequencyTracker:
def __init__(self, service_freq: float, update_freq: float, is_poll: bool):
freq = max(min(service_freq, update_freq), 1.)
if is_poll:
min_freq = max_freq = freq
else:
max_freq = min(freq, update_freq)
if service_freq >= 2 * update_freq:
min_freq = update_freq
elif update_freq >= 2* service_freq:
min_freq = freq
else:
min_freq = min(freq, freq / 2.)
self.min_freq = min_freq * 0.8
self.max_freq = max_freq * 1.2
self.avg_dt = MovingAverage(int(10 * freq))
self.recent_avg_dt = MovingAverage(int(freq))
self.prev_time = 0.0
def record_recv_time(self, cur_time: float) -> None:
# TODO: Handle case where cur_time is less than prev_time
if self.prev_time > 1e-5:
dt = cur_time - self.prev_time
self.avg_dt.add_value(dt)
self.recent_avg_dt.add_value(dt)
self.prev_time = cur_time
@property
def valid(self) -> bool:
if self.avg_dt.count == 0:
return False
avg_freq = 1.0 / self.avg_dt.get_average()
if self.min_freq <= avg_freq <= self.max_freq:
return True
avg_freq_recent = 1.0 / self.recent_avg_dt.get_average()
return self.min_freq <= avg_freq_recent <= self.max_freq
class SubMaster:
def __init__(self, services: List[str], poll: Optional[str] = None,
ignore_alive: Optional[List[str]] = None, ignore_avg_freq: Optional[List[str]] = None,
ignore_valid: Optional[List[str]] = None, addr: str = "127.0.0.1", frequency: Optional[float] = None):
self.frame = -1
self.services = services
self.seen = {s: False for s in services}
self.updated = {s: False for s in services}
self.recv_time = {s: 0. for s in services}
self.recv_frame = {s: 0 for s in services}
self.sock = {}
self.data = {}
self.logMonoTime = {s: 0 for s in services}
# zero-frequency / on-demand services are always alive and presumed valid; all others must pass checks
on_demand = {s: SERVICE_LIST[s].frequency <= 1e-5 for s in services}
self.static_freq_services = set(s for s in services if not on_demand[s])
self.alive = {s: on_demand[s] for s in services}
self.freq_ok = {s: on_demand[s] for s in services}
self.valid = {s: on_demand[s] for s in services}
self.freq_tracker: Dict[str, FrequencyTracker] = {}
self.poller = Poller()
polled_services = set([poll, ] if poll is not None else services)
self.non_polled_services = set(services) - polled_services
self.ignore_average_freq = [] if ignore_avg_freq is None else ignore_avg_freq
self.ignore_alive = [] if ignore_alive is None else ignore_alive
self.ignore_valid = [] if ignore_valid is None else ignore_valid
self.simulation = bool(int(os.getenv("SIMULATION", "0")))
# if freq and poll aren't specified, assume the max to be conservative
assert frequency is None or poll is None, "Do not specify 'frequency' - frequency of the polled service will be used."
self.update_freq = frequency or max([SERVICE_LIST[s].frequency for s in polled_services])
for s in services:
p = self.poller if s not in self.non_polled_services else None
self.sock[s] = sub_sock(s, poller=p, addr=addr, conflate=True)
try:
data = new_message(s)
except capnp.lib.capnp.KjException:
data = new_message(s, 0) # lists
self.data[s] = getattr(data.as_reader(), s)
self.freq_tracker[s] = FrequencyTracker(SERVICE_LIST[s].frequency, self.update_freq, s == poll)
def __getitem__(self, s: str) -> capnp.lib.capnp._DynamicStructReader:
return self.data[s]
def _check_avg_freq(self, s: str) -> bool:
return SERVICE_LIST[s].frequency > 0.99 and (s not in self.ignore_average_freq) and (s not in self.ignore_alive)
def update(self, timeout: int = 100) -> None:
msgs = []
for sock in self.poller.poll(timeout):
msgs.append(recv_one_or_none(sock))
# non-blocking receive for non-polled sockets
for s in self.non_polled_services:
msgs.append(recv_one_or_none(self.sock[s]))
self.update_msgs(time.monotonic(), msgs)
def update_msgs(self, cur_time: float, msgs: List[capnp.lib.capnp._DynamicStructReader]) -> None:
self.frame += 1
self.updated = dict.fromkeys(self.services, False)
for msg in msgs:
if msg is None:
continue
s = msg.which()
self.seen[s] = True
self.updated[s] = True
self.freq_tracker[s].record_recv_time(cur_time)
self.recv_time[s] = cur_time
self.recv_frame[s] = self.frame
self.data[s] = getattr(msg, s)
self.logMonoTime[s] = msg.logMonoTime
self.valid[s] = msg.valid
for s in self.static_freq_services:
# alive if delay is within 10x the expected frequency; checks relaxed in simulator
self.alive[s] = (cur_time - self.recv_time[s]) < (10. / SERVICE_LIST[s].frequency) or (self.seen[s] and self.simulation)
self.freq_ok[s] = self.freq_tracker[s].valid or self.simulation
def all_alive(self, service_list: Optional[List[str]] = None) -> bool:
return all(self.alive[s] for s in (service_list or self.services) if s not in self.ignore_alive)
def all_freq_ok(self, service_list: Optional[List[str]] = None) -> bool:
return all(self.freq_ok[s] for s in (service_list or self.services) if self._check_avg_freq(s))
def all_valid(self, service_list: Optional[List[str]] = None) -> bool:
return all(self.valid[s] for s in (service_list or self.services) if s not in self.ignore_valid)
def all_checks(self, service_list: Optional[List[str]] = None) -> bool:
return self.all_alive(service_list) and self.all_freq_ok(service_list) and self.all_valid(service_list)
class PubMaster:
def __init__(self, services: List[str]):
self.sock = {}
for s in services:
self.sock[s] = pub_sock(s)
def send(self, s: str, dat: Union[bytes, capnp.lib.capnp._DynamicStructBuilder]) -> None:
if not isinstance(dat, bytes):
dat = dat.to_bytes()
self.sock[s].send(dat)
def wait_for_readers_to_update(self, s: str, timeout: int, dt: float = 0.05) -> bool:
for _ in range(int(timeout*(1./dt))):
if self.sock[s].all_readers_updated():
return True
time.sleep(dt)
return False
def all_readers_updated(self, s: str) -> bool:
return self.sock[s].all_readers_updated() # type: ignore

72
cereal/messaging/bridge.cc Normal file
View File

@@ -0,0 +1,72 @@
#include <cassert>
#include "cereal/messaging/msgq_to_zmq.h"
#include "cereal/services.h"
#include "common/util.h"
ExitHandler do_exit;
static std::vector<std::string> get_services(const std::string &whitelist_str, bool zmq_to_msgq) {
std::vector<std::string> service_list;
for (const auto& it : services) {
std::string name = it.second.name;
bool in_whitelist = whitelist_str.find(name) != std::string::npos;
if (zmq_to_msgq && !in_whitelist) {
continue;
}
service_list.push_back(name);
}
return service_list;
}
void msgq_to_zmq(const std::vector<std::string> &endpoints, const std::string &ip) {
MsgqToZmq bridge;
bridge.run(endpoints, ip);
}
void zmq_to_msgq(const std::vector<std::string> &endpoints, const std::string &ip) {
auto poller = std::make_unique<BridgeZmqPoller>();
auto pub_context = std::make_unique<Context>();
auto sub_context = std::make_unique<BridgeZmqContext>();
std::map<BridgeZmqSubSocket *, PubSocket *> sub2pub;
for (auto endpoint : endpoints) {
auto pub_sock = new PubSocket();
auto sub_sock = new BridgeZmqSubSocket();
size_t queue_size = services.at(endpoint).queue_size;
pub_sock->connect(pub_context.get(), endpoint, true, queue_size);
sub_sock->connect(sub_context.get(), endpoint, ip, false);
poller->registerSocket(sub_sock);
sub2pub[sub_sock] = pub_sock;
}
while (!do_exit) {
for (auto sub_sock : poller->poll(100)) {
std::unique_ptr<Message> msg(sub_sock->receive(true));
if (msg) {
sub2pub[sub_sock]->sendMessage(msg.get());
}
}
}
// Clean up allocated sockets
for (auto &[sub_sock, pub_sock] : sub2pub) {
delete sub_sock;
delete pub_sock;
}
}
int main(int argc, char **argv) {
bool is_zmq_to_msgq = argc > 2;
std::string ip = is_zmq_to_msgq ? argv[1] : "127.0.0.1";
std::string whitelist_str = is_zmq_to_msgq ? std::string(argv[2]) : "";
std::vector<std::string> endpoints = get_services(whitelist_str, is_zmq_to_msgq);
if (is_zmq_to_msgq) {
zmq_to_msgq(endpoints, ip);
} else {
msgq_to_zmq(endpoints, ip);
}
return 0;
}

170
cereal/messaging/bridge_zmq.cc Normal file
View File

@@ -0,0 +1,170 @@
#include "cereal/messaging/bridge_zmq.h"
#include <cassert>
#include <cstring>
#include <unistd.h>
static size_t fnv1a_hash(const std::string &str) {
const size_t fnv_prime = 0x100000001b3;
size_t hash_value = 0xcbf29ce484222325;
for (char c : str) {
hash_value ^= (unsigned char)c;
hash_value *= fnv_prime;
}
return hash_value;
}
// FIXME: This is a hack to get the port number from the socket name, might have collisions.
static int get_port(std::string endpoint) {
size_t hash_value = fnv1a_hash(endpoint);
int start_port = 8023;
int max_port = 65535;
return start_port + (hash_value % (max_port - start_port));
}
BridgeZmqContext::BridgeZmqContext() {
context = zmq_ctx_new();
}
BridgeZmqContext::~BridgeZmqContext() {
if (context != nullptr) {
zmq_ctx_term(context);
}
}
void BridgeZmqMessage::init(size_t sz) {
size = sz;
data = new char[size];
}
void BridgeZmqMessage::init(char *d, size_t sz) {
size = sz;
data = new char[size];
memcpy(data, d, size);
}
void BridgeZmqMessage::close() {
if (size > 0) {
delete[] data;
}
data = nullptr;
size = 0;
}
BridgeZmqMessage::~BridgeZmqMessage() {
close();
}
int BridgeZmqSubSocket::connect(BridgeZmqContext *context, std::string endpoint, std::string address, bool conflate, bool check_endpoint) {
sock = zmq_socket(context->getRawContext(), ZMQ_SUB);
if (sock == nullptr) {
return -1;
}
zmq_setsockopt(sock, ZMQ_SUBSCRIBE, "", 0);
if (conflate) {
int arg = 1;
zmq_setsockopt(sock, ZMQ_CONFLATE, &arg, sizeof(int));
}
int reconnect_ivl = 500;
zmq_setsockopt(sock, ZMQ_RECONNECT_IVL_MAX, &reconnect_ivl, sizeof(reconnect_ivl));
full_endpoint = "tcp://" + address + ":";
if (check_endpoint) {
full_endpoint += std::to_string(get_port(endpoint));
} else {
full_endpoint += endpoint;
}
return zmq_connect(sock, full_endpoint.c_str());
}
void BridgeZmqSubSocket::setTimeout(int timeout) {
zmq_setsockopt(sock, ZMQ_RCVTIMEO, &timeout, sizeof(int));
}
Message *BridgeZmqSubSocket::receive(bool non_blocking) {
zmq_msg_t msg;
assert(zmq_msg_init(&msg) == 0);
int flags = non_blocking ? ZMQ_DONTWAIT : 0;
int rc = zmq_msg_recv(&msg, sock, flags);
Message *ret = nullptr;
if (rc >= 0) {
ret = new BridgeZmqMessage;
ret->init((char *)zmq_msg_data(&msg), zmq_msg_size(&msg));
}
zmq_msg_close(&msg);
return ret;
}
BridgeZmqSubSocket::~BridgeZmqSubSocket() {
if (sock != nullptr) {
zmq_close(sock);
}
}
int BridgeZmqPubSocket::connect(BridgeZmqContext *context, std::string endpoint, bool check_endpoint) {
sock = zmq_socket(context->getRawContext(), ZMQ_PUB);
if (sock == nullptr) {
return -1;
}
full_endpoint = "tcp://*:";
if (check_endpoint) {
full_endpoint += std::to_string(get_port(endpoint));
} else {
full_endpoint += endpoint;
}
// ZMQ pub sockets cannot be shared between processes, so we need to ensure pid stays the same.
pid = getpid();
return zmq_bind(sock, full_endpoint.c_str());
}
int BridgeZmqPubSocket::sendMessage(Message *message) {
assert(pid == getpid());
return zmq_send(sock, message->getData(), message->getSize(), ZMQ_DONTWAIT);
}
int BridgeZmqPubSocket::send(char *data, size_t size) {
assert(pid == getpid());
return zmq_send(sock, data, size, ZMQ_DONTWAIT);
}
BridgeZmqPubSocket::~BridgeZmqPubSocket() {
if (sock != nullptr) {
zmq_close(sock);
}
}
void BridgeZmqPoller::registerSocket(BridgeZmqSubSocket *socket) {
assert(num_polls + 1 < (sizeof(polls) / sizeof(polls[0])));
polls[num_polls].socket = socket->getRawSocket();
polls[num_polls].events = ZMQ_POLLIN;
sockets.push_back(socket);
num_polls++;
}
std::vector<BridgeZmqSubSocket *> BridgeZmqPoller::poll(int timeout) {
std::vector<BridgeZmqSubSocket *> ret;
int rc = zmq_poll(polls, num_polls, timeout);
if (rc < 0) {
return ret;
}
for (size_t i = 0; i < num_polls; i++) {
if (polls[i].revents) {
ret.push_back(sockets[i]);
}
}
return ret;
}

72
cereal/messaging/bridge_zmq.h Normal file
View File

@@ -0,0 +1,72 @@
#pragma once
#include <cstddef>
#include <string>
#include <vector>
#include <zmq.h>
#include "msgq/ipc.h"
class BridgeZmqContext {
public:
BridgeZmqContext();
void *getRawContext() { return context; }
~BridgeZmqContext();
private:
void *context = nullptr;
};
class BridgeZmqMessage : public Message {
public:
void init(size_t size);
void init(char *data, size_t size);
void close();
size_t getSize() { return size; }
char *getData() { return data; }
~BridgeZmqMessage();
private:
char *data = nullptr;
size_t size = 0;
};
class BridgeZmqSubSocket {
public:
int connect(BridgeZmqContext *context, std::string endpoint, std::string address, bool conflate = false, bool check_endpoint = true);
void setTimeout(int timeout);
Message *receive(bool non_blocking = false);
void *getRawSocket() { return sock; }
~BridgeZmqSubSocket();
private:
void *sock = nullptr;
std::string full_endpoint;
};
class BridgeZmqPubSocket {
public:
int connect(BridgeZmqContext *context, std::string endpoint, bool check_endpoint = true);
int sendMessage(Message *message);
int send(char *data, size_t size);
void *getRawSocket() { return sock; }
~BridgeZmqPubSocket();
private:
void *sock = nullptr;
std::string full_endpoint;
int pid = -1;
};
class BridgeZmqPoller {
public:
void registerSocket(BridgeZmqSubSocket *socket);
std::vector<BridgeZmqSubSocket *> poll(int timeout);
private:
static constexpr size_t MAX_BRIDGE_ZMQ_POLLERS = 128;
std::vector<BridgeZmqSubSocket *> sockets;
zmq_pollitem_t polls[MAX_BRIDGE_ZMQ_POLLERS] = {};
size_t num_polls = 0;
};

102
cereal/messaging/messaging.h Normal file
View File

@@ -0,0 +1,102 @@
#pragma once
#include <cstddef>
#include <map>
#include <string>
#include <vector>
#include <utility>
#include <capnp/serialize.h>
#include "cereal/gen/cpp/log.capnp.h"
#include "common/timing.h"
#include "msgq/ipc.h"
class SubMaster {
public:
SubMaster(const std::vector<const char *> &service_list, const std::vector<const char *> &poll = {},
const char *address = nullptr, const std::vector<const char *> &ignore_alive = {});
void update(int timeout = 1000);
void update_msgs(uint64_t current_time, const std::vector<std::pair<std::string, cereal::Event::Reader>> &messages);
inline bool allAlive(const std::vector<const char *> &service_list = {}) { return all_(service_list, false, true); }
inline bool allValid(const std::vector<const char *> &service_list = {}) { return all_(service_list, true, false); }
inline bool allAliveAndValid(const std::vector<const char *> &service_list = {}) { return all_(service_list, true, true); }
void drain();
~SubMaster();
uint64_t frame = 0;
bool updated(const char *name) const;
bool alive(const char *name) const;
bool valid(const char *name) const;
uint64_t rcv_frame(const char *name) const;
uint64_t rcv_time(const char *name) const;
cereal::Event::Reader &operator[](const char *name) const;
private:
bool all_(const std::vector<const char *> &service_list, bool valid, bool alive);
Poller *poller_ = nullptr;
struct SubMessage;
std::map<SubSocket *, SubMessage *> messages_;
std::map<std::string, SubMessage *> services_;
};
class MessageBuilder : public capnp::MallocMessageBuilder {
public:
MessageBuilder() = default;
cereal::Event::Builder initEvent(bool valid = true) {
cereal::Event::Builder event = initRoot<cereal::Event>();
event.setLogMonoTime(nanos_since_boot());
event.setValid(valid);
return event;
}
kj::ArrayPtr<capnp::byte> toBytes() {
heapArray_ = capnp::messageToFlatArray(*this);
return heapArray_.asBytes();
}
size_t getSerializedSize() {
return capnp::computeSerializedSizeInWords(*this) * sizeof(capnp::word);
}
int serializeToBuffer(unsigned char *buffer, size_t buffer_size) {
size_t serialized_size = getSerializedSize();
if (serialized_size > buffer_size) { return -1; }
kj::ArrayOutputStream out(kj::ArrayPtr<capnp::byte>(buffer, buffer_size));
capnp::writeMessage(out, *this);
return serialized_size;
}
private:
kj::Array<capnp::word> heapArray_;
};
class PubMaster {
public:
PubMaster(const std::vector<const char *> &service_list);
inline int send(const char *name, capnp::byte *data, size_t size) { return sockets_.at(name)->send((char *)data, size); }
int send(const char *name, MessageBuilder &msg);
~PubMaster();
private:
std::map<std::string, PubSocket *> sockets_;
};
class AlignedBuffer {
public:
kj::ArrayPtr<const capnp::word> align(const char *data, const size_t size) {
words_size = size / sizeof(capnp::word) + 1;
if (aligned_buf.size() < words_size) {
aligned_buf = kj::heapArray<capnp::word>(words_size < 512 ? 512 : words_size);
}
memcpy(aligned_buf.begin(), data, size);
return aligned_buf.slice(0, words_size);
}
inline kj::ArrayPtr<const capnp::word> align(Message *m) {
return align(m->getData(), m->getSize());
}
private:
kj::Array<capnp::word> aligned_buf;
size_t words_size;
};

146
cereal/messaging/msgq_to_zmq.cc Normal file
View File

@@ -0,0 +1,146 @@
#include "cereal/messaging/msgq_to_zmq.h"
#include <cassert>
#include "cereal/services.h"
#include "common/util.h"
extern ExitHandler do_exit;
// Max messages to process per socket per poll
constexpr int MAX_MESSAGES_PER_SOCKET = 50;
static std::string recv_zmq_msg(void *sock) {
zmq_msg_t msg;
zmq_msg_init(&msg);
std::string ret;
if (zmq_msg_recv(&msg, sock, 0) > 0) {
ret.assign((char *)zmq_msg_data(&msg), zmq_msg_size(&msg));
}
zmq_msg_close(&msg);
return ret;
}
void MsgqToZmq::run(const std::vector<std::string> &endpoints, const std::string &ip) {
zmq_context = std::make_unique<BridgeZmqContext>();
msgq_context = std::make_unique<Context>();
// Create ZMQPubSockets for each endpoint
for (const auto &endpoint : endpoints) {
auto &socket_pair = socket_pairs.emplace_back();
socket_pair.endpoint = endpoint;
socket_pair.pub_sock = std::make_unique<BridgeZmqPubSocket>();
int ret = socket_pair.pub_sock->connect(zmq_context.get(), endpoint);
if (ret != 0) {
printf("Failed to create ZMQ publisher for [%s]: %s\n", endpoint.c_str(), zmq_strerror(zmq_errno()));
return;
}
}
// Start ZMQ monitoring thread to monitor socket events
std::thread thread(&MsgqToZmq::zmqMonitorThread, this);
// Main loop for processing messages
while (!do_exit) {
{
std::unique_lock lk(mutex);
cv.wait(lk, [this]() { return do_exit || !sub2pub.empty(); });
if (do_exit) break;
for (auto sub_sock : msgq_poller->poll(100)) {
// Process messages for each socket
BridgeZmqPubSocket *pub_sock = sub2pub.at(sub_sock);
for (int i = 0; i < MAX_MESSAGES_PER_SOCKET; ++i) {
auto msg = std::unique_ptr<Message>(sub_sock->receive(true));
if (!msg) break;
while (pub_sock->sendMessage(msg.get()) == -1) {
if (errno != EINTR) break;
}
}
}
}
util::sleep_for(1); // Give zmqMonitorThread a chance to acquire the mutex
}
thread.join();
}
void MsgqToZmq::zmqMonitorThread() {
std::vector<zmq_pollitem_t> pollitems;
// Set up ZMQ monitor for each pub socket
for (int i = 0; i < socket_pairs.size(); ++i) {
std::string addr = "inproc://op-bridge-monitor-" + std::to_string(i);
zmq_socket_monitor(socket_pairs[i].pub_sock->getRawSocket(), addr.c_str(), ZMQ_EVENT_ACCEPTED | ZMQ_EVENT_DISCONNECTED);
void *monitor_socket = zmq_socket(zmq_context->getRawContext(), ZMQ_PAIR);
zmq_connect(monitor_socket, addr.c_str());
pollitems.emplace_back(zmq_pollitem_t{.socket = monitor_socket, .events = ZMQ_POLLIN});
}
while (!do_exit) {
int ret = zmq_poll(pollitems.data(), pollitems.size(), 1000);
if (ret < 0) {
if (errno == EINTR) {
// Due to frequent EINTR signals from msgq, introduce a brief delay (200 ms)
// to reduce CPU usage during retry attempts.
util::sleep_for(200);
}
continue;
}
for (int i = 0; i < pollitems.size(); ++i) {
if (pollitems[i].revents & ZMQ_POLLIN) {
// First frame in message contains event number and value
std::string frame = recv_zmq_msg(pollitems[i].socket);
if (frame.empty()) continue;
uint16_t event_type = *(uint16_t *)(frame.data());
// Second frame in message contains event address
frame = recv_zmq_msg(pollitems[i].socket);
if (frame.empty()) continue;
std::unique_lock lk(mutex);
auto &pair = socket_pairs[i];
if (event_type & ZMQ_EVENT_ACCEPTED) {
printf("socket [%s] connected\n", pair.endpoint.c_str());
if (++pair.connected_clients == 1) {
// Create new MSGQ subscriber socket and map to ZMQ publisher
pair.sub_sock = std::make_unique<MSGQSubSocket>();
size_t queue_size = services.at(pair.endpoint).queue_size;
pair.sub_sock->connect(msgq_context.get(), pair.endpoint, "127.0.0.1", false, true, queue_size);
sub2pub[pair.sub_sock.get()] = pair.pub_sock.get();
registerSockets();
}
} else if (event_type & ZMQ_EVENT_DISCONNECTED) {
printf("socket [%s] disconnected\n", pair.endpoint.c_str());
if (pair.connected_clients == 0 || --pair.connected_clients == 0) {
// Remove MSGQ subscriber socket from mapping and reset it
sub2pub.erase(pair.sub_sock.get());
pair.sub_sock.reset(nullptr);
registerSockets();
}
}
cv.notify_one();
}
}
}
// Clean up monitor sockets
for (int i = 0; i < pollitems.size(); ++i) {
zmq_socket_monitor(socket_pairs[i].pub_sock->getRawSocket(), nullptr, 0);
zmq_close(pollitems[i].socket);
}
cv.notify_one();
}
void MsgqToZmq::registerSockets() {
msgq_poller = std::make_unique<MSGQPoller>();
for (const auto &socket_pair : socket_pairs) {
if (socket_pair.sub_sock) {
msgq_poller->registerSocket(socket_pair.sub_sock.get());
}
}
}

36
cereal/messaging/msgq_to_zmq.h Normal file
View File

@@ -0,0 +1,36 @@
#pragma once
#include <condition_variable>
#include <map>
#include <memory>
#include <mutex>
#include <string>
#include <vector>
#include "msgq/impl_msgq.h"
#include "cereal/messaging/bridge_zmq.h"
class MsgqToZmq {
public:
MsgqToZmq() {}
void run(const std::vector<std::string> &endpoints, const std::string &ip);
protected:
void registerSockets();
void zmqMonitorThread();
struct SocketPair {
std::string endpoint;
std::unique_ptr<BridgeZmqPubSocket> pub_sock;
std::unique_ptr<MSGQSubSocket> sub_sock;
int connected_clients = 0;
};
std::unique_ptr<Context> msgq_context;
std::unique_ptr<BridgeZmqContext> zmq_context;
std::mutex mutex;
std::condition_variable cv;
std::unique_ptr<MSGQPoller> msgq_poller;
std::map<SubSocket *, BridgeZmqPubSocket *> sub2pub;
std::vector<SocketPair> socket_pairs;
};

204
cereal/messaging/socketmaster.cc Normal file
View File

@@ -0,0 +1,204 @@
#include <assert.h>
#include <stdlib.h>
#include <string>
#include <mutex>
#include "cereal/services.h"
#include "cereal/messaging/messaging.h"
const bool SIMULATION = (getenv("SIMULATION") != nullptr) && (std::string(getenv("SIMULATION")) == "1");
static inline bool inList(const std::vector<const char *> &list, const char *value) {
for (auto &v : list) {
if (strcmp(value, v) == 0) return true;
}
return false;
}
class MessageContext {
public:
MessageContext() : ctx_(nullptr) {}
~MessageContext() { delete ctx_; }
inline Context *context() {
std::call_once(init_flag, [=]() { ctx_ = Context::create(); });
return ctx_;
}
private:
Context *ctx_;
std::once_flag init_flag;
};
MessageContext message_context;
struct SubMaster::SubMessage {
std::string name;
SubSocket *socket = nullptr;
float freq = 0.0f;
bool updated = false, alive = false, valid = false, ignore_alive;
uint64_t rcv_time = 0, rcv_frame = 0;
void *allocated_msg_reader = nullptr;
bool is_polled = false;
capnp::FlatArrayMessageReader *msg_reader = nullptr;
AlignedBuffer aligned_buf;
cereal::Event::Reader event;
};
SubMaster::SubMaster(const std::vector<const char *> &service_list, const std::vector<const char *> &poll,
const char *address, const std::vector<const char *> &ignore_alive) {
poller_ = Poller::create();
for (auto name : service_list) {
assert(services.count(std::string(name)) > 0);
service serv = services.at(std::string(name));
SubSocket *socket = SubSocket::create(message_context.context(), name, address ? address : "127.0.0.1", true, true, serv.queue_size);
assert(socket != 0);
bool is_polled = inList(poll, name) || poll.empty();
if (is_polled) poller_->registerSocket(socket);
SubMessage *m = new SubMessage{
.name = name,
.socket = socket,
.freq = serv.frequency,
.ignore_alive = inList(ignore_alive, name),
.allocated_msg_reader = malloc(sizeof(capnp::FlatArrayMessageReader)),
.is_polled = is_polled};
m->msg_reader = new (m->allocated_msg_reader) capnp::FlatArrayMessageReader({});
messages_[socket] = m;
services_[name] = m;
}
}
void SubMaster::update(int timeout) {
for (auto &kv : messages_) kv.second->updated = false;
auto sockets = poller_->poll(timeout);
// add non-polled sockets for non-blocking receive
for (auto &kv : messages_) {
SubMessage *m = kv.second;
SubSocket *s = kv.first;
if (!m->is_polled) sockets.push_back(s);
}
uint64_t current_time = nanos_since_boot();
std::vector<std::pair<std::string, cereal::Event::Reader>> messages;
for (auto s : sockets) {
Message *msg = s->receive(true);
if (msg == nullptr) continue;
SubMessage *m = messages_.at(s);
m->msg_reader->~FlatArrayMessageReader();
capnp::ReaderOptions options;
options.traversalLimitInWords = kj::maxValue; // Don't limit
m->msg_reader = new (m->allocated_msg_reader) capnp::FlatArrayMessageReader(m->aligned_buf.align(msg), options);
delete msg;
messages.push_back({m->name, m->msg_reader->getRoot<cereal::Event>()});
}
update_msgs(current_time, messages);
}
void SubMaster::update_msgs(uint64_t current_time, const std::vector<std::pair<std::string, cereal::Event::Reader>> &messages){
if (++frame == UINT64_MAX) frame = 1;
for (auto &kv : messages) {
auto m_find = services_.find(kv.first);
if (m_find == services_.end()){
continue;
}
SubMessage *m = m_find->second;
m->event = kv.second;
m->updated = true;
m->rcv_time = current_time;
m->rcv_frame = frame;
m->valid = m->event.getValid();
if (SIMULATION) m->alive = true;
}
if (!SIMULATION) {
for (auto &kv : messages_) {
SubMessage *m = kv.second;
m->alive = (m->freq <= (1e-5) || ((current_time - m->rcv_time) * (1e-9)) < (10.0 / m->freq));
}
}
}
bool SubMaster::all_(const std::vector<const char *> &service_list, bool valid, bool alive) {
int found = 0;
for (auto &kv : messages_) {
SubMessage *m = kv.second;
if (service_list.size() == 0 || inList(service_list, m->name.c_str())) {
found += (!valid || m->valid) && (!alive || (m->alive || m->ignore_alive));
}
}
return service_list.size() == 0 ? found == messages_.size() : found == service_list.size();
}
void SubMaster::drain() {
while (true) {
auto polls = poller_->poll(0);
if (polls.size() == 0)
break;
for (auto sock : polls) {
Message *msg = sock->receive(true);
delete msg;
}
}
}
bool SubMaster::updated(const char *name) const {
return services_.at(name)->updated;
}
bool SubMaster::alive(const char *name) const {
return services_.at(name)->alive;
}
bool SubMaster::valid(const char *name) const {
return services_.at(name)->valid;
}
uint64_t SubMaster::rcv_frame(const char *name) const {
return services_.at(name)->rcv_frame;
}
uint64_t SubMaster::rcv_time(const char *name) const {
return services_.at(name)->rcv_time;
}
cereal::Event::Reader &SubMaster::operator[](const char *name) const {
return services_.at(name)->event;
}
SubMaster::~SubMaster() {
delete poller_;
for (auto &kv : messages_) {
SubMessage *m = kv.second;
m->msg_reader->~FlatArrayMessageReader();
free(m->allocated_msg_reader);
delete m->socket;
delete m;
}
}
PubMaster::PubMaster(const std::vector<const char *> &service_list) {
for (auto name : service_list) {
assert(services.count(name) > 0);
service serv = services.at(std::string(name));
PubSocket *socket = PubSocket::create(message_context.context(), name, true, serv.queue_size);
assert(socket);
sockets_[name] = socket;
}
}
int PubMaster::send(const char *name, MessageBuilder &msg) {
auto bytes = msg.toBytes();
return send(name, bytes.begin(), bytes.size());
}
PubMaster::~PubMaster() {
for (auto s : sockets_) delete s.second;
}

0
cereal/messaging/tests/__init__.py Normal file
View File

185
cereal/messaging/tests/test_messaging.py Normal file
View File

@@ -0,0 +1,185 @@
import os
import capnp
import multiprocessing
import numbers
import random
import threading
import time
from openpilot.common.parameterized import parameterized
import pytest
from cereal import log, car
import cereal.messaging as messaging
from cereal.services import SERVICE_LIST
events = [evt for evt in log.Event.schema.union_fields if evt in SERVICE_LIST.keys()]
def random_sock():
return random.choice(events)
def random_socks(num_socks=10):
return list({random_sock() for _ in range(num_socks)})
def random_bytes(length=1000):
return bytes([random.randrange(0xFF) for _ in range(length)])
def zmq_sleep(t=1):
if "ZMQ" in os.environ:
time.sleep(t)
# TODO: this should take any capnp struct and returrn a msg with random populated data
def random_carstate():
fields = ["vEgo", "aEgo", "brake", "steeringAngleDeg"]
msg = messaging.new_message("carState")
cs = msg.carState
for f in fields:
setattr(cs, f, random.random() * 10)
return msg
# TODO: this should compare any capnp structs
def assert_carstate(cs1, cs2):
for f in car.CarState.schema.non_union_fields:
# TODO: check all types
val1, val2 = getattr(cs1, f), getattr(cs2, f)
if isinstance(val1, numbers.Number):
assert val1 == val2, f"{f}: sent '{val1}' vs recvd '{val2}'"
def delayed_send(delay, sock, dat):
def send_func():
sock.send(dat)
threading.Timer(delay, send_func).start()
class TestMessaging:
def setUp(self):
# TODO: ZMQ tests are too slow; all sleeps will need to be
# replaced with logic to block on the necessary condition
if "ZMQ" in os.environ:
pytest.skip()
# ZMQ pub socket takes too long to die
# sleep to prevent multiple publishers error between tests
zmq_sleep()
@parameterized.expand(events)
def test_new_message(self, evt):
try:
msg = messaging.new_message(evt)
except capnp.lib.capnp.KjException:
msg = messaging.new_message(evt, random.randrange(200))
assert (time.monotonic() - msg.logMonoTime) < 0.1
assert not msg.valid
assert evt == msg.which()
@parameterized.expand(events)
def test_pub_sock(self, evt):
messaging.pub_sock(evt)
@parameterized.expand(events)
def test_sub_sock(self, evt):
messaging.sub_sock(evt)
@parameterized.expand([
(messaging.drain_sock, capnp._DynamicStructReader),
(messaging.drain_sock_raw, bytes),
])
def test_drain_sock(self, func, expected_type):
sock = "carState"
pub_sock = messaging.pub_sock(sock)
sub_sock = messaging.sub_sock(sock, timeout=1000)
zmq_sleep()
# no wait and no msgs in queue
msgs = func(sub_sock)
assert isinstance(msgs, list)
assert len(msgs) == 0
# no wait but msgs are queued up
num_msgs = random.randrange(3, 10)
for _ in range(num_msgs):
pub_sock.send(messaging.new_message(sock).to_bytes())
time.sleep(0.1)
msgs = func(sub_sock)
assert isinstance(msgs, list)
assert all(isinstance(msg, expected_type) for msg in msgs)
assert len(msgs) == num_msgs
def test_recv_sock(self):
sock = "carState"
pub_sock = messaging.pub_sock(sock)
sub_sock = messaging.sub_sock(sock, timeout=100)
zmq_sleep()
# no wait and no msg in queue, socket should timeout
recvd = messaging.recv_sock(sub_sock)
assert recvd is None
# no wait and one msg in queue
msg = random_carstate()
pub_sock.send(msg.to_bytes())
time.sleep(0.01)
recvd = messaging.recv_sock(sub_sock)
assert isinstance(recvd, capnp._DynamicStructReader)
# https://github.com/python/mypy/issues/13038
assert_carstate(msg.carState, recvd.carState)
def test_recv_one(self):
sock = "carState"
pub_sock = messaging.pub_sock(sock)
sub_sock = messaging.sub_sock(sock, timeout=1000)
zmq_sleep()
# no msg in queue, socket should timeout
recvd = messaging.recv_one(sub_sock)
assert recvd is None
# one msg in queue
msg = random_carstate()
pub_sock.send(msg.to_bytes())
recvd = messaging.recv_one(sub_sock)
assert isinstance(recvd, capnp._DynamicStructReader)
assert_carstate(msg.carState, recvd.carState)
@pytest.mark.xfail(condition="ZMQ" in os.environ, reason='ZMQ detected')
def test_recv_one_or_none(self):
sock = "carState"
pub_sock = messaging.pub_sock(sock)
sub_sock = messaging.sub_sock(sock)
zmq_sleep()
# no msg in queue, socket shouldn't block
recvd = messaging.recv_one_or_none(sub_sock)
assert recvd is None
# one msg in queue
msg = random_carstate()
pub_sock.send(msg.to_bytes())
recvd = messaging.recv_one_or_none(sub_sock)
assert isinstance(recvd, capnp._DynamicStructReader)
assert_carstate(msg.carState, recvd.carState)
def test_recv_one_retry(self):
sock = "carState"
sock_timeout = 0.1
pub_sock = messaging.pub_sock(sock)
sub_sock = messaging.sub_sock(sock, timeout=round(sock_timeout*1000))
zmq_sleep()
# this test doesn't work with ZMQ since multiprocessing interrupts it
if "ZMQ" not in os.environ:
# wait 5 socket timeouts and make sure it's still retrying
p = multiprocessing.Process(target=messaging.recv_one_retry, args=(sub_sock,))
p.start()
time.sleep(sock_timeout*5)
assert p.is_alive()
p.terminate()
# wait 5 socket timeouts before sending
msg = random_carstate()
start_time = time.monotonic()
delayed_send(sock_timeout*5, pub_sock, msg.to_bytes())
recvd = messaging.recv_one_retry(sub_sock)
assert (time.monotonic() - start_time) >= sock_timeout*5
assert isinstance(recvd, capnp._DynamicStructReader)
assert_carstate(msg.carState, recvd.carState)

160
cereal/messaging/tests/test_pub_sub_master.py Normal file
View File

@@ -0,0 +1,160 @@
import random
import time
from typing import Sized, cast
import cereal.messaging as messaging
from cereal.messaging.tests.test_messaging import events, random_sock, random_socks, \
random_bytes, random_carstate, assert_carstate, \
zmq_sleep
from cereal.services import SERVICE_LIST
class TestSubMaster:
def setup_method(self):
# ZMQ pub socket takes too long to die
# sleep to prevent multiple publishers error between tests
zmq_sleep(3)
def test_init(self):
sm = messaging.SubMaster(events)
for p in [sm.updated, sm.recv_time, sm.recv_frame, sm.alive,
sm.sock, sm.data, sm.logMonoTime, sm.valid]:
assert len(cast(Sized, p)) == len(events)
def test_init_state(self):
socks = random_socks()
sm = messaging.SubMaster(socks)
assert sm.frame == -1
assert not any(sm.updated.values())
assert not any(sm.seen.values())
on_demand = {s: SERVICE_LIST[s].frequency <= 1e-5 for s in sm.services}
assert all(sm.alive[s] == sm.valid[s] == sm.freq_ok[s] == on_demand[s] for s in sm.services)
assert all(t == 0. for t in sm.recv_time.values())
assert all(f == 0 for f in sm.recv_frame.values())
assert all(t == 0 for t in sm.logMonoTime.values())
for p in [sm.updated, sm.recv_time, sm.recv_frame, sm.alive,
sm.sock, sm.data, sm.logMonoTime, sm.valid]:
assert len(cast(Sized, p)) == len(socks)
def test_getitem(self):
sock = "carState"
pub_sock = messaging.pub_sock(sock)
sm = messaging.SubMaster([sock,])
zmq_sleep()
msg = random_carstate()
pub_sock.send(msg.to_bytes())
sm.update(1000)
assert_carstate(msg.carState, sm[sock])
# TODO: break this test up to individually test SubMaster.update and SubMaster.update_msgs
def test_update(self):
sock = "carState"
pub_sock = messaging.pub_sock(sock)
sm = messaging.SubMaster([sock,])
zmq_sleep()
for i in range(10):
msg = messaging.new_message(sock)
pub_sock.send(msg.to_bytes())
sm.update(1000)
assert sm.frame == i
assert all(sm.updated.values())
def test_update_timeout(self):
sock = random_sock()
sm = messaging.SubMaster([sock,])
timeout = random.randrange(1000, 3000)
start_time = time.monotonic()
sm.update(timeout)
t = time.monotonic() - start_time
assert t >= timeout/1000.
assert t < 3
assert not any(sm.updated.values())
def test_avg_frequency_checks(self):
for poll in (True, False):
sm = messaging.SubMaster(["modelV2", "carParams", "carState", "cameraOdometry", "liveCalibration"],
poll=("modelV2" if poll else None),
frequency=(20. if not poll else None))
checks = {
"carState": (20, 20),
"modelV2": (20, 20 if poll else 10),
"cameraOdometry": (20, 10),
"liveCalibration": (4, 4),
"carParams": (None, None),
"userBookmark": (None, None),
}
for service, (max_freq, min_freq) in checks.items():
if max_freq is not None:
assert sm._check_avg_freq(service)
assert sm.freq_tracker[service].max_freq == max_freq*1.2
assert sm.freq_tracker[service].min_freq == min_freq*0.8
else:
assert not sm._check_avg_freq(service)
def test_alive(self):
pass
def test_ignore_alive(self):
pass
def test_valid(self):
pass
# SubMaster should always conflate
def test_conflate(self):
sock = "carState"
pub_sock = messaging.pub_sock(sock)
sm = messaging.SubMaster([sock,])
n = 10
for i in range(n+1):
msg = messaging.new_message(sock)
msg.carState.vEgo = i
pub_sock.send(msg.to_bytes())
time.sleep(0.01)
sm.update(1000)
assert sm[sock].vEgo == n
class TestPubMaster:
def setup_method(self):
# ZMQ pub socket takes too long to die
# sleep to prevent multiple publishers error between tests
zmq_sleep(3)
def test_init(self):
messaging.PubMaster(events)
def test_send(self):
socks = random_socks()
pm = messaging.PubMaster(socks)
sub_socks = {s: messaging.sub_sock(s, conflate=True, timeout=1000) for s in socks}
zmq_sleep()
# PubMaster accepts either a capnp msg builder or bytes
for capnp in [True, False]:
for i in range(100):
sock = socks[i % len(socks)]
if capnp:
try:
msg = messaging.new_message(sock)
except Exception:
msg = messaging.new_message(sock, random.randrange(50))
else:
msg = random_bytes()
pm.send(sock, msg)
recvd = sub_socks[sock].receive()
if capnp:
msg.clear_write_flag()
msg = msg.to_bytes()
assert msg == recvd, i

21
cereal/messaging/tests/test_services.py Normal file
View File

@@ -0,0 +1,21 @@
import os
import tempfile
from typing import Dict
from openpilot.common.parameterized import parameterized
import cereal.services as services
from cereal.services import SERVICE_LIST
class TestServices:
@parameterized.expand(SERVICE_LIST.keys())
def test_services(self, s):
service = SERVICE_LIST[s]
assert service.frequency <= 104
assert service.decimation != 0
def test_generated_header(self):
with tempfile.NamedTemporaryFile(suffix=".h") as f:
ret = os.system(f"python3 {services.__file__} > {f.name} && clang++ {f.name} -std=c++11")
assert ret == 0, "generated services header is not valid C"

256
cereal/messaging/tests/validate_sp_cereal_upstream.py Executable file
View File

@@ -0,0 +1,256 @@
#!/usr/bin/env python3
"""Schema-level cereal compat check between sunnypilot and upstream openpilot.
Rules (per struct matched across sides by typeId):
R1 shared ordinal must reference the same type.
R2 sunnypilot-only ordinal in a union -> FAIL (unknown discriminant upstream).
R3 sunnypilot-only ordinal on a regular field -> OK (additive struct evolution).
R4 upstream-only ordinal -> OK.
R5 sunnypilot-only struct referenced via an upstream-shared field -> FAIL.
"""
from __future__ import annotations
import argparse
import json
import os
import sys
from typing import Any
NO_DISCRIMINANT = 0xFFFF
def hex_id(value: int) -> str:
return f"0x{value:016x}"
def encode_type(type_node: Any) -> dict:
which = type_node.which()
if which == "struct":
return {"kind": "struct", "typeId": hex_id(type_node.struct.typeId)}
if which == "enum":
return {"kind": "enum", "typeId": hex_id(type_node.enum.typeId)}
if which == "interface":
return {"kind": "interface", "typeId": hex_id(type_node.interface.typeId)}
if which == "list":
return {"kind": "list", "element": encode_type(type_node.list.elementType)}
if which == "anyPointer":
return {"kind": "anyPointer"}
return {"kind": which}
def encode_field(name: str, field: Any) -> dict:
proto = field.proto
ordinal = proto.ordinal.explicit if proto.ordinal.which() == "explicit" else None
discriminant = proto.discriminantValue if proto.discriminantValue != NO_DISCRIMINANT else None
if proto.which() == "group":
type_desc = {"kind": "group", "typeId": hex_id(proto.group.typeId)}
else:
type_desc = encode_type(proto.slot.type)
return {
"name": name,
"ordinal": ordinal,
"discriminant": discriminant,
"type": type_desc,
}
def encode_struct(schema: Any) -> dict:
node = schema.node
return {
"typeId": hex_id(node.id),
"displayName": node.displayName,
"hasUnion": node.struct.discriminantCount > 0,
"fields": [encode_field(name, field) for name, field in schema.fields.items()],
}
def _child_struct_schema(field: Any) -> Any:
proto = field.proto
if proto.which() == "group":
return field.schema
type_node = proto.slot.type
which = type_node.which()
if which == "struct":
return field.schema
if which == "list":
container = field.schema
element_type = type_node.list.elementType
while element_type.which() == "list":
container = container.elementType
element_type = element_type.list.elementType
if element_type.which() == "struct":
return container.elementType
return None
def collect_schema(root: Any) -> dict[str, dict]:
structs: dict[str, dict] = {}
stack = [root]
while stack:
schema = stack.pop()
type_id = hex_id(schema.node.id)
if type_id in structs:
continue
structs[type_id] = encode_struct(schema)
for _name, field in schema.fields.items():
try:
child = _child_struct_schema(field)
except Exception:
child = None
if child is not None:
stack.append(child)
return structs
def load_log(cereal_dir: str) -> Any:
import capnp
cereal_dir = os.path.abspath(cereal_dir)
capnp.remove_import_hook()
return capnp.load(os.path.join(cereal_dir, "log.capnp"), imports=[cereal_dir])
def dump_schema(cereal_dir: str, path: str) -> None:
log = load_log(cereal_dir)
payload = {
"root": hex_id(log.Event.schema.node.id),
"structs": collect_schema(log.Event.schema),
}
with open(path, "w", encoding="utf-8") as handle:
json.dump(payload, handle, indent=2, sort_keys=True)
print(f"wrote schema dump with {len(payload['structs'])} structs to {path}")
def types_equal(a: dict, b: dict) -> bool:
if a.get("kind") != b.get("kind"):
return False
kind = a["kind"]
if kind in ("struct", "enum", "interface", "group"):
return a.get("typeId") == b.get("typeId")
if kind == "list":
return types_equal(a["element"], b["element"])
return True
def type_repr(t: dict) -> str:
kind = t.get("kind", "?")
if kind in ("struct", "enum", "interface", "group"):
return f"{kind}({t.get('typeId')})"
if kind == "list":
return f"list<{type_repr(t['element'])}>"
return kind
def field_is_union_variant(field: dict) -> bool:
return field.get("discriminant") is not None
def index_fields_by_ordinal(struct: dict) -> dict[int, dict]:
indexed: dict[int, dict] = {}
for field in struct["fields"]:
ordinal = field.get("ordinal")
if ordinal is None:
continue
indexed[ordinal] = field
return indexed
def compare(sunnypilot_dump: dict, upstream_dump: dict) -> list[str]:
violations: list[str] = []
sunnypilot_structs: dict[str, dict] = sunnypilot_dump["structs"]
upstream_structs: dict[str, dict] = upstream_dump["structs"]
sunnypilot_struct_referenced_from_shared: set[str] = set()
for type_id, sunnypilot_struct in sunnypilot_structs.items():
upstream_struct = upstream_structs.get(type_id)
if upstream_struct is None:
continue
sunnypilot_fields = index_fields_by_ordinal(sunnypilot_struct)
upstream_fields = index_fields_by_ordinal(upstream_struct)
display = sunnypilot_struct["displayName"]
for ordinal, sunnypilot_field in sunnypilot_fields.items():
upstream_field = upstream_fields.get(ordinal)
if upstream_field is None:
if field_is_union_variant(sunnypilot_field):
violations.append(
f"[R2] {display} @{ordinal} ('{sunnypilot_field['name']}', {type_repr(sunnypilot_field['type'])}): "
f"union variant not present upstream. upstream cannot parse this discriminant."
)
continue
if not types_equal(sunnypilot_field["type"], upstream_field["type"]):
violations.append(
f"[R1] {display} @{ordinal}: type mismatch. "
f"sunnypilot='{sunnypilot_field['name']}' {type_repr(sunnypilot_field['type'])} vs "
f"upstream='{upstream_field['name']}' {type_repr(upstream_field['type'])}."
)
continue
cursor = sunnypilot_field["type"]
while cursor.get("kind") == "list":
cursor = cursor["element"]
if cursor.get("kind") in ("struct", "group", "interface") and cursor.get("typeId"):
sunnypilot_struct_referenced_from_shared.add(cursor["typeId"])
for type_id, sunnypilot_struct in sunnypilot_structs.items():
if type_id in upstream_structs:
continue
if type_id in sunnypilot_struct_referenced_from_shared:
violations.append(
f"[R5] struct {sunnypilot_struct['displayName']} ({type_id}) exists only on sunnypilot "
f"but is referenced from an upstream-shared field. upstream cannot resolve this type."
)
return violations
def load_peer(path: str) -> dict:
with open(path, "r", encoding="utf-8") as handle:
return json.load(handle)
def run_read(cereal_dir: str, peer_path: str) -> int:
log = load_log(cereal_dir)
peer_dump = load_peer(peer_path)
local_dump = {
"root": hex_id(log.Event.schema.node.id),
"structs": collect_schema(log.Event.schema),
}
violations = compare(sunnypilot_dump=peer_dump, upstream_dump=local_dump)
if not violations:
print("cereal compat OK: upstream openpilot can parse sunnypilot routes "
"(no leaked structs, no ordinal collisions).")
return 0
print(f"cereal compat FAIL: upstream openpilot would misparse sunnypilot routes "
f"({len(violations)} violation(s)):")
for v in violations:
print(f" {v}")
return 1
def main() -> int:
parser = argparse.ArgumentParser(
description="sunnypilot <-> upstream cereal compatibility validator (schema-level)."
)
mode = parser.add_mutually_exclusive_group(required=True)
mode.add_argument("-g", "--generate", action="store_true", help="dump local schema to JSON")
mode.add_argument("-r", "--read", action="store_true", help="load peer JSON and diff against local")
parser.add_argument("-f", "--file", default="schema.json", help="JSON file path (default: schema.json)")
parser.add_argument("--cereal-dir", required=True, help="path to cereal directory containing log.capnp")
args = parser.parse_args()
if args.generate:
dump_schema(args.cereal_dir, args.file)
return 0
return run_read(args.cereal_dir, args.file)
if __name__ == "__main__":
sys.exit(main())

147
cereal/services.py Executable file
View File

@@ -0,0 +1,147 @@
#!/usr/bin/env python3
from enum import IntEnum
from typing import Optional
# TODO: this should be automatically determined using the capnp schema
class QueueSize(IntEnum):
BIG = 10 * 1024 * 1024 # 10MB - video frames, large AI outputs
MEDIUM = 2 * 1024 * 1024 # 2MB - high freq (CAN), livestream
SMALL = 250 * 1024 # 250KB - most services
class Service:
def __init__(self, should_log: bool, frequency: float, decimation: Optional[int] = None,
queue_size: QueueSize = QueueSize.SMALL):
self.should_log = should_log
self.frequency = frequency
self.decimation = decimation
self.queue_size = queue_size
_services: dict[str, tuple] = {
# service: (should_log, frequency, qlog decimation (optional))
# note: the "EncodeIdx" packets will still be in the log
"gyroscope": (True, 104., 104),
"accelerometer": (True, 104., 104),
"magnetometer": (True, 25.),
"lightSensor": (True, 100., 100),
"temperatureSensor": (True, 2., 200),
"gpsNMEA": (True, 9.),
"deviceState": (True, 2., 1),
"touch": (True, 20., 1),
"can": (True, 100., 2053, QueueSize.BIG), # decimation gives ~3 msgs in a full segment
"controlsState": (True, 100., 10, QueueSize.MEDIUM),
"selfdriveState": (True, 100., 10),
"pandaStates": (True, 10., 1),
"peripheralState": (True, 2., 1),
"radarState": (True, 20., 5),
"roadEncodeIdx": (False, 20., 1),
"liveTracks": (True, 20.),
"sendcan": (True, 100., 139, QueueSize.MEDIUM),
"logMessage": (True, 0., None, QueueSize.BIG),
"errorLogMessage": (True, 0., 1, QueueSize.BIG),
"liveCalibration": (True, 4., 4),
"liveTorqueParameters": (True, 4., 1),
"liveDelay": (True, 4., 1),
"androidLog": (True, 0.),
"carState": (True, 100., 10),
"carControl": (True, 100., 10),
"carOutput": (True, 100., 10),
"longitudinalPlan": (True, 20., 10),
"lateralManeuverPlan": (True, 20.),
"driverAssistance": (True, 20., 20),
"procLog": (True, 0.5, 15, QueueSize.BIG),
"gpsLocationExternal": (True, 10., 10),
"gpsLocation": (True, 1., 1),
"ubloxGnss": (True, 10.),
"qcomGnss": (True, 2.),
"gnssMeasurements": (True, 10., 10),
"clocks": (True, 0.1, 1),
"ubloxRaw": (True, 20.),
"livePose": (True, 20., 4),
"liveParameters": (True, 20., 5),
"cameraOdometry": (True, 20., 10),
"thumbnail": (True, 1 / 60., 1),
"onroadEvents": (True, 1., 1),
"carParams": (True, 0.02, 1),
"roadCameraState": (True, 20., 20),
"driverCameraState": (True, 20., 20),
"driverEncodeIdx": (False, 20., 1),
"driverStateV2": (True, 20., 10),
"driverMonitoringState": (True, 20., 10),
"wideRoadEncodeIdx": (False, 20., 1),
"wideRoadCameraState": (True, 20., 20),
"drivingModelData": (True, 20., 10),
"modelV2": (True, 20., None, QueueSize.BIG),
"managerState": (True, 2., 1),
"uploaderState": (True, 0., 1),
"navInstruction": (True, 1., 10),
"navRoute": (True, 0.),
"navThumbnail": (True, 0.),
"qRoadEncodeIdx": (False, 20.),
"userBookmark": (True, 0., 1),
"soundPressure": (True, 10., 10),
"rawAudioData": (False, 20.),
"bookmarkButton": (True, 0., 1),
"audioFeedback": (True, 0., 1),
"roadEncodeData": (False, 20., None, QueueSize.BIG),
"driverEncodeData": (False, 20., None, QueueSize.BIG),
"wideRoadEncodeData": (False, 20., None, QueueSize.BIG),
"qRoadEncodeData": (False, 20., None, QueueSize.BIG),
# sunnypilot
"modelManagerSP": (False, 1., 1, QueueSize.BIG),
"backupManagerSP": (False, 1., 1, QueueSize.BIG),
"selfdriveStateSP": (True, 100., 10),
"longitudinalPlanSP": (True, 20., 10),
"onroadEventsSP": (True, 1., 1),
"carParamsSP": (True, 0.02, 1),
"carControlSP": (True, 100., 10),
"carStateSP": (True, 100., 10),
"liveMapDataSP": (True, 1., 1),
"modelDataV2SP": (True, 20., None, QueueSize.BIG),
"liveLocationKalman": (True, 20.),
# debug
"uiDebug": (True, 0., 1),
"testJoystick": (True, 0.),
"alertDebug": (True, 20., 5),
"livestreamWideRoadEncodeIdx": (False, 20.),
"livestreamRoadEncodeIdx": (False, 20.),
"livestreamDriverEncodeIdx": (False, 20.),
"livestreamWideRoadEncodeData": (False, 20., None, QueueSize.MEDIUM),
"livestreamRoadEncodeData": (False, 20., None, QueueSize.MEDIUM),
"livestreamDriverEncodeData": (False, 20., None, QueueSize.MEDIUM),
"customReservedRawData0": (True, 0.),
"customReservedRawData1": (True, 0.),
"customReservedRawData2": (True, 0.),
}
SERVICE_LIST = {name: Service(*vals) for
idx, (name, vals) in enumerate(_services.items())}
def build_header():
h = ""
h += "/* THIS IS AN AUTOGENERATED FILE, PLEASE EDIT services.py */\n"
h += "#ifndef __SERVICES_H\n"
h += "#define __SERVICES_H\n"
h += "#include <map>\n"
h += "#include <string>\n"
h += "struct service { std::string name; bool should_log; float frequency; int decimation; size_t queue_size; };\n"
h += "static std::map<std::string, service> services = {\n"
for k, v in SERVICE_LIST.items():
should_log = "true" if v.should_log else "false"
decimation = -1 if v.decimation is None else v.decimation
h += ' { "%s", {"%s", %s, %f, %d, %d}},\n' % \
(k, k, should_log, v.frequency, decimation, v.queue_size)
h += "};\n"
h += "#endif\n"
return h
if __name__ == "__main__":
print(build_header())

23
common/SConscript Normal file
View File

@@ -0,0 +1,23 @@
Import('env', 'envCython')
common_libs = [
'params.cc',
'swaglog.cc',
'util.cc',
'ratekeeper.cc',
]
_common = env.Library('common', common_libs, LIBS="json11")
Export('_common')
if GetOption('extras'):
env.Program('tests/test_common',
['tests/test_runner.cc', 'tests/test_params.cc', 'tests/test_util.cc', 'tests/test_swaglog.cc'],
LIBS=[_common, 'json11', 'zmq', 'pthread'])
# Cython bindings
params_python = envCython.Program('params_pyx.so', 'params_pyx.pyx', LIBS=envCython['LIBS'] + [_common, 'zmq', 'json11'])
common_python = [params_python]
Export('common_python')

0
common/__init__.py Normal file
View File

26
common/api/__init__.py Normal file
View File

@@ -0,0 +1,26 @@
from openpilot.common.api.comma_connect import CommaConnectApi
class Api:
def __init__(self, dongle_id):
self.service = CommaConnectApi(dongle_id)
def request(self, method, endpoint, **params):
return self.service.request(method, endpoint, **params)
def get(self, *args, **kwargs):
return self.service.get(*args, **kwargs)
def post(self, *args, **kwargs):
return self.service.post(*args, **kwargs)
def get_token(self, payload_extra=None, expiry_hours=1):
return self.service.get_token(payload_extra, expiry_hours)
def api_get(endpoint, method='GET', timeout=None, access_token=None, session=None, **params):
return CommaConnectApi(None).api_get(endpoint, method, timeout, access_token, session, **params)
def get_key_pair() -> tuple[str, str, str] | tuple[None, None, None]:
return CommaConnectApi(None).get_key_pair()

72
common/api/base.py Normal file
View File

@@ -0,0 +1,72 @@
import jwt
import os
import requests
import unicodedata
from datetime import datetime, timedelta, UTC
from openpilot.system.hardware.hw import Paths
from openpilot.system.version import get_version
# name: jwt signature algorithm
KEYS = {"id_rsa": "RS256",
"id_ecdsa": "ES256"}
class BaseApi:
def __init__(self, dongle_id, api_host, user_agent="openpilot-"):
self.dongle_id = dongle_id
self.api_host = api_host
self.user_agent = user_agent
self.jwt_algorithm, self.private_key, _ = self.get_key_pair()
def get(self, *args, **kwargs):
return self.request('GET', *args, **kwargs)
def post(self, *args, **kwargs):
return self.request('POST', *args, **kwargs)
def request(self, method, endpoint, timeout=None, access_token=None, **params):
return self.api_get(endpoint, method=method, timeout=timeout, access_token=access_token, **params)
def _get_token(self, payload_extra=None, expiry_hours=1, **extra_payload):
now = datetime.now(UTC).replace(tzinfo=None)
payload = {
'identity': self.dongle_id,
'nbf': now,
'iat': now,
'exp': now + timedelta(hours=expiry_hours),
**extra_payload
}
if payload_extra is not None:
payload.update(payload_extra)
token = jwt.encode(payload, self.private_key, algorithm=self.jwt_algorithm)
if isinstance(token, bytes):
token = token.decode('utf8')
return token
def get_token(self, payload_extra=None, expiry_hours=1):
return self._get_token(payload_extra, expiry_hours)
def remove_non_ascii_chars(self, text):
normalized_text = unicodedata.normalize('NFD', text)
ascii_encoded_text = normalized_text.encode('ascii', 'ignore')
return ascii_encoded_text.decode()
def api_get(self, endpoint, method='GET', timeout=None, access_token=None, session=None, json=None, **params):
headers = {}
if access_token is not None:
headers['Authorization'] = "JWT " + access_token
version = self.remove_non_ascii_chars(get_version())
headers['User-Agent'] = self.user_agent + version
# TODO: add session to Api
req = requests if session is None else session
return req.request(method, f"{self.api_host}/{endpoint}", timeout=timeout, headers=headers, json=json, params=params)
@staticmethod
def get_key_pair() -> tuple[str, str, str] | tuple[None, None, None]:
for key in KEYS:
if os.path.isfile(Paths.persist_root() + f'/comma/{key}') and os.path.isfile(Paths.persist_root() + f'/comma/{key}.pub'):
with open(Paths.persist_root() + f'/comma/{key}') as private, open(Paths.persist_root() + f'/comma/{key}.pub') as public:
return KEYS[key], private.read(), public.read()
return None, None, None

11
common/api/comma_connect.py Normal file
View File

@@ -0,0 +1,11 @@
import os
from openpilot.common.api.base import BaseApi
API_HOST = os.getenv('API_HOST', 'https://api.commadotai.com')
class CommaConnectApi(BaseApi):
def __init__(self, dongle_id):
super().__init__(dongle_id, API_HOST)
self.user_agent = "openpilot-"

4
common/basedir.py Normal file
View File

@@ -0,0 +1,4 @@
import os
BASEDIR = os.path.abspath(os.path.join(os.path.dirname(os.path.realpath(__file__)), "../"))

23
common/constants.py Normal file
View File

@@ -0,0 +1,23 @@
import numpy as np
# conversions
class CV:
# Speed
MPH_TO_KPH = 1.609344
KPH_TO_MPH = 1. / MPH_TO_KPH
MS_TO_KPH = 3.6
KPH_TO_MS = 1. / MS_TO_KPH
MS_TO_MPH = MS_TO_KPH * KPH_TO_MPH
MPH_TO_MS = MPH_TO_KPH * KPH_TO_MS
MS_TO_KNOTS = 1.9438
KNOTS_TO_MS = 1. / MS_TO_KNOTS
# Angle
DEG_TO_RAD = np.pi / 180.
RAD_TO_DEG = 1. / DEG_TO_RAD
# Mass
LB_TO_KG = 0.453592
ACCELERATION_DUE_TO_GRAVITY = 9.81 # m/s^2

37
common/file_chunker.py Normal file
View File

@@ -0,0 +1,37 @@
import math
import os
from pathlib import Path
CHUNK_SIZE = 45 * 1024 * 1024 # 45MB, under GitHub's 50MB limit
def get_chunk_name(name, idx, num_chunks):
return f"{name}.chunk{idx+1:02d}of{num_chunks:02d}"
def get_manifest_path(name):
return f"{name}.chunkmanifest"
def get_chunk_paths(path, file_size):
num_chunks = math.ceil(file_size / CHUNK_SIZE)
return [get_manifest_path(path)] + [get_chunk_name(path, i, num_chunks) for i in range(num_chunks)]
def chunk_file(path, targets):
manifest_path, *chunk_paths = targets
with open(path, 'rb') as f:
data = f.read()
actual_num_chunks = max(1, math.ceil(len(data) / CHUNK_SIZE))
assert len(chunk_paths) >= actual_num_chunks, f"Allowed {len(chunk_paths)} chunks but needs at least {actual_num_chunks}, for path {path}"
for i, chunk_path in enumerate(chunk_paths):
with open(chunk_path, 'wb') as f:
f.write(data[i * CHUNK_SIZE:(i + 1) * CHUNK_SIZE])
Path(manifest_path).write_text(str(len(chunk_paths)))
os.remove(path)
def read_file_chunked(path):
manifest_path = get_manifest_path(path)
if os.path.isfile(manifest_path):
num_chunks = int(Path(manifest_path).read_text().strip())
return b''.join(Path(get_chunk_name(path, i, num_chunks)).read_bytes() for i in range(num_chunks))
if os.path.isfile(path):
return Path(path).read_bytes()
raise FileNotFoundError(path)

34
common/filter_simple.py Normal file
View File

@@ -0,0 +1,34 @@
class FirstOrderFilter:
def __init__(self, x0, rc, dt, initialized=True):
self.x = x0
self.dt = dt
self.update_alpha(rc)
self.initialized = initialized
def update_alpha(self, rc):
self.alpha = self.dt / (rc + self.dt)
def update(self, x):
if self.initialized:
self.x = (1. - self.alpha) * self.x + self.alpha * x
else:
self.initialized = True
self.x = x
return self.x
class BounceFilter(FirstOrderFilter):
def __init__(self, x0, rc, dt, initialized=True, bounce=2):
self.velocity = FirstOrderFilter(0.0, 0.15, dt)
self.bounce = bounce
super().__init__(x0, rc, dt, initialized)
def update(self, x):
super().update(x)
scale = self.dt / (1.0 / 60.0) # tuned at 60 fps
self.velocity.x += (x - self.x) * self.bounce * scale * self.dt
self.velocity.update(0.0)
if abs(self.velocity.x) < 1e-3:
self.velocity.x = 0.0
self.x += self.velocity.x
return self.x

42
common/git.py Normal file
View File

@@ -0,0 +1,42 @@
from functools import cache
import subprocess
from openpilot.common.utils import run_cmd, run_cmd_default
@cache
def get_commit(cwd: str | None = None, branch: str = "HEAD") -> str:
return run_cmd_default(["git", "rev-parse", branch], cwd=cwd)
@cache
def get_commit_date(cwd: str | None = None, commit: str = "HEAD") -> str:
return run_cmd_default(["git", "show", "--no-patch", "--format='%ct %ci'", commit], cwd=cwd)
@cache
def get_short_branch(cwd: str | None = None) -> str:
return run_cmd_default(["git", "rev-parse", "--abbrev-ref", "HEAD"], cwd=cwd)
@cache
def get_branch(cwd: str | None = None) -> str:
return run_cmd_default(["git", "rev-parse", "--abbrev-ref", "--symbolic-full-name", "@{u}"], cwd=cwd)
@cache
def get_origin(cwd: str | None = None) -> str:
try:
local_branch = run_cmd(["git", "name-rev", "--name-only", "HEAD"], cwd=cwd)
tracking_remote = run_cmd(["git", "config", "branch." + local_branch + ".remote"], cwd=cwd)
return run_cmd(["git", "config", "remote." + tracking_remote + ".url"], cwd=cwd)
except subprocess.CalledProcessError: # Not on a branch, fallback
return run_cmd_default(["git", "config", "--get", "remote.origin.url"], cwd=cwd)
@cache
def get_normalized_origin(cwd: str | None = None) -> str:
return get_origin(cwd) \
.replace("git@", "", 1) \
.replace(".git", "", 1) \
.replace("https://", "", 1) \
.replace(":", "/", 1)

89
common/gpio.py Normal file
View File

@@ -0,0 +1,89 @@
import os
import fcntl
import ctypes
from functools import cache
def gpio_init(pin: int, output: bool) -> None:
try:
with open(f"/sys/class/gpio/gpio{pin}/direction", 'wb') as f:
f.write(b"out" if output else b"in")
except Exception as e:
print(f"Failed to set gpio {pin} direction: {e}")
def gpio_set(pin: int, high: bool) -> None:
try:
with open(f"/sys/class/gpio/gpio{pin}/value", 'wb') as f:
f.write(b"1" if high else b"0")
except Exception as e:
print(f"Failed to set gpio {pin} value: {e}")
def gpio_read(pin: int) -> bool | None:
val = None
try:
with open(f"/sys/class/gpio/gpio{pin}/value", 'rb') as f:
val = bool(int(f.read().strip()))
except Exception as e:
print(f"Failed to set gpio {pin} value: {e}")
return val
def gpio_export(pin: int) -> None:
if os.path.isdir(f"/sys/class/gpio/gpio{pin}"):
return
try:
with open("/sys/class/gpio/export", 'w') as f:
f.write(str(pin))
except Exception:
print(f"Failed to export gpio {pin}")
@cache
def get_irq_action(irq: int) -> list[str]:
try:
with open(f"/sys/kernel/irq/{irq}/actions") as f:
actions = f.read().strip().split(',')
return actions
except FileNotFoundError:
return []
def get_irqs_for_action(action: str) -> list[str]:
ret = []
with open("/proc/interrupts") as f:
for l in f.readlines():
irq = l.split(':')[0].strip()
if irq.isdigit() and action in get_irq_action(irq):
ret.append(irq)
return ret
# *** gpiochip ***
class gpioevent_data(ctypes.Structure):
_fields_ = [
("timestamp", ctypes.c_uint64),
("id", ctypes.c_uint32),
]
class gpioevent_request(ctypes.Structure):
_fields_ = [
("lineoffset", ctypes.c_uint32),
("handleflags", ctypes.c_uint32),
("eventflags", ctypes.c_uint32),
("label", ctypes.c_char * 32),
("fd", ctypes.c_int)
]
def gpiochip_get_ro_value_fd(label: str, gpiochip_id: int, pin: int) -> int:
GPIOEVENT_REQUEST_BOTH_EDGES = 0x3
GPIOHANDLE_REQUEST_INPUT = 0x1
GPIO_GET_LINEEVENT_IOCTL = 0xc030b404
rq = gpioevent_request()
rq.lineoffset = pin
rq.handleflags = GPIOHANDLE_REQUEST_INPUT
rq.eventflags = GPIOEVENT_REQUEST_BOTH_EDGES
rq.label = label.encode('utf-8')[:31] + b'\0'
fd = os.open(f"/dev/gpiochip{gpiochip_id}", os.O_RDONLY)
fcntl.ioctl(fd, GPIO_GET_LINEEVENT_IOCTL, rq)
os.close(fd)
return int(rq.fd)

8
common/gps.py Normal file
View File

@@ -0,0 +1,8 @@
from openpilot.common.params import Params
def get_gps_location_service(params: Params) -> str:
if params.get_bool("UbloxAvailable"):
return "gpsLocationExternal"
else:
return "gpsLocation"

81
common/i2c.py Normal file
View File

@@ -0,0 +1,81 @@
import os
import fcntl
import ctypes
# I2C constants from /usr/include/linux/i2c-dev.h
I2C_SLAVE = 0x0703
I2C_SLAVE_FORCE = 0x0706
I2C_SMBUS = 0x0720
# SMBus transfer types
I2C_SMBUS_READ = 1
I2C_SMBUS_WRITE = 0
I2C_SMBUS_BYTE_DATA = 2
I2C_SMBUS_I2C_BLOCK_DATA = 8
I2C_SMBUS_BLOCK_MAX = 32
class _I2cSmbusData(ctypes.Union):
_fields_ = [
("byte", ctypes.c_uint8),
("word", ctypes.c_uint16),
("block", ctypes.c_uint8 * (I2C_SMBUS_BLOCK_MAX + 2)),
]
class _I2cSmbusIoctlData(ctypes.Structure):
_fields_ = [
("read_write", ctypes.c_uint8),
("command", ctypes.c_uint8),
("size", ctypes.c_uint32),
("data", ctypes.POINTER(_I2cSmbusData)),
]
class SMBus:
def __init__(self, bus: int):
self._fd = os.open(f'/dev/i2c-{bus}', os.O_RDWR)
def __enter__(self) -> 'SMBus':
return self
def __exit__(self, *args) -> None:
self.close()
def close(self) -> None:
if hasattr(self, '_fd') and self._fd >= 0:
os.close(self._fd)
self._fd = -1
def _set_address(self, addr: int, force: bool = False) -> None:
ioctl_arg = I2C_SLAVE_FORCE if force else I2C_SLAVE
fcntl.ioctl(self._fd, ioctl_arg, addr)
def _smbus_access(self, read_write: int, command: int, size: int, data: _I2cSmbusData) -> None:
ioctl_data = _I2cSmbusIoctlData(read_write, command, size, ctypes.pointer(data))
fcntl.ioctl(self._fd, I2C_SMBUS, ioctl_data)
def read_byte_data(self, addr: int, register: int, force: bool = False) -> int:
self._set_address(addr, force)
data = _I2cSmbusData()
self._smbus_access(I2C_SMBUS_READ, register, I2C_SMBUS_BYTE_DATA, data)
return int(data.byte)
def write_byte_data(self, addr: int, register: int, value: int, force: bool = False) -> None:
self._set_address(addr, force)
data = _I2cSmbusData()
data.byte = value & 0xFF
self._smbus_access(I2C_SMBUS_WRITE, register, I2C_SMBUS_BYTE_DATA, data)
def read_i2c_block_data(self, addr: int, register: int, length: int, force: bool = False) -> list[int]:
self._set_address(addr, force)
if not (0 <= length <= I2C_SMBUS_BLOCK_MAX):
raise ValueError(f"length must be 0..{I2C_SMBUS_BLOCK_MAX}")
data = _I2cSmbusData()
data.block[0] = length
self._smbus_access(I2C_SMBUS_READ, register, I2C_SMBUS_I2C_BLOCK_DATA, data)
read_len = int(data.block[0]) or length
read_len = min(read_len, length)
return [int(b) for b in data.block[1 : read_len + 1]]

249
common/logging_extra.py Normal file
View File

@@ -0,0 +1,249 @@
import io
import os
import sys
import copy
import json
import time
import uuid
import socket
import logging
import traceback
import numpy as np
from threading import local
from collections import OrderedDict
from contextlib import contextmanager
LOG_TIMESTAMPS = "LOG_TIMESTAMPS" in os.environ
def json_handler(obj):
if isinstance(obj, np.bool_):
return bool(obj)
# if isinstance(obj, (datetime.date, datetime.time)):
# return obj.isoformat()
return repr(obj)
def json_robust_dumps(obj):
return json.dumps(obj, default=json_handler)
class NiceOrderedDict(OrderedDict):
def __str__(self):
return json_robust_dumps(self)
class SwagFormatter(logging.Formatter):
def __init__(self, swaglogger):
logging.Formatter.__init__(self, None, '%a %b %d %H:%M:%S %Z %Y')
self.swaglogger = swaglogger
self.host = socket.gethostname()
def format_dict(self, record):
record_dict = NiceOrderedDict()
if isinstance(record.msg, dict):
record_dict['msg'] = record.msg
else:
try:
record_dict['msg'] = record.getMessage()
except (ValueError, TypeError):
record_dict['msg'] = [record.msg]+record.args
record_dict['ctx'] = self.swaglogger.get_ctx()
if record.exc_info:
record_dict['exc_info'] = self.formatException(record.exc_info)
record_dict['level'] = record.levelname
record_dict['levelnum'] = record.levelno
record_dict['name'] = record.name
record_dict['filename'] = record.filename
record_dict['lineno'] = record.lineno
record_dict['pathname'] = record.pathname
record_dict['module'] = record.module
record_dict['funcName'] = record.funcName
record_dict['host'] = self.host
record_dict['process'] = record.process
record_dict['thread'] = record.thread
record_dict['threadName'] = record.threadName
record_dict['created'] = record.created
return record_dict
def format(self, record):
if self.swaglogger is None:
raise Exception("must set swaglogger before calling format()")
return json_robust_dumps(self.format_dict(record))
class SwagLogFileFormatter(SwagFormatter):
def fix_kv(self, k, v):
# append type to names to preserve legacy naming in logs
# avoids overlapping key namespaces with different types
# e.g. log.info() creates 'msg' -> 'msg$s'
# log.event() creates 'msg.health.logMonoTime' -> 'msg.health.logMonoTime$i'
# because overlapping namespace 'msg' caused problems
if isinstance(v, (str, bytes)):
k += "$s"
elif isinstance(v, float):
k += "$f"
elif isinstance(v, bool):
k += "$b"
elif isinstance(v, int):
k += "$i"
elif isinstance(v, dict):
nv = {}
for ik, iv in v.items():
ik, iv = self.fix_kv(ik, iv)
nv[ik] = iv
v = nv
elif isinstance(v, list):
k += "$a"
return k, v
def format(self, record):
if isinstance(record, str):
v = json.loads(record)
else:
v = self.format_dict(record)
mk, mv = self.fix_kv('msg', v['msg'])
del v['msg']
v[mk] = mv
v['id'] = uuid.uuid4().hex
return json_robust_dumps(v)
class SwagErrorFilter(logging.Filter):
def filter(self, record):
return record.levelno < logging.ERROR
def _tmpfunc():
return 0
def _srcfile():
return os.path.normcase(_tmpfunc.__code__.co_filename)
class SwagLogger(logging.Logger):
def __init__(self):
logging.Logger.__init__(self, "swaglog")
self.global_ctx = {}
self.log_local = local()
self.log_local.ctx = {}
def local_ctx(self):
try:
return self.log_local.ctx
except AttributeError:
self.log_local.ctx = {}
return self.log_local.ctx
def get_ctx(self):
return dict(self.local_ctx(), **self.global_ctx)
@contextmanager
def ctx(self, **kwargs):
old_ctx = self.local_ctx()
self.log_local.ctx = copy.copy(old_ctx) or {}
self.log_local.ctx.update(kwargs)
try:
yield
finally:
self.log_local.ctx = old_ctx
def bind(self, **kwargs):
self.local_ctx().update(kwargs)
def bind_global(self, **kwargs):
self.global_ctx.update(kwargs)
def event(self, event, *args, **kwargs):
evt = NiceOrderedDict()
evt['event'] = event
if args:
evt['args'] = args
evt.update(kwargs)
if 'error' in kwargs:
self.error(evt)
elif 'debug' in kwargs:
self.debug(evt)
else:
self.info(evt)
def timestamp(self, event_name):
if LOG_TIMESTAMPS:
t = time.monotonic()
tstp = NiceOrderedDict()
tstp['timestamp'] = NiceOrderedDict()
tstp['timestamp']["event"] = event_name
tstp['timestamp']["time"] = t*1e9
self.debug(tstp)
def findCaller(self, stack_info=False, stacklevel=1):
"""
Find the stack frame of the caller so that we can note the source
file name, line number and function name.
"""
f = sys._getframe(3)
#On some versions of IronPython, currentframe() returns None if
#IronPython isn't run with -X:Frames.
if f is not None:
f = f.f_back
orig_f = f
while f and stacklevel > 1:
f = f.f_back
stacklevel -= 1
if not f:
f = orig_f
rv = "(unknown file)", 0, "(unknown function)", None
while hasattr(f, "f_code"):
co = f.f_code
filename = os.path.normcase(co.co_filename)
if filename == _srcfile:
f = f.f_back
continue
sinfo = None
if stack_info:
sio = io.StringIO()
sio.write('Stack (most recent call last):\n')
traceback.print_stack(f, file=sio)
sinfo = sio.getvalue()
if sinfo[-1] == '\n':
sinfo = sinfo[:-1]
sio.close()
rv = (co.co_filename, f.f_lineno, co.co_name, sinfo)
break
return rv
if __name__ == "__main__":
log = SwagLogger()
stdout_handler = logging.StreamHandler(sys.stdout)
stdout_handler.setLevel(logging.INFO)
stdout_handler.addFilter(SwagErrorFilter())
log.addHandler(stdout_handler)
stderr_handler = logging.StreamHandler(sys.stderr)
stderr_handler.setLevel(logging.ERROR)
log.addHandler(stderr_handler)
log.info("asdasd %s", "a")
log.info({'wut': 1})
log.warning("warning")
log.error("error")
log.critical("critical")
log.event("test", x="y")
with log.ctx():
stdout_handler.setFormatter(SwagFormatter(log))
stderr_handler.setFormatter(SwagFormatter(log))
log.bind(user="some user")
log.info("in req")
print("")
log.warning("warning")
print("")
log.error("error")
print("")
log.critical("critical")
print("")
log.event("do_req", a=1, b="c")

45
common/markdown.py Normal file
View File

@@ -0,0 +1,45 @@
HTML_REPLACEMENTS = [
(r'&', r'&amp;'),
(r'"', r'&quot;'),
]
def parse_markdown(text: str, tab_length: int = 2) -> str:
lines = text.split("\n")
output: list[str] = []
list_level = 0
def end_outstanding_lists(level: int, end_level: int) -> int:
while level > end_level:
level -= 1
output.append("</ul>")
if level > 0:
output.append("</li>")
return end_level
for i, line in enumerate(lines):
if i + 1 < len(lines) and lines[i + 1].startswith("==="): # heading
output.append(f"<h1>{line}</h1>")
elif line.startswith("==="):
pass
elif line.lstrip().startswith("* "): # list
line_level = 1 + line.count(" " * tab_length, 0, line.index("*"))
if list_level >= line_level:
list_level = end_outstanding_lists(list_level, line_level)
else:
list_level += 1
if list_level > 1:
output[-1] = output[-1].replace("</li>", "")
output.append("<ul>")
output.append(f"<li>{line.replace('*', '', 1).lstrip()}</li>")
else:
list_level = end_outstanding_lists(list_level, 0)
if len(line) > 0:
output.append(line)
end_outstanding_lists(list_level, 0)
output_str = "\n".join(output) + "\n"
for (fr, to) in HTML_REPLACEMENTS:
output_str = output_str.replace(fr, to)
return output_str

50
common/mock/__init__.py Normal file
View File

@@ -0,0 +1,50 @@
"""
Utilities for generating mock messages for testing.
example in common/tests/test_mock.py
"""
import functools
import threading
from cereal.messaging import PubMaster
from cereal.services import SERVICE_LIST
from openpilot.common.mock.generators import generate_livePose
from openpilot.common.realtime import Ratekeeper
MOCK_GENERATOR = {
"livePose": generate_livePose
}
def generate_messages_loop(services: list[str], done: threading.Event):
pm = PubMaster(services)
rk = Ratekeeper(100)
i = 0
while not done.is_set():
for s in services:
should_send = i % (100/SERVICE_LIST[s].frequency) == 0
if should_send:
message = MOCK_GENERATOR[s]()
pm.send(s, message)
i += 1
rk.keep_time()
def mock_messages(services: list[str] | str):
if isinstance(services, str):
services = [services]
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
done = threading.Event()
t = threading.Thread(target=generate_messages_loop, args=(services, done))
t.start()
try:
return func(*args, **kwargs)
finally:
done.set()
t.join()
return wrapper
return decorator

14
common/mock/generators.py Normal file
View File

@@ -0,0 +1,14 @@
from cereal import messaging
def generate_livePose():
msg = messaging.new_message('livePose')
meas = {'x': 0.0, 'y': 0.0, 'z': 0.0, 'xStd': 0.0, 'yStd': 0.0, 'zStd': 0.0, 'valid': True}
msg.livePose.orientationNED = meas
msg.livePose.velocityDevice = meas
msg.livePose.angularVelocityDevice = meas
msg.livePose.accelerationDevice = meas
msg.livePose.inputsOK = True
msg.livePose.posenetOK = True
msg.livePose.sensorsOK = True
return msg

47
common/parameterized.py Normal file
View File

@@ -0,0 +1,47 @@
import sys
import pytest
import inspect
class parameterized:
@staticmethod
def expand(cases):
cases = list(cases)
if not cases:
return lambda func: pytest.mark.skip("no parameterized cases")(func)
def decorator(func):
params = [p for p in inspect.signature(func).parameters if p != 'self']
normalized = [c if isinstance(c, tuple) else (c,) for c in cases]
# Infer arg count from first case so extra params (e.g. from @given) are left untouched
expand_params = params[: len(normalized[0])]
if len(expand_params) == 1:
return pytest.mark.parametrize(expand_params[0], [c[0] for c in normalized])(func)
return pytest.mark.parametrize(', '.join(expand_params), normalized)(func)
return decorator
def parameterized_class(attrs, input_list=None):
if isinstance(attrs, list) and (not attrs or isinstance(attrs[0], dict)):
params_list = attrs
else:
assert input_list is not None
attr_names = (attrs,) if isinstance(attrs, str) else tuple(attrs)
params_list = [dict(zip(attr_names, v if isinstance(v, (tuple, list)) else (v,), strict=False)) for v in input_list]
def decorator(cls):
globs = sys._getframe(1).f_globals
for i, params in enumerate(params_list):
name = f"{cls.__name__}_{i}"
new_cls = type(name, (cls,), dict(params))
new_cls.__module__ = cls.__module__
new_cls.__test__ = True # override inherited False so pytest collects this subclass
globs[name] = new_cls
# Don't collect the un-parametrised base, but return it so outer decorators
# (e.g. @pytest.mark.skip) land on it and propagate to subclasses via MRO.
cls.__test__ = False
return cls
return decorator

244
common/params.cc Normal file
View File

@@ -0,0 +1,244 @@
#include "common/params.h"
#include <dirent.h>
#include <sys/file.h>
#include <algorithm>
#include <cassert>
#include <csignal>
#include <unordered_map>
#include "common/params_keys.h"
#include "common/queue.h"
#include "common/swaglog.h"
#include "common/util.h"
#include "system/hardware/hw.h"
namespace {
volatile sig_atomic_t params_do_exit = 0;
void params_sig_handler(int signal) {
params_do_exit = 1;
}
int fsync_dir(const std::string &path) {
int result = -1;
int fd = HANDLE_EINTR(open(path.c_str(), O_RDONLY, 0755));
if (fd >= 0) {
result = HANDLE_EINTR(fsync(fd));
HANDLE_EINTR(close(fd));
}
return result;
}
bool create_params_path(const std::string &param_path, const std::string &key_path) {
// Make sure params path exists
if (!util::file_exists(param_path) && !util::create_directories(param_path, 0775)) {
return false;
}
// See if the symlink exists, otherwise create it
if (!util::file_exists(key_path)) {
// 1) Create temp folder
// 2) Symlink it to temp link
// 3) Move symlink to <params>/d
std::string tmp_path = param_path + "/.tmp_XXXXXX";
// this should be OK since mkdtemp just replaces characters in place
char *tmp_dir = mkdtemp((char *)tmp_path.c_str());
if (tmp_dir == NULL) {
return false;
}
std::string link_path = std::string(tmp_dir) + ".link";
if (symlink(tmp_dir, link_path.c_str()) != 0) {
return false;
}
// don't return false if it has been created by other
if (rename(link_path.c_str(), key_path.c_str()) != 0 && errno != EEXIST) {
return false;
}
}
return true;
}
std::string ensure_params_path(const std::string &prefix, const std::string &path = {}) {
std::string params_path = path.empty() ? Path::params() : path;
if (!create_params_path(params_path, params_path + prefix)) {
throw std::runtime_error(util::string_format(
"Failed to ensure params path, errno=%d, path=%s, param_prefix=%s",
errno, params_path.c_str(), prefix.c_str()));
}
return params_path;
}
class FileLock {
public:
FileLock(const std::string &fn) {
fd_ = HANDLE_EINTR(open(fn.c_str(), O_CREAT, 0775));
if (fd_ < 0 || HANDLE_EINTR(flock(fd_, LOCK_EX)) < 0) {
LOGE("Failed to lock file %s, errno=%d", fn.c_str(), errno);
}
}
~FileLock() { close(fd_); }
private:
int fd_ = -1;
};
} // namespace
Params::Params(const std::string &path) {
params_prefix = "/" + util::getenv("OPENPILOT_PREFIX", "d");
params_path = ensure_params_path(params_prefix, path);
}
Params::~Params() {
if (future.valid()) {
future.wait();
}
assert(queue.empty());
}
std::vector<std::string> Params::allKeys(ParamKeyFlag flag) const {
std::vector<std::string> ret;
for (auto &p : keys) {
if (flag == ALL || (p.second.flags & flag)) {
ret.push_back(p.first);
}
}
return ret;
}
bool Params::checkKey(const std::string &key) {
return keys.find(key) != keys.end();
}
ParamKeyFlag Params::getKeyFlag(const std::string &key) {
return static_cast<ParamKeyFlag>(keys[key].flags);
}
ParamKeyType Params::getKeyType(const std::string &key) {
return keys[key].type;
}
std::optional<std::string> Params::getKeyDefaultValue(const std::string &key) {
return keys[key].default_value;
}
int Params::put(const char* key, const char* value, size_t value_size) {
// Information about safely and atomically writing a file: https://lwn.net/Articles/457667/
// 1) Create temp file
// 2) Write data to temp file
// 3) fsync() the temp file
// 4) rename the temp file to the real name
// 5) fsync() the containing directory
std::string tmp_path = params_path + "/.tmp_value_XXXXXX";
int tmp_fd = mkstemp((char*)tmp_path.c_str());
if (tmp_fd < 0) return -1;
int result = -1;
do {
// Write value to temp.
ssize_t bytes_written = HANDLE_EINTR(write(tmp_fd, value, value_size));
if (bytes_written < 0 || (size_t)bytes_written != value_size) {
result = -20;
break;
}
// fsync to force persist the changes.
if ((result = HANDLE_EINTR(fsync(tmp_fd))) < 0) break;
FileLock file_lock(params_path + "/.lock");
// Move temp into place.
if ((result = rename(tmp_path.c_str(), getParamPath(key).c_str())) < 0) break;
// fsync parent directory
result = fsync_dir(getParamPath());
} while (false);
close(tmp_fd);
if (result != 0) {
::unlink(tmp_path.c_str());
}
return result;
}
int Params::remove(const std::string &key) {
FileLock file_lock(params_path + "/.lock");
int result = unlink(getParamPath(key).c_str());
if (result != 0) {
return result;
}
return fsync_dir(getParamPath());
}
std::string Params::get(const std::string &key, bool block) {
if (!block) {
return util::read_file(getParamPath(key));
} else {
// blocking read until successful
params_do_exit = 0;
void (*prev_handler_sigint)(int) = std::signal(SIGINT, params_sig_handler);
void (*prev_handler_sigterm)(int) = std::signal(SIGTERM, params_sig_handler);
std::string value;
while (!params_do_exit) {
if (value = util::read_file(getParamPath(key)); !value.empty()) {
break;
}
util::sleep_for(100); // 0.1 s
}
std::signal(SIGINT, prev_handler_sigint);
std::signal(SIGTERM, prev_handler_sigterm);
return value;
}
}
std::map<std::string, std::string> Params::readAll() {
FileLock file_lock(params_path + "/.lock");
return util::read_files_in_dir(getParamPath());
}
void Params::clearAll(ParamKeyFlag key_flag) {
FileLock file_lock(params_path + "/.lock");
// 1) delete params of key_flag
// 2) delete files that are not defined in the keys.
if (DIR *d = opendir(getParamPath().c_str())) {
struct dirent *de = NULL;
while ((de = readdir(d))) {
if (de->d_type != DT_DIR) {
auto it = keys.find(de->d_name);
if (it == keys.end() || (it->second.flags & key_flag)) {
unlink(getParamPath(de->d_name).c_str());
}
}
}
closedir(d);
}
fsync_dir(getParamPath());
}
void Params::putNonBlocking(const std::string &key, const std::string &val) {
queue.push(std::make_pair(key, val));
// start thread on demand
if (!future.valid() || future.wait_for(std::chrono::milliseconds(0)) == std::future_status::ready) {
future = std::async(std::launch::async, &Params::asyncWriteThread, this);
}
}
void Params::asyncWriteThread() {
// TODO: write the latest one if a key has multiple values in the queue.
std::pair<std::string, std::string> p;
while (queue.try_pop(p, 0)) {
// Params::put is Thread-Safe
put(p.first, p.second);
}
}

91
common/params.h Normal file
View File

@@ -0,0 +1,91 @@
#pragma once
#include <future>
#include <map>
#include <optional>
#include <string>
#include <tuple>
#include <utility>
#include <vector>
#include "common/queue.h"
enum ParamKeyFlag {
PERSISTENT = 0x02,
CLEAR_ON_MANAGER_START = 0x04,
CLEAR_ON_ONROAD_TRANSITION = 0x08,
CLEAR_ON_OFFROAD_TRANSITION = 0x10,
DONT_LOG = 0x20,
DEVELOPMENT_ONLY = 0x40,
CLEAR_ON_IGNITION_ON = 0x80,
BACKUP = 0x100,
ALL = 0xFFFFFFFF
};
enum ParamKeyType {
STRING = 0, // must be utf-8 decodable
BOOL = 1,
INT = 2,
FLOAT = 3,
TIME = 4, // ISO 8601
JSON = 5,
BYTES = 6
};
struct ParamKeyAttributes {
uint32_t flags;
ParamKeyType type;
std::optional<std::string> default_value = std::nullopt;
};
class Params {
public:
explicit Params(const std::string &path = {});
~Params();
// Not copyable.
Params(const Params&) = delete;
Params& operator=(const Params&) = delete;
std::vector<std::string> allKeys(ParamKeyFlag flag = ALL) const;
bool checkKey(const std::string &key);
ParamKeyFlag getKeyFlag(const std::string &key);
ParamKeyType getKeyType(const std::string &key);
std::optional<std::string> getKeyDefaultValue(const std::string &key);
inline std::string getParamPath(const std::string &key = {}) {
return params_path + params_prefix + (key.empty() ? "" : "/" + key);
}
// Delete a value
int remove(const std::string &key);
void clearAll(ParamKeyFlag flag);
// helpers for reading values
std::string get(const std::string &key, bool block = false);
inline bool getBool(const std::string &key, bool block = false) {
return get(key, block) == "1";
}
std::map<std::string, std::string> readAll();
// helpers for writing values
int put(const char *key, const char *val, size_t value_size);
inline int put(const std::string &key, const std::string &val) {
return put(key.c_str(), val.data(), val.size());
}
inline int putBool(const std::string &key, bool val) {
return put(key.c_str(), val ? "1" : "0", 1);
}
void putNonBlocking(const std::string &key, const std::string &val);
inline void putBoolNonBlocking(const std::string &key, bool val) {
putNonBlocking(key, val ? "1" : "0");
}
private:
void asyncWriteThread();
std::string params_path;
std::string params_prefix;
// for nonblocking write
std::future<void> future;
SafeQueue<std::pair<std::string, std::string>> queue;
};

19
common/params.py Normal file
View File

@@ -0,0 +1,19 @@
from openpilot.common.params_pyx import Params, ParamKeyFlag, ParamKeyType, UnknownKeyName
assert Params
assert ParamKeyFlag
assert ParamKeyType
assert UnknownKeyName
if __name__ == "__main__":
import sys
params = Params()
key = sys.argv[1]
assert params.check_key(key), f"unknown param: {key}"
if len(sys.argv) == 3:
val = sys.argv[2]
print(f"SET: {key} = {val}")
params.put(key, val)
elif len(sys.argv) == 2:
print(f"GET: {key} = {params.get(key)}")

280
common/params_keys.h Normal file
View File

@@ -0,0 +1,280 @@
#pragma once
#include <string>
#include <unordered_map>
#include "cereal/gen/cpp/log.capnp.h"
inline static std::unordered_map<std::string, ParamKeyAttributes> keys = {
{"AccessToken", {CLEAR_ON_MANAGER_START | DONT_LOG, STRING}},
{"AdbEnabled", {PERSISTENT | BACKUP, BOOL}},
{"AlwaysOnDM", {PERSISTENT | BACKUP, BOOL}},
{"ApiCache_Device", {PERSISTENT, STRING}},
{"ApiCache_FirehoseStats", {PERSISTENT, JSON}},
{"AssistNowToken", {PERSISTENT, STRING}},
{"AthenadPid", {PERSISTENT, INT}},
{"AthenadUploadQueue", {PERSISTENT, JSON}},
{"AthenadRecentlyViewedRoutes", {PERSISTENT, STRING}},
{"BootCount", {PERSISTENT, INT}},
{"CalibrationParams", {PERSISTENT, BYTES}},
{"CameraDebugExpGain", {CLEAR_ON_MANAGER_START, STRING}},
{"CameraDebugExpTime", {CLEAR_ON_MANAGER_START, STRING}},
{"CarBatteryCapacity", {PERSISTENT, INT}},
{"CarParams", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, BYTES}},
{"CarParamsCache", {CLEAR_ON_MANAGER_START, BYTES}},
{"CarParamsPersistent", {PERSISTENT, BYTES}},
{"CarParamsPrevRoute", {PERSISTENT, BYTES}},
{"CompletedTrainingVersion", {PERSISTENT, STRING, "0"}},
{"ControlsReady", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, BOOL}},
{"CurrentBootlog", {PERSISTENT, STRING}},
{"CurrentRoute", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, STRING}},
{"DisableLogging", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, BOOL}},
{"DisablePowerDown", {PERSISTENT | BACKUP, BOOL}},
{"DisableUpdates", {PERSISTENT | BACKUP, BOOL, "0"}},
{"DisengageOnAccelerator", {PERSISTENT | BACKUP, BOOL, "0"}},
{"DongleId", {PERSISTENT, STRING}},
{"DoReboot", {CLEAR_ON_MANAGER_START, BOOL}},
{"DoShutdown", {CLEAR_ON_MANAGER_START, BOOL}},
{"DoUninstall", {CLEAR_ON_MANAGER_START, BOOL}},
{"DriverTooDistracted", {CLEAR_ON_MANAGER_START | CLEAR_ON_IGNITION_ON, BOOL}},
{"AlphaLongitudinalEnabled", {PERSISTENT | DEVELOPMENT_ONLY | BACKUP, BOOL}},
{"ExperimentalMode", {PERSISTENT | BACKUP, BOOL}},
{"ExperimentalModeConfirmed", {PERSISTENT | BACKUP, BOOL}},
{"FirmwareQueryDone", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, BOOL}},
{"ForcePowerDown", {PERSISTENT, BOOL}},
{"GitBranch", {PERSISTENT, STRING}},
{"GitCommit", {PERSISTENT, STRING}},
{"GitCommitDate", {PERSISTENT, STRING}},
{"GitDiff", {PERSISTENT, STRING}},
{"GithubSshKeys", {PERSISTENT | BACKUP, STRING}},
{"GithubUsername", {PERSISTENT | BACKUP, STRING}},
{"GitRemote", {PERSISTENT, STRING}},
{"GsmApn", {PERSISTENT | BACKUP, STRING}},
{"GsmMetered", {PERSISTENT | BACKUP, BOOL, "1"}},
{"GsmRoaming", {PERSISTENT | BACKUP, BOOL}},
{"HardwareSerial", {PERSISTENT, STRING}},
{"HasAcceptedTerms", {PERSISTENT, STRING, "0"}},
{"InstallDate", {PERSISTENT, TIME}},
{"IsDriverViewEnabled", {CLEAR_ON_MANAGER_START, BOOL}},
{"IsEngaged", {PERSISTENT, BOOL}},
{"IsLdwEnabled", {PERSISTENT | BACKUP, BOOL}},
{"IsMetric", {PERSISTENT | BACKUP, BOOL}},
{"IsOffroad", {CLEAR_ON_MANAGER_START, BOOL}},
{"IsOnroad", {PERSISTENT, BOOL}},
{"IsRhdDetected", {PERSISTENT, BOOL}},
{"IsReleaseBranch", {CLEAR_ON_MANAGER_START, BOOL}},
{"IsTakingSnapshot", {CLEAR_ON_MANAGER_START, BOOL}},
{"IsTestedBranch", {CLEAR_ON_MANAGER_START, BOOL}},
{"JoystickDebugMode", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, BOOL}},
{"LanguageSetting", {PERSISTENT | BACKUP, STRING, "en"}},
{"LastAthenaPingTime", {CLEAR_ON_MANAGER_START, INT}},
{"LastGPSPosition", {PERSISTENT, STRING}},
{"LastManagerExitReason", {CLEAR_ON_MANAGER_START, STRING}},
{"LastOffroadStatusPacket", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, JSON}},
{"LastAgnosPowerMonitorShutdown", {CLEAR_ON_MANAGER_START, STRING}},
{"LastPowerDropDetected", {CLEAR_ON_MANAGER_START, STRING}},
{"LastUpdateException", {CLEAR_ON_MANAGER_START, STRING}},
{"LastUpdateRouteCount", {PERSISTENT, INT, "0"}},
{"LastUpdateTime", {PERSISTENT, TIME}},
{"LastUpdateUptimeOnroad", {PERSISTENT, FLOAT, "0.0"}},
{"LiveDelay", {PERSISTENT | BACKUP, BYTES}},
{"LiveParameters", {PERSISTENT, JSON}},
{"LiveParametersV2", {PERSISTENT, BYTES}},
{"LiveTorqueParameters", {PERSISTENT | DONT_LOG, BYTES}},
{"LocationFilterInitialState", {PERSISTENT, BYTES}},
{"LateralManeuverMode", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, BOOL}},
{"LongitudinalManeuverMode", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, BOOL}},
{"LongitudinalPersonality", {PERSISTENT | BACKUP, INT, std::to_string(static_cast<int>(cereal::LongitudinalPersonality::STANDARD))}},
{"NetworkMetered", {PERSISTENT | BACKUP, BOOL}},
{"ObdMultiplexingChanged", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, BOOL}},
{"ObdMultiplexingEnabled", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, BOOL}},
{"Offroad_CarUnrecognized", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, JSON}},
{"Offroad_ConnectivityNeeded", {CLEAR_ON_MANAGER_START, JSON}},
{"Offroad_ConnectivityNeededPrompt", {CLEAR_ON_MANAGER_START, JSON}},
{"Offroad_ExcessiveActuation", {PERSISTENT, JSON}},
{"Offroad_IsTakingSnapshot", {CLEAR_ON_MANAGER_START, JSON}},
{"Offroad_NeosUpdate", {CLEAR_ON_MANAGER_START, JSON}},
{"Offroad_NoFirmware", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, JSON}},
{"Offroad_Recalibration", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, JSON}},
{"Offroad_TemperatureTooHigh", {CLEAR_ON_MANAGER_START, JSON}},
{"Offroad_UnregisteredHardware", {CLEAR_ON_MANAGER_START, JSON}},
{"Offroad_UpdateFailed", {CLEAR_ON_MANAGER_START, JSON}},
{"Offroad_DriverMonitoringUncertain", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, JSON}},
{"OnroadCycleRequested", {CLEAR_ON_MANAGER_START, BOOL}},
{"OpenpilotEnabledToggle", {PERSISTENT | BACKUP, BOOL, "1"}},
{"PandaHeartbeatLost", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, BOOL}},
{"PandaSomResetTriggered", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, BOOL}},
{"PandaSignatures", {CLEAR_ON_MANAGER_START, BYTES}},
{"PrimeType", {PERSISTENT, INT}},
{"RecordAudio", {PERSISTENT | BACKUP, BOOL}},
{"RecordAudioFeedback", {PERSISTENT | BACKUP, BOOL, "0"}},
{"RecordFront", {PERSISTENT | BACKUP, BOOL}},
{"RecordFrontLock", {PERSISTENT, BOOL}}, // for the internal fleet
{"SecOCKey", {PERSISTENT | DONT_LOG | BACKUP, STRING}},
{"ShowDebugInfo", {PERSISTENT, BOOL}},
{"RouteCount", {PERSISTENT, INT, "0"}},
{"SnoozeUpdate", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, BOOL}},
{"SshEnabled", {PERSISTENT | BACKUP, BOOL}},
{"TermsVersion", {PERSISTENT, STRING}},
{"TorqueBar", {PERSISTENT | BACKUP, BOOL, "0"}},
{"TrainingVersion", {PERSISTENT, STRING}},
{"UbloxAvailable", {PERSISTENT, BOOL}},
{"UpdateAvailable", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, BOOL}},
{"UpdateFailedCount", {CLEAR_ON_MANAGER_START, INT}},
{"UpdaterAvailableBranches", {PERSISTENT, STRING}},
{"UpdaterCurrentDescription", {CLEAR_ON_MANAGER_START, STRING}},
{"UpdaterCurrentReleaseNotes", {CLEAR_ON_MANAGER_START, BYTES}},
{"UpdaterFetchAvailable", {CLEAR_ON_MANAGER_START, BOOL}},
{"UpdaterNewDescription", {CLEAR_ON_MANAGER_START, STRING}},
{"UpdaterNewReleaseNotes", {CLEAR_ON_MANAGER_START, BYTES}},
{"UpdaterState", {CLEAR_ON_MANAGER_START, STRING}},
{"UpdaterTargetBranch", {CLEAR_ON_MANAGER_START, STRING}},
{"UpdaterLastFetchTime", {PERSISTENT, TIME}},
{"UptimeOffroad", {PERSISTENT, FLOAT, "0.0"}},
{"UptimeOnroad", {PERSISTENT, FLOAT, "0.0"}},
{"Version", {PERSISTENT, STRING}},
// --- sunnypilot params --- //
{"ApiCache_DriveStats", {PERSISTENT, JSON}},
{"AutoLaneChangeBsmDelay", {PERSISTENT | BACKUP, BOOL, "0"}},
{"AutoLaneChangeTimer", {PERSISTENT | BACKUP, INT, "0"}},
{"BlinkerLateralReengageDelay", {PERSISTENT | BACKUP, INT, "0"}}, // seconds
{"BlinkerMinLateralControlSpeed", {PERSISTENT | BACKUP, INT, "20"}}, // MPH or km/h
{"BlinkerPauseLateralControl", {PERSISTENT | BACKUP, INT, "0"}},
{"Brightness", {PERSISTENT | BACKUP, INT, "0"}},
{"CarList", {PERSISTENT, JSON}},
{"CarParamsSP", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, BYTES}},
{"CarParamsSPCache", {CLEAR_ON_MANAGER_START, BYTES}},
{"CarParamsSPPersistent", {PERSISTENT, BYTES}},
{"CarPlatformBundle", {PERSISTENT | BACKUP, JSON}},
{"ChevronInfo", {PERSISTENT | BACKUP, INT, "4"}},
{"CompletedSunnylinkConsentVersion", {PERSISTENT, STRING, "0"}},
{"CustomAccIncrementsEnabled", {PERSISTENT | BACKUP, BOOL, "0"}},
{"CustomAccLongPressIncrement", {PERSISTENT | BACKUP, INT, "5"}},
{"CustomAccShortPressIncrement", {PERSISTENT | BACKUP, INT, "1"}},
{"DeviceBootMode", {PERSISTENT | BACKUP, INT, "0"}},
{"DevUIInfo", {PERSISTENT | BACKUP, INT, "0"}},
{"EnableCopyparty", {PERSISTENT | BACKUP, BOOL}},
{"EnableGithubRunner", {PERSISTENT | BACKUP, BOOL}},
{"GreenLightAlert", {PERSISTENT | BACKUP, BOOL, "0"}},
{"GithubRunnerSufficientVoltage", {CLEAR_ON_MANAGER_START , BOOL}},
{"HasAcceptedTermsSP", {PERSISTENT, STRING, "0"}},
{"HideVEgoUI", {PERSISTENT | BACKUP, BOOL, "0"}},
{"IntelligentCruiseButtonManagement", {PERSISTENT | BACKUP , BOOL}},
{"InteractivityTimeout", {PERSISTENT | BACKUP, INT, "0"}},
{"IsDevelopmentBranch", {CLEAR_ON_MANAGER_START, BOOL}},
{"IsReleaseSpBranch", {CLEAR_ON_MANAGER_START, BOOL}},
{"LastGPSPositionLLK", {PERSISTENT, STRING}},
{"LeadDepartAlert", {PERSISTENT | BACKUP, BOOL, "0"}},
{"MaxTimeOffroad", {PERSISTENT | BACKUP, INT, "1800"}},
{"ModelRunnerTypeCache", {CLEAR_ON_ONROAD_TRANSITION, INT}},
{"OffroadMode", {CLEAR_ON_MANAGER_START, BOOL}},
{"Offroad_TiciSupport", {CLEAR_ON_MANAGER_START, JSON}},
{"OnroadScreenOffBrightness", {PERSISTENT | BACKUP, INT, "0"}},
{"OnroadScreenOffBrightnessMigrated", {PERSISTENT | BACKUP, STRING, "0.0"}},
{"OnroadScreenOffTimer", {PERSISTENT | BACKUP, INT, "15"}},
{"OnroadScreenOffTimerMigrated", {PERSISTENT | BACKUP, STRING, "0.0"}},
{"OnroadUploads", {PERSISTENT | BACKUP, BOOL, "1"}},
{"QuickBootToggle", {PERSISTENT | BACKUP, BOOL, "0"}},
{"QuietMode", {PERSISTENT | BACKUP, BOOL, "0"}},
{"RainbowMode", {PERSISTENT | BACKUP, BOOL, "0"}},
{"RocketFuel", {PERSISTENT | BACKUP, BOOL, "0"}},
{"ShowAdvancedControls", {PERSISTENT | BACKUP, BOOL, "0"}},
{"ShowTurnSignals", {PERSISTENT | BACKUP, BOOL, "0"}},
{"StandstillTimer", {PERSISTENT | BACKUP, BOOL, "0"}},
{"TrueVEgoUI", {PERSISTENT | BACKUP, BOOL, "0"}},
// MADS params
{"Mads", {PERSISTENT | BACKUP, BOOL, "1"}},
{"MadsMainCruiseAllowed", {PERSISTENT | BACKUP, BOOL, "1"}},
{"MadsSteeringMode", {PERSISTENT | BACKUP, INT, "0"}},
{"MadsUnifiedEngagementMode", {PERSISTENT | BACKUP, BOOL, "1"}},
// Model Manager params
{"ModelManager_ActiveBundle", {PERSISTENT, JSON}},
{"ModelManager_ClearCache", {CLEAR_ON_MANAGER_START, BOOL}},
{"ModelManager_DownloadIndex", {CLEAR_ON_MANAGER_START | CLEAR_ON_ONROAD_TRANSITION, INT}},
{"ModelManager_Favs", {PERSISTENT | BACKUP, STRING}},
{"ModelManager_LastSyncTime", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, INT, "0"}},
{"ModelManager_ModelsCache", {PERSISTENT | BACKUP, JSON}},
// Neural Network Lateral Control
{"NeuralNetworkLateralControl", {PERSISTENT | BACKUP, BOOL, "0"}},
// sunnylink params
{"EnableSunnylinkUploader", {PERSISTENT | BACKUP, BOOL}},
{"LastSunnylinkPingTime", {CLEAR_ON_MANAGER_START, INT}},
{"ParamsVersion", {PERSISTENT, INT}},
{"SunnylinkCache_Roles", {PERSISTENT, STRING}},
{"SunnylinkCache_Users", {PERSISTENT, STRING}},
{"SunnylinkDongleId", {PERSISTENT, STRING}},
{"SunnylinkdPid", {PERSISTENT, INT}},
{"SunnylinkEnabled", {PERSISTENT, BOOL, "1"}},
{"SunnylinkTempFault", {CLEAR_ON_MANAGER_START | CLEAR_ON_OFFROAD_TRANSITION, BOOL, "0"}},
// Backup Manager params
{"BackupManager_CreateBackup", {PERSISTENT, BOOL}},
{"BackupManager_RestoreVersion", {PERSISTENT, STRING}},
// sunnypilot car specific params
{"HyundaiLongitudinalTuning", {PERSISTENT | BACKUP, INT, "0"}},
{"SubaruStopAndGo", {PERSISTENT | BACKUP, BOOL, "0"}},
{"SubaruStopAndGoManualParkingBrake", {PERSISTENT | BACKUP, BOOL, "0"}},
{"TeslaCoopSteering", {PERSISTENT | BACKUP, BOOL, "0"}},
{"ToyotaEnforceStockLongitudinal", {PERSISTENT | BACKUP, BOOL, "0"}},
{"ToyotaStopAndGoHack", {PERSISTENT | BACKUP, BOOL, "0"}},
{"DynamicExperimentalControl", {PERSISTENT | BACKUP, BOOL, "0"}},
{"BlindSpot", {PERSISTENT | BACKUP, BOOL, "0"}},
// sunnypilot model params
{"CameraOffset", {PERSISTENT | BACKUP, FLOAT, "0.0"}},
{"LagdToggle", {PERSISTENT | BACKUP, BOOL, "1"}},
{"LagdToggleDelay", {PERSISTENT | BACKUP, FLOAT, "0.2"}},
{"LagdValueCache", {PERSISTENT, FLOAT, "0.2"}},
{"LaneTurnDesire", {PERSISTENT | BACKUP, BOOL, "0"}},
{"LaneTurnValue", {PERSISTENT | BACKUP, FLOAT, "19.0"}},
{"PlanplusControl", {PERSISTENT | BACKUP, FLOAT, "1.0"}},
// mapd
{"MapAdvisorySpeedLimit", {CLEAR_ON_ONROAD_TRANSITION, FLOAT}},
{"MapdVersion", {PERSISTENT, STRING}},
{"MapSpeedLimit", {CLEAR_ON_ONROAD_TRANSITION, FLOAT, "0.0"}},
{"NextMapSpeedLimit", {CLEAR_ON_ONROAD_TRANSITION, JSON}},
{"Offroad_OSMUpdateRequired", {CLEAR_ON_MANAGER_START, JSON}},
{"OsmDbUpdatesCheck", {CLEAR_ON_MANAGER_START, BOOL}}, // mapd database update happens with device ON, reset on boot
{"OSMDownloadBounds", {PERSISTENT, STRING}},
{"OsmDownloadedDate", {PERSISTENT, STRING, "0.0"}},
{"OSMDownloadLocations", {PERSISTENT, JSON}},
{"OSMDownloadProgress", {CLEAR_ON_MANAGER_START, JSON}},
{"OsmLocal", {PERSISTENT, BOOL}},
{"OsmLocationName", {PERSISTENT, STRING}},
{"OsmLocationTitle", {PERSISTENT, STRING}},
{"OsmLocationUrl", {PERSISTENT, STRING}},
{"OsmStateName", {PERSISTENT, STRING, "All"}},
{"OsmStateTitle", {PERSISTENT, STRING}},
{"OsmWayTest", {PERSISTENT, STRING}},
{"RoadName", {CLEAR_ON_ONROAD_TRANSITION, STRING}},
{"RoadNameToggle", {PERSISTENT | BACKUP, BOOL, "0"}},
// Speed Limit
{"SpeedLimitMode", {PERSISTENT | BACKUP, INT, "1"}},
{"SpeedLimitOffsetType", {PERSISTENT | BACKUP, INT, "0"}},
{"SpeedLimitPolicy", {PERSISTENT | BACKUP, INT, "3"}},
{"SpeedLimitValueOffset", {PERSISTENT | BACKUP, INT, "0"}},
// Smart Cruise Control
{"MapTargetVelocities", {CLEAR_ON_ONROAD_TRANSITION, STRING}},
{"SmartCruiseControlMap", {PERSISTENT | BACKUP, BOOL, "0"}},
{"SmartCruiseControlVision", {PERSISTENT | BACKUP, BOOL, "0"}},
// Torque lateral control custom params
{"CustomTorqueParams", {PERSISTENT | BACKUP , BOOL}},
{"EnforceTorqueControl", {PERSISTENT | BACKUP, BOOL}},
{"LiveTorqueParamsToggle", {PERSISTENT | BACKUP , BOOL}},
{"LiveTorqueParamsRelaxedToggle", {PERSISTENT | BACKUP , BOOL}},
{"TorqueControlTune", {PERSISTENT | BACKUP, FLOAT, "0.0"}},
{"TorqueParamsOverrideEnabled", {PERSISTENT | BACKUP, BOOL, "0"}},
{"TorqueParamsOverrideFriction", {PERSISTENT | BACKUP, FLOAT, "0.1"}},
{"TorqueParamsOverrideLatAccelFactor", {PERSISTENT | BACKUP, FLOAT, "2.5"}},
};

197
common/params_pyx.pyx Normal file
View File

@@ -0,0 +1,197 @@
# distutils: language = c++
# cython: language_level = 3
import builtins
import datetime
import json
from libcpp cimport bool
from libcpp.string cimport string
from libcpp.vector cimport vector
from libcpp.optional cimport optional
from openpilot.common.swaglog import cloudlog
cdef extern from "common/params.h":
cpdef enum ParamKeyFlag:
PERSISTENT
CLEAR_ON_MANAGER_START
CLEAR_ON_ONROAD_TRANSITION
CLEAR_ON_OFFROAD_TRANSITION
DEVELOPMENT_ONLY
CLEAR_ON_IGNITION_ON
BACKUP
ALL
cpdef enum ParamKeyType:
STRING
BOOL
INT
FLOAT
TIME
JSON
BYTES
cdef cppclass c_Params "Params":
c_Params(string) except + nogil
string get(string, bool) nogil
bool getBool(string, bool) nogil
int remove(string) nogil
int put(string, string) nogil
void putNonBlocking(string, string) nogil
void putBoolNonBlocking(string, bool) nogil
int putBool(string, bool) nogil
bool checkKey(string) nogil
ParamKeyType getKeyType(string) nogil
optional[string] getKeyDefaultValue(string) nogil
string getParamPath(string) nogil
void clearAll(ParamKeyFlag)
vector[string] allKeys(ParamKeyFlag)
PYTHON_2_CPP = {
(str, STRING): lambda v: v,
(builtins.bool, BOOL): lambda v: "1" if v else "0",
(int, INT): str,
(float, FLOAT): str,
(datetime.datetime, TIME): lambda v: v.isoformat(),
(dict, JSON): json.dumps,
(list, JSON): json.dumps,
(bytes, BYTES): lambda v: v,
}
CPP_2_PYTHON = {
STRING: lambda v: v.decode("utf-8"),
BOOL: lambda v: v == b"1",
INT: int,
FLOAT: float,
TIME: lambda v: datetime.datetime.fromisoformat(v.decode("utf-8")),
JSON: json.loads,
BYTES: lambda v: v,
}
def ensure_bytes(v):
return v.encode() if isinstance(v, str) else v
class UnknownKeyName(Exception):
pass
cdef class Params:
cdef c_Params* p
cdef str d
def __cinit__(self, d=""):
cdef string path = <string>d.encode()
with nogil:
self.p = new c_Params(path)
self.d = d
def __reduce__(self):
return (type(self), (self.d,))
def __dealloc__(self):
del self.p
def clear_all(self, tx_flag=ParamKeyFlag.ALL):
self.p.clearAll(tx_flag)
def check_key(self, key):
key = ensure_bytes(key)
if not self.p.checkKey(key):
raise UnknownKeyName(key)
return key
def python2cpp(self, proposed_type, expected_type, value, key):
cast = PYTHON_2_CPP.get((proposed_type, expected_type))
if cast:
return cast(value)
raise TypeError(f"Type mismatch while writing param {key}: {proposed_type=} {expected_type=} {value=}")
def _cpp2python(self, t, value, default, key):
if value is None:
return None
try:
return CPP_2_PYTHON[t](value)
except (KeyError, TypeError, ValueError):
cloudlog.warning(f"Failed to cast param {key} with {value=} from type {t=}")
return self._cpp2python(t, default, None, key)
def get(self, key, bool block=False, bool return_default=False):
cdef string k = self.check_key(key)
cdef ParamKeyType t = self.p.getKeyType(k)
cdef optional[string] default = self.p.getKeyDefaultValue(k)
cdef string val
with nogil:
val = self.p.get(k, block)
default_val = (default.value() if default.has_value() else None) if return_default else None
if val == b"":
if block:
# If we got no value while running in blocked mode
# it means we got an interrupt while waiting
raise KeyboardInterrupt
else:
return self._cpp2python(t, default_val, None, key)
return self._cpp2python(t, val, default_val, key)
def get_bool(self, key, bool block=False):
cdef string k = self.check_key(key)
cdef bool r
with nogil:
r = self.p.getBool(k, block)
return r
def _put_cast(self, key, dat):
cdef string k = self.check_key(key)
cdef ParamKeyType t = self.p.getKeyType(k)
return ensure_bytes(self.python2cpp(type(dat), t, dat, key))
def put(self, key, dat):
"""
Warning: This function blocks until the param is written to disk!
In very rare cases this can take over a second, and your code will hang.
Use the put_nonblocking, put_bool_nonblocking in time sensitive code, but
in general try to avoid writing params as much as possible.
"""
cdef string k = self.check_key(key)
cdef string dat_bytes = self._put_cast(key, dat)
with nogil:
self.p.put(k, dat_bytes)
def put_bool(self, key, bool val):
cdef string k = self.check_key(key)
with nogil:
self.p.putBool(k, val)
def put_nonblocking(self, key, dat):
cdef string k = self.check_key(key)
cdef string dat_bytes = self._put_cast(key, dat)
with nogil:
self.p.putNonBlocking(k, dat_bytes)
def put_bool_nonblocking(self, key, bool val):
cdef string k = self.check_key(key)
with nogil:
self.p.putBoolNonBlocking(k, val)
def remove(self, key):
cdef string k = self.check_key(key)
with nogil:
self.p.remove(k)
def get_param_path(self, key=""):
cdef string key_bytes = ensure_bytes(key)
return self.p.getParamPath(key_bytes).decode("utf-8")
def get_type(self, key):
return self.p.getKeyType(self.check_key(key))
def all_keys(self, flag=ParamKeyFlag.ALL):
return self.p.allKeys(flag)
def get_default_value(self, key):
cdef string k = self.check_key(key)
cdef ParamKeyType t = self.p.getKeyType(k)
cdef optional[string] default = self.p.getKeyDefaultValue(k)
return self._cpp2python(t, default.value(), None, key) if default.has_value() else None
def cpp2python(self, key, value):
cdef string k = self.check_key(key)
cdef ParamKeyType t = self.p.getKeyType(k)
return self._cpp2python(t, value, None, key)

57
common/pid.py Normal file
View File

@@ -0,0 +1,57 @@
import numpy as np
from numbers import Number
class PIDController:
def __init__(self, k_p, k_i, k_d=0., pos_limit=1e308, neg_limit=-1e308, rate=100):
self._k_p: list[list[float]] = [[0], [k_p]] if isinstance(k_p, Number) else k_p
self._k_i: list[list[float]] = [[0], [k_i]] if isinstance(k_i, Number) else k_i
self._k_d: list[list[float]] = [[0], [k_d]] if isinstance(k_d, Number) else k_d
self.set_limits(pos_limit, neg_limit)
self.i_dt = 1.0 / rate
self.speed = 0.0
self.reset()
@property
def k_p(self):
return np.interp(self.speed, self._k_p[0], self._k_p[1])
@property
def k_i(self):
return np.interp(self.speed, self._k_i[0], self._k_i[1])
@property
def k_d(self):
return np.interp(self.speed, self._k_d[0], self._k_d[1])
def reset(self):
self.p = 0.0
self.i = 0.0
self.d = 0.0
self.f = 0.0
self.control = 0
def set_limits(self, pos_limit, neg_limit):
self.pos_limit = pos_limit
self.neg_limit = neg_limit
def update(self, error, error_rate=0.0, speed=0.0, feedforward=0., freeze_integrator=False):
self.speed = speed
self.p = self.k_p * float(error)
self.d = self.k_d * error_rate
self.f = feedforward
if not freeze_integrator:
i = self.i + self.k_i * self.i_dt * error
# Don't allow windup if already clipping
test_control = self.p + i + self.d + self.f
i_upperbound = self.i if test_control > self.pos_limit else self.pos_limit
i_lowerbound = self.i if test_control < self.neg_limit else self.neg_limit
self.i = np.clip(i, i_lowerbound, i_upperbound)
control = self.p + self.i + self.d + self.f
self.control = np.clip(control, self.neg_limit, self.pos_limit)
return self.control

43
common/prefix.h Normal file
View File

@@ -0,0 +1,43 @@
#pragma once
#include <cassert>
#include <string>
#include "common/params.h"
#include "common/util.h"
#include "system/hardware/hw.h"
class OpenpilotPrefix {
public:
OpenpilotPrefix(std::string prefix = {}) {
if (prefix.empty()) {
prefix = util::random_string(15);
}
#ifdef __APPLE__
msgq_path = "/tmp/msgq_" + prefix;
#else
msgq_path = "/dev/shm/msgq_" + prefix;
#endif
bool ret = util::create_directories(msgq_path, 0777);
assert(ret);
setenv("OPENPILOT_PREFIX", prefix.c_str(), 1);
}
~OpenpilotPrefix() {
auto param_path = Params().getParamPath();
if (util::file_exists(param_path)) {
std::string real_path = util::readlink(param_path);
util::check_system(util::string_format("rm %s -rf", real_path.c_str()));
unlink(param_path.c_str());
}
if (getenv("COMMA_CACHE") == nullptr) {
util::check_system(util::string_format("rm %s -rf", Path::download_cache_root().c_str()));
}
util::check_system(util::string_format("rm %s -rf", Path::comma_home().c_str()));
util::check_system(util::string_format("rm %s -rf", msgq_path.c_str()));
unsetenv("OPENPILOT_PREFIX");
}
private:
std::string msgq_path;
};

61
common/prefix.py Normal file
View File

@@ -0,0 +1,61 @@
import os
import platform
import shutil
import uuid
from openpilot.common.params import Params
from openpilot.system.hardware import PC
from openpilot.system.hardware.hw import Paths
from openpilot.system.hardware.hw import DEFAULT_DOWNLOAD_CACHE_ROOT
class OpenpilotPrefix:
def __init__(self, prefix: str | None = None, create_dirs_on_enter: bool = True, clean_dirs_on_exit: bool = True, shared_download_cache: bool = False):
self.prefix = prefix if prefix else str(uuid.uuid4().hex[0:15])
shm_path = "/tmp" if platform.system() == "Darwin" else "/dev/shm"
self.msgq_path = os.path.join(shm_path, "msgq_" + self.prefix)
self.create_dirs_on_enter = create_dirs_on_enter
self.clean_dirs_on_exit = clean_dirs_on_exit
self.shared_download_cache = shared_download_cache
def __enter__(self):
self.original_prefix = os.environ.get('OPENPILOT_PREFIX', None)
os.environ['OPENPILOT_PREFIX'] = self.prefix
if self.create_dirs_on_enter:
self.create_dirs()
if self.shared_download_cache:
os.environ["COMMA_CACHE"] = DEFAULT_DOWNLOAD_CACHE_ROOT
return self
def __exit__(self, exc_type, exc_obj, exc_tb):
if self.clean_dirs_on_exit:
self.clean_dirs()
try:
del os.environ['OPENPILOT_PREFIX']
if self.original_prefix is not None:
os.environ['OPENPILOT_PREFIX'] = self.original_prefix
except KeyError:
pass
return False
def create_dirs(self):
try:
os.mkdir(self.msgq_path)
except FileExistsError:
pass
os.makedirs(Paths.log_root(), exist_ok=True)
def clean_dirs(self):
symlink_path = Params().get_param_path()
if os.path.exists(symlink_path):
shutil.rmtree(os.path.realpath(symlink_path), ignore_errors=True)
os.remove(symlink_path)
shutil.rmtree(self.msgq_path, ignore_errors=True)
if PC:
shutil.rmtree(Paths.log_root(), ignore_errors=True)
if not os.environ.get("COMMA_CACHE", False):
shutil.rmtree(Paths.download_cache_root(), ignore_errors=True)
shutil.rmtree(Paths.comma_home(), ignore_errors=True)

52
common/queue.h Normal file
View File

@@ -0,0 +1,52 @@
#pragma once
#include <condition_variable>
#include <mutex>
#include <queue>
template <class T>
class SafeQueue {
public:
SafeQueue() = default;
void push(const T& v) {
{
std::unique_lock lk(m);
q.push(v);
}
cv.notify_one();
}
T pop() {
std::unique_lock lk(m);
cv.wait(lk, [this] { return !q.empty(); });
T v = q.front();
q.pop();
return v;
}
bool try_pop(T& v, int timeout_ms = 0) {
std::unique_lock lk(m);
if (!cv.wait_for(lk, std::chrono::milliseconds(timeout_ms), [this] { return !q.empty(); })) {
return false;
}
v = q.front();
q.pop();
return true;
}
bool empty() const {
std::scoped_lock lk(m);
return q.empty();
}
size_t size() const {
std::scoped_lock lk(m);
return q.size();
}
private:
mutable std::mutex m;
std::condition_variable cv;
std::queue<T> q;
};

40
common/ratekeeper.cc Normal file
View File

@@ -0,0 +1,40 @@
#include "common/ratekeeper.h"
#include <algorithm>
#include "common/swaglog.h"
#include "common/timing.h"
#include "common/util.h"
RateKeeper::RateKeeper(const std::string &name_, float rate, float print_delay_threshold_)
: name(name_),
print_delay_threshold(std::max(0.f, print_delay_threshold_)) {
interval = 1 / rate;
last_monitor_time = seconds_since_boot();
next_frame_time = last_monitor_time + interval;
}
bool RateKeeper::keepTime() {
bool lagged = monitorTime();
if (remaining_ > 0) {
util::sleep_for(remaining_ * 1000);
}
return lagged;
}
bool RateKeeper::monitorTime() {
++frame_;
last_monitor_time = seconds_since_boot();
remaining_ = next_frame_time - last_monitor_time;
bool lagged = remaining_ < 0;
if (lagged) {
if (print_delay_threshold > 0 && remaining_ < -print_delay_threshold) {
LOGW("%s lagging by %.2f ms", name.c_str(), -remaining_ * 1000);
}
next_frame_time = last_monitor_time + interval;
} else {
next_frame_time += interval;
}
return lagged;
}

23
common/ratekeeper.h Normal file
View File

@@ -0,0 +1,23 @@
#pragma once
#include <cstdint>
#include <string>
class RateKeeper {
public:
RateKeeper(const std::string &name, float rate, float print_delay_threshold = 0);
~RateKeeper() {}
bool keepTime();
bool monitorTime();
inline uint64_t frame() const { return frame_; }
inline double remaining() const { return remaining_; }
private:
double interval;
double next_frame_time;
double last_monitor_time;
double remaining_ = 0;
float print_delay_threshold = 0;
uint64_t frame_ = 0;
std::string name;
};

96
common/realtime.py Normal file
View File

@@ -0,0 +1,96 @@
"""Utilities for reading real time clocks and keeping soft real time constraints."""
import gc
import os
import sys
import time
from setproctitle import getproctitle
from openpilot.common.utils import MovingAverage
from openpilot.system.hardware import PC
# time step for each process
DT_CTRL = 0.01 # controlsd
DT_MDL = 0.05 # model
DT_HW = 0.5 # hardwared and manager
DT_DMON = 0.05 # driver monitoring
class Priority:
# CORE 2
# - modeld = 55
# - camerad = 54
CTRL_LOW = 51 # plannerd & radard
# CORE 3
# - pandad = 55
CTRL_HIGH = 53
def set_core_affinity(cores: list[int]) -> None:
if sys.platform == 'linux' and not PC:
os.sched_setaffinity(0, cores)
def config_realtime_process(cores: int | list[int], priority: int) -> None:
gc.disable()
if sys.platform == 'linux' and not PC:
os.sched_setscheduler(0, os.SCHED_FIFO, os.sched_param(priority))
c = cores if isinstance(cores, list) else [cores, ]
set_core_affinity(c)
class Ratekeeper:
def __init__(self, rate: float, print_delay_threshold: float | None = 0.0) -> None:
"""Rate in Hz for ratekeeping. print_delay_threshold must be nonnegative."""
self._interval = 1. / rate
self._print_delay_threshold = print_delay_threshold
self._frame = 0
self._remaining = 0.0
self._process_name = getproctitle()
self._last_monitor_time = -1.
self._next_frame_time = -1.
self.avg_dt = MovingAverage(100)
self.avg_dt.add_value(self._interval)
@property
def frame(self) -> int:
return self._frame
@property
def remaining(self) -> float:
return self._remaining
@property
def lagging(self) -> bool:
expected_dt = self._interval * (1 / 0.9)
return self.avg_dt.get_average() > expected_dt
# Maintain loop rate by calling this at the end of each loop
def keep_time(self) -> bool:
lagged = self.monitor_time()
if self._remaining > 0:
time.sleep(self._remaining)
return lagged
# Monitors the cumulative lag, but does not enforce a rate
def monitor_time(self) -> bool:
if self._last_monitor_time < 0:
self._next_frame_time = time.monotonic() + self._interval
self._last_monitor_time = time.monotonic()
prev = self._last_monitor_time
self._last_monitor_time = time.monotonic()
self.avg_dt.add_value(self._last_monitor_time - prev)
lagged = False
remaining = self._next_frame_time - time.monotonic()
self._next_frame_time += self._interval
if self._print_delay_threshold is not None and remaining < -self._print_delay_threshold:
print(f"{self._process_name} lagging by {-remaining * 1000:.2f} ms")
lagged = True
self._frame += 1
self._remaining = remaining
return lagged

54
common/simple_kalman.py Normal file
View File

@@ -0,0 +1,54 @@
import numpy as np
def get_kalman_gain(dt, A, C, Q, R, iterations=100):
P = np.zeros_like(Q)
for _ in range(iterations):
P = A.dot(P).dot(A.T) + dt * Q
S = C.dot(P).dot(C.T) + R
K = P.dot(C.T).dot(np.linalg.inv(S))
P = (np.eye(len(P)) - K.dot(C)).dot(P)
return K
class KF1D:
# this EKF assumes constant covariance matrix, so calculations are much simpler
# the Kalman gain also needs to be precomputed using the control module
def __init__(self, x0, A, C, K):
self.x0_0 = x0[0][0]
self.x1_0 = x0[1][0]
self.A0_0 = A[0][0]
self.A0_1 = A[0][1]
self.A1_0 = A[1][0]
self.A1_1 = A[1][1]
self.C0_0 = C[0]
self.C0_1 = C[1]
self.K0_0 = K[0][0]
self.K1_0 = K[1][0]
self.A_K_0 = self.A0_0 - self.K0_0 * self.C0_0
self.A_K_1 = self.A0_1 - self.K0_0 * self.C0_1
self.A_K_2 = self.A1_0 - self.K1_0 * self.C0_0
self.A_K_3 = self.A1_1 - self.K1_0 * self.C0_1
# K matrix needs to be pre-computed as follow:
# import control
# (x, l, K) = control.dare(np.transpose(self.A), np.transpose(self.C), Q, R)
# self.K = np.transpose(K)
def update(self, meas):
#self.x = np.dot(self.A_K, self.x) + np.dot(self.K, meas)
x0_0 = self.A_K_0 * self.x0_0 + self.A_K_1 * self.x1_0 + self.K0_0 * meas
x1_0 = self.A_K_2 * self.x0_0 + self.A_K_3 * self.x1_0 + self.K1_0 * meas
self.x0_0 = x0_0
self.x1_0 = x1_0
return [self.x0_0, self.x1_0]
@property
def x(self):
return [[self.x0_0], [self.x1_0]]
def set_x(self, x):
self.x0_0 = x[0][0]
self.x1_0 = x[1][0]

52
common/spinner.py Executable file
View File

@@ -0,0 +1,52 @@
import os
import subprocess
from openpilot.common.basedir import BASEDIR
class Spinner:
def __init__(self):
try:
self.spinner_proc = subprocess.Popen(["./spinner.py"],
stdin=subprocess.PIPE,
cwd=os.path.join(BASEDIR, "system", "ui"),
close_fds=True)
except OSError:
self.spinner_proc = None
def __enter__(self):
return self
def update(self, spinner_text: str):
if self.spinner_proc is not None:
self.spinner_proc.stdin.write(spinner_text.encode('utf8') + b"\n")
try:
self.spinner_proc.stdin.flush()
except BrokenPipeError:
pass
def update_progress(self, cur: float, total: float):
self.update(str(round(100 * cur / total)))
def close(self):
if self.spinner_proc is not None:
self.spinner_proc.kill()
try:
self.spinner_proc.communicate(timeout=2.)
except subprocess.TimeoutExpired:
print("WARNING: failed to kill spinner")
self.spinner_proc = None
def __del__(self):
self.close()
def __exit__(self, exc_type, exc_value, traceback):
self.close()
if __name__ == "__main__":
import time
with Spinner() as s:
s.update("Spinner text")
time.sleep(5.0)
print("gone")
time.sleep(5.0)

73
common/stat_live.py Normal file
View File

@@ -0,0 +1,73 @@
import numpy as np
class RunningStat:
# tracks realtime mean and standard deviation without storing any data
def __init__(self, priors=None, max_trackable=-1):
self.max_trackable = max_trackable
if priors is not None:
# initialize from history
self.M = priors[0]
self.S = priors[1]
self.n = priors[2]
self.M_last = self.M
self.S_last = self.S
else:
self.reset()
def reset(self):
self.M = 0.
self.S = 0.
self.M_last = 0.
self.S_last = 0.
self.n = 0
def push_data(self, new_data):
# short term memory hack
if self.max_trackable < 0 or self.n < self.max_trackable:
self.n += 1
if self.n == 0:
self.M_last = new_data
self.M = self.M_last
self.S_last = 0.
else:
self.M = self.M_last + (new_data - self.M_last) / self.n
self.S = self.S_last + (new_data - self.M_last) * (new_data - self.M)
self.M_last = self.M
self.S_last = self.S
def mean(self):
return self.M
def variance(self):
if self.n >= 2:
return self.S / (self.n - 1.)
else:
return 0
def std(self):
return np.sqrt(self.variance())
def params_to_save(self):
return [self.M, self.S, self.n]
class RunningStatFilter:
def __init__(self, raw_priors=None, filtered_priors=None, max_trackable=-1):
self.raw_stat = RunningStat(raw_priors, -1)
self.filtered_stat = RunningStat(filtered_priors, max_trackable)
def reset(self):
self.raw_stat.reset()
self.filtered_stat.reset()
def push_and_update(self, new_data):
_std_last = self.raw_stat.std()
self.raw_stat.push_data(new_data)
_delta_std = self.raw_stat.std() - _std_last
if _delta_std <= 0:
self.filtered_stat.push_data(new_data)
else:
pass
# self.filtered_stat.push_data(self.filtered_stat.mean())
# class SequentialBayesian():

157
common/swaglog.cc Normal file
View File

@@ -0,0 +1,157 @@
#ifndef _GNU_SOURCE
#define _GNU_SOURCE
#endif
#include "common/swaglog.h"
#include <cassert>
#include <limits>
#include <mutex>
#include <string>
#include <zmq.h>
#include <stdarg.h>
#include "third_party/json11/json11.hpp"
#include "common/version.h"
#include "system/hardware/hw.h"
#include "sunnypilot/common/version.h"
class SwaglogState {
public:
SwaglogState() {
zctx = zmq_ctx_new();
sock = zmq_socket(zctx, ZMQ_PUSH);
// Timeout on shutdown for messages to be received by the logging process
int timeout = 100;
zmq_setsockopt(sock, ZMQ_LINGER, &timeout, sizeof(timeout));
zmq_connect(sock, Path::swaglog_ipc().c_str());
// workaround for https://github.com/dropbox/json11/issues/38
setlocale(LC_NUMERIC, "C");
print_level = CLOUDLOG_WARNING;
if (const char* print_lvl = getenv("LOGPRINT")) {
if (strcmp(print_lvl, "debug") == 0) {
print_level = CLOUDLOG_DEBUG;
} else if (strcmp(print_lvl, "info") == 0) {
print_level = CLOUDLOG_INFO;
} else if (strcmp(print_lvl, "warning") == 0) {
print_level = CLOUDLOG_WARNING;
}
}
ctx_j = json11::Json::object{};
if (char* dongle_id = getenv("DONGLE_ID")) {
ctx_j["dongle_id"] = dongle_id;
}
if (char* git_origin = getenv("GIT_ORIGIN")) {
ctx_j["origin"] = git_origin;
}
if (char* git_branch = getenv("GIT_BRANCH")) {
ctx_j["branch"] = git_branch;
}
if (char* git_commit = getenv("GIT_COMMIT")) {
ctx_j["commit"] = git_commit;
}
if (char* daemon_name = getenv("MANAGER_DAEMON")) {
ctx_j["daemon"] = daemon_name;
}
ctx_j["version"] = SUNNYPILOT_VERSION;
ctx_j["dirty"] = !getenv("CLEAN");
ctx_j["device"] = Hardware::get_name();
}
~SwaglogState() {
zmq_close(sock);
zmq_ctx_destroy(zctx);
}
void log(int levelnum, const char* filename, int lineno, const char* func, const char* msg, const std::string& log_s) {
std::lock_guard lk(lock);
if (levelnum >= print_level) {
printf("%s: %s\n", filename, msg);
}
zmq_send(sock, log_s.data(), log_s.length(), ZMQ_NOBLOCK);
}
std::mutex lock;
void* zctx = nullptr;
void* sock = nullptr;
int print_level;
json11::Json::object ctx_j;
};
bool LOG_TIMESTAMPS = getenv("LOG_TIMESTAMPS");
uint32_t NO_FRAME_ID = std::numeric_limits<uint32_t>::max();
static void cloudlog_common(int levelnum, const char* filename, int lineno, const char* func,
char* msg_buf, const json11::Json::object &msg_j={}) {
static SwaglogState s;
json11::Json::object log_j = json11::Json::object {
{"ctx", s.ctx_j},
{"levelnum", levelnum},
{"filename", filename},
{"lineno", lineno},
{"funcname", func},
{"created", seconds_since_epoch()}
};
if (msg_j.empty()) {
log_j["msg"] = msg_buf;
} else {
log_j["msg"] = msg_j;
}
std::string log_s;
log_s += (char)levelnum;
((json11::Json)log_j).dump(log_s);
s.log(levelnum, filename, lineno, func, msg_buf, log_s);
free(msg_buf);
}
void cloudlog_e(int levelnum, const char* filename, int lineno, const char* func,
const char* fmt, ...) {
va_list args;
va_start(args, fmt);
char* msg_buf = nullptr;
int ret = vasprintf(&msg_buf, fmt, args);
va_end(args);
if (ret <= 0 || !msg_buf) return;
cloudlog_common(levelnum, filename, lineno, func, msg_buf);
}
void cloudlog_t_common(int levelnum, const char* filename, int lineno, const char* func,
uint32_t frame_id, const char* fmt, va_list args) {
if (!LOG_TIMESTAMPS) return;
char* msg_buf = nullptr;
int ret = vasprintf(&msg_buf, fmt, args);
if (ret <= 0 || !msg_buf) return;
json11::Json::object tspt_j = json11::Json::object{
{"event", msg_buf},
{"time", std::to_string(nanos_since_boot())}
};
if (frame_id < NO_FRAME_ID) {
tspt_j["frame_id"] = std::to_string(frame_id);
}
tspt_j = json11::Json::object{{"timestamp", tspt_j}};
cloudlog_common(levelnum, filename, lineno, func, msg_buf, tspt_j);
}
void cloudlog_te(int levelnum, const char* filename, int lineno, const char* func,
const char* fmt, ...) {
va_list args;
va_start(args, fmt);
cloudlog_t_common(levelnum, filename, lineno, func, NO_FRAME_ID, fmt, args);
va_end(args);
}
void cloudlog_te(int levelnum, const char* filename, int lineno, const char* func,
uint32_t frame_id, const char* fmt, ...) {
va_list args;
va_start(args, fmt);
cloudlog_t_common(levelnum, filename, lineno, func, frame_id, fmt, args);
va_end(args);
}

76
common/swaglog.h Normal file
View File

@@ -0,0 +1,76 @@
#pragma once
#include "common/timing.h"
#define CLOUDLOG_DEBUG 10
#define CLOUDLOG_INFO 20
#define CLOUDLOG_WARNING 30
#define CLOUDLOG_ERROR 40
#define CLOUDLOG_CRITICAL 50
#ifdef __GNUC__
#define SWAG_LOG_CHECK_FMT(a, b) __attribute__ ((format (printf, a, b)))
#else
#define SWAG_LOG_CHECK_FMT(a, b)
#endif
void cloudlog_e(int levelnum, const char* filename, int lineno, const char* func,
const char* fmt, ...) SWAG_LOG_CHECK_FMT(5, 6);
void cloudlog_te(int levelnum, const char* filename, int lineno, const char* func,
const char* fmt, ...) SWAG_LOG_CHECK_FMT(5, 6);
void cloudlog_te(int levelnum, const char* filename, int lineno, const char* func,
uint32_t frame_id, const char* fmt, ...) SWAG_LOG_CHECK_FMT(6, 7);
#define cloudlog(lvl, fmt, ...) cloudlog_e(lvl, __FILE__, __LINE__, \
__func__, \
fmt, ## __VA_ARGS__)
#define cloudlog_t(lvl, ...) cloudlog_te(lvl, __FILE__, __LINE__, \
__func__, \
__VA_ARGS__)
#define cloudlog_rl(burst, millis, lvl, fmt, ...) \
{ \
static uint64_t __begin = 0; \
static int __printed = 0; \
static int __missed = 0; \
\
int __burst = (burst); \
int __millis = (millis); \
uint64_t __ts = nanos_since_boot(); \
\
if (!__begin) { __begin = __ts; } \
\
if (__begin + __millis*1000000ULL < __ts) { \
if (__missed) { \
cloudlog(CLOUDLOG_WARNING, "cloudlog: %d messages suppressed", __missed); \
} \
__begin = 0; \
__printed = 0; \
__missed = 0; \
} \
\
if (__printed < __burst) { \
cloudlog(lvl, fmt, ## __VA_ARGS__); \
__printed++; \
} else { \
__missed++; \
} \
}
#define LOGT(...) cloudlog_t(CLOUDLOG_DEBUG, __VA_ARGS__)
#define LOGD(fmt, ...) cloudlog(CLOUDLOG_DEBUG, fmt, ## __VA_ARGS__)
#define LOG(fmt, ...) cloudlog(CLOUDLOG_INFO, fmt, ## __VA_ARGS__)
#define LOGW(fmt, ...) cloudlog(CLOUDLOG_WARNING, fmt, ## __VA_ARGS__)
#define LOGE(fmt, ...) cloudlog(CLOUDLOG_ERROR, fmt, ## __VA_ARGS__)
#define LOGD_100(fmt, ...) cloudlog_rl(2, 100, CLOUDLOG_DEBUG, fmt, ## __VA_ARGS__)
#define LOG_100(fmt, ...) cloudlog_rl(2, 100, CLOUDLOG_INFO, fmt, ## __VA_ARGS__)
#define LOGW_100(fmt, ...) cloudlog_rl(2, 100, CLOUDLOG_WARNING, fmt, ## __VA_ARGS__)
#define LOGE_100(fmt, ...) cloudlog_rl(2, 100, CLOUDLOG_ERROR, fmt, ## __VA_ARGS__)

144
common/swaglog.py Normal file
View File

@@ -0,0 +1,144 @@
import logging
import os
import time
import warnings
from pathlib import Path
from logging.handlers import BaseRotatingHandler
import zmq
from openpilot.common.logging_extra import SwagLogger, SwagFormatter, SwagLogFileFormatter
from openpilot.system.hardware.hw import Paths
def get_file_handler():
Path(Paths.swaglog_root()).mkdir(parents=True, exist_ok=True)
base_filename = os.path.join(Paths.swaglog_root(), "swaglog")
handler = SwaglogRotatingFileHandler(base_filename)
return handler
class SwaglogRotatingFileHandler(BaseRotatingHandler):
def __init__(self, base_filename, interval=60, max_bytes=1024*256, backup_count=2500, encoding=None):
super().__init__(base_filename, mode="a", encoding=encoding, delay=True)
self.base_filename = base_filename
self.interval = interval # seconds
self.max_bytes = max_bytes
self.backup_count = backup_count
self.log_files = self.get_existing_logfiles()
log_indexes = [f.split(".")[-1] for f in self.log_files]
self.last_file_idx = max([int(i) for i in log_indexes if i.isdigit()] or [-1])
self.last_rollover = None
self.doRollover()
def _open(self):
self.last_rollover = time.monotonic()
self.last_file_idx += 1
next_filename = f"{self.base_filename}.{self.last_file_idx:010}"
stream = open(next_filename, self.mode, encoding=self.encoding)
self.log_files.insert(0, next_filename)
return stream
def get_existing_logfiles(self):
log_files = list()
base_dir = os.path.dirname(self.base_filename)
for fn in os.listdir(base_dir):
fp = os.path.join(base_dir, fn)
if fp.startswith(self.base_filename) and os.path.isfile(fp):
log_files.append(fp)
return sorted(log_files)
def shouldRollover(self, record):
size_exceeded = self.max_bytes > 0 and self.stream.tell() >= self.max_bytes
time_exceeded = self.interval > 0 and self.last_rollover + self.interval <= time.monotonic()
return size_exceeded or time_exceeded
def doRollover(self):
if self.stream:
self.stream.close()
self.stream = self._open()
if self.backup_count > 0:
while len(self.log_files) > self.backup_count:
to_delete = self.log_files.pop()
if os.path.exists(to_delete): # just being safe, should always exist
os.remove(to_delete)
class UnixDomainSocketHandler(logging.Handler):
def __init__(self, formatter):
logging.Handler.__init__(self)
self.setFormatter(formatter)
self.pid = None
self.zctx = None
self.sock = None
def __del__(self):
self.close()
def close(self):
if self.sock is not None:
self.sock.close()
if self.zctx is not None:
self.zctx.term()
def connect(self):
self.zctx = zmq.Context()
self.sock = self.zctx.socket(zmq.PUSH)
self.sock.setsockopt(zmq.LINGER, 10)
self.sock.connect(Paths.swaglog_ipc())
self.pid = os.getpid()
def emit(self, record):
if os.getpid() != self.pid:
# TODO suppresses warning about forking proc with zmq socket, fix root cause
warnings.filterwarnings("ignore", category=ResourceWarning, message="unclosed.*<zmq.*>")
self.connect()
msg = self.format(record).rstrip('\n')
# print("SEND".format(repr(msg)))
try:
s = chr(record.levelno)+msg
self.sock.send(s.encode('utf8'), zmq.NOBLOCK)
except zmq.error.Again:
# drop :/
pass
class ForwardingHandler(logging.Handler):
def __init__(self, target_logger):
super().__init__()
self.target_logger = target_logger
def emit(self, record):
self.target_logger.handle(record)
def add_file_handler(log):
"""
Function to add the file log handler to swaglog.
This can be used to store logs when logmessaged is not running.
"""
handler = get_file_handler()
handler.setFormatter(SwagLogFileFormatter(log))
log.addHandler(handler)
cloudlog = log = SwagLogger()
log.setLevel(logging.DEBUG)
outhandler = logging.StreamHandler()
print_level = os.environ.get('LOGPRINT', 'warning')
if print_level == 'debug':
outhandler.setLevel(logging.DEBUG)
elif print_level == 'info':
outhandler.setLevel(logging.INFO)
elif print_level == 'warning':
outhandler.setLevel(logging.WARNING)
ipchandler = UnixDomainSocketHandler(SwagFormatter(log))
log.addHandler(outhandler)
# logs are sent through IPC before writing to disk to prevent disk I/O blocking
log.addHandler(ipchandler)

1
common/tests/.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
test_common

0
common/tests/__init__.py Normal file
View File

19
common/tests/test_file_helpers.py Normal file
View File

@@ -0,0 +1,19 @@
import os
from uuid import uuid4
from openpilot.common.utils import atomic_write
class TestFileHelpers:
def run_atomic_write_func(self, atomic_write_func):
path = f"/tmp/tmp{uuid4()}"
with atomic_write_func(path) as f:
f.write("test")
assert not os.path.exists(path)
with open(path) as f:
assert f.read() == "test"
os.remove(path)
def test_atomic_write(self):
self.run_atomic_write_func(atomic_write)

15
common/tests/test_markdown.py Normal file
View File

@@ -0,0 +1,15 @@
import os
from openpilot.common.basedir import BASEDIR
from openpilot.common.markdown import parse_markdown
class TestMarkdown:
def test_all_release_notes(self):
with open(os.path.join(BASEDIR, "CHANGELOG.md")) as f:
release_notes = f.read().split("\n\n")
assert len(release_notes) > 10
for rn in release_notes:
md = parse_markdown(rn)
assert len(md) > 0

27
common/tests/test_params.cc Normal file
View File

@@ -0,0 +1,27 @@
#include "catch2/catch.hpp"
#define private public
#include "common/params.h"
#include "common/util.h"
TEST_CASE("params_nonblocking_put") {
char tmp_path[] = "/tmp/asyncWriter_XXXXXX";
const std::string param_path = mkdtemp(tmp_path);
auto param_names = {"CarParams", "IsMetric"};
{
Params params(param_path);
for (const auto &name : param_names) {
params.putNonBlocking(name, "1");
// param is empty
REQUIRE(params.get(name).empty());
}
// check if thread is running
REQUIRE(params.future.valid());
REQUIRE(params.future.wait_for(std::chrono::milliseconds(0)) == std::future_status::timeout);
}
// check results
Params p(param_path);
for (const auto &name : param_names) {
REQUIRE(p.get(name) == "1");
}
}

141
common/tests/test_params.py Normal file
View File

@@ -0,0 +1,141 @@
import pytest
import datetime
import os
import threading
import time
import uuid
from openpilot.common.params import Params, ParamKeyFlag, UnknownKeyName
class TestParams:
def setup_method(self):
self.params = Params()
def test_params_put_and_get(self):
self.params.put("DongleId", "cb38263377b873ee")
assert self.params.get("DongleId") == "cb38263377b873ee"
def test_params_non_ascii(self):
st = b"\xe1\x90\xff"
self.params.put("CarParams", st)
assert self.params.get("CarParams") == st
def test_params_get_cleared_manager_start(self):
self.params.put("CarParams", b"test")
self.params.put("DongleId", "cb38263377b873ee")
assert self.params.get("CarParams") == b"test"
undefined_param = self.params.get_param_path(uuid.uuid4().hex)
with open(undefined_param, "w") as f:
f.write("test")
assert os.path.isfile(undefined_param)
self.params.clear_all(ParamKeyFlag.CLEAR_ON_MANAGER_START)
assert self.params.get("CarParams") is None
assert self.params.get("DongleId") is not None
assert not os.path.isfile(undefined_param)
def test_params_two_things(self):
self.params.put("DongleId", "bob")
self.params.put("AthenadPid", 123)
assert self.params.get("DongleId") == "bob"
assert self.params.get("AthenadPid") == 123
def test_params_get_block(self):
def _delayed_writer():
time.sleep(0.1)
self.params.put("CarParams", b"test")
threading.Thread(target=_delayed_writer).start()
assert self.params.get("CarParams") is None
assert self.params.get("CarParams", block=True) == b"test"
def test_params_unknown_key_fails(self):
with pytest.raises(UnknownKeyName):
self.params.get("swag")
with pytest.raises(UnknownKeyName):
self.params.get_bool("swag")
with pytest.raises(UnknownKeyName):
self.params.put("swag", "abc")
with pytest.raises(UnknownKeyName):
self.params.put_bool("swag", True)
def test_remove_not_there(self):
assert self.params.get("CarParams") is None
self.params.remove("CarParams")
assert self.params.get("CarParams") is None
def test_get_bool(self):
self.params.remove("IsMetric")
assert not self.params.get_bool("IsMetric")
self.params.put_bool("IsMetric", True)
assert self.params.get_bool("IsMetric")
self.params.put_bool("IsMetric", False)
assert not self.params.get_bool("IsMetric")
self.params.put("IsMetric", True)
assert self.params.get_bool("IsMetric")
self.params.put("IsMetric", False)
assert not self.params.get_bool("IsMetric")
def test_put_non_blocking_with_get_block(self):
q = Params()
def _delayed_writer():
time.sleep(0.1)
Params().put_nonblocking("CarParams", b"test")
threading.Thread(target=_delayed_writer).start()
assert q.get("CarParams") is None
assert q.get("CarParams", True) == b"test"
def test_put_bool_non_blocking_with_get_block(self):
q = Params()
def _delayed_writer():
time.sleep(0.1)
Params().put_bool_nonblocking("CarParams", True)
threading.Thread(target=_delayed_writer).start()
assert q.get("CarParams") is None
assert q.get("CarParams", True) == b"1"
def test_params_all_keys(self):
keys = Params().all_keys()
# sanity checks
assert len(keys) > 20
assert len(keys) == len(set(keys))
assert b"CarParams" in keys
def test_params_default_value(self):
self.params.remove("LanguageSetting")
self.params.remove("LongitudinalPersonality")
self.params.remove("LiveParameters")
assert self.params.get("LanguageSetting") is None
assert self.params.get("LanguageSetting", return_default=False) is None
assert isinstance(self.params.get("LanguageSetting", return_default=True), str)
assert isinstance(self.params.get("LongitudinalPersonality", return_default=True), int)
assert self.params.get("LiveParameters") is None
assert self.params.get("LiveParameters", return_default=True) is None
def test_params_get_type(self):
# json
self.params.put("ApiCache_FirehoseStats", {"a": 0})
assert self.params.get("ApiCache_FirehoseStats") == {"a": 0}
# int
self.params.put("BootCount", 1441)
assert self.params.get("BootCount") == 1441
# bool
self.params.put("AdbEnabled", True)
assert self.params.get("AdbEnabled")
assert isinstance(self.params.get("AdbEnabled"), bool)
# time
now = datetime.datetime.now(datetime.UTC)
self.params.put("InstallDate", now)
assert self.params.get("InstallDate") == now

2
common/tests/test_runner.cc Normal file
View File

@@ -0,0 +1,2 @@
#define CATCH_CONFIG_MAIN
#include "catch2/catch.hpp"

29
common/tests/test_simple_kalman.py Normal file
View File

@@ -0,0 +1,29 @@
from openpilot.common.simple_kalman import KF1D
class TestSimpleKalman:
def setup_method(self):
dt = 0.01
x0_0 = 0.0
x1_0 = 0.0
A0_0 = 1.0
A0_1 = dt
A1_0 = 0.0
A1_1 = 1.0
C0_0 = 1.0
C0_1 = 0.0
K0_0 = 0.12287673
K1_0 = 0.29666309
self.kf = KF1D(x0=[[x0_0], [x1_0]],
A=[[A0_0, A0_1], [A1_0, A1_1]],
C=[C0_0, C0_1],
K=[[K0_0], [K1_0]])
def test_getter_setter(self):
self.kf.set_x([[1.0], [1.0]])
assert self.kf.x == [[1.0], [1.0]]
def test_update_returns_state(self):
x = self.kf.update(100)
assert x == [i[0] for i in self.kf.x]

90
common/tests/test_swaglog.cc Normal file
View File

@@ -0,0 +1,90 @@
#include <zmq.h>
#include <iostream>
#include "catch2/catch.hpp"
#include "common/swaglog.h"
#include "common/util.h"
#include "common/version.h"
#include "system/hardware/hw.h"
#include "third_party/json11/json11.hpp"
#include "sunnypilot/common/version.h"
std::string daemon_name = "testy";
std::string dongle_id = "test_dongle_id";
int LINE_NO = 0;
void log_thread(int thread_id, int msg_cnt) {
for (int i = 0; i < msg_cnt; ++i) {
LOGD("%d", thread_id);
LINE_NO = __LINE__ - 1;
usleep(1);
}
}
void recv_log(int thread_cnt, int thread_msg_cnt) {
void *zctx = zmq_ctx_new();
void *sock = zmq_socket(zctx, ZMQ_PULL);
zmq_bind(sock, Path::swaglog_ipc().c_str());
std::vector<int> thread_msgs(thread_cnt);
int total_count = 0;
for (auto start = std::chrono::steady_clock::now(), now = start;
now < start + std::chrono::seconds{1} && total_count < (thread_cnt * thread_msg_cnt);
now = std::chrono::steady_clock::now()) {
char buf[4096] = {};
if (zmq_recv(sock, buf, sizeof(buf), ZMQ_DONTWAIT) <= 0) {
if (errno == EAGAIN || errno == EINTR || errno == EFSM) continue;
break;
}
REQUIRE(buf[0] == CLOUDLOG_DEBUG);
std::string err;
auto msg = json11::Json::parse(buf + 1, err);
REQUIRE(!msg.is_null());
REQUIRE(msg["levelnum"].int_value() == CLOUDLOG_DEBUG);
REQUIRE_THAT(msg["filename"].string_value(), Catch::Contains("test_swaglog.cc"));
REQUIRE(msg["funcname"].string_value() == "log_thread");
REQUIRE(msg["lineno"].int_value() == LINE_NO);
auto ctx = msg["ctx"];
REQUIRE(ctx["daemon"].string_value() == daemon_name);
REQUIRE(ctx["dongle_id"].string_value() == dongle_id);
REQUIRE(ctx["dirty"].bool_value() == true);
REQUIRE(ctx["version"].string_value() == SUNNYPILOT_VERSION);
std::string device = Hardware::get_name();
REQUIRE(ctx["device"].string_value() == device);
int thread_id = atoi(msg["msg"].string_value().c_str());
REQUIRE((thread_id >= 0 && thread_id < thread_cnt));
thread_msgs[thread_id]++;
total_count++;
}
for (int i = 0; i < thread_cnt; ++i) {
INFO("thread :" << i);
REQUIRE(thread_msgs[i] == thread_msg_cnt);
}
zmq_close(sock);
zmq_ctx_destroy(zctx);
}
TEST_CASE("swaglog") {
setenv("MANAGER_DAEMON", daemon_name.c_str(), 1);
setenv("DONGLE_ID", dongle_id.c_str(), 1);
setenv("dirty", "1", 1);
const int thread_cnt = 5;
const int thread_msg_cnt = 100;
std::vector<std::thread> log_threads;
for (int i = 0; i < thread_cnt; ++i) {
log_threads.push_back(std::thread(log_thread, i, thread_msg_cnt));
}
for (auto &t : log_threads) t.join();
recv_log(thread_cnt, thread_msg_cnt);
}

147
common/tests/test_util.cc Normal file
View File

@@ -0,0 +1,147 @@
#include <dirent.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <algorithm>
#include <climits>
#include <fstream>
#include <random>
#include <string>
#include "catch2/catch.hpp"
#include "common/util.h"
std::string random_bytes(int size) {
std::random_device rd;
std::independent_bits_engine<std::default_random_engine, CHAR_BIT, unsigned char> rbe(rd());
std::string bytes(size + 1, '\0');
std::generate(bytes.begin(), bytes.end(), std::ref(rbe));
return bytes;
}
TEST_CASE("util::read_file") {
SECTION("read /proc/version") {
std::string ret = util::read_file("/proc/version");
REQUIRE(ret.find("Linux version") != std::string::npos);
}
SECTION("read from sysfs") {
std::string ret = util::read_file("/sys/power/wakeup_count");
REQUIRE(!ret.empty());
}
SECTION("read file") {
char filename[] = "/tmp/test_read_XXXXXX";
int fd = mkstemp(filename);
REQUIRE(util::read_file(filename).empty());
std::string content = random_bytes(64 * 1024);
REQUIRE(write(fd, content.c_str(), content.size()) == (ssize_t)content.size());
std::string ret = util::read_file(filename);
bool equal = (ret == content);
REQUIRE(equal);
close(fd);
}
SECTION("read directory") {
REQUIRE(util::read_file(".").empty());
}
SECTION("read non-existent file") {
std::string ret = util::read_file("does_not_exist");
REQUIRE(ret.empty());
}
SECTION("read non-permission") {
REQUIRE(util::read_file("/proc/kmsg").empty());
}
}
TEST_CASE("util::file_exists") {
char filename[] = "/tmp/test_file_exists_XXXXXX";
int fd = mkstemp(filename);
REQUIRE(fd != -1);
close(fd);
SECTION("existent file") {
REQUIRE(util::file_exists(filename));
REQUIRE(util::file_exists("/tmp"));
}
SECTION("nonexistent file") {
std::string fn = filename;
REQUIRE(!util::file_exists(fn + "/nonexistent"));
}
SECTION("file has no access permissions") {
std::string fn = "/proc/kmsg";
std::ifstream f(fn);
REQUIRE(f.good() == false);
REQUIRE(util::file_exists(fn));
}
::remove(filename);
}
TEST_CASE("util::read_files_in_dir") {
char tmp_path[] = "/tmp/test_XXXXXX";
const std::string test_path = mkdtemp(tmp_path);
const std::string files[] = {".test1", "'test2'", "test3"};
for (auto fn : files) {
std::ofstream{test_path + "/" + fn} << fn;
}
mkdir((test_path + "/dir").c_str(), 0777);
std::map<std::string, std::string> result = util::read_files_in_dir(test_path);
REQUIRE(result.find("dir") == result.end());
REQUIRE(result.size() == std::size(files));
for (auto& [k, v] : result) {
REQUIRE(k == v);
}
}
TEST_CASE("util::safe_fwrite") {
char filename[] = "/tmp/XXXXXX";
int fd = mkstemp(filename);
close(fd);
std::string dat = random_bytes(1024 * 1024);
FILE *f = util::safe_fopen(filename, "wb");
REQUIRE(f != nullptr);
size_t size = util::safe_fwrite(dat.data(), 1, dat.size(), f);
REQUIRE(size == dat.size());
int ret = util::safe_fflush(f);
REQUIRE(ret == 0);
ret = fclose(f);
REQUIRE(ret == 0);
bool equal = (dat == util::read_file(filename));
REQUIRE(equal);
}
TEST_CASE("util::create_directories") {
REQUIRE(system("rm /tmp/test_create_directories -rf") == 0);
std::string dir = "/tmp/test_create_directories/a/b/c/d/e/f";
auto check_dir_permissions = [](const std::string &path, mode_t mode) -> bool {
struct stat st = {};
return stat(path.c_str(), &st) == 0 && (st.st_mode & S_IFMT) == S_IFDIR && (st.st_mode & (S_IRWXU | S_IRWXG | S_IRWXO)) == mode;
};
SECTION("create_directories") {
REQUIRE(util::create_directories(dir, 0755));
REQUIRE(check_dir_permissions(dir, 0755));
}
SECTION("dir already exists") {
REQUIRE(util::create_directories(dir, 0755));
REQUIRE(util::create_directories(dir, 0755));
}
SECTION("a file exists with the same name") {
REQUIRE(util::create_directories(dir, 0755));
int f = open((dir + "/file").c_str(), O_RDWR | O_CREAT, 0644);
REQUIRE(f != -1);
close(f);
REQUIRE(util::create_directories(dir + "/file", 0755) == false);
REQUIRE(util::create_directories(dir + "/file/1/2/3", 0755) == false);
}
SECTION("end with slashes") {
REQUIRE(util::create_directories(dir + "/", 0755));
}
SECTION("empty") {
REQUIRE(util::create_directories("", 0755) == false);
}
}

63
common/text_window.py Executable file
View File

@@ -0,0 +1,63 @@
#!/usr/bin/env python3
import os
import time
import subprocess
from openpilot.common.basedir import BASEDIR
class TextWindow:
def __init__(self, text):
try:
self.text_proc = subprocess.Popen(["./text.py", text],
stdin=subprocess.PIPE,
cwd=os.path.join(BASEDIR, "system", "ui"),
close_fds=True)
except OSError:
self.text_proc = None
def get_status(self):
if self.text_proc is not None:
self.text_proc.poll()
return self.text_proc.returncode
return None
def __enter__(self):
return self
def close(self):
if self.text_proc is not None:
self.text_proc.terminate()
self.text_proc = None
def wait_for_exit(self):
if self.text_proc is not None:
while True:
if self.get_status() == 1:
return
time.sleep(0.1)
def __del__(self):
self.close()
def __exit__(self, exc_type, exc_value, traceback):
self.close()
if __name__ == "__main__":
text = """Traceback (most recent call last):
File "./controlsd.py", line 608, in <module>
main()
File "./controlsd.py", line 604, in main
controlsd_thread(sm, pm, logcan)
File "./controlsd.py", line 455, in controlsd_thread
1/0
ZeroDivisionError: division by zero"""
print(text)
with TextWindow(text) as s:
for _ in range(100):
if s.get_status() == 1:
print("Got exit button")
break
time.sleep(0.1)
print("gone")

16
common/time_helpers.py Normal file
View File

@@ -0,0 +1,16 @@
import datetime
from pathlib import Path
MIN_DATE = datetime.datetime(year=2025, month=2, day=21)
MAX_DATE = datetime.datetime(year=2035, month=1, day=1)
def min_date():
# on systemd systems, the default time is the systemd build time
systemd_path = Path("/lib/systemd/systemd")
if systemd_path.exists():
d = datetime.datetime.fromtimestamp(systemd_path.stat().st_mtime)
return max(MIN_DATE, d + datetime.timedelta(days=1))
return MIN_DATE
def system_time_valid():
return min_date() < datetime.datetime.now() < MAX_DATE

27
common/timeout.py Normal file
View File

@@ -0,0 +1,27 @@
import signal
class TimeoutException(Exception):
pass
class Timeout:
"""
Timeout context manager.
For example this code will raise a TimeoutException:
with Timeout(seconds=5, error_msg="Sleep was too long"):
time.sleep(10)
"""
def __init__(self, seconds, error_msg=None):
if error_msg is None:
error_msg = f'Timed out after {seconds} seconds'
self.seconds = seconds
self.error_msg = error_msg
def handle_timeout(self, signume, frame):
raise TimeoutException(self.error_msg)
def __enter__(self):
signal.signal(signal.SIGALRM, self.handle_timeout)
signal.alarm(self.seconds)
def __exit__(self, exc_type, exc_val, exc_tb):
signal.alarm(0)

51
common/timing.h Normal file
View File

@@ -0,0 +1,51 @@
#pragma once
#include <cstdint>
#include <ctime>
#ifdef __APPLE__
#define CLOCK_BOOTTIME CLOCK_MONOTONIC
#endif
static inline uint64_t nanos_since_boot() {
struct timespec t;
clock_gettime(CLOCK_BOOTTIME, &t);
return t.tv_sec * 1000000000ULL + t.tv_nsec;
}
static inline double millis_since_boot() {
struct timespec t;
clock_gettime(CLOCK_BOOTTIME, &t);
return t.tv_sec * 1000.0 + t.tv_nsec * 1e-6;
}
static inline double seconds_since_boot() {
struct timespec t;
clock_gettime(CLOCK_BOOTTIME, &t);
return (double)t.tv_sec + t.tv_nsec * 1e-9;
}
static inline uint64_t nanos_since_epoch() {
struct timespec t;
clock_gettime(CLOCK_REALTIME, &t);
return t.tv_sec * 1000000000ULL + t.tv_nsec;
}
static inline double seconds_since_epoch() {
struct timespec t;
clock_gettime(CLOCK_REALTIME, &t);
return (double)t.tv_sec + t.tv_nsec * 1e-9;
}
// you probably should use nanos_since_boot instead
static inline uint64_t nanos_monotonic() {
struct timespec t;
clock_gettime(CLOCK_MONOTONIC, &t);
return t.tv_sec * 1000000000ULL + t.tv_nsec;
}
static inline uint64_t nanos_monotonic_raw() {
struct timespec t;
clock_gettime(CLOCK_MONOTONIC_RAW, &t);
return t.tv_sec * 1000000000ULL + t.tv_nsec;
}

70
common/transformations/README.md Normal file
View File

@@ -0,0 +1,70 @@
Reference Frames
------
Many reference frames are used throughout. This
folder contains all helper functions needed to
transform between them. Generally this is done
by generating a rotation matrix and multiplying.
| Name | [x, y, z] | Units | Notes |
| :-------------: |:-------------:| :-----:| :----: |
| Geodetic | [Latitude, Longitude, Altitude] | geodetic coordinates | Sometimes used as [lon, lat, alt], avoid this frame. |
| ECEF | [x, y, z] | meters | We use **ITRF14 (IGS14)**, NOT NAD83. <br> This is the global Mesh3D frame. |
| NED | [North, East, Down] | meters | Relative to earth's surface, useful for visualizing. |
| Device | [Forward, Right, Down] | meters | This is the Mesh3D local frame. <br> Relative to camera, **not imu.** <br> ![img](http://upload.wikimedia.org/wikipedia/commons/thumb/2/2f/RPY_angles_of_airplanes.png/440px-RPY_angles_of_airplanes.png)|
| Calibrated | [Forward, Right, Down] | meters | This is the frame the model outputs are in. <br> More details below. <br>|
| Car | [Forward, Right, Down] | meters | This is useful for estimating position of points on the road. <br> More details below. <br>|
| View | [Right, Down, Forward] | meters | Like device frame, but according to camera conventions. |
| Camera | [u, v, focal] | pixels | Like view frame, but 2d on the camera image.|
| Normalized Camera | [u / focal, v / focal, 1] | / | |
| Model | [u, v, focal] | pixels | The sampled rectangle of the full camera frame the model uses. |
| Normalized Model | [u / focal, v / focal, 1] | / | |
Orientation Conventions
------
Quaternions, rotation matrices and euler angles are three
equivalent representations of orientation and all three are
used throughout the code base.
For euler angles the preferred convention is [roll, pitch, yaw]
which corresponds to rotations around the [x, y, z] axes. All
euler angles should always be in radians or radians/s unless
for plotting or display purposes. For quaternions the hamilton
notations is preferred which is [q<sub>w</sub>, q<sub>x</sub>, q<sub>y</sub>, q<sub>z</sub>]. All quaternions
should always be normalized with a strictly positive q<sub>w</sub>. **These
quaternions are a unique representation of orientation whereas euler angles
or rotation matrices are not.**
To rotate from one frame into another with euler angles the
convention is to rotate around roll, then pitch and then yaw,
while rotating around the rotated axes, not the original axes.
Car frame
------
Device frame is aligned with the road-facing camera used by openpilot. However, when controlling the vehicle it is helpful to think in a reference frame aligned with the vehicle. These two reference frames can be different.
The orientation of car frame is defined to be aligned with the car's direction of travel and the road plane when the vehicle is driving on a flat road and not turning. The origin of car frame is defined to be directly below device frame (in car frame), such that it is on the road plane. The position and orientation of this frame is not necessarily always aligned with the direction of travel or the road plane due to suspension movements and other effects.
Calibrated frame
------
It is helpful for openpilot's driving model to take in images that look similar when mounted differently in different cars. To achieve this we "calibrate" the images by transforming it into calibrated frame. Calibrated frame is defined to be aligned with car frame in pitch and yaw, and aligned with device frame in roll. It also has the same origin as device frame.
Example
------
To transform global Mesh3D positions and orientations (positions_ecef, quats_ecef) into the local frame described by the
first position and orientation from Mesh3D one would do:
```
ecef_from_local = rot_from_quat(quats_ecef[0])
local_from_ecef = ecef_from_local.T
positions_local = np.einsum('ij,kj->ki', local_from_ecef, postions_ecef - positions_ecef[0])
rotations_global = rot_from_quat(quats_ecef)
rotations_local = np.einsum('ij,kjl->kil', local_from_ecef, rotations_global)
eulers_local = euler_from_rot(rotations_local)
```

0
common/transformations/__init__.py Normal file
View File

179
common/transformations/camera.py Normal file
View File

@@ -0,0 +1,179 @@
import itertools
import numpy as np
from dataclasses import dataclass
import openpilot.common.transformations.orientation as orient
## -- hardcoded hardware params --
@dataclass(frozen=True)
class CameraConfig:
width: int
height: int
focal_length: float
@property
def size(self):
return (self.width, self.height)
@property
def intrinsics(self):
# aka 'K' aka camera_frame_from_view_frame
return np.array([
[self.focal_length, 0.0, float(self.width)/2],
[0.0, self.focal_length, float(self.height)/2],
[0.0, 0.0, 1.0]
])
@property
def intrinsics_inv(self):
# aka 'K_inv' aka view_frame_from_camera_frame
return np.linalg.inv(self.intrinsics)
@dataclass(frozen=True)
class _NoneCameraConfig(CameraConfig):
width: int = 0
height: int = 0
focal_length: float = 0
@dataclass(frozen=True)
class DeviceCameraConfig:
fcam: CameraConfig
dcam: CameraConfig
ecam: CameraConfig
def all_cams(self):
for cam in ['fcam', 'dcam', 'ecam']:
if not isinstance(getattr(self, cam), _NoneCameraConfig):
yield cam, getattr(self, cam)
_ar_ox_fisheye = CameraConfig(1928, 1208, 567.0) # focal length probably wrong? magnification is not consistent across frame
_os_fisheye = CameraConfig(2688 // 2, 1520 // 2, 567.0 / 4 * 3)
_ar_ox_config = DeviceCameraConfig(CameraConfig(1928, 1208, 2648.0), _ar_ox_fisheye, _ar_ox_fisheye)
_os_config = DeviceCameraConfig(CameraConfig(2688 // 2, 1520 // 2, 1522.0 * 3 / 4), _os_fisheye, _os_fisheye)
_neo_config = DeviceCameraConfig(CameraConfig(1164, 874, 910.0), CameraConfig(816, 612, 650.0), _NoneCameraConfig())
DEVICE_CAMERAS = {
# A "device camera" is defined by a device type and sensor
# sensor type was never set on eon/neo/two
("neo", "unknown"): _neo_config,
# unknown here is AR0231, field was added with OX03C10 support
("tici", "unknown"): _ar_ox_config,
# before deviceState.deviceType was set, assume tici AR config
("unknown", "ar0231"): _ar_ox_config,
("unknown", "ox03c10"): _ar_ox_config,
# simulator (emulates a tici)
("pc", "unknown"): _ar_ox_config,
}
prods = itertools.product(('tici', 'tizi', 'mici'), (('ar0231', _ar_ox_config), ('ox03c10', _ar_ox_config), ('os04c10', _os_config)))
DEVICE_CAMERAS.update({(d, c[0]): c[1] for d, c in prods})
# device/mesh : x->forward, y-> right, z->down
# view : x->right, y->down, z->forward
device_frame_from_view_frame = np.array([
[ 0., 0., 1.],
[ 1., 0., 0.],
[ 0., 1., 0.]
])
view_frame_from_device_frame = device_frame_from_view_frame.T
# aka 'extrinsic_matrix'
# road : x->forward, y -> left, z->up
def get_view_frame_from_road_frame(roll, pitch, yaw, height):
device_from_road = orient.rot_from_euler([roll, pitch, yaw]).dot(np.diag([1, -1, -1]))
view_from_road = view_frame_from_device_frame.dot(device_from_road)
return np.hstack((view_from_road, [[0], [height], [0]]))
# aka 'extrinsic_matrix'
def get_view_frame_from_calib_frame(roll, pitch, yaw, height):
device_from_calib= orient.rot_from_euler([roll, pitch, yaw])
view_from_calib = view_frame_from_device_frame.dot(device_from_calib)
return np.hstack((view_from_calib, [[0], [height], [0]]))
def vp_from_ke(m):
"""
Computes the vanishing point from the product of the intrinsic and extrinsic
matrices C = KE.
The vanishing point is defined as lim x->infinity C (x, 0, 0, 1).T
"""
return (m[0, 0]/m[2, 0], m[1, 0]/m[2, 0])
def roll_from_ke(m):
# note: different from calibration.h/RollAnglefromKE: i think that one's just wrong
return np.arctan2(-(m[1, 0] - m[1, 1] * m[2, 0] / m[2, 1]),
-(m[0, 0] - m[0, 1] * m[2, 0] / m[2, 1]))
def normalize(img_pts, intrinsics):
# normalizes image coordinates
# accepts single pt or array of pts
intrinsics_inv = np.linalg.inv(intrinsics)
img_pts = np.array(img_pts)
input_shape = img_pts.shape
img_pts = np.atleast_2d(img_pts)
img_pts = np.hstack((img_pts, np.ones((img_pts.shape[0], 1))))
img_pts_normalized = img_pts.dot(intrinsics_inv.T)
img_pts_normalized[(img_pts < 0).any(axis=1)] = np.nan
return img_pts_normalized[:, :2].reshape(input_shape)
def denormalize(img_pts, intrinsics, width=np.inf, height=np.inf):
# denormalizes image coordinates
# accepts single pt or array of pts
img_pts = np.array(img_pts)
input_shape = img_pts.shape
img_pts = np.atleast_2d(img_pts)
img_pts = np.hstack((img_pts, np.ones((img_pts.shape[0], 1), dtype=img_pts.dtype)))
img_pts_denormalized = img_pts.dot(intrinsics.T)
if np.isfinite(width):
img_pts_denormalized[img_pts_denormalized[:, 0] > width] = np.nan
img_pts_denormalized[img_pts_denormalized[:, 0] < 0] = np.nan
if np.isfinite(height):
img_pts_denormalized[img_pts_denormalized[:, 1] > height] = np.nan
img_pts_denormalized[img_pts_denormalized[:, 1] < 0] = np.nan
return img_pts_denormalized[:, :2].reshape(input_shape)
def get_calib_from_vp(vp, intrinsics):
vp_norm = normalize(vp, intrinsics)
yaw_calib = np.arctan(vp_norm[0])
pitch_calib = -np.arctan(vp_norm[1]*np.cos(yaw_calib))
roll_calib = 0
return roll_calib, pitch_calib, yaw_calib
def device_from_ecef(pos_ecef, orientation_ecef, pt_ecef):
# device from ecef frame
# device frame is x -> forward, y-> right, z -> down
# accepts single pt or array of pts
input_shape = pt_ecef.shape
pt_ecef = np.atleast_2d(pt_ecef)
ecef_from_device_rot = orient.rotations_from_quats(orientation_ecef)
device_from_ecef_rot = ecef_from_device_rot.T
pt_ecef_rel = pt_ecef - pos_ecef
pt_device = np.einsum('jk,ik->ij', device_from_ecef_rot, pt_ecef_rel)
return pt_device.reshape(input_shape)
def img_from_device(pt_device):
# img coordinates from pts in device frame
# first transforms to view frame, then to img coords
# accepts single pt or array of pts
input_shape = pt_device.shape
pt_device = np.atleast_2d(pt_device)
pt_view = np.einsum('jk,ik->ij', view_frame_from_device_frame, pt_device)
# This function should never return negative depths
pt_view[pt_view[:, 2] < 0] = np.nan
pt_img = pt_view/pt_view[:, 2:3]
return pt_img.reshape(input_shape)[:, :2]

Some files were not shown because too many files have changed in this diff Show More