Refactor flag handling in ESCC and introduce sunnyParams

Refactored ESCC flag handling to use sunnyParams for clarity and better structure. Updated related tests and configurations to align with the new sunnyParams structure. Also, added the sunnyParams definition to car.capnp.
Update Hyundai carcontroller for ESCC and adjust SP flags
2026-06-24 14:22:07 +08:00 · 2024-11-10 21:26:16 +01:00 · 2024-11-10 16:48:49 +01:00 · 2024-11-10 15:37:43 +01:00 · 2024-11-10 15:19:26 +01:00 · 2024-11-08 17:45:31 +01:00
195 changed files with 3457 additions and 4808 deletions
@@ -0,0 +1,3 @@
+.Xauthority
+.env
+.host/
@@ -0,0 +1,18 @@
+FROM ghcr.io/commaai/openpilot-base:latest
+
+RUN apt update && apt install -y vim net-tools usbutils htop ripgrep tmux wget mesa-utils xvfb libxtst6 libxv1 libglu1-mesa gdb bash-completion
+RUN python3 -m ensurepip --upgrade
+RUN pip3 install ipython jupyter jupyterlab
+
+RUN cd /tmp && \
+    ARCH=$(arch | sed s/aarch64/arm64/ | sed s/x86_64/amd64/) && \
+    curl -L -o virtualgl.deb "https://github.com/VirtualGL/virtualgl/releases/download/3.1.1/virtualgl_3.1.1_$ARCH.deb" && \
+    dpkg -i virtualgl.deb
+
+RUN usermod -aG video batman
+
+USER batman
+
+RUN cd $HOME && \
+    curl -O https://raw.githubusercontent.com/commaai/agnos-builder/master/userspace/home/.tmux.conf && \
+    curl -O https://raw.githubusercontent.com/commaai/agnos-builder/master/userspace/home/.vimrc
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+
+TARGET_USER=batman
+source .devcontainer/.host/.env
+
+# override display flag for mac hosts
+if [[ $HOST_OS == darwin ]]; then
+  echo "Setting up DISPLAY override for macOS..."
+  cat <<EOF >> /home/$TARGET_USER/.bashrc
+source .devcontainer/.host/.env
+if [ -n "\$HOST_DISPLAY" ]; then
+  DISPLAY_NUM=\$(echo "\$HOST_DISPLAY" | awk -F: '{print \$NF}')
+  export DISPLAY=host.docker.internal:\$DISPLAY_NUM
+fi
+EOF
+fi
+
+# setup virtualgl for mac hosts
+if [[ $HOST_OS == darwin ]]; then
+  echo "Setting up virtualgl for macOS..."
+  cat <<EOF >> /home/$TARGET_USER/.bashrc
+if [ -n "\$HOST_DISPLAY" ]; then
+  export VGL_PORT=10000
+  export VGL_CLIENT=host.docker.internal
+  export VGL_COMPRESS=rgb
+  export VGL_DISPLAY=:99
+  export VGL_FPS=60
+  # prevent vglrun from running exec
+  alias exec=:; source vglrun :; unalias exec
+fi
+EOF
+fi
+
+# These lines are temporary, to remain backwards compatible with old devcontainers
+# that were running as root and therefore had their caches written as root
+sudo chown -R $TARGET_USER: /tmp/scons_cache
+sudo chown -R $TARGET_USER: /tmp/comma_download_cache
+sudo chown -R $TARGET_USER: /home/batman/.comma
@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+source .devcontainer/.host/.env
+
+# setup safe directories for submodules
+SUBMODULE_DIRS=$(git config --file .gitmodules --get-regexp path | awk '{ print $2 }')
+for DIR in $SUBMODULE_DIRS; do 
+  git config --global --add safe.directory "$PWD/$DIR"
+done
+
+# virtual display for virtualgl
+if [[ "$HOST_OS" == "darwin" ]] && [[ -n "$HOST_DISPLAY" ]]; then
+  echo "Starting virtual display at :99 ..."
+  tmux new-session -d -s fakedisplay Xvfb :99 -screen 0 1920x1080x24
+fi
@@ -0,0 +1,53 @@
+{
+  "name": "openpilot devcontainer",
+  "build": {
+    "dockerfile": "Dockerfile"
+  },
+  "postCreateCommand": ".devcontainer/container_post_create.sh",
+  "postStartCommand": ".devcontainer/container_post_start.sh",
+  "initializeCommand": [".devcontainer/host_setup"],
+  "privileged": true,
+  "containerEnv": {
+    "DISPLAY": "${localEnv:DISPLAY}",
+    "PYTHONPATH": "${containerWorkspaceFolder}",
+    "TERM": "xterm-256color",
+    "force_color_prompt": "1"
+  },
+  "runArgs": [
+    "--volume=/dev:/dev",
+    "--volume=/tmp/.X11-unix:/tmp/.X11-unix",
+    "--volume=${localWorkspaceFolder}/.devcontainer/.host/.Xauthority:/home/batman/.Xauthority",
+    "--volume=${localEnv:HOME}/.comma:/home/batman/.comma",
+    "--volume=${localEnv:HOME}/.azure:/home/batman/.azure",
+    "--volume=/tmp/comma_download_cache:/tmp/comma_download_cache",
+    "--shm-size=1G",
+    "--add-host=host.docker.internal:host-gateway", // required to use host.docker.internal on linux
+    "--publish=0.0.0.0:8070-8079:8070-8079"  // body ZMQ services
+  ],
+  "features": {
+    "ghcr.io/devcontainers/features/common-utils:2": {
+      "installZsh": false,
+      "installOhMyZsh": false,
+      "upgradePackages": false,
+      "username": "batman"
+    },
+    "ghcr.io/devcontainers-contrib/features/gh-cli:1": {},
+    "ghcr.io/devcontainers/features/azure-cli:1": {}
+  },
+  "containerUser": "batman",
+  "remoteUser": "batman",
+  "customizations": {
+    "vscode": {
+      "extensions": [
+        "ms-python.python",
+        "ms-vscode.cpptools",
+        "ms-toolsai.jupyter",
+        "guyskk.language-cython",
+        "lharri73.dbc"
+      ]
+    }
+  },
+  "mounts": [
+    "type=volume,source=scons_cache,target=/tmp/scons_cache"
+  ]
+}
@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+
+# pull base image
+if [[ -z $USE_LOCAL_IMAGE ]]; then
+  echo "Updating openpilot_base image if needed..."
+  docker pull ghcr.io/commaai/openpilot-base:latest
+fi
+
+# setup .host dir
+mkdir -p .devcontainer/.host
+
+# setup links to Xauthority
+XAUTHORITY_LINK=".devcontainer/.host/.Xauthority"
+rm -f $XAUTHORITY_LINK
+if [[ -z $XAUTHORITY ]]; then
+  echo "XAUTHORITY not set. Fallback to ~/.Xauthority ..."
+  if ! [[ -f $HOME/.Xauthority ]]; then
+    echo "~/.XAuthority file does not exist. GUI tools may not work properly."
+    touch $XAUTHORITY_LINK # dummy file to satisfy container volume mount
+  else
+    ln -sf $HOME/.Xauthority $XAUTHORITY_LINK
+  fi
+else
+    ln -sf $XAUTHORITY $XAUTHORITY_LINK
+fi
+
+# setup host env file
+HOST_INFO_FILE=".devcontainer/.host/.env"
+SYSTEM=$(uname -s | tr '[:upper:]' '[:lower:]')
+echo "HOST_OS=\"$SYSTEM\"" > $HOST_INFO_FILE
+echo "HOST_DISPLAY=\"$DISPLAY\"" >> $HOST_INFO_FILE
+
+# run virtualgl if macos
+if [[ $SYSTEM == "darwin" ]]; then
+  echo
+  if [[ -f /opt/VirtualGL/bin/vglclient ]]; then
+    echo "Starting VirtualGL client at port 10000..."
+    VGL_LOG_FILE=".devcontainer/.host/.vgl/vglclient.log"
+    mkdir -p "$(dirname $VGL_LOG_FILE)"
+    /opt/VirtualGL/bin/vglclient -l "$VGL_LOG_FILE" -display "$DISPLAY" -port 10000 -detach
+  else
+    echo "VirtualGL not found. GUI tools may not work properly. Some GUI tools require OpenGL to work properly. To use them with XQuartz on mac, VirtualGL needs to be installed. To install it run:"
+    echo
+    echo "  brew install --cask virtualgl"
+    echo
+  fi
+fi
@@ -0,0 +1,10 @@
+:: pull base image
+IF NOT DEFINED USE_LOCAL_IMAGE ^
+echo "Updating openpilot_base image if needed..." && ^
+docker pull ghcr.io/commaai/openpilot-base:latest
+
+:: setup .host dir
+mkdir .devcontainer\.host
+
+:: setup host env file
+echo "" > .devcontainer\.host\.env
@@ -8,7 +8,7 @@ assignees: ''

 **Checklist**

- [ ] added entry to CAR in selfdrive/car/*/values.py and ran `selfdrive/car/docs.py` to generate new docs
+- [ ] added entry to CAR in selfdrive/car/*/values.py and ran `selfdrive/opcar/docs.py` to generate new docs
 - [ ] test route added to [routes.py](https://github.com/commaai/openpilot/blob/master/selfdrive/car/tests/routes.py)
 - [ ] route with openpilot:
 - [ ] route with stock system:
@@ -44,7 +44,7 @@ Explain how you tested this bug fix.

 **Checklist**

- [ ] added entry to CAR in selfdrive/car/*/values.py and ran `selfdrive/car/docs.py` to generate new docs
+- [ ] added entry to CAR in selfdrive/car/*/values.py and ran `selfdrive/opcar/docs.py` to generate new docs
 - [ ] test route added to [routes.py](https://github.com/commaai/openpilot/blob/master/selfdrive/car/tests/routes.py)
 - [ ] route with openpilot:
 - [ ] route with stock system:
@@ -14,25 +14,17 @@ inputs:
    description: 'whether to save the cache'
    default: 'false'
    required: false
-outputs:
-  cache-hit:
-    description: 'cache hit occurred'
-    value: ${{ (contains(runner.name, 'nsc') && steps.ns-cache.outputs.cache-hit) ||
-               (!contains(runner.name, 'nsc') && inputs.save != 'false' && steps.gha-cache.outputs.cache-hit) ||
-               (!contains(runner.name, 'nsc') && inputs.save == 'false' && steps.gha-cache-ro.outputs.cache-hit) }}

 runs:
  using: "composite"
  steps:
    - name: setup namespace cache
-      id: ns-cache
      if: ${{ contains(runner.name, 'nsc') }}
      uses: namespacelabs/nscloud-cache-action@v1
      with:
        path: ${{ inputs.path }}

    - name: setup github cache
-      id: gha-cache
      if: ${{ !contains(runner.name, 'nsc') && inputs.save != 'false' }}
      uses: 'actions/cache@v4'
      with:
@@ -41,7 +33,6 @@ runs:
        restore-keys: ${{ inputs.restore-keys }}

    - name: setup github cache
-      id: gha-cache-ro
      if: ${{ !contains(runner.name, 'nsc') && inputs.save == 'false' }}
      uses: 'actions/cache/restore@v4'
      with:
@@ -15,3 +15,7 @@ jobs:
    uses: commaai/openpilot/.github/workflows/selfdrive_tests.yaml@master
    with:
      run_number: ${{ inputs.run_number }}
+  tools_tests:
+    uses: commaai/openpilot/.github/workflows/tools_tests.yaml@master
+    with:
+      run_number: ${{ inputs.run_number }}
@@ -19,9 +19,8 @@ jobs:
  docs:
    name: build docs
    runs-on: ubuntu-latest
+    timeout-minutes: 1
    steps:
-    - uses: commaai/timeout@v1
-
    - uses: actions/checkout@v4
      with:
        submodules: true
@@ -1,7 +1,7 @@
 name: release
 on:
  schedule:
-    - cron: '0 9 * * *'
+    - cron: '0 10 * * *'
  workflow_dispatch:

 jobs:
@@ -28,7 +28,7 @@ jobs:
        git add .
    - name: update car docs
      run: |
-        scons -j$(nproc) --minimal opendbc_repo
+        scons -j$(nproc) --minimal opendbc
        PYTHONPATH=. python selfdrive/car/docs.py
        git add docs/CARS.md
    - name: Create Pull Request
@@ -32,9 +32,9 @@ env:
 jobs:
  build_release:
    name: build release
-    runs-on:
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
+    runs-on: ${{ ((github.repository == 'commaai/openpilot') &&
+                   ((github.event_name != 'pull_request') ||
+                    (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-latest' }}
    env:
      STRIPPED_DIR: /tmp/releasepilot
    steps:
@@ -52,7 +52,7 @@ jobs:
      run: TARGET_DIR=$STRIPPED_DIR release/build_devel.sh
    - uses: ./.github/workflows/setup-with-retry
    - name: Check submodules
-      if: github.repository == 'commaai/openpilot'
+      if: github.ref == 'refs/heads/master' && github.repository == 'commaai/openpilot'
      timeout-minutes: 3
      run: release/check-submodules.sh
    - name: Build openpilot and run checks
@@ -75,9 +75,15 @@ jobs:
        ${{ env.RUN }} "scripts/lint/lint.sh --skip check_added_large_files"

  build:
-    runs-on:
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
+    strategy:
+      matrix:
+        arch: ${{ fromJson(
+           ((github.repository == 'commaai/openpilot') &&
+              ((github.event_name != 'pull_request') ||
+               (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && '["x86_64", "aarch64"]' || '["x86_64"]' ) }}
+    runs-on: ${{ ((matrix.arch == 'aarch64') && 'namespace-profile-arm64-2x8') ||
+                 ((matrix.arch == 'x86_64') && ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16') ||
+                   'ubuntu-latest'}}
    steps:
    - uses: actions/checkout@v4
      with:
@@ -86,41 +92,61 @@ jobs:
      if: github.ref == 'refs/heads/master' && github.event_name != 'pull_request' && github.repository == 'commaai/openpilot'
      run: |
        echo "PUSH_IMAGE=true" >> "$GITHUB_ENV"
+        echo "TARGET_ARCHITECTURE=${{ matrix.arch }}" >> "$GITHUB_ENV"
        $DOCKER_LOGIN
    - uses: ./.github/workflows/setup-with-retry
+      with:
+        docker_hub_pat: ${{ secrets.DOCKER_HUB_PAT }}
    - uses: ./.github/workflows/compile-openpilot
-      timeout-minutes: 30
+      timeout-minutes: ${{ ((steps.restore-scons-cache.outputs.cache-hit == 'true') && 15 || 30) }} # allow more time when we missed the scons cache

  build_mac:
    name: build macOS
-    runs-on: ${{ github.repository == 'commaai/openpilot' && 'namespace-profile-macos-8x14' || 'macos-latest' }}
+    runs-on: macos-latest
    steps:
    - uses: actions/checkout@v4
      with:
        submodules: true
-    - name: Homebrew cache
-      uses: ./.github/workflows/auto-cache
-      with:
-        path: ~/Library/Caches/Homebrew
+    - run: git lfs pull
    - name: Install dependencies
      run: ./tools/mac_setup.sh
      env:
        # package install has DeprecationWarnings
        PYTHONWARNINGS: default
-    - run: git lfs pull
    - run: echo "CACHE_COMMIT_DATE=$(git log -1 --pretty='format:%cd' --date=format:'%Y-%m-%d-%H:%M')" >> $GITHUB_ENV
    - name: Getting scons cache
-      uses: ./.github/workflows/auto-cache
+      uses: 'actions/cache@v4'
      with:
        path: /tmp/scons_cache
+        key: scons-${{ runner.arch }}-macos-${{ env.CACHE_COMMIT_DATE }}-${{ github.sha }}
+        restore-keys: |
+          scons-${{ runner.arch }}-macos-${{ env.CACHE_COMMIT_DATE }}
+          scons-${{ runner.arch }}-macos
    - name: Building openpilot
      run: . .venv/bin/activate && scons -j$(nproc)

+  docker_push_multiarch:
+    name: docker push multiarch tag
+    runs-on: ubuntu-latest
+    if: github.ref == 'refs/heads/master' && github.event_name != 'pull_request' && github.repository == 'commaai/openpilot'
+    needs: [build]
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        submodules: false
+    - name: Setup docker
+      run: |
+        $DOCKER_LOGIN
+    - name: Merge x64 and arm64 tags
+      run: |
+        export PUSH_IMAGE=true
+        scripts/retry.sh selfdrive/test/docker_tag_multiarch.sh base x86_64 aarch64
+
  static_analysis:
    name: static analysis
-    runs-on:
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
+    runs-on: ${{ ((github.repository == 'commaai/openpilot') &&
+                   ((github.event_name != 'pull_request') ||
+                    (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
    env:
      PYTHONWARNINGS: default
    steps:
@@ -134,24 +160,27 @@ jobs:

  unit_tests:
    name: unit tests
-    runs-on:
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
+    runs-on: ${{ ((github.repository == 'commaai/openpilot') &&
+                   ((github.event_name != 'pull_request') ||
+                    (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-latest' }}
    steps:
    - uses: actions/checkout@v4
      with:
        submodules: true
    - uses: ./.github/workflows/setup-with-retry
+      with:
+        docker_hub_pat: ${{ secrets.DOCKER_HUB_PAT }}
    - name: Build openpilot
+      timeout-minutes: ${{ ((steps.restore-scons-cache.outputs.cache-hit == 'true') && 10 || 30) }} # allow more time when we missed the scons cache
      run: ${{ env.RUN }} "scons -j$(nproc)"
    - name: Run unit tests
-      timeout-minutes: ${{ contains(runner.name, 'nsc') && 1 || 20 }}
+      timeout-minutes: 15
      run: |
-        ${{ env.RUN }} "$PYTEST --collect-only -m 'not slow' &> /dev/null && \
-                        MAX_EXAMPLES=1 $PYTEST -m 'not slow' && \
+        ${{ env.RUN }} "source selfdrive/test/setup_xvfb.sh && \
+                        $PYTEST --timeout 60 -m 'not slow' && \
                        ./selfdrive/ui/tests/create_test_translations.sh && \
                        QT_QPA_PLATFORM=offscreen ./selfdrive/ui/tests/test_translations && \
-                        chmod -R 777 /tmp/comma_download_cache"
+                        pytest ./selfdrive/ui/tests/test_translations.py"
    - name: "Upload coverage to Codecov"
      uses: codecov/codecov-action@v4
      with:
@@ -161,25 +190,27 @@ jobs:

  process_replay:
    name: process replay
-    runs-on:
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
+    runs-on: ${{ ((github.repository == 'commaai/openpilot') &&
+                   ((github.event_name != 'pull_request') ||
+                    (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-latest' }}
    steps:
    - uses: actions/checkout@v4
      with:
        submodules: true
    - uses: ./.github/workflows/setup-with-retry
+      with:
+        docker_hub_pat: ${{ secrets.DOCKER_HUB_PAT }}
    - name: Cache test routes
      id: dependency-cache
      uses: actions/cache@v4
      with:
        path: .ci_cache/comma_download_cache
-        key: proc-replay-${{ hashFiles('selfdrive/test/process_replay/ref_commit', 'selfdrive/test/process_replay/test_processes.py') }}
+        key: proc-replay-${{ hashFiles('.github/workflows/selfdrive_tests.yaml', 'selfdrive/test/process_replay/ref_commit', 'selfdrive/test/process_replay/test_regen.py') }}
    - name: Build openpilot
      run: |
        ${{ env.RUN }} "scons -j$(nproc)"
    - name: Run replay
-      timeout-minutes: ${{ contains(runner.name, 'nsc') && (steps.dependency-cache.outputs.cache-hit == 'true') && 1 || 20 }}
+      timeout-minutes: 30
      run: |
        ${{ env.RUN }} "coverage run selfdrive/test/process_replay/test_processes.py -j$(nproc) && \
                        chmod -R 777 /tmp/comma_download_cache && \
@@ -214,33 +245,33 @@ jobs:

  test_cars:
    name: cars
-    runs-on:
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
+    runs-on: ${{ ((github.repository == 'commaai/openpilot') &&
+                   ((github.event_name != 'pull_request') ||
+                    (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-latest' }}
    strategy:
      fail-fast: false
      matrix:
-        job: [0, 1, 2, 3]
+        job: [0, 1]
    steps:
    - uses: actions/checkout@v4
      with:
        submodules: true
    - uses: ./.github/workflows/setup-with-retry
    - name: Cache test routes
-      id: routes-cache
-      uses: actions/cache@v4
+      id: dependency-cache
+      uses: ./.github/workflows/auto-cache
      with:
        path: .ci_cache/comma_download_cache
        key: car_models-${{ hashFiles('selfdrive/car/tests/test_models.py', 'selfdrive/car/tests/routes.py') }}-${{ matrix.job }}
    - name: Build openpilot
      run: ${{ env.RUN }} "scons -j$(nproc)"
    - name: Test car models
-      timeout-minutes: ${{ contains(runner.name, 'nsc') && (steps.routes-cache.outputs.cache-hit == 'true') && 1 || 20 }}
+      timeout-minutes: 20
      run: |
-        ${{ env.RUN }} "MAX_EXAMPLES=1 $PYTEST selfdrive/car/tests/test_models.py && \
+        ${{ env.RUN }} "$PYTEST selfdrive/car/tests/test_models.py && \
                        chmod -R 777 /tmp/comma_download_cache"
      env:
-        NUM_JOBS: 4
+        NUM_JOBS: 2
        JOB_ID: ${{ matrix.job }}
    - name: "Upload coverage to Codecov"
      uses: codecov/codecov-action@v4
@@ -306,9 +337,9 @@ jobs:

  simulator_driving:
    name: simulator driving
-    runs-on:
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
+    runs-on: ${{ ((github.repository == 'commaai/openpilot') &&
+                   ((github.event_name != 'pull_request') ||
+                    (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
    steps:
    - uses: actions/checkout@v4
      with:
@@ -327,9 +358,9 @@ jobs:
  create_ui_report:
    # This job name needs to be the same as UI_JOB_NAME in ui_preview.yaml
    name: Create UI Report
-    runs-on:
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
-      - ${{ ((github.repository == 'commaai/openpilot') && ((github.event_name != 'pull_request') || (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-experiments:docker.builds.local-cache=separate' || 'ubuntu-24.04' }}
+    runs-on: ${{ ((github.repository == 'commaai/openpilot') &&
+                   ((github.event_name != 'pull_request') ||
+                    (github.event.pull_request.head.repo.full_name == 'commaai/openpilot'))) && 'namespace-profile-amd64-8x16' || 'ubuntu-24.04' }}
    steps:
      - uses: actions/checkout@v4
        with:
@@ -17,6 +17,7 @@ runs:
      uses: ./.github/workflows/setup
      continue-on-error: true
      with:
+        docker_hub_pat: ${{ inputs.docker_hub_pat }}
        is_retried: true
    - if: steps.setup1.outcome == 'failure'
      shell: bash
@@ -26,6 +27,7 @@ runs:
      uses: ./.github/workflows/setup
      continue-on-error: true
      with:
+        docker_hub_pat: ${{ inputs.docker_hub_pat }}
        is_retried: true
    - if: steps.setup2.outcome == 'failure'
      shell: bash
@@ -34,4 +36,5 @@ runs:
      if: steps.setup2.outcome == 'failure'
      uses: ./.github/workflows/setup
      with:
+        docker_hub_pat: ${{ inputs.docker_hub_pat }}
        is_retried: true
@@ -1,6 +1,10 @@
 name: 'openpilot env setup'

 inputs:
+  docker_hub_pat:
+    description: 'Auth token for Docker Hub, required for BuildJet jobs'
+    required: true
+    default: ''
  is_retried:
    description: 'A mock param that asserts that we use the setup-with-retry instead of this action directly'
    required: false
@@ -16,20 +20,23 @@ runs:
        echo "You should not run this action directly. Use setup-with-retry instead"
        exit 1

-    - shell: bash
-      name: No retries!
-      run: |
-        if [ "${{ github.run_attempt }}" -gt 1 ]; then
-          echo -e "\033[0;31m##################################################"
-          echo -e "\033[0;31m    Retries not allowed! Fix the flaky test!      "
-          echo -e "\033[0;31m##################################################\033[0m"
-          exit 1
-        fi
-
    # do this after checkout to ensure our custom LFS config is used to pull from GitLab
    - shell: bash
      run: git lfs pull

+    # on BuildJet runners, must be logged into DockerHub to avoid rate limiting
+    # https://buildjet.com/for-github-actions/docs/guides/docker
+    - shell: bash
+      if: ${{ contains(runner.name, 'buildjet') && inputs.docker_hub_pat == '' }}
+      run: |
+        echo "Need to set the Docker Hub PAT secret as an input to this action"
+        exit 1
+    - name: Login to Docker Hub
+      if: contains(runner.name, 'buildjet')
+      shell: bash
+      run: |
+        docker login -u adeebshihadeh -p ${{ inputs.docker_hub_pat }}
+
    # build cache
    - id: date
      shell: bash
@@ -0,0 +1,54 @@
+name: tools
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+  workflow_call:
+    inputs:
+      run_number:
+        default: '1'
+        required: true
+        type: string
+concurrency:
+  group: tools-tests-ci-run-${{ inputs.run_number }}-${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && github.run_id || github.head_ref || github.ref }}-${{ github.workflow }}-${{ github.event_name }}
+  cancel-in-progress: true
+
+env:
+  BASE_IMAGE: openpilot-base
+  DOCKER_LOGIN: docker login ghcr.io -u ${{ github.actor }} -p ${{ secrets.GITHUB_TOKEN }}
+
+  BUILD: selfdrive/test/docker_build.sh base
+
+  RUN: docker run --shm-size 2G -v $GITHUB_WORKSPACE:/tmp/openpilot -w /tmp/openpilot -e FILEREADER_CACHE=1 -e PYTHONPATH=/tmp/openpilot -e NUM_JOBS -e JOB_ID -e GITHUB_ACTION -e GITHUB_REF -e GITHUB_HEAD_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_RUN_ID -v $GITHUB_WORKSPACE/.ci_cache/scons_cache:/tmp/scons_cache -v $GITHUB_WORKSPACE/.ci_cache/comma_download_cache:/tmp/comma_download_cache -v $GITHUB_WORKSPACE/.ci_cache/openpilot_cache:/tmp/openpilot_cache $BASE_IMAGE /bin/bash -c
+
+
+jobs:
+  devcontainer:
+    name: devcontainer
+    runs-on: ubuntu-latest
+    if: false  # we can re-enable once this is faster
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        submodules: true
+    - uses: ./.github/workflows/setup-with-retry
+    - name: Use local image for testing devcontainer with latest base image
+      run: |
+        echo "USE_LOCAL_IMAGE=true" >> "$GITHUB_ENV"
+    - name: Setup Dev Container CLI
+      run: npm install -g @devcontainers/cli
+    - name: Build dev container image
+      run: ./scripts/retry.sh devcontainer build --workspace-folder .
+    - name: Run dev container
+      run: |
+        mkdir -p /tmp/devcontainer_scons_cache/
+        cp -r $GITHUB_WORKSPACE/.ci_cache/scons_cache/. /tmp/devcontainer_scons_cache/
+        devcontainer up --workspace-folder .
+    - name: Test environment
+      run: |
+        devcontainer exec --workspace-folder . scons -j$(nproc) cereal/ common/
+        devcontainer exec --workspace-folder . pip3 install pip-install-test
+        devcontainer exec --workspace-folder . touch /home/batman/.comma/auth.json
+        devcontainer exec --workspace-folder . sudo touch /root/test.txt
@@ -84,7 +84,7 @@ jobs:
        run: >-
          sudo apt-get install -y imagemagick

-          scenes="homescreen settings_device settings_software settings_toggles settings_developer offroad_alert update_available prime onroad onroad_disengaged onroad_override onroad_sidebar onroad_wide onroad_wide_sidebar onroad_alert_small onroad_alert_mid onroad_alert_full driver_camera body keyboard"
+          scenes="homescreen settings_device settings_toggles settings_developer offroad_alert update_available prime onroad onroad_disengaged onroad_override onroad_sidebar onroad_wide onroad_wide_sidebar onroad_alert_small onroad_alert_mid onroad_alert_full driver_camera body keyboard"
          A=($scenes)

          DIFF=""
@@ -76,7 +76,6 @@ selfdrive/modeld/models/*.thneed
 selfdrive/modeld/models/*.pkl

 *.bz2
-*.zst

 build/

@@ -1,18 +1,18 @@
 [submodule "panda"]
  path = panda
-  url = ../../commaai/panda.git
+  url = https://github.com/sunnyhaibin/panda.git
 [submodule "opendbc"]
  path = opendbc_repo
-  url = ../../commaai/opendbc.git
+  url = https://github.com/sunnypilot/opendbc.git
 [submodule "msgq"]
  path = msgq_repo
-  url = ../../commaai/msgq.git
+  url = https://github.com/sunnypilot/msgq.git
 [submodule "rednose_repo"]
  path = rednose_repo
-  url = ../../commaai/rednose.git
+  url = https://github.com/commaai/rednose.git
 [submodule "teleoprtc_repo"]
  path = teleoprtc_repo
-  url = ../../commaai/teleoprtc
+  url = https://github.com/commaai/teleoprtc
 [submodule "tinygrad"]
  path = tinygrad_repo
-  url = https://github.com/commaai/tinygrad.git
+  url = https://github.com/tinygrad/tinygrad.git
@@ -68,14 +68,17 @@ RUN usermod -aG sudo $USER
 RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
 USER $USER

-COPY --chown=$USER pyproject.toml uv.lock /home/$USER
-COPY --chown=$USER tools/install_python_dependencies.sh /home/$USER/tools/
+COPY --chown=$USER pyproject.toml uv.lock /tmp/
+COPY --chown=$USER tools/install_python_dependencies.sh /tmp/tools/

 ENV VIRTUAL_ENV=/home/$USER/.venv
 ENV PATH="$VIRTUAL_ENV/bin:$PATH"
-RUN cd /home/$USER && \
+RUN cd /tmp && \
    tools/install_python_dependencies.sh && \
-    rm -rf tools/ pyproject.toml uv.lock .cache
+    mkdir -p $VIRTUAL_ENV && \
+    cp -r /tmp/.venv/* $VIRTUAL_ENV && \
+    rm -rf /tmp/* && \
+    rm -rf /home/$USER/.cache

 USER root
 RUN sudo git config --global --add safe.directory /tmp/openpilot
@@ -66,7 +66,7 @@ fi
 ln -snf ${env.TEST_DIR} /data/pythonpath

 cd ${env.TEST_DIR} || true
-time ${cmd}
+${cmd}
 END"""

    sh script: ssh_cmd, label: step_label
@@ -79,10 +79,6 @@ def deviceStage(String stageName, String deviceType, List extra_env, def steps)
        return
    }

-    if (isReplay()) {
-      error("REPLAYING TESTS IS NOT ALLOWED. FIX THEM INSTEAD.")
-    }
-
    def extra = extra_env.collect { "export ${it}" }.join('\n');
    def branch = env.BRANCH_NAME ?: 'master';
    def gitDiff = sh returnStdout: true, script: 'curl -s -H "Authorization: Bearer ${GITHUB_COMMENTS_TOKEN}" https://api.github.com/repos/commaai/openpilot/compare/master...${GIT_BRANCH} | jq .files[].filename || echo "/"', label: 'Getting changes'
@@ -127,11 +123,6 @@ def hasPathChanged(String gitDiff, List<String> paths) {
  return false
 }

-def isReplay() {
-  def replayClass = "org.jenkinsci.plugins.workflow.cps.replay.ReplayCause"
-  return currentBuild.rawBuild.getCauses().any{ cause -> cause.toString().contains(replayClass) }
-}
-
 def setupCredentials() {
  withCredentials([
    string(credentialsId: 'azure_token', variable: 'AZURE_TOKEN'),
@@ -202,22 +193,22 @@ node {
    parallel (
      // tici tests
      'onroad tests': {
-        deviceStage("onroad", "tici-needs-can", ["UNSAFE=1"], [
+        deviceStage("onroad", "tici-needs-can", [], [
          // TODO: ideally, this test runs in master-ci, but it takes 5+m to build it
          //["build master-ci", "cd $SOURCE_DIR/release && TARGET_DIR=$TEST_DIR $SOURCE_DIR/scripts/retry.sh ./build_devel.sh"],
          step("build openpilot", "cd system/manager && ./build.py"),
          step("check dirty", "release/check-dirty.sh"),
-          step("onroad tests", "pytest selfdrive/test/test_onroad.py -s", [timeout: 60]),
+          step("onroad tests", "pytest selfdrive/test/test_onroad.py -s"),
          //["time to onroad", "pytest selfdrive/test/test_time_to_onroad.py"],
        ])
      },
      'HW + Unit Tests': {
        deviceStage("tici-hardware", "tici-common", ["UNSAFE=1"], [
          step("build", "cd system/manager && ./build.py"),
-          step("test pandad", "pytest selfdrive/pandad/tests/test_pandad.py", [diffPaths: ["panda", "selfdrive/pandad/"]]),
+          step("test pandad", "pytest selfdrive/pandad/tests/test_pandad.py", [diffPaths: ["panda/", "selfdrive/pandad/"]]),
          step("test power draw", "pytest -s system/hardware/tici/tests/test_power_draw.py"),
-          step("test encoder", "LD_LIBRARY_PATH=/usr/local/lib pytest system/loggerd/tests/test_encoder.py", [diffPaths: ["system/loggerd/"]]),
-          step("test pigeond", "pytest system/ubloxd/tests/test_pigeond.py", [diffPaths: ["system/ubloxd/"]]),
+          step("test encoder", "LD_LIBRARY_PATH=/usr/local/lib pytest system/loggerd/tests/test_encoder.py"),
+          step("test pigeond", "pytest system/ubloxd/tests/test_pigeond.py"),
          step("test manager", "pytest system/manager/test/test_manager.py"),
        ])
      },
@@ -230,12 +221,12 @@ node {
      'camerad': {
        deviceStage("AR0231", "tici-ar0231", ["UNSAFE=1"], [
          step("build", "cd system/manager && ./build.py"),
-          step("test camerad", "pytest system/camerad/test/test_camerad.py", [timeout: 60]),
+          step("test camerad", "pytest system/camerad/test/test_camerad.py"),
          step("test exposure", "pytest system/camerad/test/test_exposure.py"),
        ])
        deviceStage("OX03C10", "tici-ox03c10", ["UNSAFE=1"], [
          step("build", "cd system/manager && ./build.py"),
-          step("test camerad", "pytest system/camerad/test/test_camerad.py", [timeout: 60]),
+          step("test camerad", "pytest system/camerad/test/test_camerad.py"),
          step("test exposure", "pytest system/camerad/test/test_exposure.py"),
        ])
      },
@@ -251,8 +242,8 @@ node {
      },
      'replay': {
        deviceStage("model-replay", "tici-replay", ["UNSAFE=1"], [
-          step("build", "cd system/manager && ./build.py", [diffPaths: ["selfdrive/modeld/", "tinygrad_repo", "selfdrive/test/process_replay/model_replay.py"]]),
-          step("model replay", "selfdrive/test/process_replay/model_replay.py", [diffPaths: ["selfdrive/modeld/", "tinygrad_repo", "selfdrive/test/process_replay/model_replay.py"]]),
+          step("build", "cd system/manager && ./build.py", [diffPaths: ["selfdrive/modeld/"]]),
+          step("model replay", "selfdrive/test/process_replay/model_replay.py", [diffPaths: ["selfdrive/modeld/"]]),
        ])
      },
      'tizi': {
@@ -260,9 +251,9 @@ node {
          step("build openpilot", "cd system/manager && ./build.py"),
          step("test pandad loopback", "SINGLE_PANDA=1 pytest selfdrive/pandad/tests/test_pandad_loopback.py"),
          step("test pandad spi", "pytest selfdrive/pandad/tests/test_pandad_spi.py"),
-          step("test pandad", "pytest selfdrive/pandad/tests/test_pandad.py", [diffPaths: ["panda", "selfdrive/pandad/"]]),
+          step("test pandad", "pytest selfdrive/pandad/tests/test_pandad.py", [diffPaths: ["panda/", "selfdrive/pandad/"]]),
          step("test amp", "pytest system/hardware/tici/tests/test_amplifier.py"),
-          step("test qcomgpsd", "pytest system/qcomgpsd/tests/test_qcomgpsd.py", [diffPaths: ["system/qcomgpsd/"]]),
+          step("test qcomgpsd", "pytest system/qcomgpsd/tests/test_qcomgpsd.py"),
        ])
      },

@@ -0,0 +1,21 @@
+# Custom MIT License
+
+Copyright (c) 2024, Haibin Wen, SUNNYPILOT LLC
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to view and modify the Software, subject to the following conditions:
+
+1. **Permission Required**: Permission Required for Commercial, For-Profit, or Closed Source Use: Use of the Software, in whole or in part, for any commercial purposes, for-profit projects, or in closed source projects requires explicit written permission from the original author(s).
+
+2. **Redistribution**: Any redistribution of the Software, modified or unmodified, must retain this license notice and the following acknowledgment:
+   "This software is licensed under a custom license requiring permission for use."
+
+3. **Visibility**: Any project that uses the Software must visibly mention the following acknowledgment:
+   "This project uses software from Haibin Wen and SUNNYPILOT LLC and is licensed under a custom license requiring permission for use."
+
+4. **No Warranty**: THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+Contact sunnypilot Support <support@sunnypilot.ai> for permission requests.
+
+---
+
+Haibin Wen, SUNNYPILOT LLC
@@ -4,12 +4,8 @@ Version 0.9.8 (2024-XX-XX)
  * Trained in brand new ML simulator
  * Model now gates applying positive accel in Chill mode
 * New driving monitoring model
-  * Reduced false positives related to passengers
-* Image processing pipeline moved to the ISP
-  * More GPU time for driving models
-  * Power draw reduced 0.5W, which means your device runs cooler 
+  * Reduced false positives related to passengers 
 * Added toggle to enable driver monitoring even when openpilot is not engaged
-* Enable openpilot longitudinal control for Ford Q3 vehicles
 * New Toyota TSS2 longitudinal tune

 Version 0.9.7 (2024-06-13)
@@ -349,7 +349,7 @@ Export('common', 'gpucommon')
 env_swaglog = env.Clone()
 env_swaglog['CXXFLAGS'].append('-DSWAGLOG="\\"common/swaglog.h\\""')
 SConscript(['msgq_repo/SConscript'], exports={'env': env_swaglog})
-SConscript(['opendbc_repo/SConscript'], exports={'env': env_swaglog})
+SConscript(['opendbc/can/SConscript'], exports={'env': env_swaglog})

 SConscript(['cereal/SConscript'])

@@ -366,7 +366,6 @@ SConscript(['rednose/SConscript'])

 # Build system services
 SConscript([
-  'system/ui/SConscript',
  'system/proclogd/SConscript',
  'system/ubloxd/SConscript',
  'system/loggerd/SConscript',
@@ -486,9 +486,6 @@ struct DeviceState @0xa4d8b5af2aa492eb {
  nvmeTempC @35 :List(Float32);
  modemTempC @36 :List(Float32);
  pmicTempC @39 :List(Float32);
-  intakeTempC @46 :Float32;
-  exhaustTempC @47 :Float32;
-  caseTempC @48 :Float32;
  maxTempC @44 :Float32;  # max of other temps, used to control fan
  thermalZones @38 :List(ThermalZone);
  thermalStatus @14 :ThermalStatus;
@@ -168,18 +168,18 @@ class TestMessaging:

    # this test doesn't work with ZMQ since multiprocessing interrupts it
    if "ZMQ" not in os.environ:
-      # wait 5 socket timeouts and make sure it's still retrying
+      # wait 15 socket timeouts and make sure it's still retrying
      p = multiprocessing.Process(target=messaging.recv_one_retry, args=(sub_sock,))
      p.start()
-      time.sleep(sock_timeout*5)
+      time.sleep(sock_timeout*15)
      assert p.is_alive()
      p.terminate()

-    # wait 5 socket timeouts before sending
+    # wait 15 socket timeouts before sending
    msg = random_carstate()
-    delayed_send(sock_timeout*5, pub_sock, msg.to_bytes())
+    delayed_send(sock_timeout*15, pub_sock, msg.to_bytes())
    start_time = time.monotonic()
    recvd = messaging.recv_one_retry(sub_sock)
-    assert (time.monotonic() - start_time) >= sock_timeout*5
+    assert (time.monotonic() - start_time) >= sock_timeout*15
    assert isinstance(recvd, capnp._DynamicStructReader)
    assert_carstate(msg.carState, recvd.carState)
@@ -63,13 +63,14 @@ class TestSubMaster:
  def test_update_timeout(self):
    sock = random_sock()
    sm = messaging.SubMaster([sock,])
-    timeout = random.randrange(1000, 3000)
-    start_time = time.monotonic()
-    sm.update(timeout)
-    t = time.monotonic() - start_time
-    assert t >= timeout/1000.
-    assert t < 3
-    assert not any(sm.updated.values())
+    for _ in range(5):
+      timeout = random.randrange(1000, 5000)
+      start_time = time.monotonic()
+      sm.update(timeout)
+      t = time.monotonic() - start_time
+      assert t >= timeout/1000.
+      assert t < 5
+      assert not any(sm.updated.values())

  def test_avg_frequency_checks(self):
    for poll in (True, False):
@@ -17,5 +17,5 @@ class TestServices:

  def test_generated_header(self):
    with tempfile.NamedTemporaryFile(suffix=".h") as f:
-      ret = os.system(f"python3 {services.__file__} > {f.name} && clang++ {f.name} -std=c++11")
+      ret = os.system(f"python3 {services.__file__} > {f.name} && clang++ {f.name}")
      assert ret == 0, "generated services header is not valid C"
@@ -112,6 +112,7 @@ std::unordered_map<std::string, uint32_t> keys = {
    {"DisablePowerDown", PERSISTENT},
    {"DisableUpdates", PERSISTENT},
    {"DisengageOnAccelerator", PERSISTENT},
+    {"DmModelInitialized", CLEAR_ON_ONROAD_TRANSITION},
    {"DongleId", PERSISTENT},
    {"DoReboot", CLEAR_ON_MANAGER_START},
    {"DoShutdown", CLEAR_ON_MANAGER_START},
@@ -36,16 +36,11 @@ class UnknownKeyName(Exception):

 cdef class Params:
  cdef c_Params* p
-  cdef str d

  def __cinit__(self, d=""):
    cdef string path = <string>d.encode()
    with nogil:
      self.p = new c_Params(path)
-    self.d = d
-
-  def __reduce__(self):
-    return (type(self), (self.d,))

  def __dealloc__(self):
    del self.p
@@ -59,13 +59,15 @@ class PIDController:
    if override:
      self.i -= self.i_unwind_rate * float(np.sign(self.i))
    else:
-      if not freeze_integrator:
-        self.i = self.i + error * self.k_i * self.i_rate
+      i = self.i + error * self.k_i * self.i_rate
+      control = self.p + i + self.d + self.f

-        # Clip i to prevent exceeding control limits
-        control_no_i = self.p + self.d + self.f
-        control_no_i = clip(control_no_i, self.neg_limit, self.pos_limit)
-        self.i = clip(self.i, self.neg_limit - control_no_i, self.pos_limit - control_no_i)
+      # Update when changing i will move the control away from the limits
+      # or when i will move towards the sign of the error
+      if ((error >= 0 and (control <= self.pos_limit or i < 0.0)) or
+          (error <= 0 and (control >= self.neg_limit or i > 0.0))) and \
+         not freeze_integrator:
+        self.i = i

    control = self.p + self.i + self.d + self.f

@@ -13,7 +13,7 @@ public:
    if (prefix.empty()) {
      prefix = util::random_string(15);
    }
-    msgq_path = Path::shm_path() + "/" + prefix;
+    msgq_path = "/dev/shm/" + prefix;
    bool ret = util::create_directories(msgq_path, 0777);
    assert(ret);
    setenv("OPENPILOT_PREFIX", prefix.c_str(), 1);
@@ -11,7 +11,7 @@ from openpilot.system.hardware.hw import DEFAULT_DOWNLOAD_CACHE_ROOT
 class OpenpilotPrefix:
  def __init__(self, prefix: str = None, clean_dirs_on_exit: bool = True, shared_download_cache: bool = False):
    self.prefix = prefix if prefix else str(uuid.uuid4().hex[0:15])
-    self.msgq_path = os.path.join(Paths.shm_path(), self.prefix)
+    self.msgq_path = os.path.join('/dev/shm', self.prefix)
    self.clean_dirs_on_exit = clean_dirs_on_exit
    self.shared_download_cache = shared_download_cache

@@ -48,13 +48,13 @@ class Ratekeeper:
  def __init__(self, rate: float, print_delay_threshold: float | None = 0.0) -> None:
    """Rate in Hz for ratekeeping. print_delay_threshold must be nonnegative."""
    self._interval = 1. / rate
+    self._next_frame_time = time.monotonic() + self._interval
    self._print_delay_threshold = print_delay_threshold
    self._frame = 0
    self._remaining = 0.0
    self._process_name = getproctitle()
    self._dts = deque([self._interval], maxlen=100)
-    self._last_monitor_time = -1.
-    self._next_frame_time = -1.
+    self._last_monitor_time = time.monotonic()

  @property
  def frame(self) -> int:
@@ -79,10 +79,6 @@ class Ratekeeper:

  # Monitors the cumulative lag, but does not enforce a rate
  def monitor_time(self) -> bool:
-    if self._last_monitor_time < 0:
-      self._next_frame_time = time.monotonic() + self._interval
-      self._last_monitor_time = time.monotonic()
-
    prev = self._last_monitor_time
    self._last_monitor_time = time.monotonic()
    self._dts.append(self._last_monitor_time - prev)
@@ -2,9 +2,8 @@

 #include "common/watchdog.h"
 #include "common/util.h"
-#include "system/hardware/hw.h"

-const std::string watchdog_fn_prefix = Path::shm_path() + "/wd_";  // + <pid>
+const std::string watchdog_fn_prefix = "/dev/shm/wd_";  // + <pid>

 bool watchdog_kick(uint64_t ts) {
  static std::string fn = watchdog_fn_prefix + std::to_string(getpid());
@@ -30,21 +30,21 @@ A supported vehicle is one that just works when you install a comma device. All
 |comma|body|All|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|None||
 |CUPRA|Ateca 2018-23|Adaptive Cruise Control (ACC) & Lane Assist|openpilot available[<sup>1,12</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 USB-C coupler<br>- 1 VW J533 connector<br>- 1 comma 3X<br>- 1 harness box<br>- 1 long OBD-C cable<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=CUPRA&model=Ateca 2018-23">Buy Here</a></sub></details>||
 |Dodge|Durango 2020-21|Adaptive Cruise Control (ACC)|Stock|0 mph|39 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 FCA connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Dodge&model=Durango 2020-21">Buy Here</a></sub></details>||
-|Ford|Bronco Sport 2021-24|Co-Pilot360 Assist+|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 angled mount (8 degrees)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Bronco Sport 2021-24">Buy Here</a></sub></details>||
-|Ford|Escape 2020-22|Co-Pilot360 Assist+|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Escape 2020-22">Buy Here</a></sub></details>||
-|Ford|Escape Hybrid 2020-22|Co-Pilot360 Assist+|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Escape Hybrid 2020-22">Buy Here</a></sub></details>||
-|Ford|Escape Plug-in Hybrid 2020-22|Co-Pilot360 Assist+|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Escape Plug-in Hybrid 2020-22">Buy Here</a></sub></details>||
-|Ford|Explorer 2020-24|Co-Pilot360 Assist+|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Explorer 2020-24">Buy Here</a></sub></details>||
-|Ford|Explorer Hybrid 2020-24|Co-Pilot360 Assist+|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Explorer Hybrid 2020-24">Buy Here</a></sub></details>||
-|Ford|Focus 2018[<sup>3</sup>](#footnotes)|Adaptive Cruise Control with Lane Centering|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Focus 2018">Buy Here</a></sub></details>||
-|Ford|Focus Hybrid 2018[<sup>3</sup>](#footnotes)|Adaptive Cruise Control with Lane Centering|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Focus Hybrid 2018">Buy Here</a></sub></details>||
-|Ford|Kuga 2020-22|Adaptive Cruise Control with Lane Centering|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Kuga 2020-22">Buy Here</a></sub></details>||
-|Ford|Kuga Hybrid 2020-22|Adaptive Cruise Control with Lane Centering|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Kuga Hybrid 2020-22">Buy Here</a></sub></details>||
-|Ford|Kuga Plug-in Hybrid 2020-22|Adaptive Cruise Control with Lane Centering|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Kuga Plug-in Hybrid 2020-22">Buy Here</a></sub></details>||
-|Ford|Maverick 2022|LARIAT Luxury|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 angled mount (8 degrees)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Maverick 2022">Buy Here</a></sub></details>||
-|Ford|Maverick 2023-24|Co-Pilot360 Assist|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 angled mount (8 degrees)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Maverick 2023-24">Buy Here</a></sub></details>||
-|Ford|Maverick Hybrid 2022|LARIAT Luxury|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 angled mount (8 degrees)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Maverick Hybrid 2022">Buy Here</a></sub></details>||
-|Ford|Maverick Hybrid 2023-24|Co-Pilot360 Assist|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 angled mount (8 degrees)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Maverick Hybrid 2023-24">Buy Here</a></sub></details>||
+|Ford|Bronco Sport 2021-24|Co-Pilot360 Assist+|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 angled mount (8 degrees)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Bronco Sport 2021-24">Buy Here</a></sub></details>||
+|Ford|Escape 2020-22|Co-Pilot360 Assist+|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Escape 2020-22">Buy Here</a></sub></details>||
+|Ford|Escape Hybrid 2020-22|Co-Pilot360 Assist+|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Escape Hybrid 2020-22">Buy Here</a></sub></details>||
+|Ford|Escape Plug-in Hybrid 2020-22|Co-Pilot360 Assist+|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Escape Plug-in Hybrid 2020-22">Buy Here</a></sub></details>||
+|Ford|Explorer 2020-23|Co-Pilot360 Assist+|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Explorer 2020-23">Buy Here</a></sub></details>||
+|Ford|Explorer Hybrid 2020-23|Co-Pilot360 Assist+|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Explorer Hybrid 2020-23">Buy Here</a></sub></details>||
+|Ford|Focus 2018[<sup>3</sup>](#footnotes)|Adaptive Cruise Control with Lane Centering|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Focus 2018">Buy Here</a></sub></details>||
+|Ford|Focus Hybrid 2018[<sup>3</sup>](#footnotes)|Adaptive Cruise Control with Lane Centering|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Focus Hybrid 2018">Buy Here</a></sub></details>||
+|Ford|Kuga 2020-22|Adaptive Cruise Control with Lane Centering|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Kuga 2020-22">Buy Here</a></sub></details>||
+|Ford|Kuga Hybrid 2020-22|Adaptive Cruise Control with Lane Centering|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Kuga Hybrid 2020-22">Buy Here</a></sub></details>||
+|Ford|Kuga Plug-in Hybrid 2020-22|Adaptive Cruise Control with Lane Centering|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Kuga Plug-in Hybrid 2020-22">Buy Here</a></sub></details>||
+|Ford|Maverick 2022|LARIAT Luxury|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 angled mount (8 degrees)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Maverick 2022">Buy Here</a></sub></details>||
+|Ford|Maverick 2023-24|Co-Pilot360 Assist|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 angled mount (8 degrees)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Maverick 2023-24">Buy Here</a></sub></details>||
+|Ford|Maverick Hybrid 2022|LARIAT Luxury|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 angled mount (8 degrees)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Maverick Hybrid 2022">Buy Here</a></sub></details>||
+|Ford|Maverick Hybrid 2023-24|Co-Pilot360 Assist|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 angled mount (8 degrees)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Ford&model=Maverick Hybrid 2023-24">Buy Here</a></sub></details>||
 |Genesis|G70 2018|All|Stock|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai F connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Genesis&model=G70 2018">Buy Here</a></sub></details>||
 |Genesis|G70 2019-21|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai F connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Genesis&model=G70 2019-21">Buy Here</a></sub></details>||
 |Genesis|G70 2022-23|All|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Hyundai L connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Genesis&model=G70 2022-23">Buy Here</a></sub></details>||
@@ -167,7 +167,7 @@ A supported vehicle is one that just works when you install a comma device. All
 |Lexus|ES 2017-18|All|openpilot available[<sup>2</sup>](#footnotes)|19 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 RJ45 cable (7 ft)<br>- 1 Toyota A connector<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Lexus&model=ES 2017-18">Buy Here</a></sub></details>||
 |Lexus|ES 2019-24|All|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 RJ45 cable (7 ft)<br>- 1 Toyota A connector<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Lexus&model=ES 2019-24">Buy Here</a></sub></details>||
 |Lexus|ES Hybrid 2017-18|All|openpilot available[<sup>2</sup>](#footnotes)|19 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 RJ45 cable (7 ft)<br>- 1 Toyota A connector<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Lexus&model=ES Hybrid 2017-18">Buy Here</a></sub></details>||
-|Lexus|ES Hybrid 2019-25|All|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 RJ45 cable (7 ft)<br>- 1 Toyota A connector<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Lexus&model=ES Hybrid 2019-25">Buy Here</a></sub></details>|<a href="https://youtu.be/BZ29osRVJeg?t=12" target="_blank"><img height="18px" src="assets/icon-youtube.svg"></img></a>|
+|Lexus|ES Hybrid 2019-24|All|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 RJ45 cable (7 ft)<br>- 1 Toyota A connector<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Lexus&model=ES Hybrid 2019-24">Buy Here</a></sub></details>|<a href="https://youtu.be/BZ29osRVJeg?t=12" target="_blank"><img height="18px" src="assets/icon-youtube.svg"></img></a>|
 |Lexus|GS F 2016|All|Stock|19 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 RJ45 cable (7 ft)<br>- 1 Toyota A connector<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Lexus&model=GS F 2016">Buy Here</a></sub></details>||
 |Lexus|IS 2017-19|All|Stock|19 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 RJ45 cable (7 ft)<br>- 1 Toyota A connector<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Lexus&model=IS 2017-19">Buy Here</a></sub></details>||
 |Lexus|IS 2022-23|All|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 RJ45 cable (7 ft)<br>- 1 Toyota A connector<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Lexus&model=IS 2022-23">Buy Here</a></sub></details>||
@@ -184,11 +184,11 @@ A supported vehicle is one that just works when you install a comma device. All
 |Lexus|RX Hybrid 2017-19|All|openpilot available[<sup>2</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 RJ45 cable (7 ft)<br>- 1 Toyota A connector<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Lexus&model=RX Hybrid 2017-19">Buy Here</a></sub></details>||
 |Lexus|RX Hybrid 2020-22|All|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 RJ45 cable (7 ft)<br>- 1 Toyota A connector<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Lexus&model=RX Hybrid 2020-22">Buy Here</a></sub></details>||
 |Lexus|UX Hybrid 2019-23|All|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 RJ45 cable (7 ft)<br>- 1 Toyota A connector<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Lexus&model=UX Hybrid 2019-23">Buy Here</a></sub></details>||
-|Lincoln|Aviator 2020-24|Co-Pilot360 Plus|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Lincoln&model=Aviator 2020-24">Buy Here</a></sub></details>||
-|Lincoln|Aviator Plug-in Hybrid 2020-24|Co-Pilot360 Plus|openpilot|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Lincoln&model=Aviator Plug-in Hybrid 2020-24">Buy Here</a></sub></details>||
+|Lincoln|Aviator 2020-24|Co-Pilot360 Plus|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Lincoln&model=Aviator 2020-24">Buy Here</a></sub></details>||
+|Lincoln|Aviator Plug-in Hybrid 2020-24|Co-Pilot360 Plus|openpilot available[<sup>1</sup>](#footnotes)|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Ford Q3 connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Lincoln&model=Aviator Plug-in Hybrid 2020-24">Buy Here</a></sub></details>||
 |MAN|eTGE 2020-24|Adaptive Cruise Control (ACC) & Lane Assist|openpilot available[<sup>1,12</sup>](#footnotes)|0 mph|31 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 USB-C coupler<br>- 1 VW J533 connector<br>- 1 angled mount (8 degrees)<br>- 1 comma 3X<br>- 1 harness box<br>- 1 long OBD-C cable<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=MAN&model=eTGE 2020-24">Buy Here</a></sub></details>|<a href="https://youtu.be/4100gLeabmo" target="_blank"><img height="18px" src="assets/icon-youtube.svg"></img></a>|
 |MAN|TGE 2017-24|Adaptive Cruise Control (ACC) & Lane Assist|openpilot available[<sup>1,12</sup>](#footnotes)|0 mph|31 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 USB-C coupler<br>- 1 VW J533 connector<br>- 1 angled mount (8 degrees)<br>- 1 comma 3X<br>- 1 harness box<br>- 1 long OBD-C cable<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=MAN&model=TGE 2017-24">Buy Here</a></sub></details>|<a href="https://youtu.be/4100gLeabmo" target="_blank"><img height="18px" src="assets/icon-youtube.svg"></img></a>|
-|Mazda|CX-5 2022-25|All|Stock|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Mazda connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Mazda&model=CX-5 2022-25">Buy Here</a></sub></details>||
+|Mazda|CX-5 2022-24|All|Stock|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Mazda connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Mazda&model=CX-5 2022-24">Buy Here</a></sub></details>||
 |Mazda|CX-9 2021-23|All|Stock|0 mph|28 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-full.svg)](##)|<details><summary>Parts</summary><sub>- 1 Mazda connector<br>- 1 RJ45 cable (7 ft)<br>- 1 comma 3X<br>- 1 comma power v2<br>- 1 harness box<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Mazda&model=CX-9 2021-23">Buy Here</a></sub></details>|<a href="https://youtu.be/dA3duO4a0O4" target="_blank"><img height="18px" src="assets/icon-youtube.svg"></img></a>|
 |Nissan|Altima 2019-20|ProPILOT Assist|Stock|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 Nissan B connector<br>- 1 RJ45 cable (7 ft)<br>- 1 USB-C coupler<br>- 1 comma 3X<br>- 1 harness box<br>- 1 long OBD-C cable<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Nissan&model=Altima 2019-20">Buy Here</a></sub></details>||
 |Nissan|Leaf 2018-23|ProPILOT Assist|Stock|0 mph|0 mph|[![star](assets/icon-star-full.svg)](##)|[![star](assets/icon-star-empty.svg)](##)|<details><summary>Parts</summary><sub>- 1 Nissan A connector<br>- 1 RJ45 cable (7 ft)<br>- 1 USB-C coupler<br>- 1 comma 3X<br>- 1 harness box<br>- 1 long OBD-C cable<br>- 1 mount<br>- 1 right angle OBD-C cable (1.5 ft)<br><a href="https://comma.ai/shop/comma-3x.html?make=Nissan&model=Leaf 2018-23">Buy Here</a></sub></details>|<a href="https://youtu.be/vaMbtAh_0cY" target="_blank"><img height="18px" src="assets/icon-youtube.svg"></img></a>|
@@ -1,9 +1,8 @@
 # openpilot glossary

-* **onroad**: openpilot's system state while ignition is on
-* **offroad**: openpilot's system state while ignition is off
-* **route**: a route is a recording of an onroad session
+* **route**:
 * **segment**: routes are split into one minute chunks called segments.
-* **comma connect**: the web viewer for all your routes; check it out at [connect.comma.ai](https://connect.comma.ai).
-* **panda**: this is the secondary processor on the device that implements the functional safety and directly talks to the car over CAN. See the [panda repo](https://github.com/commaai/panda).
-* **comma 3X**: the latest hardware by comma.ai for running openpilot. more info at [comma.ai/shop](https://comma.ai/shop).
+* **panda**: this is . See the repo.
+* **onroad**:
+* **offroad**:
+* **comma 3X**:
@@ -11,9 +11,6 @@ On the comma three, the serial console is exposed through a UART-to-USB chip, an

 On the comma 3X, the serial console is accessible through the [panda](https://github.com/commaai/panda) using the `panda/tests/som_debug.sh` script.

-  * Username: `comma`
-  * Password: `comma`
-
 ## SSH

 In order to SSH into your device, you'll need a GitHub account with SSH keys. See this [GitHub article](https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh) for getting your account setup with SSH keys.
@@ -7,7 +7,7 @@ export OPENBLAS_NUM_THREADS=1
 export VECLIB_MAXIMUM_THREADS=1

 if [ -z "$AGNOS_VERSION" ]; then
-  export AGNOS_VERSION="11.3"
+  export AGNOS_VERSION="11.2"
 fi

 export STAGING_ROOT="/data/safe_staging"
@@ -18,12 +18,10 @@ nav:
  - How-to:
    - Turn the speed blue: how-to/turn-the-speed-blue.md
    - Connect to a comma 3/3X: how-to/connect-to-comma.md
-    # - Make your first pull request: how-to/make-first-pr.md
    #- Replay a drive: how-to/replay-a-drive.md
  - Concepts:
    - Logs: concepts/logs.md
    - Safety: concepts/safety.md
-    - Glossary: concepts/glossary.md
  - Car Porting:
    - What is a car port?: car-porting/what-is-a-car-port.md
    - Porting a car brand: car-porting/brand-port.md
@@ -31,9 +31,6 @@ dependencies = [
  # body / webrtcd
  "aiohttp",
  "aiortc",
-  # aiortc does not put an upper bound on pyopenssl and is now incompatible
-  # with the latest release
-  "pyopenssl < 24.3.0",
  "pyaudio",

  # panda
@@ -101,6 +98,7 @@ dev = [
  "azure-identity",
  "azure-storage-blob",
  "dictdiffer",
+  "flaky",
  "lru-dict",
  "matplotlib",
  "parameterized >=0.8, <0.9",
@@ -241,7 +239,6 @@ exclude = [
  "cereal",
  "panda",
  "opendbc",
-  "opendbc_repo",
  "rednose_repo",
  "tinygrad_repo",
  "teleoprtc",
@@ -1,12 +1,7 @@
 #!/usr/bin/env bash

 while read hash submodule ref; do
-  if [ "$submodule" = "tinygrad_repo" ]; then
-    echo "Skipping $submodule"
-    continue
-  fi
-
-  git -C $submodule fetch --depth 100 origin master
+  git -C $submodule fetch --depth 4000 origin master
  git -C $submodule branch -r --contains $hash | grep "origin/master"
  if [ "$?" -eq 0 ]; then
    echo "$submodule ok"
@@ -32,6 +32,7 @@ blacklist = [

  ".git/",
  ".github/",
+  ".devcontainer/",
  "Darwin/",
  ".vscode",

@@ -37,7 +37,7 @@ for f in sorted(pyf):
  lns = len(src.split("\n"))
  tree = ast.parse(src)
  Analyzer().visit(tree)
-  print(f"{lns:5d} {f} {xbit}")
+  print("%5d %s %s" % (lns, f, xbit))
  if 'test' in f:
    testlns += lns
  elif f.startswith(('tools/', 'scripts/', 'selfdrive/debug')):
@@ -47,8 +47,8 @@ for f in sorted(pyf):
  else:
    tlns += lns

-print(f"{tlns} lines of openpilot python")
-print(f"{carlns} lines of car ports")
-print(f"{scriptlns} lines of tools/scripts/debug")
-print(f"{testlns} lines of tests")
+print("%d lines of openpilot python" % tlns)
+print("%d lines of car ports" % carlns)
+print("%d lines of tools/scripts/debug" % scriptlns)
+print("%d lines of tests" % testlns)
 #print(sorted(list(imps)))
@@ -1,10 +0,0 @@
-#!/usr/bin/env bash
-
-FAIL=0
-
-if grep -n '#include "third_party/raylib/include/raylib\.h"' $@ | grep -v '^system/ui/raylib/raylib\.h'; then
-  echo -e "Bad raylib include found! Use '#include \"system/ui/raylib/raylib.h\"' instead\n"
-  FAIL=1
-fi
-
-exit $FAIL
@@ -53,7 +53,6 @@ function run_tests() {
  run "check_shebang_scripts_are_executable" python3 -m pre_commit_hooks.check_shebang_scripts_are_executable $ALL_FILES
  run "check_shebang_format" $DIR/check_shebang_format.sh $ALL_FILES
  run "check_nomerge_comments" $DIR/check_nomerge_comments.sh $ALL_FILES
-  run "check_raylib_includes" $DIR/check_raylib_includes.sh $ALL_FILES

  if [[ -z "$FAST" ]]; then
    run "mypy" mypy $PYTHON_FILES
@@ -16,9 +16,9 @@ def waste(core):
  j = 0
  while 1:
    if (i % 100) == 0:
-      setproctitle(f"{core:3d}: {i:8d}")
+      setproctitle("%3d: %8d" % (core, i))
      lt = time.monotonic()
-      print(f"{core:3d}: {i:8d} {lt-st:f}  {j:.2f}")
+      print("%3d: %8d %f  %.2f" % (core, i, lt-st, j))
      st = lt
    i += 1
    j = np.sum(np.matmul(m1, m2))
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f7565541b4e6213221174839b9b2b67397ced0b9807ea56413989fd37325b3b6
+size 4908
@@ -17,7 +17,7 @@ class TestCarDocs:
    with open(CARS_MD_OUT) as f:
      current_cars_md = f.read()

-    assert generated_cars_md == current_cars_md, "Run selfdrive/car/docs.py to update the compatibility documentation"
+    assert generated_cars_md == current_cars_md, "Run selfdrive/opcar/docs.py to update the compatibility documentation"

  def test_docs_diff(self):
    dump_path = os.path.join(BASEDIR, "selfdrive", "car", "tests", "cars_dump")
@@ -4,7 +4,6 @@ import pytest
 import random
 import unittest # noqa: TID251
 from collections import defaultdict, Counter
-from functools import partial
 import hypothesis.strategies as st
 from hypothesis import Phase, given, settings
 from parameterized import parameterized_class
@@ -23,8 +22,7 @@ from openpilot.selfdrive.selfdrived.selfdrived import SelfdriveD
 from openpilot.selfdrive.pandad import can_capnp_to_list
 from openpilot.selfdrive.test.helpers import read_segment_list
 from openpilot.system.hardware.hw import DEFAULT_DOWNLOAD_CACHE_ROOT
-from openpilot.tools.lib.logreader import LogReader, LogsUnavailable, openpilotci_source_zst, openpilotci_source, internal_source, \
-                                          internal_source_zst, comma_api_source, auto_source
+from openpilot.tools.lib.logreader import LogReader, LogsUnavailable
 from openpilot.tools.lib.route import SegmentName

 from panda.tests.libpanda import libpanda_py
@@ -128,9 +126,7 @@ class TestCarModelBase(unittest.TestCase):
      segment_range = f"{cls.test_route.route}/{seg}"

      try:
-        source = partial(auto_source, sources=[internal_source, internal_source_zst] if len(INTERNAL_SEG_LIST) else \
-                                              [openpilotci_source_zst, openpilotci_source, comma_api_source])
-        lr = LogReader(segment_range, source=source)
+        lr = LogReader(segment_range)
        return cls.get_testing_data_from_logreader(lr)
      except (LogsUnavailable, AssertionError):
        pass
@@ -394,7 +390,7 @@ class TestCarModelBase(unittest.TestCase):
      for msg in filter(lambda m: m.src in range(64), can.can):
        to_send = libpanda_py.make_CANPacket(msg.address, msg.src % 4, msg.dat)
        ret = self.safety.safety_rx_hook(to_send)
-        self.assertEqual(1, ret, f"safety rx failed ({ret=}): {(msg.address, msg.src % 4)}")
+        self.assertEqual(1, ret, f"safety rx failed ({ret=}): {to_send}")

      # Skip first frame so CS_prev is properly initialized
      if idx == 0:
@@ -209,7 +209,7 @@ class Controls:
      self.update()
      CC, lac_log = self.state_control()
      self.publish(CC, lac_log)
-      rk.monitor_time()
+      rk.keep_time()

 def main():
  config_realtime_process(4, Priority.CTRL_HIGH)
@@ -28,7 +28,7 @@ def can_printer(bus, max_msg, addr, ascii_decode):
        x = binascii.hexlify(msgs[_addr][-1]).decode('ascii')
        freq = len(msgs[_addr]) / (time.monotonic() - start)
        if max_msg is None or _addr < max_msg:
-          dd += f"{_addr:04X}({_addr:4d})({len(msgs[_addr]):6d})({freq:3}dHz) {x.ljust(20)} {a}\n"
+          dd += "%04X(%4d)(%6d)(%3dHz) %s %s\n" % (_addr, _addr, len(msgs[_addr]), freq, x.ljust(20), a)
      print(dd)
      lp = time.monotonic()

@@ -29,7 +29,7 @@ if __name__ == '__main__':
    start_t = time.process_time_ns()
    for msg in msgs:
      can_list = can_capnp_to_list([msg])
-      for cp in tm.CI.can_parsers.values():
+      for cp in tm.CI.can_parsers:
        if cp is not None:
          cp.update_strings(can_list)
    ets.append((time.process_time_ns() - start_t) * 1e-6)
@@ -8,7 +8,6 @@ from typing import cast

 from cereal.services import SERVICE_LIST
 from openpilot.tools.lib.logreader import LogReader, ReadMode
-from openpilot.selfdrive.test.process_replay.migration import migrate_all

 if __name__ == "__main__":
  cnt_events: Counter = Counter()
@@ -21,7 +20,7 @@ if __name__ == "__main__":
  start_time = math.inf
  end_time = -math.inf
  ignition_off = None
-  for msg in migrate_all(LogReader(sys.argv[1], ReadMode.QLOG)):
+  for msg in LogReader(sys.argv[1], ReadMode.QLOG):
    t = (msg.logMonoTime - start_time) / 1e9
    end_time = max(end_time, msg.logMonoTime)
    start_time = min(start_time, msg.logMonoTime)
@@ -22,7 +22,7 @@ def get_fingerprint(lr):
          msgs[c.address] = len(c.dat)

  # show CAN fingerprint
-  fingerprint = ', '.join(f"{v[0]}: {v[1]}" for v in sorted(msgs.items()))
+  fingerprint = ', '.join("%d: %d" % v for v in sorted(msgs.items()))
  print(f"\nfound {len(msgs)} messages. CAN fingerprint:\n")
  print(fingerprint)

@@ -25,7 +25,7 @@ while True:
    if c.src % 0x80 == 0 and c.address < 0x800 and c.address not in (0x7df, 0x7e0, 0x7e8):
      msgs[c.address] = len(c.dat)

-  fingerprint = ', '.join(f"{v[0]}: {v[1]}" for v in sorted(msgs.items()))
+  fingerprint = ', '.join("%d: %d" % v for v in sorted(msgs.items()))

  print(f"number of messages {len(msgs)}:")
  print(f"fingerprint {fingerprint}")
@@ -8,7 +8,6 @@ from enum import Enum
 from collections import defaultdict

 from cereal import log, messaging
-from cereal.services import SERVICE_LIST
 from openpilot.common.transformations.orientation import rot_from_euler
 from openpilot.common.realtime import config_realtime_process
 from openpilot.common.params import Params
@@ -24,10 +23,8 @@ MIN_STD_SANITY_CHECK = 1e-5  # m or rad
 MAX_FILTER_REWIND_TIME = 0.8  # s
 MAX_SENSOR_TIME_DIFF = 0.1  # s
 YAWRATE_CROSS_ERR_CHECK_FACTOR = 30
-INPUT_INVALID_THRESHOLD = 0.5  # 0 bad inputs ignored
-TIMING_INVALID_THRESHOLD = 2.5  # 2 bad timings ignored
-INPUT_INVALID_DECAY = 0.9993  # ~10 secs to resume after exceeding allowed bad inputs by one (at 100hz)
-TIMING_INVALID_DECAY = 0.9990  # ~2 secs to resume after exceeding allowed bad timings by one (at 100hz)
+INPUT_INVALID_THRESHOLD = 0.5
+INPUT_INVALID_DECAY = 0.9993  # ~10 secs to resume after a bad input
 POSENET_STD_INITIAL_VALUE = 10.0
 POSENET_STD_HIST_HALF = 20

@@ -268,13 +265,10 @@ def main():
  estimator = LocationEstimator(DEBUG)

  filter_initialized = False
-  critcal_services = ["accelerometer", "gyroscope", "cameraOdometry"]
-  observation_timing_invalid = defaultdict(int)
+  critcal_services = ["accelerometer", "gyroscope", "liveCalibration", "cameraOdometry"]
+  observation_timing_invalid = False
  observation_input_invalid = defaultdict(int)

-  input_invalid_decay = {s: INPUT_INVALID_DECAY ** (100. / SERVICE_LIST[s].frequency) for s in critcal_services}
-  timing_invalid_decay = {s: TIMING_INVALID_DECAY ** (100. / SERVICE_LIST[s].frequency) for s in critcal_services}
-
  initial_pose = params.get("LocationFilterInitialState")
  if initial_pose is not None:
    initial_pose = json.loads(initial_pose)
@@ -288,6 +282,8 @@ def main():
    acc_msgs, gyro_msgs = (messaging.drain_sock(sock) for sock in sensor_sockets)

    if filter_initialized:
+      observation_timing_invalid = False
+
      msgs = []
      for msg in acc_msgs + gyro_msgs:
        t, valid, which, data = msg.logMonoTime, msg.valid, msg.which(), getattr(msg, msg.which())
@@ -302,23 +298,18 @@ def main():
        if valid:
          t = log_mono_time * 1e-9
          res = estimator.handle_log(t, which, msg)
-          if which not in critcal_services:
-            continue
-
          if res == HandleLogResult.TIMING_INVALID:
-            observation_timing_invalid[which] += 1
+            observation_timing_invalid = True
          elif res == HandleLogResult.INPUT_INVALID:
            observation_input_invalid[which] += 1
          else:
-            observation_input_invalid[which] *= input_invalid_decay[which]
-            observation_timing_invalid[which] *= timing_invalid_decay[which]
+            observation_input_invalid[which] *= INPUT_INVALID_DECAY
    else:
      filter_initialized = sm.all_checks() and sensor_all_checks(acc_msgs, gyro_msgs, sensor_valid, sensor_recv_time, sensor_alive, SIMULATION)

    if sm.updated["cameraOdometry"]:
      critical_service_inputs_valid = all(observation_input_invalid[s] < INPUT_INVALID_THRESHOLD for s in critcal_services)
-      critical_service_timing_valid = all(observation_timing_invalid[s] < TIMING_INVALID_THRESHOLD for s in critcal_services)
-      inputs_valid = sm.all_valid() and critical_service_inputs_valid and critical_service_timing_valid
+      inputs_valid = sm.all_valid() and critical_service_inputs_valid and not observation_timing_invalid
      sensors_valid = sensor_all_checks(acc_msgs, gyro_msgs, sensor_valid, sensor_recv_time, sensor_alive, SIMULATION)

      msg = estimator.get_msg(sensors_valid, inputs_valid, filter_initialized)
@@ -1,3 +1,4 @@
+import pytest
 import numpy as np
 from collections import defaultdict
 from enum import Enum
@@ -16,7 +17,6 @@ SELECT_COMPARE_FIELDS = {
  'sensors_flag': ['sensorsOK'],
 }
 JUNK_IDX = 100
-CONSISTENT_SPIKES_COUNT = 10


 class Scenario(Enum):
@@ -25,8 +25,6 @@ class Scenario(Enum):
  GYRO_SPIKE_MIDWAY = 'gyro_spike_midway'
  ACCEL_OFF = 'accel_off'
  ACCEL_SPIKE_MIDWAY = 'accel_spike_midway'
-  SENSOR_TIMING_SPIKE_MIDWAY = 'timing_spikes'
-  SENSOR_TIMING_CONSISTENT_SPIKES = 'timing_consistent_spikes'


 def get_select_fields_data(logs):
@@ -45,17 +43,6 @@ def get_select_fields_data(logs):
  return data


-def modify_logs_midway(logs, which, count, fn):
-  non_which = [x for x in logs if x.which() != which]
-  which = [x for x in logs if x.which() == which]
-  temps = which[len(which) // 2:len(which) // 2 + count]
-  for i, temp in enumerate(temps):
-    temp = temp.as_builder()
-    fn(temp)
-    which[len(which) // 2 + i] = temp.as_reader()
-  return sorted(non_which + which, key=lambda x: x.logMonoTime)
-
-
 def run_scenarios(scenario, logs):
  if scenario == Scenario.BASE:
    pass
@@ -64,28 +51,30 @@ def run_scenarios(scenario, logs):
    logs = sorted([x for x in logs if x.which() != 'gyroscope'], key=lambda x: x.logMonoTime)

  elif scenario == Scenario.GYRO_SPIKE_MIDWAY:
-    def gyro_spike(msg):
-      msg.gyroscope.gyroUncalibrated.v[0] += 3.0
-    logs = modify_logs_midway(logs, 'gyroscope', 1, gyro_spike)
+    non_gyro = [x for x in logs if x.which() not in 'gyroscope']
+    gyro = [x for x in logs if x.which() in 'gyroscope']
+    temp = gyro[len(gyro) // 2].as_builder()
+    temp.gyroscope.gyroUncalibrated.v[0] += 3.0
+    gyro[len(gyro) // 2] = temp.as_reader()
+    logs = sorted(non_gyro + gyro, key=lambda x: x.logMonoTime)

  elif scenario == Scenario.ACCEL_OFF:
    logs = sorted([x for x in logs if x.which() != 'accelerometer'], key=lambda x: x.logMonoTime)

  elif scenario == Scenario.ACCEL_SPIKE_MIDWAY:
-    def acc_spike(msg):
-      msg.accelerometer.acceleration.v[0] += 10.0
-    logs = modify_logs_midway(logs, 'accelerometer', 1, acc_spike)
-
-  elif scenario == Scenario.SENSOR_TIMING_SPIKE_MIDWAY or scenario == Scenario.SENSOR_TIMING_CONSISTENT_SPIKES:
-    def timing_spike(msg):
-      msg.accelerometer.timestamp -= int(0.150 * 1e9)
-    count = 1 if scenario == Scenario.SENSOR_TIMING_SPIKE_MIDWAY else CONSISTENT_SPIKES_COUNT
-    logs = modify_logs_midway(logs, 'accelerometer', count, timing_spike)
+    non_accel = [x for x in logs if x.which() not in 'accelerometer']
+    accel = [x for x in logs if x.which() in 'accelerometer']
+    temp = accel[len(accel) // 2].as_builder()
+    temp.accelerometer.acceleration.v[0] += 10.0
+    accel[len(accel) // 2] = temp.as_reader()
+    logs = sorted(non_accel + accel, key=lambda x: x.logMonoTime)

  replayed_logs = replay_process_with_name(name='locationd', lr=logs)
  return get_select_fields_data(logs), get_select_fields_data(replayed_logs)


+@pytest.mark.xdist_group("test_locationd_scenarios")
+@pytest.mark.shared_download_cache
 class TestLocationdScenarios:
  """
  Test locationd with different scenarios. In all these scenarios, we expect the following:
@@ -133,7 +122,7 @@ class TestLocationdScenarios:
    assert np.allclose(orig_data['yaw_rate'], replayed_data['yaw_rate'], atol=np.radians(0.35))
    assert np.allclose(orig_data['roll'], replayed_data['roll'], atol=np.radians(0.55))
    assert np.diff(replayed_data['inputs_flag'])[499] == -1.0
-    assert np.diff(replayed_data['inputs_flag'])[704] == 1.0
+    assert np.diff(replayed_data['inputs_flag'])[696] == 1.0

  def test_accel_off(self):
    """
@@ -157,21 +146,3 @@ class TestLocationdScenarios:
    orig_data, replayed_data = run_scenarios(Scenario.ACCEL_SPIKE_MIDWAY, self.logs)
    assert np.allclose(orig_data['yaw_rate'], replayed_data['yaw_rate'], atol=np.radians(0.35))
    assert np.allclose(orig_data['roll'], replayed_data['roll'], atol=np.radians(0.55))
-
-  def test_single_timing_spike(self):
-    """
-    Test: timing of 150ms off for the single accelerometer message in the middle of the segment
-    Expected Result: the message is ignored, and inputsOK is False for that time
-    """
-    orig_data, replayed_data = run_scenarios(Scenario.SENSOR_TIMING_SPIKE_MIDWAY, self.logs)
-    assert np.all(replayed_data['inputs_flag'] == orig_data['inputs_flag'])
-    assert np.all(replayed_data['sensors_flag'] == orig_data['sensors_flag'])
-
-  def test_consistent_timing_spikes(self):
-    """
-    Test: consistent timing spikes for N accelerometer messages in the middle of the segment
-    Expected Result: inputsOK becomes False after N of bad measurements
-    """
-    orig_data, replayed_data = run_scenarios(Scenario.SENSOR_TIMING_CONSISTENT_SPIKES, self.logs)
-    assert np.diff(replayed_data['inputs_flag'])[500] == -1.0
-    assert np.diff(replayed_data['inputs_flag'])[787] == 1.0
@@ -12,6 +12,7 @@ from cereal import messaging
 from cereal.messaging import PubMaster, SubMaster
 from msgq.visionipc import VisionIpcClient, VisionStreamType, VisionBuf
 from openpilot.common.swaglog import cloudlog
+from openpilot.common.params import Params
 from openpilot.common.realtime import set_realtime_priority
 from openpilot.selfdrive.modeld.runners import ModelRunner, Runtime
 from openpilot.selfdrive.modeld.models.commonmodel_pyx import CLContext
@@ -125,6 +126,7 @@ def main():
  cl_context = CLContext()
  model = ModelState(cl_context)
  cloudlog.warning("models loaded, dmonitoringmodeld starting")
+  Params().put_bool("DmModelInitialized", True)

  cloudlog.warning("connecting to driver stream")
  vipc_client = VisionIpcClient("camerad", VisionStreamType.VISION_STREAM_DRIVER, True, cl_context)
@@ -137,6 +139,7 @@ def main():
  pm = PubMaster(["driverStateV2"])

  calib = np.zeros(CALIB_LEN, dtype=np.float32)
+  # last = 0

  while True:
    buf = vipc_client.recv()
@@ -152,6 +155,8 @@ def main():
    t2 = time.perf_counter()

    pm.send("driverStateV2", get_driverstate_packet(model_output, vipc_client.frame_id, vipc_client.timestamp_sof, t2 - t1, gpu_execution_time))
+    # print("dmonitoring process: %.2fms, from last %.2fms\n" % (t2 - t1, t1 - last))
+    # last = t1


 if __name__ == "__main__":
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9dc64f5d1e7d6b67f1d4659a3483f03b4324b4c7b969a5ba90c4e37e62bf6fce
-size 50320584
+oid sha256:2a845fd16d6482222c574db833d2badb37ebcdf9c7d2987ab347ef63e728a146
+size 50309976
@@ -145,10 +145,6 @@ void Panda::set_can_speed_kbps(uint16_t bus, uint16_t speed) {
  handle->control_write(0xde, bus, (speed * 10));
 }

-void Panda::set_can_fd_auto(uint16_t bus, bool enabled) {
-  handle->control_write(0xe8, bus, enabled);
-}
-
 void Panda::set_data_speed_kbps(uint16_t bus, uint16_t speed) {
  handle->control_write(0xf9, bus, (speed * 10));
 }
@@ -77,7 +77,6 @@ public:
  void enable_deepsleep();
  void send_heartbeat(bool engaged);
  void set_can_speed_kbps(uint16_t bus, uint16_t speed);
-  void set_can_fd_auto(uint16_t bus, bool enabled);
  void set_data_speed_kbps(uint16_t bus, uint16_t speed);
  void set_canfd_non_iso(uint16_t bus, bool non_iso);
  void can_send(const capnp::List<cereal::CanData>::Reader &can_data_list);
@@ -67,10 +67,6 @@ Panda *connect(std::string serial="", uint32_t index=0) {
  }
  //panda->enable_deepsleep();

-  for (int i = 0; i < PANDA_BUS_CNT; i++) {
-    panda->set_can_fd_auto(i, true);
-  }
-
  if (!panda->up_to_date() && !getenv("BOARDD_SKIP_FW_CHECK")) {
    throw std::runtime_error("Panda firmware out of date. Run pandad.py to update.");
  }
@@ -310,16 +306,6 @@ void send_peripheral_state(Panda *panda, PubMaster *pm) {
    LOGW("reading hwmon took %lfms", read_time);
  }

-  // fall back to panda's voltage and current measurement
-  if (ps.getVoltage() == 0 && ps.getCurrent() == 0) {
-    auto health_opt = panda->get_state();
-    if (health_opt) {
-      health_t health = *health_opt;
-      ps.setVoltage(health.voltage_pkt);
-      ps.setCurrent(health.current_pkt);
-    }
-  }
-
  uint16_t fan_speed_rpm = panda->get_fan_speed();
  ps.setFanSpeedRpm(fan_speed_rpm);

@@ -239,7 +239,7 @@ int PandaSpiHandle::spi_transfer_retry(uint8_t endpoint, uint8_t *tx_data, uint1
        // due to full TX buffers
        nack_count += 1;
        if (nack_count > 3) {
-          SPILOG(LOGD, "NACK sleep %d", nack_count);
+          SPILOG(LOGE, "NACK sleep %d", nack_count);
          usleep(std::clamp(nack_count*10, 200, 2000));
        }
      }
@@ -418,7 +418,7 @@ fail:
    }
  }

-  if (ret >= 0) ret = -1;
+  if (ret > 0) ret = -1;
  return ret;
 }
 #endif
@@ -6,7 +6,6 @@ import cereal.messaging as messaging
 from cereal import log
 from openpilot.common.gpio import gpio_set, gpio_init
 from panda import Panda, PandaDFU, PandaProtocolMismatch
-from openpilot.common.retry import retry
 from openpilot.system.manager.process_config import managed_processes
 from openpilot.system.hardware import HARDWARE
 from openpilot.system.hardware.tici.pins import GPIO
@@ -52,7 +51,6 @@ class TestPandad:
    assert not Panda.wait_for_dfu(None, 3)
    assert not Panda.wait_for_panda(None, 3)

-  @retry(attempts=3)
  def _flash_bootstub_and_test(self, fn, expect_mismatch=False):
    self._go_to_dfu()
    pd = PandaDFU(None)
@@ -14,7 +14,7 @@ from openpilot.common.params import Params
 from openpilot.common.timeout import Timeout
 from openpilot.selfdrive.pandad import can_list_to_can_capnp
 from openpilot.system.hardware import TICI
-from openpilot.selfdrive.test.helpers import with_processes
+from openpilot.selfdrive.test.helpers import phone_only, with_processes


@retry(attempts=3)
@@ -72,6 +72,7 @@ class TestBoarddLoopback:
    os.environ['STARTED'] = '1'
    os.environ['BOARDD_LOOPBACK'] = '1'

+  @phone_only
  @with_processes(['pandad'])
  def test_loopback(self):
    num_pandas = 2 if TICI and "SINGLE_PANDA" not in os.environ else 1
@@ -7,7 +7,7 @@ import random
 import cereal.messaging as messaging
 from cereal.services import SERVICE_LIST
 from openpilot.system.hardware import HARDWARE
-from openpilot.selfdrive.test.helpers import with_processes
+from openpilot.selfdrive.test.helpers import phone_only, with_processes
 from openpilot.selfdrive.pandad.tests.test_pandad_loopback import setup_pandad, send_random_can_messages

 JUNGLE_SPAM = "JUNGLE_SPAM" in os.environ
@@ -23,6 +23,7 @@ class TestBoarddSpi:
    if not JUNGLE_SPAM:
      os.environ['BOARDD_LOOPBACK'] = '1'

+  @phone_only
  @with_processes(['pandad'])
  def test_spi_corruption(self, subtests):
    setup_pandad(1)
@@ -42,7 +42,7 @@ class TestAlerts:

    for name, e in events.items():
      if not name.endswith("DEPRECATED"):
-        fail_msg = f"{name} @{e} not in EVENTS"
+        fail_msg = "%s @%d not in EVENTS" % (name, e)
        assert e in EVENTS.keys(), fail_msg

  # ensure alert text doesn't exceed allowed width
@@ -10,6 +10,7 @@ from functools import wraps
 import cereal.messaging as messaging
 from openpilot.common.params import Params
 from openpilot.system.manager.process_config import managed_processes
+from openpilot.system.hardware import PC
 from openpilot.system.version import training_version, terms_version


@@ -28,6 +29,14 @@ def set_params_enabled():
  msg.liveCalibration.rpyCalib = [0.0, 0.0, 0.0]
  params.put("CalibrationParams", msg.to_bytes())

+def phone_only(f):
+  @wraps(f)
+  def wrap(self, *args, **kwargs):
+    if PC:
+      pytest.skip("This test is not meant to run on PC")
+    return f(self, *args, **kwargs)
+  return wrap
+
 def release_only(f):
  @wraps(f)
  def wrap(self, *args, **kwargs):
@@ -6,7 +6,6 @@ import capnp
 from cereal import messaging, car, log
 from opendbc.car.fingerprints import MIGRATION
 from opendbc.car.toyota.values import EPS_SCALE
-from opendbc.car.ford.values import CAR as FORD, FordFlags
 from openpilot.selfdrive.modeld.constants import ModelConstants
 from openpilot.selfdrive.modeld.fill_model_msg import fill_xyz_poly, fill_lane_line_meta
 from openpilot.selfdrive.test.process_replay.vision_meta import meta_from_encode_index
@@ -271,8 +270,6 @@ def migrate_pandaStates(msgs):
    "TOYOTA_RAV4": EPS_SCALE["TOYOTA_RAV4"] | Panda.FLAG_TOYOTA_ALT_BRAKE,
    "KIA_EV6": Panda.FLAG_HYUNDAI_EV_GAS | Panda.FLAG_HYUNDAI_CANFD_HDA2,
  }
-  # TODO: get new Ford route
-  safety_param_migration |= {car: Panda.FLAG_FORD_LONG_CONTROL for car in (set(FORD) - FORD.with_flags(FordFlags.CANFD))}

  # Migrate safety param base on carParams
  CP = next((m.carParams for _, m in msgs if m.which() == 'carParams'), None)
@@ -7,20 +7,19 @@ import tempfile
 from itertools import zip_longest

 import matplotlib.pyplot as plt
-import numpy as np

 from openpilot.common.git import get_commit
 from openpilot.system.hardware import PC
 from openpilot.tools.lib.openpilotci import get_url
 from openpilot.selfdrive.test.process_replay.compare_logs import compare_logs, format_diff
 from openpilot.selfdrive.test.process_replay.process_replay import get_process_config, replay_process
-from openpilot.tools.lib.framereader import FrameReader, NumpyFrameReader
+from openpilot.tools.lib.framereader import FrameReader
 from openpilot.tools.lib.logreader import LogReader, save_log
 from openpilot.tools.lib.github_utils import GithubUtils

 TEST_ROUTE = "2f4452b03ccb98f0|2022-12-03--13-45-30"
 SEGMENT = 6
-MAX_FRAMES = 100 if PC else 400
+MAX_FRAMES = 100 if PC else 600

 NO_MODEL = "NO_MODEL" in os.environ
 SEND_EXTRA_INPUTS = bool(int(os.getenv("SEND_EXTRA_INPUTS", "0")))
@@ -32,14 +31,14 @@ GITHUB = GithubUtils(API_TOKEN, DATA_TOKEN)


 def get_log_fn(test_route, ref="master"):
-  return f"{test_route}_model_tici_{ref}.zst"
+  return f"{test_route}_model_tici_{ref}.bz2"

 def plot(proposed, master, title, tmp):
  proposed = list(proposed)
  master = list(master)
  fig, ax = plt.subplots()
-  ax.plot(master, label='MASTER')
  ax.plot(proposed, label='PROPOSED')
+  ax.plot(master, label='MASTER')
  plt.legend(loc='best')
  plt.title(title)
  plt.savefig(f'{tmp}/{title}.png')
@@ -59,18 +58,10 @@ def generate_report(proposed, master, tmp, commit):
                     (lambda x: x.laneLines[1].y[0], "laneLines.y"),
                     (lambda x: x.meta.disengagePredictions.gasPressProbs[1], "gasPressProbs")
                    ], "modelV2")
-  DriverStateV2_Plots = zl([
-                     (lambda x: x.wheelOnRightProb, "wheelOnRightProb"),
-                     (lambda x: x.leftDriverData.faceProb, "leftDriverData.faceProb"),
-                     (lambda x: x.leftDriverData.faceOrientation[0], "leftDriverData.faceOrientation0"),
-                     (lambda x: x.leftDriverData.leftBlinkProb, "leftDriverData.leftBlinkProb"),
-                     (lambda x: x.leftDriverData.notReadyProb[0], "leftDriverData.notReadyProb0"),
-                     (lambda x: x.rightDriverData.faceProb, "rightDriverData.faceProb"),
-                    ], "driverStateV2")

  return [plot(map(v[0], get_event(proposed, event)), \
               map(v[0], get_event(master, event)), f"{v[1]}_{commit[:7]}", tmp) \
-               for v,event in ([*ModelV2_Plots] + [*DriverStateV2_Plots])]
+               for v,event in [*ModelV2_Plots]]

 def create_table(title, files, link, open_table=False):
  if not files:
@@ -152,44 +143,21 @@ def model_replay(lr, frs):
  dmonitoringmodeld = get_process_config("dmonitoringmodeld")

  modeld_msgs = replay_process(modeld, modeld_logs, frs)
-  if isinstance(frs['roadCameraState'], NumpyFrameReader):
-    del frs['roadCameraState'].frames
-    del frs['wideRoadCameraState'].frames
  dmonitoringmodeld_msgs = replay_process(dmonitoringmodeld, dmodeld_logs, frs)
  return modeld_msgs + dmonitoringmodeld_msgs


-def get_frames():
-  regen_cache = "--regen-cache" in sys.argv
-  cache = "--cache" in sys.argv or not PC or regen_cache
-  videos = ('fcamera.hevc', 'dcamera.hevc', 'ecamera.hevc')
-  cams = ('roadCameraState', 'driverCameraState', 'wideRoadCameraState')
-
-  if cache:
-    frames_cache = '/tmp/model_replay_cache' if PC else '/data/model_replay_cache'
-    os.makedirs(frames_cache, exist_ok=True)
-
-    cache_size = 200
-    for v in videos:
-      if not all(os.path.isfile(f'{frames_cache}/{TEST_ROUTE}_{v}_{i}.npy') for i in range(MAX_FRAMES//cache_size)) or regen_cache:
-        f = FrameReader(get_url(TEST_ROUTE, SEGMENT, v)).get(0, MAX_FRAMES + 1, pix_fmt="nv12")
-        print(f'Caching {v}...')
-        for i in range(MAX_FRAMES//cache_size):
-          np.save(f'{frames_cache}/{TEST_ROUTE}_{v}_{i}', f[(i * cache_size) + 1:((i + 1) * cache_size) + 1])
-        del f
-
-    return {c : NumpyFrameReader(f"{frames_cache}/{TEST_ROUTE}_{v}", 1928, 1208, cache_size) for c,v in zip(cams, videos, strict=True)}
-  else:
-    return {c : FrameReader(get_url(TEST_ROUTE, SEGMENT, v), readahead=True) for c,v in zip(cams, videos, strict=True)}
-
-
 if __name__ == "__main__":
  update = "--update" in sys.argv or (os.getenv("GIT_BRANCH", "") == 'master')
  replay_dir = os.path.dirname(os.path.abspath(__file__))

  # load logs
-  lr = list(LogReader(get_url(TEST_ROUTE, SEGMENT, "rlog.zst")))
-  frs = get_frames()
+  lr = list(LogReader(get_url(TEST_ROUTE, SEGMENT, "rlog.bz2")))
+  frs = {
+    'roadCameraState': FrameReader(get_url(TEST_ROUTE, SEGMENT, "fcamera.hevc"), readahead=True),
+    'driverCameraState': FrameReader(get_url(TEST_ROUTE, SEGMENT, "dcamera.hevc"), readahead=True),
+    'wideRoadCameraState': FrameReader(get_url(TEST_ROUTE, SEGMENT, "ecamera.hevc"), readahead=True)
+  }

  log_msgs = []
  # run replays
@@ -5,13 +5,13 @@ import copy
 import json
 import heapq
 import signal
+import platform
 from collections import Counter, OrderedDict
 from dataclasses import dataclass, field
 from typing import Any
 from collections.abc import Callable, Iterable
 from tqdm import tqdm
 import capnp
-from openpilot.system.hardware.hw import Paths

 import cereal.messaging as messaging
 from cereal import car
@@ -780,7 +780,8 @@ def generate_params_config(lr=None, CP=None, fingerprint=None, custom_params=Non

 def generate_environ_config(CP=None, fingerprint=None, log_dir=None) -> dict[str, Any]:
  environ_dict = {}
-  environ_dict["PARAMS_ROOT"] = f"{Paths.shm_path()}/params"
+  if platform.system() != "Darwin":
+    environ_dict["PARAMS_ROOT"] = "/dev/shm/params"
  if log_dir is not None:
    environ_dict["LOG_ROOT"] = log_dir

@@ -1 +1 @@
-255ceb08c75bc85379da5ec247e612be3716fb43
+992ac80ef848afb85562ca24b1c5a3d410aacd05
@@ -1,5 +1,4 @@
 import copy
-import os
 from hypothesis import given, HealthCheck, Phase, settings
 import hypothesis.strategies as st
 from parameterized import parameterized
@@ -15,15 +14,13 @@ import openpilot.selfdrive.test.process_replay.process_replay as pr
 NOT_TESTED = ['selfdrived', 'controlsd', 'card', 'plannerd', 'calibrationd', 'dmonitoringd', 'paramsd', 'dmonitoringmodeld', 'modeld']

 TEST_CASES = [(cfg.proc_name, copy.deepcopy(cfg)) for cfg in pr.CONFIGS if cfg.proc_name not in NOT_TESTED]
-MAX_EXAMPLES = int(os.environ.get("MAX_EXAMPLES", "10"))

 class TestFuzzProcesses:

  # TODO: make this faster and increase examples
  @parameterized.expand(TEST_CASES)
  @given(st.data())
-  @settings(phases=[Phase.generate, Phase.target], max_examples=MAX_EXAMPLES, deadline=1000,
-            suppress_health_check=[HealthCheck.too_slow, HealthCheck.data_too_large])
+  @settings(phases=[Phase.generate, Phase.target], max_examples=10, deadline=1000, suppress_health_check=[HealthCheck.too_slow, HealthCheck.data_too_large])
  def test_fuzz_process(self, proc_name, cfg, data):
    msgs = FuzzyGenerator.get_random_event_msg(data.draw, events=cfg.pubs, real_floats=True)
    lr = [log.Event.new_message(**m).as_reader() for m in msgs]
@@ -56,7 +56,7 @@ segments = [
  ("NISSAN", "regen58464878D07|2024-08-30--03-15-31--0"),
  ("VOLKSWAGEN", "regenED976DEB757|2024-08-30--03-18-02--0"),
  ("MAZDA", "regenACF84CCF482|2024-08-30--03-21-55--0"),
-  ("FORD", "regen756F8230C21|2024-11-07--00-08-24--0"),
+  ("FORD", "regen6ECC59A6307|2024-08-30--03-25-42--0"),
 ]

 # dashcamOnly makes don't need to be tested until a full port is done
@@ -193,10 +193,6 @@ if __name__ == "__main__":
        if cfg.proc_name not in tested_procs:
          continue

-        # to speed things up, we only test all segments on card
-        if cfg.proc_name != 'card' and car_brand not in ('HYUNDAI', 'TOYOTA', 'HONDA', 'SUBARU', 'FORD'):
-          continue
-
        cur_log_fn = os.path.join(FAKEDATA, f"{segment}_{cfg.proc_name}_{cur_commit}.zst")
        if args.update_refs:  # reference logs will not exist if routes were just regenerated
          ref_log_path = get_url(*segment.rsplit("--", 1,), "rlog.zst")
@@ -86,7 +86,7 @@ safe_checkout() {
  rsync -a --delete $SOURCE_DIR $TEST_DIR
 }

-unsafe_checkout() {( set -e
+unsafe_checkout() {
  # checkout directly in test dir, leave old build products

  cd $TEST_DIR
@@ -105,7 +105,7 @@ unsafe_checkout() {( set -e

  git lfs pull
  (ulimit -n 65535 && git lfs prune)
-)}
+}

 export GIT_PACK_THREADS=8

@@ -115,13 +115,8 @@ if [ ! -d "$SOURCE_DIR" ]; then
 fi

 if [ ! -z "$UNSAFE" ]; then
-  echo "trying unsafe checkout"
-  set +e
+  echo "doing unsafe checkout"
  unsafe_checkout
-  if [[ "$?" -ne 0 ]]; then
-    safe_checkout
-  fi
-  set -e
 else
  echo "doing safe checkout"
  safe_checkout
@@ -10,8 +10,8 @@ import time
 import numpy as np
 import zstandard as zstd
 from collections import Counter, defaultdict
+from functools import cached_property
 from pathlib import Path
-from tabulate import tabulate

 from cereal import car, log
 import cereal.messaging as messaging
@@ -33,9 +33,6 @@ CPU usage budget
  should not exceed MAX_TOTAL_CPU
 """

-TEST_DURATION = 25
-LOG_OFFSET = 8
-
 MAX_TOTAL_CPU = 265.  # total for all 8 cores
 PROCS = {
  # Baseline CPU usage by process
@@ -52,28 +49,28 @@ PROCS = {
  "selfdrive.controls.radard": 2.0,
  "selfdrive.modeld.modeld": 17.0,
  "selfdrive.modeld.dmonitoringmodeld": 11.0,
-  "system.hardware.hardwared": 4.0,
+  "system.hardware.hardwared": 3.87,
  "selfdrive.locationd.calibrationd": 2.0,
  "selfdrive.locationd.torqued": 5.0,
  "selfdrive.locationd.locationd": 25.0,
  "selfdrive.ui.soundd": 3.0,
  "selfdrive.monitoring.dmonitoringd": 4.0,
-  "./proclogd": 2.0,
-  "system.logmessaged": 1.0,
+  "./proclogd": 1.54,
+  "system.logmessaged": 0.2,
  "system.tombstoned": 0,
-  "./logcatd": 1.0,
+  "./logcatd": 0,
  "system.micd": 5.0,
  "system.timed": 0,
  "selfdrive.pandad.pandad": 0,
-  "system.statsd": 1.0,
-  "system.loggerd.uploader": 15.0,
-  "system.loggerd.deleter": 1.0,
+  "system.statsd": 0.4,
+  "system.loggerd.uploader": (0.5, 15.0),
+  "system.loggerd.deleter": 0.1,
 }

 PROCS.update({
  "tici": {
-    "./pandad": 5.0,
-    "./ubloxd": 1.0,
+    "./pandad": 4.0,
+    "./ubloxd": 0.02,
    "system.ubloxd.pigeond": 6.0,
  },
  "tizi": {
@@ -101,13 +98,6 @@ TIMINGS = {
  "wideRoadCameraState": [1.5, 0.35],
 }

-LOGS_SIZE_RATE = {
-  "qlog": 0.0083,
-  "rlog": 0.1528,
-  "qcamera.ts": 0.03828,
-}
-LOGS_SIZE_RATE.update(dict.fromkeys(['ecamera.hevc', 'fcamera.hevc'], 1.2740))
-

 def cputime_total(ct):
  return ct.cpuUser + ct.cpuSystem + ct.cpuChildrenUser + ct.cpuChildrenSystem
@@ -134,11 +124,10 @@ class TestOnroad:
    if os.path.exists(Paths.log_root()):
      shutil.rmtree(Paths.log_root())

-    # start manager and run openpilot for TEST_DURATION
+    # start manager and run openpilot for a minute
    proc = None
    try:
      manager_path = os.path.join(BASEDIR, "system/manager/manager.py")
-      cls.manager_st = time.monotonic()
      proc = subprocess.Popen(["python", manager_path])

      sm = messaging.SubMaster(['carState'])
@@ -146,24 +135,26 @@ class TestOnroad:
        while sm.recv_frame['carState'] < 0:
          sm.update(1000)

+      # make sure we get at least two full segments
      route = None
      cls.segments = []
      with Timeout(300, "timed out waiting for logs"):
        while route is None:
          route = params.get("CurrentRoute", encoding="utf-8")
-          time.sleep(0.01)
+          time.sleep(0.1)

        # test car params caching
        params.put("CarParamsCache", car.CarParams().to_bytes())

-        while len(cls.segments) < 1:
+        while len(cls.segments) < 3:
          segs = set()
          if Path(Paths.log_root()).exists():
            segs = set(Path(Paths.log_root()).glob(f"{route}--*"))
          cls.segments = sorted(segs, key=lambda s: int(str(s).rsplit('--')[-1]))
-          time.sleep(0.01)
+          time.sleep(2)

-      time.sleep(TEST_DURATION)
+      # chop off last, incomplete segment
+      cls.segments = cls.segments[:-1]

    finally:
      cls.gpu_procs = {psutil.Process(int(f.name)).name() for f in pathlib.Path('/sys/devices/virtual/kgsl/kgsl/proc/').iterdir() if f.is_dir()}
@@ -175,8 +166,9 @@ class TestOnroad:

    cls.lrs = [list(LogReader(os.path.join(str(s), "rlog"))) for s in cls.segments]

-    cls.lr = list(LogReader(os.path.join(str(cls.segments[0]), "rlog")))
-    cls.log_path = cls.segments[0]
+    # use the second segment by default as it's the first full segment
+    cls.lr = list(LogReader(os.path.join(str(cls.segments[1]), "rlog")))
+    cls.log_path = cls.segments[1]

    cls.log_sizes = {}
    for f in cls.log_path.iterdir():
@@ -186,13 +178,16 @@ class TestOnroad:
        with open(f, 'rb') as ff:
          cls.log_sizes[f] = len(zstd.compress(ff.read(), LOG_COMPRESSION_LEVEL)) / 1e6

-    cls.msgs = defaultdict(list)
-    for m in cls.lr:
-      cls.msgs[m.which()].append(m)

+  @cached_property
+  def service_msgs(self):
+    msgs = defaultdict(list)
+    for m in self.lr:
+      msgs[m.which()].append(m)
+    return msgs

  def test_service_frequencies(self, subtests):
-    for s, msgs in self.msgs.items():
+    for s, msgs in self.service_msgs.items():
      if s in ('initData', 'sentinel'):
        continue

@@ -201,14 +196,10 @@ class TestOnroad:
        continue

      with subtests.test(service=s):
-        assert len(msgs) >= math.floor(SERVICE_LIST[s].frequency*int(TEST_DURATION*0.8))
-
-  def test_manager_starting_time(self):
-    st = self.msgs['managerState'][0].logMonoTime / 1e9
-    assert (st - self.manager_st) < 10, f"manager.py took {st - self.manager_st}s to publish the first 'managerState' msg"
+        assert len(msgs) >= math.floor(SERVICE_LIST[s].frequency*55)

  def test_cloudlog_size(self):
-    msgs = self.msgs['logMessage']
+    msgs = [m for m in self.lr if m.which() == 'logMessage']

    total_size = sum(len(m.as_builder().to_bytes()) for m in msgs)
    assert total_size < 3.5e5
@@ -219,10 +210,16 @@ class TestOnroad:

  def test_log_sizes(self):
    for f, sz in self.log_sizes.items():
-      rate = LOGS_SIZE_RATE[f.name]
-      minn = rate * TEST_DURATION * 0.8
-      maxx = rate * TEST_DURATION * 1.2
-      assert minn < sz <  maxx
+      if f.name == "qcamera.ts":
+        assert 2.15 < sz < 2.6
+      elif f.name == "qlog":
+        assert 0.4 < sz < 0.55
+      elif f.name == "rlog":
+        assert 5 < sz < 50
+      elif f.name.endswith('.hevc'):
+        assert 70 < sz < 80
+      else:
+        raise NotImplementedError

  def test_ui_timings(self):
    result = "\n"
@@ -230,7 +227,7 @@ class TestOnroad:
    result += "-------------- UI Draw Timing ------------------\n"
    result += "------------------------------------------------\n"

-    ts = [m.uiDebug.drawTimeMillis for m in self.msgs['uiDebug']]
+    ts = [m.uiDebug.drawTimeMillis for m in self.service_msgs['uiDebug']]
    result += f"min  {min(ts):.2f}ms\n"
    result += f"max  {max(ts):.2f}ms\n"
    result += f"std  {np.std(ts):.2f}ms\n"
@@ -247,44 +244,53 @@ class TestOnroad:
    assert len(veryslow) < 5, f"Too many slow frame draw times: {veryslow}"

  def test_cpu_usage(self, subtests):
-    print("\n------------------------------------------------")
-    print("------------------ CPU Usage -------------------")
-    print("------------------------------------------------")
+    result = "\n"
+    result += "------------------------------------------------\n"
+    result += "------------------ CPU Usage -------------------\n"
+    result += "------------------------------------------------\n"

    plogs_by_proc = defaultdict(list)
-    for pl in self.msgs['procLog']:
+    for pl in self.service_msgs['procLog']:
      for x in pl.procLog.procs:
        if len(x.cmdline) > 0:
          n = list(x.cmdline)[0]
          plogs_by_proc[n].append(x)
+    print(plogs_by_proc.keys())

    cpu_ok = True
-    dt = (self.msgs['procLog'][-1].logMonoTime - self.msgs['procLog'][0].logMonoTime) / 1e9
-    header = ['process', 'usage', 'expected', 'max allowed', 'test result']
-    rows = []
-    for proc_name, expected in PROCS.items():
+    dt = (self.service_msgs['procLog'][-1].logMonoTime - self.service_msgs['procLog'][0].logMonoTime) / 1e9
+    for proc_name, expected_cpu in PROCS.items():

-      error = ""
-      usage = 0.
+      err = ""
+      exp = "???"
+      cpu_usage = 0.
      x = plogs_by_proc[proc_name]
      if len(x) > 2:
        cpu_time = cputime_total(x[-1]) - cputime_total(x[0])
-        usage = cpu_time / dt * 100.
+        cpu_usage = cpu_time / dt * 100.

-        max_allowed = max(expected * 1.8, expected + 5.0)
-        if usage > max_allowed:
-          error = "❌ USING MORE CPU THAN EXPECTED ❌"
-          cpu_ok = False
+        if isinstance(expected_cpu, tuple):
+          exp = str(expected_cpu)
+          minn, maxx = expected_cpu
+        else:
+          exp = f"{expected_cpu:5.2f}"
+          minn = min(expected_cpu * 0.65, max(expected_cpu - 1.0, 0.0))
+          maxx = max(expected_cpu * 1.15, expected_cpu + 5.0)

+        if cpu_usage > maxx:
+          err = "using more CPU than expected"
+        elif cpu_usage < minn:
+          err = "using less CPU than expected"
      else:
-        error = "❌ NO METRICS FOUND ❌"
-        cpu_ok = False
+        err = "NO METRICS FOUND"

-      rows.append([proc_name, usage, expected, max_allowed, error or "✅"])
-    print(tabulate(rows, header, tablefmt="simple_grid", stralign="center", numalign="center", floatfmt=".2f"))
+      result += f"{proc_name.ljust(35)}  {cpu_usage:5.2f}% ({exp}%) {err}\n"
+      if len(err) > 0:
+        cpu_ok = False
+    result += "------------------------------------------------\n"

    # Ensure there's no missing procs
-    all_procs = {p.name for p in self.msgs['managerState'][0].managerState.processes if p.shouldBeRunning}
+    all_procs = {p.name for p in self.service_msgs['managerState'][0].managerState.processes if p.shouldBeRunning}
    for p in all_procs:
      with subtests.test(proc=p):
        assert any(p in pp for pp in PROCS.keys()), f"Expected CPU usage missing for {p}"
@@ -293,25 +299,21 @@ class TestOnroad:
    procs_tot = sum([(max(x) if isinstance(x, tuple) else x) for x in PROCS.values()])
    with subtests.test(name="total CPU"):
      assert procs_tot < MAX_TOTAL_CPU, "Total CPU budget exceeded"
-    print("------------------------------------------------")
-    print(f"Total allocated CPU usage is {procs_tot}%, budget is {MAX_TOTAL_CPU}%, {MAX_TOTAL_CPU-procs_tot:.1f}% left")
-    print("------------------------------------------------")
+    result +=  "------------------------------------------------\n"
+    result += f"Total allocated CPU usage is {procs_tot}%, budget is {MAX_TOTAL_CPU}%, {MAX_TOTAL_CPU-procs_tot:.1f}% left\n"
+    result +=  "------------------------------------------------\n"
+
+    print(result)

    assert cpu_ok

  def test_memory_usage(self):
-    print("\n------------------------------------------------")
-    print("--------------- Memory Usage -------------------")
-    print("------------------------------------------------")
-    offset = int(SERVICE_LIST['deviceState'].frequency * LOG_OFFSET)
-    mems = [m.deviceState.memoryUsagePercent for m in self.msgs['deviceState'][offset:]]
+    mems = [m.deviceState.memoryUsagePercent for m in self.service_msgs['deviceState']]
    print("Memory usage: ", mems)

    # check for big leaks. note that memory usage is
    # expected to go up while the MSGQ buffers fill up
-    assert np.average(mems) <= 65, "Average memory usage above 65%"
-    assert np.max(np.diff(mems)) <= 4, "Max memory increase too high"
-    assert np.average(np.diff(mems)) <= 1, "Average memory increase too high"
+    assert max(mems) - min(mems) <= 3.0

  def test_gpu_usage(self):
    assert self.gpu_procs == {"weston", "ui", "camerad", "selfdrive.modeld.modeld", "selfdrive.modeld.dmonitoringmodeld"}
@@ -322,9 +324,7 @@ class TestOnroad:
    result += "-------------- ImgProc Timing ------------------\n"
    result += "------------------------------------------------\n"

-    ts = []
-    for s in ['roadCameraState', 'driverCameraState', 'wideCameraState']:
-      ts.extend(getattr(m, s).processingTime for m in self.msgs[s])
+    ts = [getattr(m, m.which()).processingTime for m in self.lr if 'CameraState' in m.which()]
    assert min(ts) < 0.025, f"high execution time: {min(ts)}"
    result += f"execution time: min  {min(ts):.5f}s\n"
    result += f"execution time: max  {max(ts):.5f}s\n"
@@ -357,7 +357,7 @@ class TestOnroad:

    cfgs = [("longitudinalPlan", 0.05, 0.05),]
    for (s, instant_max, avg_max) in cfgs:
-      ts = [getattr(m, s).solverExecutionTime for m in self.msgs[s]]
+      ts = [getattr(m, s).solverExecutionTime for m in self.service_msgs[s]]
      assert max(ts) < instant_max, f"high '{s}' execution time: {max(ts)}"
      assert np.mean(ts) < avg_max, f"high avg '{s}' execution time: {np.mean(ts)}"
      result += f"'{s}' execution time: min  {min(ts):.5f}s\n"
@@ -377,7 +377,7 @@ class TestOnroad:
      ("driverStateV2", 0.050, 0.026),
    ]
    for (s, instant_max, avg_max) in cfgs:
-      ts = [getattr(m, s).modelExecutionTime for m in self.msgs[s]]
+      ts = [getattr(m, s).modelExecutionTime for m in self.service_msgs[s]]
      assert max(ts) < instant_max, f"high '{s}' execution time: {max(ts)}"
      assert np.mean(ts) < avg_max, f"high avg '{s}' execution time: {np.mean(ts)}"
      result += f"'{s}' execution time: min  {min(ts):.5f}s\n"
@@ -388,32 +388,33 @@ class TestOnroad:

  def test_timings(self):
    passed = True
-    print("\n------------------------------------------------")
-    print("----------------- Service Timings --------------")
-    print("------------------------------------------------")
-
-    header = ['service', 'max', 'min', 'mean', 'expected mean', 'rsd', 'max allowed rsd', 'test result']
-    rows = []
+    result = "\n"
+    result += "------------------------------------------------\n"
+    result += "----------------- Service Timings --------------\n"
+    result += "------------------------------------------------\n"
    for s, (maxmin, rsd) in TIMINGS.items():
-      offset = int(SERVICE_LIST[s].frequency * LOG_OFFSET)
-      msgs = [m.logMonoTime for m in self.msgs[s][offset:]]
+      msgs = [m.logMonoTime for m in self.service_msgs[s]]
      if not len(msgs):
        raise Exception(f"missing {s}")

      ts = np.diff(msgs) / 1e9
      dt = 1 / SERVICE_LIST[s].frequency

-      errors = []
-      if not np.allclose(np.mean(ts), dt, rtol=0.03, atol=0):
-        errors.append("❌ FAILED MEAN TIMING CHECK ❌")
-      if not np.allclose([np.max(ts), np.min(ts)], dt, rtol=maxmin, atol=0):
-        errors.append("❌ FAILED MAX/MIN TIMING CHECK ❌")
-      if (np.std(ts)/dt) > rsd:
-        errors.append("❌ FAILED RSD TIMING CHECK ❌")
-      passed = not errors and passed
-      rows.append([s, *(np.array([np.max(ts), np.min(ts), np.mean(ts), dt])*1e3), np.std(ts)/dt, rsd, "\n".join(errors) or "✅"])
+      try:
+        np.testing.assert_allclose(np.mean(ts), dt, rtol=0.03, err_msg=f"{s} - failed mean timing check")
+        np.testing.assert_allclose([np.max(ts), np.min(ts)], dt, rtol=maxmin, err_msg=f"{s} - failed max/min timing check")
+      except Exception as e:
+        result += str(e) + "\n"
+        passed = False

-    print(tabulate(rows, header, tablefmt="simple_grid", stralign="center", numalign="center", floatfmt=".2f"))
+      if np.std(ts) / dt > rsd:
+        result += f"{s} - failed RSD timing check\n"
+        passed = False
+
+      result += f"{s.ljust(40)}: {np.array([np.mean(ts), np.max(ts), np.min(ts)])*1e3}\n"
+      result += f"{''.ljust(40)}  {np.max(np.absolute([np.max(ts)/dt, np.min(ts)/dt]))} {np.std(ts)/dt}\n"
+    result += "="*67
+    print(result)
    assert passed

  @release_only
@@ -429,12 +430,11 @@ class TestOnroad:

  def test_engagable(self):
    no_entries = Counter()
-    for m in self.msgs['onroadEvents']:
+    for m in self.service_msgs['onroadEvents']:
      for evt in m.onroadEvents:
        if evt.noEntry:
          no_entries[evt.name] += 1

-    offset = int(SERVICE_LIST['selfdriveState'].frequency * LOG_OFFSET)
-    eng = [m.selfdriveState.engageable for m in self.msgs['selfdriveState'][offset:]]
+    eng = [m.selfdriveState.engageable for m in self.service_msgs['selfdriveState']]
    assert all(eng), \
           f"Not engageable for whole segment:\n- selfdriveState.engageable: {Counter(eng)}\n- No entry events: {no_entries}"
@@ -92,7 +92,7 @@ if GetOption('extras') and arch != "Darwin":
    ("openpilot", release),
    ("openpilot_test", f"{release}-staging"),
    ("openpilot_nightly", "nightly"),
-    ("openpilot_internal", "nightly-dev"),
+    ("openpilot_internal", "master"),
  ]

  cont = senv.Command(f"installer/continue_openpilot.o", f"installer/continue_openpilot.sh",
@@ -124,7 +124,6 @@ OffroadHome::OffroadHome(QWidget* parent) : QFrame(parent) {
    // left: PrimeAdWidget
    QStackedWidget *left_widget = new QStackedWidget(this);
    QVBoxLayout *left_prime_layout = new QVBoxLayout();
-    left_prime_layout->setContentsMargins(0, 0, 0, 0);
    QWidget *prime_user = new PrimeUserWidget();
    prime_user->setStyleSheet(R"(
    border-radius: 10px;
@@ -3,7 +3,6 @@
 #include "selfdrive/ui/qt/offroad/developer_panel.h"
 #include "selfdrive/ui/qt/widgets/ssh_keys.h"
 #include "selfdrive/ui/qt/widgets/controls.h"
-#include "common/util.h"

 DeveloperPanel::DeveloperPanel(SettingsWindow *parent) : ListWidget(parent) {
  // SSH keys
@@ -25,32 +24,13 @@ DeveloperPanel::DeveloperPanel(SettingsWindow *parent) : ListWidget(parent) {
  addItem(longManeuverToggle);

  // Joystick and longitudinal maneuvers should be hidden on release branches
-  is_release = params.getBool("IsReleaseBranch");
+  // also the toggles should be not available to change in onroad state
+  const bool is_release = params.getBool("IsReleaseBranch");
+  QObject::connect(uiState(), &UIState::offroadTransition, [=](bool offroad) {
+    for (auto btn : findChildren<ParamControl *>()) {
+      btn->setVisible(!is_release);
+      btn->setEnabled(offroad);
+    }
+  });

-  // Toggles should be not available to change in onroad state
-  QObject::connect(uiState(), &UIState::offroadTransition, this, &DeveloperPanel::updateToggles);
-}
-
-void DeveloperPanel::updateToggles(bool _offroad) {
-  for (auto btn : findChildren<ParamControl *>()) {
-    btn->setVisible(!is_release);
-    btn->setEnabled(_offroad);
-  }
-
-  // longManeuverToggle should not be toggleable if the car don't have longitudinal control
-  auto cp_bytes = params.get("CarParamsPersistent");
-  if (!cp_bytes.empty()) {
-    AlignedBuffer aligned_buf;
-    capnp::FlatArrayMessageReader cmsg(aligned_buf.align(cp_bytes.data(), cp_bytes.size()));
-    cereal::CarParams::Reader CP = cmsg.getRoot<cereal::CarParams>();
-    longManeuverToggle->setEnabled(hasLongitudinalControl(CP) && _offroad);
-  } else {
-    longManeuverToggle->setEnabled(false);
-  }
-
-  offroad = _offroad;
-}
-
-void DeveloperPanel::showEvent(QShowEvent *event) {
-  updateToggles(offroad);
 }
@@ -6,15 +6,9 @@ class DeveloperPanel : public ListWidget {
  Q_OBJECT
 public:
  explicit DeveloperPanel(SettingsWindow *parent);
-  void showEvent(QShowEvent *event) override;

 private:
  Params params;
  ParamControl* joystickToggle;
  ParamControl* longManeuverToggle;
-  bool is_release;
-  bool offroad;
-
-private slots:
-  void updateToggles(bool _offroad);
 };
@@ -54,7 +54,7 @@ SoftwarePanel::SoftwarePanel(QWidget* parent) : ListWidget(parent) {
  connect(targetBranchBtn, &ButtonControl::clicked, [=]() {
    auto current = params.get("GitBranch");
    QStringList branches = QString::fromStdString(params.get("UpdaterAvailableBranches")).split(",");
-    for (QString b : {current.c_str(), "devel-staging", "devel", "nightly", "nightly-dev", "master-ci", "master"}) {
+    for (QString b : {current.c_str(), "devel-staging", "devel", "nightly", "master-ci", "master"}) {
      auto i = branches.indexOf(b);
      if (i >= 0) {
        branches.removeAt(i);
@@ -25,7 +25,6 @@ void ModelRenderer::draw(QPainter &painter, const QRect &surface_rect) {
  clip_region = surface_rect.adjusted(-CLIP_MARGIN, -CLIP_MARGIN, CLIP_MARGIN, CLIP_MARGIN);
  experimental_mode = sm["selfdriveState"].getSelfdriveState().getExperimentalMode();
  longitudinal_control = sm["carParams"].getCarParams().getOpenpilotLongitudinalControl();
-  path_offset_z = sm["liveCalibration"].getLiveCalibration().getHeight()[0];

  painter.save();

@@ -56,7 +55,7 @@ void ModelRenderer::update_leads(const cereal::RadarState::Reader &radar_state,
    const auto &lead_data = (i == 0) ? radar_state.getLeadOne() : radar_state.getLeadTwo();
    if (lead_data.getStatus()) {
      float z = line.getZ()[get_path_length_idx(line, lead_data.getDRel())];
-      mapToScreen(lead_data.getDRel(), -lead_data.getYRel(), z + path_offset_z, &lead_vertices[i]);
+      mapToScreen(lead_data.getDRel(), -lead_data.getYRel(), z + 1.22, &lead_vertices[i]);
    }
  }
 }
@@ -88,7 +87,7 @@ void ModelRenderer::update_model(const cereal::ModelDataV2::Reader &model, const
    max_distance = std::clamp((float)(lead_d - fmin(lead_d * 0.35, 10.)), 0.0f, max_distance);
  }
  max_idx = get_path_length_idx(model_position, max_distance);
-  mapLineToPolygon(model_position, 0.9, path_offset_z, &track_vertices, max_idx, false);
+  mapLineToPolygon(model_position, 0.9, 1.22, &track_vertices, max_idx, false);
 }

 void ModelRenderer::drawLaneLines(QPainter &painter) {
@@ -29,7 +29,6 @@ private:
  bool prev_allow_throttle = true;
  float lane_line_probs[4] = {};
  float road_edge_stds[2] = {};
-  float path_offset_z = 1.22f;
  QPolygonF track_vertices;
  QPolygonF lane_line_vertices[4] = {};
  QPolygonF road_edge_vertices[2] = {};
@@ -262,7 +262,7 @@ class ListWidget : public QWidget {
    outer_layout.addLayout(&inner_layout);
    inner_layout.setMargin(0);
    inner_layout.setSpacing(25); // default spacing is 25
-    outer_layout.addStretch(1);
+    outer_layout.addStretch();
  }
  inline void addItem(QWidget *w) { inner_layout.addWidget(w); }
  inline void addItem(QLayout *layout) { inner_layout.addLayout(layout); }
@@ -45,11 +45,6 @@ def setup_settings_toggles(click, pm: PubMaster):
  click(278, 650)
  time.sleep(UI_DELAY)

-def setup_settings_software(click, pm: PubMaster):
-  setup_settings_device(click, pm)
-  click(278, 800)
-  time.sleep(UI_DELAY)
-
 def setup_settings_developer(click, pm: PubMaster):
  setup_settings_device(click, pm)
  click(278, 960)
@@ -122,8 +117,11 @@ def setup_body(click, pm: PubMaster):

 def setup_keyboard(click, pm: PubMaster):
  setup_settings_device(click, pm)
-  click(250, 965)
-  click(1930, 228)
+  click(250, 575)
+  click(2020, 218)
+  click(1830, 80)
+  click(2035, 808)
+  click(90, 480)

 def setup_driver_camera(click, pm: PubMaster):
  setup_settings_device(click, pm)
@@ -182,7 +180,6 @@ CASES = {
  "pair_device": setup_pair_device,
  "settings_device": setup_settings_device,
  "settings_toggles": setup_settings_toggles,
-  "settings_software": setup_settings_software,
  "settings_developer": setup_settings_developer,
  "onroad": setup_onroad,
  "onroad_disengaged": setup_onroad_disengaged,
@@ -117,11 +117,11 @@
    <name>DeveloperPanel</name>
    <message>
        <source>Joystick Debug Mode</source>
-        <translation>وضع تصحيح أخطاء عصا التحكم</translation>
+        <translation type="unfinished"></translation>
    </message>
    <message>
        <source>Longitudinal Maneuver Mode</source>
-        <translation>وضع المناورة الطولية</translation>
+        <translation type="unfinished"></translation>
    </message>
 </context>
 <context>
@@ -435,11 +435,11 @@
    </message>
    <message>
        <source>Waiting to start</source>
-        <translation>في انتظار البدء</translation>
+        <translation type="unfinished"></translation>
    </message>
    <message>
        <source>System Unresponsive</source>
-        <translation>النظام لا يستجيب</translation>
+        <translation type="unfinished"></translation>
    </message>
 </context>
 <context>
@@ -631,7 +631,7 @@ This may take up to a minute.</source>
    </message>
    <message>
        <source>Developer</source>
-        <translation>المطور</translation>
+        <translation type="unfinished"></translation>
    </message>
 </context>
 <context>
@@ -117,11 +117,11 @@
    <name>DeveloperPanel</name>
    <message>
        <source>Joystick Debug Mode</source>
-        <translation>Modo de depuración de joystick</translation>
+        <translation type="unfinished"></translation>
    </message>
    <message>
        <source>Longitudinal Maneuver Mode</source>
-        <translation>Modo de maniobra longitudinal</translation>
+        <translation type="unfinished"></translation>
    </message>
 </context>
 <context>
@@ -615,7 +615,7 @@ Esto puede tardar un minuto.</translation>
    </message>
    <message>
        <source>Developer</source>
-        <translation>Desarrollador</translation>
+        <translation type="unfinished"></translation>
    </message>
 </context>
 <context>
@@ -117,11 +117,11 @@
    <name>DeveloperPanel</name>
    <message>
        <source>Joystick Debug Mode</source>
-        <translation>조이스틱 디버그 모드</translation>
+        <translation type="unfinished"></translation>
    </message>
    <message>
        <source>Longitudinal Maneuver Mode</source>
-        <translation>롱컨 기동 모드</translation>
+        <translation type="unfinished"></translation>
    </message>
 </context>
 <context>
@@ -611,7 +611,7 @@ This may take up to a minute.</source>
    </message>
    <message>
        <source>Developer</source>
-        <translation>개발자</translation>
+        <translation type="unfinished"></translation>
    </message>
 </context>
 <context>
@@ -145,7 +145,7 @@ void Device::resetInteractiveTimeout(int timeout) {

 void Device::updateBrightness(const UIState &s) {
  float clipped_brightness = offroad_brightness;
-  if (s.scene.started && s.scene.light_sensor >= 0) {
+  if (s.scene.started && s.scene.light_sensor > 0) {
    clipped_brightness = s.scene.light_sensor;

    // CIE 1931 - https://www.photonstophotos.net/GeneralTopics/Exposure/Psychometric_Lightness_and_Gamma.htm
@@ -54,7 +54,6 @@ RETRY_DELAY = 10  # seconds
 MAX_RETRY_COUNT = 30  # Try for at most 5 minutes if upload fails immediately
 MAX_AGE = 31 * 24 * 3600  # seconds
 WS_FRAME_SIZE = 4096
-DEVICE_STATE_UPDATE_INTERVAL = 1.0  # in seconds

 NetworkType = log.DeviceState.NetworkType

@@ -100,9 +99,9 @@ send_queue: Queue[str] = queue.Queue()
 upload_queue: Queue[UploadItem] = queue.Queue()
 low_priority_send_queue: Queue[str] = queue.Queue()
 log_recv_queue: Queue[str] = queue.Queue()
+cancelled_uploads: set[str] = set()

 cur_upload_items: dict[int, UploadItem | None] = {}
-cur_upload_items_lock = threading.Lock()


 def strip_zst_extension(fn: str) -> str:
@@ -130,9 +129,8 @@ class UploadQueueCache:
  @staticmethod
  def cache(upload_queue: Queue[UploadItem]) -> None:
    try:
-      with upload_queue.mutex:
-        items = [asdict(item) for item in upload_queue.queue]
-
+      queue: list[UploadItem | None] = list(upload_queue.queue)
+      items = [asdict(i) for i in queue if i is not None and (i.id not in cancelled_uploads)]
      Params().put("AthenadUploadQueue", json.dumps(items))
    except Exception:
      cloudlog.exception("athena.UploadQueueCache.cache.exception")
@@ -199,13 +197,11 @@ def retry_upload(tid: int, end_event: threading.Event, increase_count: bool = Tr
      progress=0,
      current=False
    )
-
-    with cur_upload_items_lock:
-      upload_queue.put_nowait(item)
-      cur_upload_items[tid] = None
-
+    upload_queue.put_nowait(item)
    UploadQueueCache.cache(upload_queue)

+    cur_upload_items[tid] = None
+
    for _ in range(RETRY_DELAY):
      time.sleep(1)
      if end_event.is_set():
@@ -215,17 +211,15 @@ def retry_upload(tid: int, end_event: threading.Event, increase_count: bool = Tr
 def cb(sm, item, tid, end_event: threading.Event, sz: int, cur: int) -> None:
  # Abort transfer if connection changed to metered after starting upload
  # or if athenad is shutting down to re-connect the websocket
-  if not item.allow_cellular:
-    if (time.monotonic() - sm.recv_time['deviceState']) > DEVICE_STATE_UPDATE_INTERVAL:
-      sm.update(0)
-      if sm['deviceState'].networkMetered:
-        raise AbortTransferException
+  sm.update(0)
+  metered = sm['deviceState'].networkMetered
+  if metered and (not item.allow_cellular):
+    raise AbortTransferException

  if end_event.is_set():
    raise AbortTransferException

-  with cur_upload_items_lock:
-    cur_upload_items[tid] = replace(item, progress=cur / sz if sz else 1)
+  cur_upload_items[tid] = replace(item, progress=cur / sz if sz else 1)


 def upload_handler(end_event: threading.Event) -> None:
@@ -233,10 +227,14 @@ def upload_handler(end_event: threading.Event) -> None:
  tid = threading.get_ident()

  while not end_event.is_set():
+    cur_upload_items[tid] = None
+
    try:
-      with cur_upload_items_lock:
-        cur_upload_items[tid] = None
-        cur_upload_items[tid] = item = replace(upload_queue.get(timeout=1), current=True)
+      cur_upload_items[tid] = item = replace(upload_queue.get(timeout=1), current=True)
+
+      if item.id in cancelled_uploads:
+        cancelled_uploads.remove(item.id)
+        continue

      # Remove item if too old
      age = datetime.now() - datetime.fromtimestamp(item.created_at / 1000)
@@ -260,13 +258,13 @@ def upload_handler(end_event: threading.Event) -> None:
          sz = -1

        cloudlog.event("athena.upload_handler.upload_start", fn=fn, sz=sz, network_type=network_type, metered=metered, retry_count=item.retry_count)
+        response = _do_upload(item, partial(cb, sm, item, tid, end_event))

-        with _do_upload(item, partial(cb, sm, item, tid, end_event)) as response:
-          if response.status_code not in (200, 201, 401, 403, 412):
-            cloudlog.event("athena.upload_handler.retry", status_code=response.status_code, fn=fn, sz=sz, network_type=network_type, metered=metered)
-            retry_upload(tid, end_event)
-          else:
-            cloudlog.event("athena.upload_handler.success", fn=fn, sz=sz, network_type=network_type, metered=metered)
+        if response.status_code not in (200, 201, 401, 403, 412):
+          cloudlog.event("athena.upload_handler.retry", status_code=response.status_code, fn=fn, sz=sz, network_type=network_type, metered=metered)
+          retry_upload(tid, end_event)
+        else:
+          cloudlog.event("athena.upload_handler.success", fn=fn, sz=sz, network_type=network_type, metered=metered)

        UploadQueueCache.cache(upload_queue)
      except (requests.exceptions.Timeout, requests.exceptions.ConnectionError, requests.exceptions.SSLError):
@@ -311,16 +309,13 @@ def getMessage(service: str, timeout: int = 1000) -> dict:
    raise Exception("invalid service")

  socket = messaging.sub_sock(service, timeout=timeout)
-  try:
-    ret = messaging.recv_one(socket)
+  ret = messaging.recv_one(socket)

-    if ret is None:
-      raise TimeoutError
+  if ret is None:
+    raise TimeoutError

-    # this is because capnp._DynamicStructReader doesn't have typing information
-    return cast(dict, ret.to_dict())
-  finally:
-    del socket
+  # this is because capnp._DynamicStructReader doesn't have typing information
+  return cast(dict, ret.to_dict())


@dispatcher.add_method
@@ -415,10 +410,8 @@ def uploadFilesToUrls(files_data: list[UploadFileDict]) -> UploadFilesToUrlRespo

@dispatcher.add_method
 def listUploadQueue() -> list[UploadItemDict]:
-  with cur_upload_items_lock, upload_queue.mutex:
-    items = list(upload_queue.queue) + [item for item in cur_upload_items.values() if item is not None]
-
-  return [asdict(item) for item in items]
+  items = list(upload_queue.queue) + list(cur_upload_items.values())
+  return [asdict(i) for i in items if (i is not None) and (i.id not in cancelled_uploads)]


@dispatcher.add_method
@@ -426,14 +419,13 @@ def cancelUpload(upload_id: str | list[str]) -> dict[str, int | str]:
  if not isinstance(upload_id, list):
    upload_id = [upload_id]

-  with upload_queue.mutex:
-    remaining_items = [item for item in upload_queue.queue if item.id not in upload_id]
-    if len(remaining_items) == len(upload_queue.queue):
-      return {"success": 0, "error": "not found"}
+  uploading_ids = {item.id for item in list(upload_queue.queue)}
+  cancelled_ids = uploading_ids.intersection(upload_id)
+  if len(cancelled_ids) == 0:
+    return {"success": 0, "error": "not found"}

-    upload_queue.queue.clear()
-    upload_queue.queue.extend(remaining_items)
-    return {"success": 1}
+  cancelled_uploads.update(cancelled_ids)
+  return {"success": 1}

@dispatcher.add_method
 def setRouteViewed(route: str) -> dict[str, int | str]:
@@ -634,9 +626,8 @@ def log_handler(end_event: threading.Event) -> None:

 def stat_handler(end_event: threading.Event) -> None:
  STATS_DIR = Paths.stats_root()
-  last_scan = 0.0
-
  while not end_event.is_set():
+    last_scan = 0.
    curr_scan = time.monotonic()
    try:
      if curr_scan - last_scan > 10:
@@ -810,8 +801,6 @@ def main(exit_event: threading.Event = None):
      cur_upload_items.clear()

      handle_long_poll(ws, exit_event)
-
-      ws.close()
    except (KeyboardInterrupt, SystemExit):
      break
    except (ConnectionError, TimeoutError, WebSocketException):
@@ -78,6 +78,7 @@ class TestAthenadMethods:

    athenad.upload_queue = queue.Queue()
    athenad.cur_upload_items.clear()
+    athenad.cancelled_uploads.clear()

    for i in os.listdir(Paths.log_root()):
      p = os.path.join(Paths.log_root(), i)
@@ -239,7 +240,7 @@ class TestAthenadMethods:
  @with_upload_handler
  def test_upload_handler_retry(self, mocker, host, status, retry):
    mock_put = mocker.patch('requests.put')
-    mock_put.return_value.__enter__.return_value.status_code = status
+    mock_put.return_value.status_code = status
    fn = self._create_file('qlog.zst')
    item = athenad.UploadItem(path=fn, url=f"{host}/qlog.zst", headers={}, created_at=int(time.time()*1000), id='', allow_cellular=True)

@@ -281,10 +282,13 @@ class TestAthenadMethods:
    athenad.upload_queue.put_nowait(item)
    dispatcher["cancelUpload"](item.id)

+    assert item.id in athenad.cancelled_uploads
+
    self._wait_for_upload()
    time.sleep(0.1)

    assert athenad.upload_queue.qsize() == 0
+    assert len(athenad.cancelled_uploads) == 0

  @with_upload_handler
  def test_cancel_expiry(self):
@@ -327,7 +331,7 @@ class TestAthenadMethods:
    assert items[0] == asdict(item)
    assert not items[0]['current']

-    dispatcher["cancelUpload"](item.id)
+    athenad.cancelled_uploads.add(item.id)
    items = dispatcher["listUploadQueue"]()
    assert len(items) == 0

@@ -339,7 +343,7 @@ class TestAthenadMethods:
    athenad.upload_queue.put_nowait(item2)

    # Ensure canceled items are not persisted
-    dispatcher["cancelUpload"](item2.id)
+    athenad.cancelled_uploads.add(item2.id)

    # serialize item
    athenad.UploadQueueCache.cache(athenad.upload_queue)
@@ -65,19 +65,16 @@ void CameraBuf::init(cl_device_id device_id, cl_context context, SpectraCamera *
  const SensorInfo *sensor = cam->sensor.get();

  is_raw = cam->is_raw;
+  camera_bufs_raw = std::make_unique<VisionBuf[]>(frame_buf_count);
  frame_metadata = std::make_unique<FrameMetadata[]>(frame_buf_count);

-  // RAW frames from ISP
-  if (is_raw) {
-    camera_bufs_raw = std::make_unique<VisionBuf[]>(frame_buf_count);
-
-    const int raw_frame_size = (sensor->frame_height + sensor->extra_height) * sensor->frame_stride;
-    for (int i = 0; i < frame_buf_count; i++) {
-      camera_bufs_raw[i].allocate(raw_frame_size);
-      camera_bufs_raw[i].init_cl(device_id, context);
-    }
-    LOGD("allocated %d CL buffers", frame_buf_count);
+  // RAW + final frames from ISP
+  const int raw_frame_size = (sensor->frame_height + sensor->extra_height) * sensor->frame_stride;
+  for (int i = 0; i < frame_buf_count; i++) {
+    camera_bufs_raw[i].allocate(raw_frame_size);
+    camera_bufs_raw[i].init_cl(device_id, context);
  }
+  LOGD("allocated %d CL buffers", frame_buf_count);

  out_img_width = sensor->frame_width;
  out_img_height = sensor->hdr_offset > 0 ? (sensor->frame_height - sensor->hdr_offset) / 2 : sensor->frame_height;
@@ -86,8 +83,8 @@ void CameraBuf::init(cl_device_id device_id, cl_context context, SpectraCamera *
  // TODO: VENUS_BUFFER_SIZE should give the size, but it's too small. dependent on encoder settings?
  size_t nv12_size = (out_img_width <= 1344 ? 2900 : 2346)*cam->stride;

-  vipc_server->create_buffers_with_sizes(stream_type, VIPC_BUFFER_COUNT, out_img_width, out_img_height, nv12_size, cam->stride, cam->uv_offset);
-  LOGD("created %d YUV vipc buffers with size %dx%d", VIPC_BUFFER_COUNT, cam->stride, cam->y_height);
+  vipc_server->create_buffers_with_sizes(stream_type, YUV_BUFFER_COUNT, out_img_width, out_img_height, nv12_size, cam->stride, cam->uv_offset);
+  LOGD("created %d YUV vipc buffers with size %dx%d", YUV_BUFFER_COUNT, cam->stride, cam->y_height);

  imgproc = new ImgProc(device_id, context, this, sensor, cam->cc.camera_num, cam->stride, cam->uv_offset);
 }
@@ -117,7 +114,7 @@ bool CameraBuf::acquire(int expo_time) {
    cur_frame_data.processing_time = (millis_since_boot() - start_time) / 1000.0;
  } else {
    cur_yuv_buf = vipc_server->get_buffer(stream_type, cur_buf_idx);
-    cur_frame_data.processing_time = (double)(cur_frame_data.timestamp_end_of_isp - cur_frame_data.timestamp_eof)*1e-9;
+    cur_frame_data.processing_time = (double)(cur_frame_data.timestamp_end_of_isp - cur_frame_data.timestamp_eof)*1e-6;
  }

  VisionIpcBufExtra extra = {
@@ -8,7 +8,7 @@
 #include "common/util.h"


-const int VIPC_BUFFER_COUNT = 18;
+const int YUV_BUFFER_COUNT = 20;

 typedef struct FrameMetadata {
  uint32_t frame_id;
@@ -69,10 +69,10 @@ public:
 };

 void CameraState::init(VisionIpcServer *v, cl_device_id device_id, cl_context ctx) {
-  camera.camera_open(v, device_id, ctx);
-
  if (!camera.enabled) return;

+  camera.camera_open(v, device_id, ctx);
+
  fl_pix = camera.cc.focal_len / camera.sensor->pixel_size_mm;
  set_exposure_rect();

@@ -140,14 +140,14 @@ void CameraState::set_camera_exposure(float grey_frac) {
  // Therefore we use the target EV from 3 frames ago, the grey fraction that was just measured was the result of that control action.
  // TODO: Lower latency to 2 frames, by using the histogram outputted by the sensor we can do AE before the debayering is complete

+  const float cur_ev_ = cur_ev[camera.buf.cur_frame_data.frame_id % 3];
  const auto &sensor = camera.sensor;
-  const float cur_ev_ = cur_ev[camera.buf.cur_frame_data.frame_id % 3] * sensor->ev_scale;

  // Scale target grey between 0.1 and 0.4 depending on lighting conditions
  float new_target_grey = std::clamp(0.4 - 0.3 * log2(1.0 + sensor->target_grey_factor*cur_ev_) / log2(6000.0), 0.1, 0.4);
  float target_grey = (1.0 - k_grey) * target_grey_fraction + k_grey * new_target_grey;

-  float desired_ev = std::clamp(cur_ev_ / sensor->ev_scale * target_grey / grey_frac, sensor->min_ev, sensor->max_ev);
+  float desired_ev = std::clamp(cur_ev_ * target_grey / grey_frac, sensor->min_ev, sensor->max_ev);
  float k = (1.0 - k_ev) / 3.0;
  desired_ev = (k * cur_ev[0]) + (k * cur_ev[1]) + (k * cur_ev[2]) + (k_ev * desired_ev);

--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
DevTekVE	370c0229ae	Refactor flag handling in ESCC and introduce sunnyParams Refactored ESCC flag handling to use sunnyParams for clarity and better structure. Updated related tests and configurations to align with the new sunnyParams structure. Also, added the sunnyParams definition to car.capnp.	2024-11-10 21:26:16 +01:00
DevTekVE	0a89980a8b	Update Hyundai carcontroller for ESCC and adjust SP flags Added ESCC parameter to `create_acc_opt` call for enhanced support in Hyundai. Also realigned Sunnypilot flags to prevent conflicts with existing definitions and arranged for proper bit allocation.	2024-11-10 16:48:49 +01:00
DevTekVE	9d18dd3687	Reorder HyundaiFlagsSP bit flags to avoid conflicts. Updated the bit flags in HyundaiFlagsSP to be in reverse order to prevent conflicts with existing CarParams from OP. This ensures no overlap and maintains code clarity and functionality.	2024-11-10 15:37:43 +01:00
DevTekVE	518ee80782	bump submodule	2024-11-10 15:19:26 +01:00
DevTekVE	a36f4836c1	bump opendbc	2024-11-08 17:45:31 +01:00
DevTekVE	78ca68d3dc	Add unit tests for Hyundai ESCC and ESCC Base classes Introduced comprehensive test cases for ESCC and ESCC Base classes within the Hyundai car implementation. These tests include validating message IDs, enabled flag handling, and refreshing car state functionalities. The goal is to ensure robust and accurate behavior of the ESCC-related features.	2024-11-08 10:53:27 +01:00
DevTekVE	e62763c5ad	Remove SP_DEBUG dependency from escc as its not currently used	2024-11-08 10:42:29 +01:00
DevTekVE	8c3d2e8130	Sync up submodules for ESCC	2024-11-08 10:17:06 +01:00
DevTekVE	c05191949a	Updating submodules	2024-11-02 19:15:58 +01:00
Jason Wen	9d5ea83f5a	Add Custom MIT License (#438 )	2024-11-02 18:53:31 +01:00