Commit Graph

44 Commits

Author SHA1 Message Date
Trey Moen 71e7369174 build_system: stream build output directly into rootfs (#552)
* build_system: stream build output directly into rootfs

Pipe buildx tar output directly into the mounted rootfs instead of
loading into Docker image store and then exporting. Skips the slow
"exporting layers" step and the intermediate container.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* no

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-30 11:35:15 -07:00
Robin Reckmann 46d60babfb Use tonistiigi/binfmt for emulation to fix qemu segfaults (#468)
Use tonistiigi/binfmt for emulation
2026-01-31 14:44:44 -08:00
Maxime Desroches 41a3543df1 bump Ubuntu image version (#484)
bump
2025-08-22 17:11:44 -07:00
Maxime Desroches d8d33fa2ac increase system image size 2025-05-06 14:35:41 -07:00
Adeeb Shihadeh daa63ce943 bump base image version (#460)
* bump base image version

* hash
2025-03-03 09:07:44 -08:00
Maxime Desroches 5e5a350981 manifest with all partitions (#443)
* all

* ota

* all manif

* ab

* remove these

* hash

* userdata

* ontheflyisthebest

* on the fly

* dont need

* remove

* handle fill

* comment

* all

* all img

* rename

* read

* sparse image

* tool

* no root

* userdata

* userdata

* need this unfort

* fix

* fix

* nop

* remove

* nop

* this too

* ota check

* better for push

* fix

* again

* good now
2025-02-02 19:55:34 -08:00
Adeeb Shihadeh 13ab1819b7 Revert SquashFS (#449)
* Revert "switch to lzo, same size as gzip and boots faster"

This reverts commit 10fcd04ca5.

* Revert "Switch to gzip'd squash (#441)"

This reverts commit 2091362d56.

* Revert "Switch to squashfs for tiny rootfs! (#438)"

This reverts commit 1e16811f45.

* Revert "fix stats"

This reverts commit a590cefcde.
2025-01-28 16:36:59 -08:00
Adeeb Shihadeh 10fcd04ca5 switch to lzo, same size as gzip and boots faster 2025-01-26 17:05:54 -08:00
Adeeb Shihadeh e9a87569a2 Remove audio kernel modules (#444)
* Remove audio kernel modules

* cleanup

* lil more
2025-01-25 18:06:02 -08:00
Adeeb Shihadeh 2091362d56 Switch to gzip'd squash (#441) 2025-01-24 16:42:48 -08:00
Adeeb Shihadeh 1e16811f45 Switch to squashfs for tiny rootfs! (#438)
* try squashfs

* rw

* build squash directly

* oops

* cleanup
2025-01-24 15:41:49 -08:00
Adeeb Shihadeh 5d83b2b8b2 Remove wifi kernel module (#435)
* remove wifi kernel module

* bump kernel
2025-01-23 20:06:16 -08:00
Adeeb Shihadeh 10cde95b9b bump version to 11.3 2024-11-22 11:00:02 -08:00
Adeeb Shihadeh 0d74a7d47c fixes for remote docker buildx building 2024-11-16 19:50:34 -08:00
Adeeb Shihadeh 753de49f95 setup namespace remote builder (#398)
* setup namespace remote builder

* lil more

* use nsc build for now

* load
2024-10-31 14:40:05 -07:00
Adeeb Shihadeh 94fb9317ee compile bytecode for python packages (#389)
* compile bytecode for python packages

* bump up size

* lil bigger
2024-10-24 15:09:13 -07:00
Robin Reckmann 8759b3a563 Fix build on amd64 systems (#372) 2024-09-16 09:44:58 -07:00
Andrei Radulescu 147dce2e8f remove system-skip-chunks alt image (#367) 2024-09-13 07:51:18 -07:00
Andrei Radulescu 675fe11371 Smaller image size (#355)
* 3.7G

* [upload]

* 3700M

* 3800M

* 3900M

* 4000M

* [upload]

* dynamic ROOTFS_IMAGE_SIZE [upload]

* 10% instead [upload]

* 15% ?! [upload]

* revert dynamic size [upload]
2024-09-11 13:31:26 -07:00
Andrei Radulescu 13cd823813 Nicer version for CI builds (#351)
* correct GIT_HASH in CI

* nicer version

* [upload]
2024-09-06 09:48:21 -07:00
Robin Reckmann e45702b5b6 Use shasum for macOS compatibility (#336) 2024-09-03 14:00:35 -07:00
Andrei Radulescu 396d8e19a4 build_system: fail when ubuntu base is updated (#332)
* fetch latest ubuntu-base image and error on download failed

* Dockerfile ubuntu base is now an argument

* revert fetch latest ubuntu-base image

* ubuntu base checksum

* print only the link to releases when download/checksum fails

* simpler checksum
2024-09-02 11:50:58 -07:00
Robbe Derks fd9992c337 Default to buildkit (#331)
* use buildkit to skip adding context

* also build system with buildkit
2024-09-02 13:33:10 +02:00
Andrei Radulescu 915ff5eb09 update bash scripts to #!/usr/bin/env bash (#268)
* update bash shebang for better compatibility including macOS

* re-build
2024-08-31 10:34:25 -07:00
Andrei Radulescu 37a76e0950 image creation cleanup (#317)
* cleaning

* remove resize2fs shrinking

* [upload]

* updated ROOTFS_IMAGE_SIZE comment about resize2fs -M
2024-08-19 13:17:48 -07:00
Andrei Radulescu bdb9394d62 Ship non-sparse system image in OTA (#275)
* removed system-skip-chunks image

* removed sparsed system.img

* clean github workflow

* copy image to output

* fake system-skip-chunks [upload]

* sparse key must exist otherwise KeyError: 'sparse' in openpilot

* reduce image size to 5G otherwise MemoryError: Unable to allocate output buffer. in openpilot

* [upload]

* Revert "fake system-skip-chunks"

This reverts commit bcd11e7e32.

* recreate 10G image [upload]

* resize2fs

* resize2fs shrink image

* revert skip-chunks image

* cleanup

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2024-08-18 12:55:18 -07:00
Andrei Radulescu 4a924daa5a Ubuntu 24.04 (#262)
* Ubuntu 24.04 (#188)

* separate out qtwayland5 compilation

* ubuntu 22.04

* temp build system image

* remove bionic deps

* temp packages update

* update armhf packages

* back to libstdc++6

* replaced remove dqt5-default meta package with actual depss

* fix for build libqmi crashing with /usr/bin/env: 'python': No such file or directory

* mapbox-gl-native gcc11 fix

* openpilot_dependencies.sh change

* ubuntu-base 24.04 daily current

* apt install meson because PEP 668

* temp g++-11 for mapbox-gl-native

* bump capnproto to 1.0.2

* temp build only agnos-compiler-mapbox-gl-native

* Fixes for compiling mapbox-gl-native

* Fixes for compiling ffmpeg

* Build main system image again

* pyopencl from git since no 2023.1.5 release since https://github.com/inducer/pyopencl/issues/705

* Try to fix checkinstall - no success

* apt capnproto & ffmpeg since can't figure out issue with checkinstall

* Commented removed services disable

* Ubuntu 24.04 currently has arm-none-eabi/12.2.1 but just use wildcard to go to first folder

* pyclean is missing ?!

* ubuntu-latest in tests workflow

* Updated README

* Reverted internal/package_ota.py changes

* rebuilt kernel

* poetry.lock update

* Simplified build_system pipeline

* headers for capnproto

* Fix typo

* Fix for checkinstall

* Reverted apt capnproto & ffmpeg

* Some cleaning in Dockerfile

* 24.04 libwayland-client

* libffi6:armhf deb file

* Updated qtwayland

* Revert "libffi6:armhf deb file"

This reverts commit bb9cb5da95.

* libffi.so.6.0.4

* qt5-qmake qtbase5-dev qtbase5-dev-tools already in qt5-default

* Removed qtwayland old files

* checkinstall temp fix

* QStandardPaths: wrong permissions on runtime directory /var/tmp/weston, 0770 instead of 0700

* temp permissions fix makes loading build spinner show

* temp build optimization

* 24.04 new libs

* build kernel don't use tools on aarch64/arm64

* updated boot image and ko files

* cleaning

* switch to release 24.04

* wip - downgrade qt and libwayland

* polkitd-pkla package to support polkit *.pkla

* fixed issue with old qt needing libssl 1.1.1f

* remove /.dockerenv file

* more cleaning

---------

Co-authored-by: Justin Newberry <justin@comma.ai>
Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* Ubuntu 24.04 branch fixes (#235)

* bump version

* /lib64 is now UsrMerged

* latest pip poetry; poetry.lock update

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>

* revert README

* [upload]

* disable failing nvmf-autoconnect.service

* disable failing networking.service

* [upload]

* no replacement for ondemand.service

* snapd.service is in snapd package we don't install

* nfs is using rpc-pipefs-generator now, when used

* checkinstall is still not fixed so remove TODO

* updated comment about usr merged lib64

* added checkinstall bug link

* removed not used libffi.so.6 and libwayland-client.so.0

* cleaned compile-qtwayland5.sh and removed 5.15 patch

* removed unused qt-comma 5.15 deb

* cleaned up custom debs for qt and libwayland and added a README

* fixed typo [upload]

* change version

* build-libwayland-deb.sh

* build-qt-deb.sh

* cleaned README

* [upload]

* fix libdl.so qtwayland build [upload]

* rebuild debs

* update libdl path [upload]

* added missing systemd-resolved

* ping capability

* disable some systemd services

* etc/resolv.conf link should not be needed

* Revert "etc/resolv.conf link should not be needed"

This reverts commit b627615a9e.

* Revert "ping capability"

This reverts commit 812ca357cf.

* set ping capability outside of container

* disable systemd services: console-setup and sfsconfig

Co-authored-by: Robin Reckmann <robin.reckmann@gmail.com>

* disabled more systemd services

* bump version [upload]

* update version

---------

Co-authored-by: Justin Newberry <justin@comma.ai>
Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
Co-authored-by: Robin Reckmann <robin.reckmann@gmail.com>
2024-08-13 16:46:55 -07:00
Robin Reckmann 6d886f2357 Error on Dockerfile warnings (#298) 2024-08-11 15:13:18 -07:00
Robin Reckmann c017d6fba0 Fix InvalidBaseImagePlatform warning on aarch64 (#284) 2024-08-06 06:56:13 -07:00
Andrei Radulescu e5fae93540 Build kernel in docker (#247)
* build kernel in container

* remove ci deps

* cleaning

* more cleaning

* even more cleaning

* README.md update for masOS Case-sensitive required volume

* README.md more info about macOS

* README.md more info about macOS

* README.md more info about macOS - M-series vs Intel

* README.md more info about macOS - M-series vs Intel

* extract_tools should install and pull lfs no matter the ARCH

* raise attention on macOS Case-sensitive volume

* --privileged not needed in build_kernel.sh

* remove docker and git checks

* set -e

* always build image

* updated host user in container

* also works with sudo/root

* removed deps in ci since they are not needed anymore

* agnos-meta-builder

* show macOS support only if current path is not on a Case-sensitive APFS volume

* more succint

* just exit

* fixing caching issues

* cache kernel out

* cleaning

* debug out

* ccache-action no needed anymore

* debug

* rebuild

* remove out cache

* revert ccache action

* create-symlink not needed since it's building in the container

* simpler ccache key name

* cleaning

* rebuild

* git ignore .ccache

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2024-07-25 10:47:56 -07:00
robin-reckmann 81e84a3561 Fix image creation docker workflow in host ubuntu (#252)
* Fix image creation docker workflow in host ubuntu

* Fix missing user variable

* Only add new non-root users
2024-07-21 16:08:38 -07:00
Andrei Radulescu 8612508955 CI: add system build (#251)
* incremental caching

* increase timeout to 40min; remove if: cache-hits

* rm that

* simplified to have one kernel cache

* [upload] tag for upload-artifact

* [upload] test

* custom key/restore for ccache

* removed if in get commit message

* timeout to 60min

* multiline support in LAST_COMMIT_MESSAGE

* Update .github/workflows/build.yaml

---------

Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2024-07-21 16:07:32 -07:00
Andrei Radulescu d977eb6f22 Fix build system in docker on macOS (#249)
* build system in docker

* removed docker and git checks

* set -e

* no comments in Dockerfile.sparsify

* always build agnos-mount

* require Docker v23.0 and remove deprecated DOCKER_BUILDKIT=1 and DOCKER_CLI_EXPERIMENTAL=enabled

* move docker exec to helpers
2024-07-18 13:22:29 -07:00
Andrei Radulescu d44d1526a3 Fix setting network stuff in container (#245) 2024-07-16 07:01:43 -07:00
Cameron Clough b5f39a4b62 Build kernel in CI (#231)
* build workflow

* submodule example

* just agnos-kernel-sdm845

* filter

* ubuntu20

* unecessary

* cancel in progress

* upload

* try ccache...

* test ccache

* cache out dir

* v4

* cleanup

* ccache again...

* skip system for now

* try this

* allow override CC

* improve cache hit

* fix

* test if this matters

* don't append timestamp

* try again

* empty

* doesn't help

* test

* Revert "test"

This reverts commit e3a7db8bd5.

* test

* Revert "test"

This reverts commit 42e9ebaaac.

* build system again

* more space

* longer timeout...

* CI froze

* use namespace labs remote builder

* permissions

* try again

* nscloud

* permissions

* ubuntu 20

* wip

* upload kernel modules

* build_kernel arch fix

Co-authored-by: Andrei Radulescu <andi.radulescu@gmail.com>

* lfs clone

* .

* namespace doesn't support git filter

* no perms

* fix typo

* namespace ubuntu2004 doesn't have python2...

* update name

* oh

* no wget

* fix arm64 check

* mount container

Co-authored-by: Andrei Radulescu <andi.radulescu@gmail.com>

* missing deps

* switch to pyenv-action which caches

* update nscloud-checkout-action

* reduce timeout

* run network stuff in container

* test cache

* cache kernel: does this work?

* try again

* test kernel cache

* use github cache action

* test kernel cache

* try actions/checkout

* kernel: cache intermediate build outputs

* name

* test cache 1

* don't skip kernel build

* always set arch to arm64

* just kernel

* fix that

---------

Co-authored-by: Andrei Radulescu <andi.radulescu@gmail.com>
Co-authored-by: Adeeb Shihadeh <adeebshihadeh@gmail.com>
2024-07-14 11:00:59 -07:00
Andrei Radulescu 278b6946a4 Fixed incorrectly registering qemu on Apple silicon (#236)
Fixed incorrectly registering qemu on arm64
2024-07-06 08:53:44 -07:00
Greg Hogan bc20686ddf hostname with serial number (#180)
* hostname with serial number

* add systemd.hostname kernel param

* /etc/hostname is now a symlink

* probably shouldn't have moved that

* fix path

* force symlink

* update kernel load script to support passing in device name

* remove kernel param

* change hostname from tici to comma

* bump kernel
2023-11-08 10:53:28 -08:00
Adeeb Shihadeh 36fc190bc8 include simg2dontcare image in manifest (#170)
Co-authored-by: Cameron Clough <cameronjclough@gmail.com>
2023-08-21 17:47:37 +01:00
Adeeb Shihadeh 361c0a6714 disable serial-hostname, still not completely reliable 2022-11-10 12:51:51 -08:00
Adeeb Shihadeh 6c61d0740a SPI panda (#97)
* spidev python package

* remove this nvidia stuff

* just run it everytime

* udev rules
2022-11-09 21:24:56 -08:00
Adeeb Shihadeh 9d616d60d9 use buildkit for build_system.sh 2022-10-22 18:11:08 -07:00
Adeeb Shihadeh 9242f06cea cleanup kernel module handling (#90) 2022-09-23 13:05:32 -07:00
Adeeb Shihadeh db174b5c9c add /BUILD with git commit and build date 2022-05-26 16:08:52 -07:00
Adeeb Shihadeh 4e38c8f181 initial agnos-builder release 2021-09-23 19:08:32 -07:00