Implements a new cache-based pipeline that dramatically improves build times
through hierarchical caching with fork-specific caches.
Key changes:
- Add executor, commands, and job definitions for cache-based builds
- Add create_hashes job to generate cache digest files
- Add x86_blobs job for blob downloads with cache support
- Add x86_musl_cross_make job for toolchain builds with cache save
- Add x86_coreboot job per fork, each saves both modules and coreboot caches
- Add ppc64_musl_cross_make and ppc64_coreboot jobs (decoupled from single job)
- Add glossary documenting fan-in, workspace chain, cache layers
Fixes for cache invalidation issues:
- Remove .circleci/config.yml from cache key hashes (prevents cache invalidation
on CI config changes - was causing full rebuilds on every pipeline)
- Fix musl-cross-make module to auto-detect existing crossgcc using wildcard check
- Exclude .circleci/config.yml from all_modules_and_patches.sha256sums and
coreboot_musl-cross-make.sha256sums
Test results (multiple pipeline runs):
Pipeline 3789 (first run, cold cache):
- x86-musl-cross-make: 30 min
- ppc64-musl-cross-make: 16 min
- Result: Cache saved
Pipeline 3790 (second run, cache hit):
- x86-musl-cross-make: 4.5 min (6.6x faster than first run)
- ppc64-musl-cross-make: 4.5 min (3.5x faster than first run)
- Result: Beats baseline (14.5 min) by 3.2x
Pipeline 3791 (third run, cache hit):
- x86-musl-cross-make: ~6 min (27s Make Board + spin up variance)
- ppc64-musl-cross-make: 4.5 min
- Result: Still beats baseline
The wildcard fix for musl-cross-make module detects existing crossgcc from
cache and skips rebuild entirely (Make Board takes only 27s vs 26 min cold).
Add tests/circle-ci-simulation/ for local cache behavior verification:
- test_cache_hash.sh, test_musl_skip.sh, simulate_cold_cache.sh, etc.
Signed-off-by: Thierry Laurion <insurgo@riseup.net>
Previously coreboot-purism depended on coreboot-24.02.01's toolchain
to avoid duplicate toolchain builds. However, no maintained boards
actually use 24.02.01 directly (only unmaintained x230-legacy boards),
so this dependency was unnecessary complexity.
Now coreboot-purism builds its complete toolchain independently,
making it consistent with all other coreboot forks (Dasharo variants,
talos_2) which also build their own toolchains.
This simplifies the cache strategy: each fork's cache contains only
its own coreboot directory, with no hidden dependencies.
Signed-off-by: Thierry Laurion <insurgo@riseup.net>
Add coreboot.save_in_defconfig_format_backup target that runs savedefconfig
and saves the result as .config_defconfig alongside the original .config file.
Signed-off-by: Thierry Laurion <insurgo@riseup.net>
- config/coreboot-novacustom-v560tu.config: switch EC from SYSTEM76_EC
to DASHARO_EC (CONFIG_EC_DASHARO_EC=y, _UPDATE=y, _UPDATE_FILE="ec.rom"),
matching the v540tu change already in this branch
- modules/dasharo-ec: strip UNTESTED_/EOL_ board name prefixes before
the board model mapping so that boards not yet promoted (e.g.
UNTESTED_nitropad-ns50) resolve correctly without a separate entry
- modules/dasharo-ec: add board model mappings for novacustom-nv4x_adl
(ns5x_adl - nv4x_adl) and nitropad-ns50 (ns5x_adl)
- modules/dasharo-ec: drop redundant sh -c wrapper from cp recipe;
Make already invokes recipe lines via the shell
- modules/dasharo-ec: replace dd seek-based padding with
truncate --size=128KiB, as suggested in review (ec.rom is always
smaller than 128 KiB)
- modules/dasharo-ec: indent ifeq variable assignments with a tab for
readability
Signed-off-by: Filip Lewiński <filip.lewinski@3mdeb.com>
Signed-off-by: Thierry Laurion <insurgo@riseup.net>
- Remove leading tabs from ifeq variable assignments (Make parse error)
- Move post-build steps (cp, dd, touch) out of dasharo-ec_target into
a proper Make rule; _target is now only make arguments
- Set dasharo-ec_output to ec.rom (the actual artifact, not .built)
- Add -$(BOARD) suffix to dasharo-ec_base_dir and dasharo-ec_dir so
v540tu and v560tu maintain independent build trees and ec.rom files
- Add explicit rule to copy and zero-pad ec.rom to 128 KB
- Fix coreboot copy rule: use FORCE + cmp so the recipe always runs
but only updates the destination (and its mtime) when content differs,
preventing the mtime race between boards sharing coreboot_base_dir
Signed-off-by: Filip Lewiński <filip.lewinski@3mdeb.com>
Signed-off-by: Thierry Laurion <insurgo@riseup.net>
Introduce validation to ensure coreboot CBFS size matches IFD BIOS region:
- Add bin/validate_cbfs_ifd_fit.sh: New script to validate CBFS size against
Intel Flash Descriptor (IFD) BIOS region and provide space usage statistics
- Add Makefile targets: validate_cbfs_ifd and fix_cbfs_ifd for manual validation
and automatic fixes
- Integrate validation into coreboot module build process to catch misconfigs
at compile time
- Add validation to final ROM image generation step
- Add informational hint to Makefile when V=1 verbose flag is not set
This prevents shipping firmware with CBFS sizes that exceed the available
BIOS region, catching configuration errors early in the build process.
Signed-off-by: Thierry Laurion <insurgo@riseup.net>
config/busybox.config: add reboot command under busybox
initrd/bin/reboot: use 'busybox reboot -f' instead of 'echo b > /proc/sysrq-trigger'
modules/busybox: remove /sbin/reboot symlink to busybox so that our /bin/reboot is called, which calls busybox for reboot
Signed-off-by: Thierry Laurion <insurgo@riseup.net>
There was an issue where initrd_tmp_dir was used to create tools.cpio and wipe that dir, while data.cpio was subdirectory of initrd_tmp_dir.
This seperates tools.cpio temp dir from data.cpio temp dir:
- tools.cpio: initrd_tmp_dir/tools/{bin,lib}
- data.cpio: initrd_tmp_dir/data
So that when wiping happens of a subdir, this doesn't affect others:
- initrd_tmp_dir = mktemp -d
- initrd_tools_dir = initrd_tmp_dir/tools
- initrd_bin_dir = initrd_tools_dir/bin
- initrd_lib_dir = initrd_tools_dir/lib
- initrd_data_dir = initrd_tmp_dir/data
This commit:
- Use proper variable names everywhere, no more initrd_tmp_dir.
- initrd_tools_dir, initrd_lib_dir, initrd_bin_dir, initrd_data_dir
- change to busybox (initrd_tmp_dir/bin -> initrd_bin_dir) as well and review all other modules
Signed-off-by: Thierry Laurion <insurgo@riseup.net>
This is the actual default keymap the kernel loads by default (it is
different from any of the keymaps provided by kbd, although possibly
not in any way anyone would care about).
This makes loadkeys --default work.
Signed-off-by: Jonathon Hall <jonathon.hall@puri.sm>
user@localhost:~/heads$ ls -alh /home/user/heads/build/x86/qemu-coreboot-whiptail-tpm2/data.cpio
-rw-r--r-- 1 root root 2.4M Apr 23 12:33 /home/user/heads/build/x86/qemu-coreboot-whiptail-tpm2/data.cpio
user@localhost:~/heads$ sudo xz --check=crc32 --lzma2=dict=1MiB build/x86/qemu-coreboot-whiptail-tpm2/data.cpio -z -9
user@localhost:~/heads$ ls -alh build/x86/qemu-coreboot-whiptail-tpm2/data.cpio.xz
-rw-r--r-- 1 root root 158K Apr 23 12:33 build/x86/qemu-coreboot-whiptail-tpm2/data.cpio.xz
So adding all source based keymaps add 158Kb compressed to initrd.cpio.xz
TODO:
- init: if exported keymap_path and file exists: loadkeys it; otherwise ontinue as of now and defaults to default US keymap
- Configurator deitor: show filelist, loadkeys keymap, test input with user, confirm keymap selection, persist setting
- Add to oem-factory-reset (but adds another step: in default steps? in non-default? Desired workflow?)
- This is desired for a long time since OEM customize keyboard selected by end user prior of shipping: so should customize keymap under Heads and OS install.
Signed-off-by: Thierry Laurion <insurgo@riseup.net>
Thierry Laurion
Filip Lewiński
Maciej Pijanowski
Thomas Clarke
gaspar-ilom
Jonathon Hall