Potentially automating cuts to master and/or protecting develop from failing builds #272

As someone who's been using this project since early this year, I've been very grateful to be able to bring my Pi into a NixOS environment.

That being said, I've noticed a lag with the nixpkgs because I'm following main. I could follow develop, but a quick chat with Claude tells me that develop broke three times since the last cut from main in June. Ideally, I'd either want to improve the quality of the develop branch and switch to that, or automate cuts to main.

Having chatted with Claude, it seems there isn't a special process for cutting from develop:

Short answer: nothing is done to main that makes it more stable. It's an older snapshot of develop that the maintainer, nvmd, promotes by hand every few weeks. Its stability comes from time and judgment: things break on develop and get fixed before the next promotion. Yes, most of that can be automated. The CI that would catch breakages already exists, but it runs after merge instead of before.

What main actually is

• How it's promoted: nvmd opens a "Release v1.YYYYMMDD.N" PR from develop to main, writes a changelog by hand, and tags it. The last three were merged 15–70 seconds after being opened, and the formatter was the only check that ran.
• No commits of its own: apart from those release merges, main is just an older develop.
• Cadence: in 2026 a release came every 9–51 days. It's now 83 days since v1.20260801.0, the longest gap in over a year. That's why it feels so far behind. Since then:
  • 38 merges, about two-thirds of them bot PRs
  • kernel 6.18.34 → 6.18.52 (your Pi is still on 6.18.34)
  • firmware 20260521 → 20260915
  • about a dozen nixpkgs bumps

What the lag buys you

develop broke after a merge at least three times since June. Nobody on main ever saw it:

┌────────┬──────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────┬────────────────────────┐
│ Date │ PR │ What broke │ Fixed │
├────────┼──────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┼────────────────────────┤
│ Jun 20 │ #205 raspberrypi-utils (bot) │ The package and all 4 installer systems failed to build │ 17dfd64, ~5 min later │
├────────┼──────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┼────────────────────────┤
│ Aug 22 │ #237 rpicam-apps (bot) │ rpicam-apps failed to build │ 625dd88, ~30 min later │
├────────┼──────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┼────────────────────────┤
│ Sep 20 │ #256 kernel 6.18.52 │ Referenced raspberrypifw_20260919, which doesn't exist, so the configs failed to evaluate │ ce7da8e, 4 min later │
└────────┴──────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────┘

None of these PRs ran more than the formatting check before merge, and #256 ran no checks at all.

a commit is just taken from develop, and a release is cut from a PR with change logs. Long story short, I want newer Nix packages. I'm open to helping to achieve an automated pipeline for that that. I've spoken with Claude and come up with this workflow, but I thought I'd get your views on this first, before I try and get it some or all of it implemented!

None of these PRs ran more than the formatting check before merge, and #256 ran no checks at all.

Gaps in the current CI

• PRs only get a format check: that's all .github/workflows/ci.yaml does. The real build matrix only runs on push (.github/workflows/cachix.yaml:3), so it finds breakage once it's already on develop.
• Bot PRs never trigger a build: the flake.lock and nix-update PRs are opened with GITHUB_TOKEN, which doesn't trigger other workflows (update-packages.yml:65 notes this). There are zero build runs on those branches.
• Every Dependabot build fails: Dependabot runs don't get repo secrets, and the cache name is stored as a secret (cachix.yaml:28). The run I checked failed at the cachix step in all 14 jobs. They're false failures, and they teach people to ignore red builds.
• No tests at all: there's no checks output, and the ~600 lines of shell that write /boot/firmware have no tests. nix flake check fails today because packages.aarch64-linux.linuxPackages_rpi5 isn't a derivation (flake.nix:192).

What to automate, in order of payoff

1. Run the existing builds on PRs and require them.

   • Add pull_request to cachix.yaml, and only push to the cache on pushes to develop/main.
   • Hardcode name: nixos-raspberrypi, since the cache is public anyway. That fixes the Dependabot failures and lets PRs from forks build using the cache read-only.
   • Make these checks required on develop with a branch ruleset.
   • Most PRs hit the cache and finish in minutes. Kernel and nixpkgs bumps take about 1.5 hours, which is exactly when you want the build. This alone would have blocked all three breakages.

2. Give the bots a GitHub App token (actions/create-github-app-token) so their PRs trigger CI, then auto-merge them when green. That covers about two-thirds of develop's traffic without a person.

3. Add a checks output. First move linuxPackages_* into legacyPackages so nix flake check passes. Then add:

   • An eval matrix: every board × bootloader × optional modules, evaluated without building. Evaluating the four installer configs took 69 seconds on this Pi 5. Run against kenel update 6.18.52 + firmware #256's merge commit, it fails with attribute 'raspberrypifw_20260919' missing.
   • Bootloader tests: run each config's installBootLoader into a temp directory and check the expected files (config.txt, kernel, initrd, cmdline.txt, DTBs). This needs no Pi and no VM.

4. Automate promotion, the way the nixos-unstable channel follows nixpkgs master. A weekly job picks the newest develop commit that:

   • passed every check, including the cache build, so main users never compile a kernel
   • has been on develop for at least 7 days
   • has no open regression issues

   It then opens (or auto-merges) the release PR with generated notes, and tags it.

5. Real-hardware boot is the gap CI can't close, and it's what the manual waiting really covers. A self-hosted runner on a spare Pi could switch to the candidate, reboot, and run a health check. Run it on push or on a schedule, never on PRs from forks.

TLDR I'd love to either shore up the quality of develop, help automate cutting to main or both. Just wanna get buy-in from @nvmd first :).

PR for this here