Skip to content

fix(release): restore installs below macOS 26.2 with a baseline metallib, and stop misreporting the failure - #661

Draft
Gajesh2007 wants to merge 6 commits into
masterfrom
cursor/fix-macos-15-metallib-floor-66bb
Draft

fix(release): restore installs below macOS 26.2 with a baseline metallib, and stop misreporting the failure#661
Gajesh2007 wants to merge 6 commits into
masterfrom
cursor/fix-macos-15-metallib-floor-66bb

Conversation

@Gajesh2007

@Gajesh2007 Gajesh2007 commented Aug 22, 2026

Copy link
Copy Markdown
Member

What broke

A provider on an M4 Pro running macOS 15.7.9 got this from curl -fsSL https://api.darkbloom.dev/install.sh | bash:

  Bundle hash verified ✓
  Staging and verifying the complete app before touching the live install ...
Error: safe R1 was not latched as requested
  ✗ Packaged paged-kernel runtime smoke failed.
  Existing installation was left unchanged.

The message is wrong, and the underlying breakage is much bigger than one machine.

Root cause

Since #607 (v0.8.2), the release workflow builds the shipped mlx.metallib itself with MLX_METALLIB_DEPLOYMENT_TARGET=26.2, because the M5 _nax kernels only compile against Metal 4.0 with a macOS 26.2 deployment target (mlx/backend/metal/kernels/CMakeLists.txt). Every .air and the final link get -mmacosx-version-min=26.2.

A metallib linked for 26.2 is rejected outright by every older Metal runtime. So on macOS 15:

  1. load_default_library fails on the colocated mlx.metallib and every other candidate, and throws.
  2. That throw escapes Device::Device(), so metal::device(gpu) throws.
  3. gemma4_expert_qmm_diagnostics_snapshot catches everything and returns an all-zero struct — "diagnostics are optional".
  4. requested == false, so PackagedRuntimeSmoke.validateSafeR1 throws .safeR1NotRequested.

CI never caught it because blacksmith-12vcpu-macos-latest is on macOS 26.

So every Mac below macOS 26.2 has been unable to install or self-update since v0.8.2, while install.sh advertised "macOS 14+" and docs/architecture/hardware-support.md claimed macOS 15+.

What changed

1. Two kernel libraries, selected by MLX's own fallback

MLX's load_default_library probes the colocated mlx.metallib first and the colocated Resources/mlx.metallib second, falling through only when the first fails to load. That is exactly the macOS 15 case, so no host code or MLX-fork change is needed — the release just has to put a loadable library in the second slot.

Library Deployment target _nax Location
Primary 26.2 yes Contents/MacOS/mlx.metallib
Baseline 15.0 no Contents/MacOS/Resources/mlx.metallib

is_nax_available() is itself gated on macOS 26.2, so a host that lands on the baseline never asks for a kernel the baseline lacks.

MLX resolves those paths with dladdr against the directory of the running executable, which for a $PATH invocation through ~/.darkbloom/bin/darkbloom is bin/, not the bundle — that is why bin/mlx.metallib exists as a symlink today. So install.sh and UpdateInstallLayout.ensureCanonicalLinks(.app) mirror both libraries into bin/, and retire the baseline mirror when the installed app has none.

scripts/fetch-metallib.sh gains MLX_METALLIB_OUTPUT_NAME and derives the symbol contract from the deployment target: _nax required at ≥ 26.2, forbidden below. A baseline whose target silently did not take fails the build instead of shipping an artifact just as unloadable as the primary it exists to cover.

2. Attest the library that actually runs

template_hashes["mlx_metallib"] reports Contents/MacOS/mlx.metallib — below macOS 26.2, precisely the library the process cannot load. Left alone, this change would have put the whole sub-26.2 population on an unattested kernel set: swapping the baseline on disk changes nothing the coordinator checks, while swapping the primary they never load still deroutes them.

Providers now also report mlx_metallib_baseline; the release registers the expectation through the existing comma-separated template_hashes field (SyncRuntimeManifest already parses that, so no schema change), and verifyRuntimeHashesForBackend scopes routing verification to both keys. The baseline is enforced only when the provider reports itverifyRuntimeHashesAgainstManifest treats an omitted manifest key as a mismatch, so scoping it in unconditionally would deroute the entire existing fleet the moment a new release registered it.

3. Honest floor

install.sh checks sw_vers -productVersion (through env -u SYSTEM_VERSION_COMPAT, which otherwise reports 10.16) against MIN_MACOS before downloading, and darkbloom update pauses on sub-floor hosts instead of re-downloading a candidate that can never pass staged verification — stage failures get no quarantine backoff, so that was an unbounded loop. LSMinimumSystemVersion and Package.swift move from macOS 14 to the floor, as do the landing page, console setup page, and provider hardware doc. scripts/check-macos-floor.sh (new, wired into the Ubuntu release-integrity job) pins the floor and both deployment targets across the installers, the release workflow, fetch-metallib.sh, Package.swift, PackagedMetallib.swift, and every surface that advertises a floor.

Before / after

Behavior

flowchart TB
  subgraph Before
    B0[curl install.sh on macOS 15.7.9] --> B1[download + hash verify OK]
    B1 --> B2[runtime-smoke]
    B2 --> B3["load_default_library:<br/>mlx.metallib min-OS 26.2 rejected"]
    B3 --> B4["Device() throws"]
    B4 --> B5["diagnostics snapshot catches all<br/>-> all-zero struct"]
    B5 --> B6["requested == false"]
    B6 --> B7["Error: safe R1 was not latched as requested"]
    B7 --> B8[install aborts, no usable path forward]
  end
Loading
flowchart TB
  subgraph After
    A0[curl install.sh] --> A1{"sw_vers >= MIN_MACOS?"}
    A1 -- no --> A2["Error: requires macOS 15.0 or later (found 14.x)<br/>exit before downloading"]
    A1 -- yes --> A3[download + hash verify]
    A3 --> A4[baseline marker/library coupling check]
    A4 --> A5[runtime-smoke]
    A5 --> A6{"primary mlx.metallib loads?"}
    A6 -- "macOS 26.2+" --> A7[primary: NAX kernels available]
    A6 -- "macOS 15.x" --> A8["fallback Resources/mlx.metallib<br/>loads, NAX never requested"]
    A7 --> A9[install succeeds; bin/ mirrors both libraries]
    A8 --> A9
    A9 --> A10["register: reports mlx_metallib<br/>AND mlx_metallib_baseline"]
    A6 -- "neither loads" --> A11["MLX has no usable Metal runtime:<br/>host OS + real NSError per candidate"]
  end
Loading

Code

flowchart TB
  subgraph CodeBefore["Before"]
    C1["fetch-metallib.sh<br/>(single build, _nax always required)"] --> C2["release-swift.yml<br/>stage Contents/MacOS/mlx.metallib"]
    C2 --> C3["install.sh verify_staged_app<br/>runtime-smoke > /dev/null"]
    C3 --> C5["commit_staged_app<br/>bin/ mirrors 1 metallib"]
    C4["PackagedRuntimeSmoke.validateSafeR1<br/>guard requested else .safeR1NotRequested"]
    C6["ProviderLoop+Serve<br/>template_hashes[mlx_metallib]"] --> C7["server.go verifyRuntimeHashesForBackend<br/>scopes mlx_metallib only"]
  end
Loading
flowchart TB
  subgraph CodeAfter["After"]
    D1["fetch-metallib.sh<br/>+MLX_METALLIB_OUTPUT_NAME<br/>+NAX_CONTRACT required/forbidden by target"] --> D2["release-swift.yml<br/>build primary + baseline, sign both,<br/>smoke baseline-in-primary-slot,<br/>smoke corrupt-primary + good-baseline,<br/>inverted _nax assertion,<br/>register mlx_metallib_baseline"]
    D2 --> D3["install.sh<br/>version_at_least preflight<br/>verify_baseline_metallib_capability<br/>link_baseline_metallib into bin/<br/>surface smoke stderr"]
    D2 --> D4["BaselineMetallibCapabilityVerifier<br/>+ ensureCanonicalLinks mirrors bin/Resources<br/>+ checkForUpdate floor gate"]
    D5["PackagedRuntimeSmoke.validateSafeR1<br/>probeMetalRuntime() seam"] --> D6["MetalRuntimeProbe.classify<br/>-> .healthy / .noMetalDevice /<br/>.unknownExecutableDirectory / .noLoadableMetallib"]
    D6 --> D7["PackagedMetallib<br/>paths, targets, floor, diagnosis text"]
    D7 --> D8["check-macos-floor.sh (CI)<br/>pins all of the above + advertised floors"]
    D9["BinaryHasher.baselineMetallibHash<br/>ProviderLoop+Serve reports both keys"] --> D10["server.go verifyRuntimeHashesForBackend<br/>scopes both; baseline enforced only when reported"]
  end
Loading

Evidence

The real PackagedMetallib + MetalRuntimeProbe sources from this branch, compiled and run against the exact macOS 15.7.9 scenario (only the MTLDevice.makeLibrary(URL:) seam is stubbed, with Metal's actual rejection text):

Metal runtime diagnosis and baseline fallback

The new drift pin, including a negative control that injects a mismatch into each pinned file one at a time:

macOS floor drift pin and installer preflight

Also run here: full go build ./... && go test ./... on the coordinator (including four new attestation cases: matching baseline passes, swapped baseline caught, provider reporting no baseline still passes, swapped primary still caught), gofmt -l, npx eslint src/ on console-ui, bash -n on every touched script, both installer static assertions, YAML parse of both workflows, and Swift 6 -typecheck -strict-concurrency=complete of the new source files on a Linux 6.1 toolchain plus -parse of every modified Swift file. Each drift pin was verified to fail on an injected mismatch.

Left to CI (it has the hardware this environment does not)

  • test-provider / "Run atomic installer artifact tests" builds artifacts with a real ad-hoc-signed Contents/MacOS/Resources/mlx.metallib and runs codesign --verify --deep --strict. This is the load-bearing check on the new bundle layout: if Contents/MacOS/Resources/ is not codesign-clean, this PR goes red before any release can. Five rejection variants, each pinned to the specific baseline failure it tests, plus a pre-baseline artifact that must stay installable and a downgrade that must retire the bin/ mirror. test-install-preflight.sh now also runs there, so the version gate is exercised under macOS /bin/bash 3.2.
  • Swift suites: BaselineMetallibCapabilityTests (6 tests, including the bin/ mirror through ensureCanonicalLinks), a new MetallibHashTests case for the baseline locator, and four added cases in GemmaOptimizationReportingTests.
  • The release workflow's two extra smokes, the inverted symbol contract, and mlx_metallib_baseline registration run at tag time.

Notes for review

  • No version bump and no tag: per CLAUDE.md, releases only happen on explicit request. This needs a release to reach the affected fleet.
  • MIN_MACOS is 15.0 rather than 14.0 because Metal 3.2 (MLX_METAL_VERSION >= 320, macOS 15) gates MLX's fence kernel and fence.cpp calls get_kernel("fence_wait") unconditionally. A baseline built at 14.0 would ship without it.
  • The baseline is deliberately not staged into the release tar's flat bin/. That layout is only reachable for legacy no-.app artifacts, and UpdateInstallLayout's flat snapshot copies three files by name, so an extra file there would make a flat rollback hash-mismatch forever.
  • Known limitation, documented in PackagedMetallib: is_nax_available() keys off the OS, not off which library won the probe. The fallback is safe only because the primary is unloadable only below 26.2 — every install path hashes and code-signature-verifies it. Do not widen the fallback without also gating NAX on the library that loaded.
  • Threat Model Review is red on this PR, as it is on every recent PR and on master: the job dies with anthropic.AuthenticationError: 401 API key is invalid. Unrelated to this change.

To show artifacts inline, enable in settings.

Slack Thread

Open in Web Open in Cursor 

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…ch error

mlx_metal_gemma4_expert_qmm_diagnostics_snapshot catches every exception out
of metal::device(gpu) and returns an all-zero struct, so a host that cannot
construct a Metal device or load the packaged metallib is indistinguishable
from one that simply never requested the expert-slice route. The packaged
smoke has already proved MLX_GATHER_QMM_EXPERT_SLICES=1 is live in the
process by that point, so it now probes Metal directly on an unrequested
route and reports either 'no Metal device' or the host OS plus the real
NSError from each packaged kernel library it tried.

This is what a macOS 15 provider hit on v0.8.9: the shipped metallib is
linked for macOS 26.2 and cannot load there, and the install aborted with
'safe R1 was not latched as requested'.
…install

Since v0.8.2 CI builds the shipped mlx.metallib itself with
MLX_METALLIB_DEPLOYMENT_TARGET=26.2, which is what the M5 _nax kernels
require. A metallib linked -mmacosx-version-min=26.2 is rejected outright by
every older Metal runtime, so MLX's Device() constructor threw on any host
below macOS 26.2 and every such Mac has been unable to install or
self-update — while install.sh advertised macOS 14+ and hardware-support.md
claimed macOS 15+.

Releases now also build a NAX-free baseline library at the floor and stage it
at Contents/MacOS/Resources/mlx.metallib, which is MLX's own second colocated
probe in load_default_library and is reached only when the primary fails to
load. macOS 26.2+ keeps the NAX kernels; older hosts land on the baseline,
which they never ask NAX kernels of because is_nax_available() is itself
gated on macOS 26.2.

- fetch-metallib.sh takes an output name and derives the symbol contract from
  the deployment target: _nax required at >= 26.2, forbidden below, so a
  baseline whose target silently did not take fails the build.
- The release workflow signs the baseline like the primary, re-runs the full
  packaged smoke with the baseline in the primary slot (the runner is on
  macOS 26, so the normal run never reaches it), and asserts the inverted
  symbol contract on the final signed bytes.
- A signed baseline-metallib-v1 marker couples library and marker in
  install.sh and in self-update artifact verification; releases predating the
  layout ship neither and stay installable.
- install.sh checks sw_vers against MIN_MACOS before downloading, and prints
  the smoke child's own diagnosis on failure. LSMinimumSystemVersion moves to
  the same floor.
- scripts/check-macos-floor.sh pins the floor and both deployment targets
  across the installers, the workflow, the helper, and PackagedMetallib.swift.
@vercel

vercel Bot commented Aug 22, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
d-inference Ready Ready Preview Aug 22, 2026 7:08am
d-inference-console-ui-dev Ready Ready Preview Aug 22, 2026 7:08am
d-inference-landing Ready Ready Preview Aug 22, 2026 7:08am

Request Review

…y below the floor

A macOS 15.7.9 host that cannot load any packaged kernel library has a
packaging fault, not an old OS, and telling it to upgrade buries the bug.
PackagedMetallib.meetsMinimumMacOS mirrors the installer's version_at_least
so the diagnosis picks the remedy that would actually help.
…es keep the fallback

MLX resolves its metallib probes with dladdr on its own symbol, i.e. against
the directory of the running executable. For a $PATH invocation through
~/.darkbloom/bin/darkbloom that is bin/, not the app bundle — which is the
entire reason bin/mlx.metallib exists as a symlink today. Shipping the
baseline only inside the bundle left every foreground command (start
--foreground, local, benchmark, runtime-smoke, and the /usr/local/bin
shortcut) probing bin/mlx.metallib -> the 26.2 primary, then
bin/Resources/mlx.metallib -> nothing. On macOS 15 that is exactly the failure
the baseline exists to prevent.

Both install.sh's commit_staged_app and UpdateInstallLayout's
ensureCanonicalLinks(.app) now mirror the baseline too, and retire the mirror
when the installed app has none so a downgrade cannot leave a dangling probe.
The atomic installer suite asserts the mirror after install, its absence for a
pre-baseline artifact, and that a downgrade removes it; a new Swift test
covers the same three states through the self-update path.

Deliberately NOT staged into the release tar's flat bin/: that layout is only
reachable for legacy no-.app artifacts, and UpdateInstallLayout's flat
snapshot copies three files by name, so an extra file there would make a flat
rollback hash-mismatch forever.
…ity checks

Review follow-ups:

- The release workflow only ever exercised the primary library (the runner is
  on macOS 26), so a corrupted-primary-beside-good-baseline smoke now pins
  MLX's colocated probe order itself. A libs/mlx-swift bump that renames or
  reorders the candidates fails at tag time instead of shipping macOS 15 a
  provider that cannot start.
- BaselineMetallibCapabilityVerifier used fileExists for presence, which
  follows symlinks, so two dangling symlinks took the pre-baseline early
  return. It now matches the installer's [ -e ] || [ -L ], honors its injected
  FileManager throughout, and drops the Darwin-only branch since
  attributesOfItem already lstats.
- The atomic installer suite asserted only that a bad artifact was rejected,
  which the signature check satisfies on its own. Each variant now pins the
  specific baseline failure it is testing, and the empty-library fixture is
  re-signed so it reaches that check instead of tripping codesign first.
- A nil executable directory is its own diagnosis rather than an empty
  tried-nothing list, and a present-but-unreadable library is no longer
  reported as absent.
- landing/, the console setup page, and the provider hardware doc still
  advertised macOS 14; check-macos-floor.sh now pins every surface that quotes
  the floor to a prospective provider, not just the installers.
- Documented the invariant the fallback rests on: is_nax_available() keys off
  the OS, not off which library loaded, so widening the fallback without
  gating NAX on the winner would dispatch kernels the baseline lacks.
template_hashes["mlx_metallib"] reports Contents/MacOS/mlx.metallib, which
below macOS 26.2 is precisely the library the process cannot load. Every
provider on that population would have run an unattested kernel set: swapping
the baseline on disk changed nothing the coordinator checks, while swapping
the primary — which those hosts never load — still got them derouted.

Providers now also report mlx_metallib_baseline, the release registers it
through the existing comma-separated template_hashes field (no schema change;
SyncRuntimeManifest already parses that into the manifest), and
verifyRuntimeHashesForBackend scopes routing verification to both keys.

The baseline expectation is enforced only when the provider reports it.
verifyRuntimeHashesAgainstManifest treats a manifest key the provider omits as
a mismatch, so scoping it in unconditionally would deroute the entire existing
fleet the moment a new release registered it — and a macOS 26.2+ provider that
ships no baseline has nothing to attest either way.

Also from review:

- darkbloom update now pauses on sub-floor hosts. Stage-verification failures
  get no quarantine backoff, so a macOS 14 host on a pre-v0.8.2 release would
  otherwise re-download the tarball on every watchdog tick forever.
- install.sh reads sw_vers through env -u SYSTEM_VERSION_COMPAT; with that
  variable set it returns 10.16 on every modern macOS and the gate would have
  refused a perfectly good host. An unreadable version stays non-fatal.
- Package.swift moves to .macOS(.v15) and is pinned by check-macos-floor.sh.
- Release smokes dump captured stdout before failing (set -e was killing the
  printf), the job gets a timeout, and the metallib cache saves on failure so
  a broken second build does not discard the first.
- test-install-preflight.sh also runs on the macOS job, where the installer's
  version gate actually executes under bash 3.2.
@wronkiew

Copy link
Copy Markdown

I tried installing 0.8.1 on a M1 Max running Sonoma 14.8.9 and that doesn't work either.

% ./Darkbloom.app/Contents/MacOS/darkbloom --version    
0.8.1
% ./Darkbloom.app/Contents/MacOS/darkbloom runtime-smoke
MLX/ErrorHandler.swift:345: Fatal error: [metal::Device] Unable to load kernel rbitsc
Function rbitsc is using language version 4.0 which is incompatible with this OS.
 at /Users/runner/_work/d-inference/d-inference/libs/mlx-swift/Source/Cmlx/mlx-c/mlx/c/transforms.cpp:73

@rafaelcmrj

Copy link
Copy Markdown

+1

@soumitra24x7

Copy link
Copy Markdown

One more concrete data point from the field, on the same root cause but a different code path: the self-update path, not just install. v0.8.10 → v0.8.13 on a M3 Ultra / macOS 15.7.5 fails with failed to replace binary: process exited 1. The same gate that #663 wired into production (the staged binary's runtime-smoke, run from verifyRuntimeCapabilities in SelfUpdater.swift ~L1163) hit the same metallib load crash, but the user never sees the real error.

What the operator sees:

$ darkbloom update
darkbloom update
Current version: 0.8.10

Checking for updates...
failed to replace binary: process exited 1

What is actually happening (manually invoking the same child the gate runs, against both the v0.8.10 install and the staged v0.8.13 bundle extracted from pub-3d1cb668259340eeb2276e1d375c846d.r2.dev/releases/v0.8.13/darkbloom-bundle-macos-arm64.tar.gz):

$ ~/.darkbloom/bin/darkbloom runtime-smoke
Error: safe R1 was not latched as requested
$ DARKBLOOM_NO_UPDATE_CHECK=1 ./Darkbloom.app/Contents/MacOS/darkbloom runtime-smoke
Error: safe R1 was not latched as requested

The bundle hashes match the release manifest (ef3ab343... / 08c48889...), the bundle codesign --verify --deep --strict passes with the production DR, the Resources/mlx.metallib baseline slot is absent from the v0.8.13 tarball entirely (only MacOS/mlx.metallib is present), and the staged binary crashes the same way the installed one does.

The misreport comes from BoundedProcess.runCapturingStandardOutput / BoundedProcess.Failure.exited(status: 1, stderrTail: nil) — the wrapper's own comment at Process/BoundedProcess.swift:96-99 already calls this out: "a child that writes a short diagnostic and exits fast ... saying 'child exited 1' sent the operator into the wrong rabbit hole." On the install path that surface is the install script (>/dev/null 2>&1 || { ... }); on the update path it is verifyRuntimeCapabilities re-throwing as replaceFailed("\(error)"), which surfaces the wrapper string verbatim.

Two things this PR is uniquely positioned to fix, beyond the install path already described:

  1. MetalRuntimeProbe should also be invoked from verifyRuntimeCapabilities before the gate calls runtime-smoke, so a host that cannot construct a Metal device at all (or whose metallib fails to load) short-circuits with the real PackagedMetallib / MLX metallib loaded=false reason, instead of catching it through the silent safeR1NotRequested path. Right now the update gate re-discovers the same crash that the install gate catches and the install gate's PR description already documents.

  2. The verifyRuntimeCapabilities runtime-smoke call is host-incompatible by construction on every macOS < 26.2 install that this PR is trying to rescue. runtime-smoke cannot succeed on a host where the production metallib cannot load — that is the very class of host this PR is unblocking. So the gate is rejecting exactly the bundles it should be accepting. One option: probe the host's metallib load first, and if the host is sub-floor, downgrade the verification policy to codesign + bundle hashes + pagedattention.metal present + paged-kernel capability marker only, skipping the runtime smoke. The metallib-load probe is the only thing that actually distinguishes "this bundle is bad" from "this host is bad", and it is the probe this PR already adds.

For the operators who are on v0.8.10 today and trying to move forward: there is no path off v0.8.10 on macOS 15 short of this PR landing or a Tahoe upgrade. Every patch between 0.8.10 and the current release ships the same metallib hash 08c48889aee7a8d126e47b12528e8f3e2c43f45866de938be11a8777a952b033 (built from mlx 0a725e3000ed), so there is no patch-level fix available via update — it is exactly this PR or nothing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants