fix(release): restore installs below macOS 26.2 with a baseline metallib, and stop misreporting the failure - #661
Conversation
…ch error mlx_metal_gemma4_expert_qmm_diagnostics_snapshot catches every exception out of metal::device(gpu) and returns an all-zero struct, so a host that cannot construct a Metal device or load the packaged metallib is indistinguishable from one that simply never requested the expert-slice route. The packaged smoke has already proved MLX_GATHER_QMM_EXPERT_SLICES=1 is live in the process by that point, so it now probes Metal directly on an unrequested route and reports either 'no Metal device' or the host OS plus the real NSError from each packaged kernel library it tried. This is what a macOS 15 provider hit on v0.8.9: the shipped metallib is linked for macOS 26.2 and cannot load there, and the install aborted with 'safe R1 was not latched as requested'.
…install Since v0.8.2 CI builds the shipped mlx.metallib itself with MLX_METALLIB_DEPLOYMENT_TARGET=26.2, which is what the M5 _nax kernels require. A metallib linked -mmacosx-version-min=26.2 is rejected outright by every older Metal runtime, so MLX's Device() constructor threw on any host below macOS 26.2 and every such Mac has been unable to install or self-update — while install.sh advertised macOS 14+ and hardware-support.md claimed macOS 15+. Releases now also build a NAX-free baseline library at the floor and stage it at Contents/MacOS/Resources/mlx.metallib, which is MLX's own second colocated probe in load_default_library and is reached only when the primary fails to load. macOS 26.2+ keeps the NAX kernels; older hosts land on the baseline, which they never ask NAX kernels of because is_nax_available() is itself gated on macOS 26.2. - fetch-metallib.sh takes an output name and derives the symbol contract from the deployment target: _nax required at >= 26.2, forbidden below, so a baseline whose target silently did not take fails the build. - The release workflow signs the baseline like the primary, re-runs the full packaged smoke with the baseline in the primary slot (the runner is on macOS 26, so the normal run never reaches it), and asserts the inverted symbol contract on the final signed bytes. - A signed baseline-metallib-v1 marker couples library and marker in install.sh and in self-update artifact verification; releases predating the layout ship neither and stay installable. - install.sh checks sw_vers against MIN_MACOS before downloading, and prints the smoke child's own diagnosis on failure. LSMinimumSystemVersion moves to the same floor. - scripts/check-macos-floor.sh pins the floor and both deployment targets across the installers, the workflow, the helper, and PackagedMetallib.swift.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…y below the floor A macOS 15.7.9 host that cannot load any packaged kernel library has a packaging fault, not an old OS, and telling it to upgrade buries the bug. PackagedMetallib.meetsMinimumMacOS mirrors the installer's version_at_least so the diagnosis picks the remedy that would actually help.
…es keep the fallback MLX resolves its metallib probes with dladdr on its own symbol, i.e. against the directory of the running executable. For a $PATH invocation through ~/.darkbloom/bin/darkbloom that is bin/, not the app bundle — which is the entire reason bin/mlx.metallib exists as a symlink today. Shipping the baseline only inside the bundle left every foreground command (start --foreground, local, benchmark, runtime-smoke, and the /usr/local/bin shortcut) probing bin/mlx.metallib -> the 26.2 primary, then bin/Resources/mlx.metallib -> nothing. On macOS 15 that is exactly the failure the baseline exists to prevent. Both install.sh's commit_staged_app and UpdateInstallLayout's ensureCanonicalLinks(.app) now mirror the baseline too, and retire the mirror when the installed app has none so a downgrade cannot leave a dangling probe. The atomic installer suite asserts the mirror after install, its absence for a pre-baseline artifact, and that a downgrade removes it; a new Swift test covers the same three states through the self-update path. Deliberately NOT staged into the release tar's flat bin/: that layout is only reachable for legacy no-.app artifacts, and UpdateInstallLayout's flat snapshot copies three files by name, so an extra file there would make a flat rollback hash-mismatch forever.
…ity checks Review follow-ups: - The release workflow only ever exercised the primary library (the runner is on macOS 26), so a corrupted-primary-beside-good-baseline smoke now pins MLX's colocated probe order itself. A libs/mlx-swift bump that renames or reorders the candidates fails at tag time instead of shipping macOS 15 a provider that cannot start. - BaselineMetallibCapabilityVerifier used fileExists for presence, which follows symlinks, so two dangling symlinks took the pre-baseline early return. It now matches the installer's [ -e ] || [ -L ], honors its injected FileManager throughout, and drops the Darwin-only branch since attributesOfItem already lstats. - The atomic installer suite asserted only that a bad artifact was rejected, which the signature check satisfies on its own. Each variant now pins the specific baseline failure it is testing, and the empty-library fixture is re-signed so it reaches that check instead of tripping codesign first. - A nil executable directory is its own diagnosis rather than an empty tried-nothing list, and a present-but-unreadable library is no longer reported as absent. - landing/, the console setup page, and the provider hardware doc still advertised macOS 14; check-macos-floor.sh now pins every surface that quotes the floor to a prospective provider, not just the installers. - Documented the invariant the fallback rests on: is_nax_available() keys off the OS, not off which library loaded, so widening the fallback without gating NAX on the winner would dispatch kernels the baseline lacks.
template_hashes["mlx_metallib"] reports Contents/MacOS/mlx.metallib, which below macOS 26.2 is precisely the library the process cannot load. Every provider on that population would have run an unattested kernel set: swapping the baseline on disk changed nothing the coordinator checks, while swapping the primary — which those hosts never load — still got them derouted. Providers now also report mlx_metallib_baseline, the release registers it through the existing comma-separated template_hashes field (no schema change; SyncRuntimeManifest already parses that into the manifest), and verifyRuntimeHashesForBackend scopes routing verification to both keys. The baseline expectation is enforced only when the provider reports it. verifyRuntimeHashesAgainstManifest treats a manifest key the provider omits as a mismatch, so scoping it in unconditionally would deroute the entire existing fleet the moment a new release registered it — and a macOS 26.2+ provider that ships no baseline has nothing to attest either way. Also from review: - darkbloom update now pauses on sub-floor hosts. Stage-verification failures get no quarantine backoff, so a macOS 14 host on a pre-v0.8.2 release would otherwise re-download the tarball on every watchdog tick forever. - install.sh reads sw_vers through env -u SYSTEM_VERSION_COMPAT; with that variable set it returns 10.16 on every modern macOS and the gate would have refused a perfectly good host. An unreadable version stays non-fatal. - Package.swift moves to .macOS(.v15) and is pinned by check-macos-floor.sh. - Release smokes dump captured stdout before failing (set -e was killing the printf), the job gets a timeout, and the metallib cache saves on failure so a broken second build does not discard the first. - test-install-preflight.sh also runs on the macOS job, where the installer's version gate actually executes under bash 3.2.
|
I tried installing 0.8.1 on a M1 Max running Sonoma 14.8.9 and that doesn't work either. |
|
+1 |
|
One more concrete data point from the field, on the same root cause but a different code path: the self-update path, not just install. v0.8.10 → v0.8.13 on a M3 Ultra / macOS 15.7.5 fails with What the operator sees: What is actually happening (manually invoking the same child the gate runs, against both the v0.8.10 install and the staged v0.8.13 bundle extracted from The bundle hashes match the release manifest ( The misreport comes from Two things this PR is uniquely positioned to fix, beyond the install path already described:
For the operators who are on v0.8.10 today and trying to move forward: there is no path off v0.8.10 on macOS 15 short of this PR landing or a Tahoe upgrade. Every patch between 0.8.10 and the current release ships the same metallib hash |
What broke
A provider on an M4 Pro running macOS 15.7.9 got this from
curl -fsSL https://api.darkbloom.dev/install.sh | bash:The message is wrong, and the underlying breakage is much bigger than one machine.
Root cause
Since #607 (v0.8.2), the release workflow builds the shipped
mlx.metallibitself withMLX_METALLIB_DEPLOYMENT_TARGET=26.2, because the M5_naxkernels only compile against Metal 4.0 with a macOS 26.2 deployment target (mlx/backend/metal/kernels/CMakeLists.txt). Every.airand the final link get-mmacosx-version-min=26.2.A metallib linked for 26.2 is rejected outright by every older Metal runtime. So on macOS 15:
load_default_libraryfails on the colocatedmlx.metalliband every other candidate, and throws.Device::Device(), sometal::device(gpu)throws.gemma4_expert_qmm_diagnostics_snapshotcatches everything and returns an all-zero struct — "diagnostics are optional".requested == false, soPackagedRuntimeSmoke.validateSafeR1throws.safeR1NotRequested.CI never caught it because
blacksmith-12vcpu-macos-latestis on macOS 26.So every Mac below macOS 26.2 has been unable to install or self-update since v0.8.2, while
install.shadvertised "macOS 14+" anddocs/architecture/hardware-support.mdclaimed macOS 15+.What changed
1. Two kernel libraries, selected by MLX's own fallback
MLX's
load_default_libraryprobes the colocatedmlx.metallibfirst and the colocatedResources/mlx.metallibsecond, falling through only when the first fails to load. That is exactly the macOS 15 case, so no host code or MLX-fork change is needed — the release just has to put a loadable library in the second slot._naxContents/MacOS/mlx.metallibContents/MacOS/Resources/mlx.metallibis_nax_available()is itself gated on macOS 26.2, so a host that lands on the baseline never asks for a kernel the baseline lacks.MLX resolves those paths with
dladdragainst the directory of the running executable, which for a$PATHinvocation through~/.darkbloom/bin/darkbloomisbin/, not the bundle — that is whybin/mlx.metallibexists as a symlink today. Soinstall.shandUpdateInstallLayout.ensureCanonicalLinks(.app)mirror both libraries intobin/, and retire the baseline mirror when the installed app has none.scripts/fetch-metallib.shgainsMLX_METALLIB_OUTPUT_NAMEand derives the symbol contract from the deployment target:_naxrequired at ≥ 26.2, forbidden below. A baseline whose target silently did not take fails the build instead of shipping an artifact just as unloadable as the primary it exists to cover.2. Attest the library that actually runs
template_hashes["mlx_metallib"]reportsContents/MacOS/mlx.metallib— below macOS 26.2, precisely the library the process cannot load. Left alone, this change would have put the whole sub-26.2 population on an unattested kernel set: swapping the baseline on disk changes nothing the coordinator checks, while swapping the primary they never load still deroutes them.Providers now also report
mlx_metallib_baseline; the release registers the expectation through the existing comma-separatedtemplate_hashesfield (SyncRuntimeManifestalready parses that, so no schema change), andverifyRuntimeHashesForBackendscopes routing verification to both keys. The baseline is enforced only when the provider reports it —verifyRuntimeHashesAgainstManifesttreats an omitted manifest key as a mismatch, so scoping it in unconditionally would deroute the entire existing fleet the moment a new release registered it.3. Honest floor
install.shcheckssw_vers -productVersion(throughenv -u SYSTEM_VERSION_COMPAT, which otherwise reports10.16) againstMIN_MACOSbefore downloading, anddarkbloom updatepauses on sub-floor hosts instead of re-downloading a candidate that can never pass staged verification — stage failures get no quarantine backoff, so that was an unbounded loop.LSMinimumSystemVersionandPackage.swiftmove from macOS 14 to the floor, as do the landing page, console setup page, and provider hardware doc.scripts/check-macos-floor.sh(new, wired into the Ubunturelease-integrityjob) pins the floor and both deployment targets across the installers, the release workflow,fetch-metallib.sh,Package.swift,PackagedMetallib.swift, and every surface that advertises a floor.Before / after
Behavior
flowchart TB subgraph Before B0[curl install.sh on macOS 15.7.9] --> B1[download + hash verify OK] B1 --> B2[runtime-smoke] B2 --> B3["load_default_library:<br/>mlx.metallib min-OS 26.2 rejected"] B3 --> B4["Device() throws"] B4 --> B5["diagnostics snapshot catches all<br/>-> all-zero struct"] B5 --> B6["requested == false"] B6 --> B7["Error: safe R1 was not latched as requested"] B7 --> B8[install aborts, no usable path forward] endflowchart TB subgraph After A0[curl install.sh] --> A1{"sw_vers >= MIN_MACOS?"} A1 -- no --> A2["Error: requires macOS 15.0 or later (found 14.x)<br/>exit before downloading"] A1 -- yes --> A3[download + hash verify] A3 --> A4[baseline marker/library coupling check] A4 --> A5[runtime-smoke] A5 --> A6{"primary mlx.metallib loads?"} A6 -- "macOS 26.2+" --> A7[primary: NAX kernels available] A6 -- "macOS 15.x" --> A8["fallback Resources/mlx.metallib<br/>loads, NAX never requested"] A7 --> A9[install succeeds; bin/ mirrors both libraries] A8 --> A9 A9 --> A10["register: reports mlx_metallib<br/>AND mlx_metallib_baseline"] A6 -- "neither loads" --> A11["MLX has no usable Metal runtime:<br/>host OS + real NSError per candidate"] endCode
flowchart TB subgraph CodeBefore["Before"] C1["fetch-metallib.sh<br/>(single build, _nax always required)"] --> C2["release-swift.yml<br/>stage Contents/MacOS/mlx.metallib"] C2 --> C3["install.sh verify_staged_app<br/>runtime-smoke > /dev/null"] C3 --> C5["commit_staged_app<br/>bin/ mirrors 1 metallib"] C4["PackagedRuntimeSmoke.validateSafeR1<br/>guard requested else .safeR1NotRequested"] C6["ProviderLoop+Serve<br/>template_hashes[mlx_metallib]"] --> C7["server.go verifyRuntimeHashesForBackend<br/>scopes mlx_metallib only"] endflowchart TB subgraph CodeAfter["After"] D1["fetch-metallib.sh<br/>+MLX_METALLIB_OUTPUT_NAME<br/>+NAX_CONTRACT required/forbidden by target"] --> D2["release-swift.yml<br/>build primary + baseline, sign both,<br/>smoke baseline-in-primary-slot,<br/>smoke corrupt-primary + good-baseline,<br/>inverted _nax assertion,<br/>register mlx_metallib_baseline"] D2 --> D3["install.sh<br/>version_at_least preflight<br/>verify_baseline_metallib_capability<br/>link_baseline_metallib into bin/<br/>surface smoke stderr"] D2 --> D4["BaselineMetallibCapabilityVerifier<br/>+ ensureCanonicalLinks mirrors bin/Resources<br/>+ checkForUpdate floor gate"] D5["PackagedRuntimeSmoke.validateSafeR1<br/>probeMetalRuntime() seam"] --> D6["MetalRuntimeProbe.classify<br/>-> .healthy / .noMetalDevice /<br/>.unknownExecutableDirectory / .noLoadableMetallib"] D6 --> D7["PackagedMetallib<br/>paths, targets, floor, diagnosis text"] D7 --> D8["check-macos-floor.sh (CI)<br/>pins all of the above + advertised floors"] D9["BinaryHasher.baselineMetallibHash<br/>ProviderLoop+Serve reports both keys"] --> D10["server.go verifyRuntimeHashesForBackend<br/>scopes both; baseline enforced only when reported"] endEvidence
The real
PackagedMetallib+MetalRuntimeProbesources from this branch, compiled and run against the exact macOS 15.7.9 scenario (only theMTLDevice.makeLibrary(URL:)seam is stubbed, with Metal's actual rejection text):Metal runtime diagnosis and baseline fallback
The new drift pin, including a negative control that injects a mismatch into each pinned file one at a time:
macOS floor drift pin and installer preflight
Also run here: full
go build ./... && go test ./...on the coordinator (including four new attestation cases: matching baseline passes, swapped baseline caught, provider reporting no baseline still passes, swapped primary still caught),gofmt -l,npx eslint src/on console-ui,bash -non every touched script, both installer static assertions, YAML parse of both workflows, and Swift 6-typecheck -strict-concurrency=completeof the new source files on a Linux 6.1 toolchain plus-parseof every modified Swift file. Each drift pin was verified to fail on an injected mismatch.Left to CI (it has the hardware this environment does not)
test-provider/ "Run atomic installer artifact tests" builds artifacts with a real ad-hoc-signedContents/MacOS/Resources/mlx.metalliband runscodesign --verify --deep --strict. This is the load-bearing check on the new bundle layout: ifContents/MacOS/Resources/is not codesign-clean, this PR goes red before any release can. Five rejection variants, each pinned to the specific baseline failure it tests, plus a pre-baseline artifact that must stay installable and a downgrade that must retire thebin/mirror.test-install-preflight.shnow also runs there, so the version gate is exercised under macOS/bin/bash3.2.BaselineMetallibCapabilityTests(6 tests, including thebin/mirror throughensureCanonicalLinks), a newMetallibHashTestscase for the baseline locator, and four added cases inGemmaOptimizationReportingTests.mlx_metallib_baselineregistration run at tag time.Notes for review
CLAUDE.md, releases only happen on explicit request. This needs a release to reach the affected fleet.MIN_MACOSis 15.0 rather than 14.0 because Metal 3.2 (MLX_METAL_VERSION >= 320, macOS 15) gates MLX'sfencekernel andfence.cppcallsget_kernel("fence_wait")unconditionally. A baseline built at 14.0 would ship without it.bin/. That layout is only reachable for legacy no-.appartifacts, andUpdateInstallLayout's flat snapshot copies three files by name, so an extra file there would make a flat rollback hash-mismatch forever.PackagedMetallib:is_nax_available()keys off the OS, not off which library won the probe. The fallback is safe only because the primary is unloadable only below 26.2 — every install path hashes and code-signature-verifies it. Do not widen the fallback without also gating NAX on the library that loaded.Threat Model Reviewis red on this PR, as it is on every recent PR and on master: the job dies withanthropic.AuthenticationError: 401 API key is invalid. Unrelated to this change.To show artifacts inline, enable in settings.
Slack Thread
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.