perf(provider): Gemma 4 26B v0.8.2 — retained optimization cutover, config controls, source-matched metallib - #607
Conversation
abd1985 lands q=128 query sub-blocking in AttentionV1: prompt chunks attend in blocks, slicing K/V to each block's own visible span. Sliding executed/useful collapses from 1.499x to 1.062x at q=128, and the composed-path score tensor becomes O(1) in chunk length (2.04 GB -> 0.51 GB at 124k context on a full-attention layer). Blocked call sites are updateAndAttendRow and borrowAndAttendRow. Decode, last-query prefill, and span-bearing vision chunks keep the single-call path. DARKBLOOM_CBV2_ATTN_QUERY_BLOCK overrides the width; 0 restores the previous behavior. Verified: provider swift build clean; coordinator go build/test clean. provider swift test 1639 tests, 2 pre-existing SelfUpdaterTests failures from a missing local mlx.metallib fixture (environmental, unrelated to this bump).
Rev 1 was written against submodule b177c35. Re-verified every claim against e65b5bb / abd1985 and corrected 26 items (log in section 22). Four findings changed the plan's shape: 1. #85 already shipped Track A (query sub-blocking), so that track is deleted -- and paged does NOT inherit the win. PagedLayerCache .prefillAttend still issues one SDPA over the full [L, kL] rectangle and adds a gathered K/V copy, so the migration now regresses prefill activation memory unless new item 0.2p lands first. 2. B=8 is NOT reachable by provider config. The coordinator quality cap (concurrency_cap.go:272-299) takes MIN(reported, ceil(1.2 * floor(( solo/15 - 1)/0.27))); gemma-4's 10-18 tok/s solo yields an effective cap of 2 at both B=4 and B=8, pinned by concurrency_cap_test.go :140-147. Gate G0 as written would have measured nothing. Split into G0a/G0b, added section 8 (nine coordinator files, four of them capacity-feedback loops that can eject a B=8 box), tripled Track E. 3. The wire cannot distinguish a paged provider from a contiguous one -- BackendSlotCapacity has no backend discriminator, so a mixed canary is unobservable. Added as Gate G5. 4. CI runs no paged correctness test; four suites are compiled and discarded, and all three named silent-corruption classes have zero fireable assertions. New sections 19 (test/CI) and 20 (benchmark). Also corrected: WS-3.4's justification was false in the dangerous direction (rectangular MTP verification is live in production, so the first paged MTP slot is a process abort, not a fallback); the activation-reserve mirror is servability.go:50, not scheduler.go:1301; the DARKBLOOM_CBV2_PAGED_KV kill switch is unreachable under today's default; Track T should extend CBv2KVSharingParityTests rather than start over. Rev 1's 27,136 donation floor was right and the findings doc's 26,624 was wrong. Also commits the two companion reports the plan cites, so its cross-references resolve in-tree.
…onfig-backed controls, source-matched metallib Production integration of the retained v0.8.2 Gemma 4 slices; pins libs/mlx-swift-lm (53e8ccd) and libs/mlx-swift (db9b77d, nested 9dd10bef). - Shared-tower cutover: production/benchmark resolution unwraps the outer MLXVLM.Gemma4 once and threads its owned textModel through engine, cache, sizing, media, prefix/frozen-replay, and MTP paths. EngineV2VLMTextExtraction and its parity-probe plumbing are deleted. - Config authority: GemmaOptimizationEnvironment/Settings resolve the retained layer-18 and coupled weighted-unsort + safe-R1 requests once per process before any MLX device access (launchd/foreground/local/ benchmark), with requested/effective reporting; raw env sampling stays experimental-only. - Version surfaces: ProviderCore.version 0.8.2; coordinator LatestProviderVersion fallback 0.8.2 in sync. - Source-matched metallib: fetch-metallib builds from libs/mlx-swift/Source/Cmlx/mlx (JIT off), refuses artifacts missing NAX/GEMV/the R1 builder/tile symbols; ci/integration/release workflows replace PyPI metallibs and purge cache-restored ones; Makefile provider-build/test stages the metallib for runner and xctest bundles; PackagedRuntimeSmoke/darkbloom runtime-smoke cover it. - e2e/testbed + scripts/gemma_contbatch: benchmark harness environment threading; docs (beta-features, cli-reference, installation, quickstart) and READMEs updated. Measured effect (gemma-4-26B-A4B-it-qat-4bit, M4 Max, immutable pre-edit vs retained weighted+R1 profile, median/geomean): arrival TTFT +44.8%, arrival end-to-end TPS +19.6%, single-shot TTFT +15.2% (2048-tok 1715ms -> ~1389ms), decode aggregate TPS +1.3%, prefill TPS +2.4%, peak memory unchanged. Dropped: expert gate/up packing, dense gate/up packing, standalone weighted-unsort, standalone R1, historical accepted R1 control (noisy/negative). R1 kernel armed but recorded zero production hits; retained opt-in under the coupled gate. Benchmark artifacts predate the final kernel-body source edit; rebuild and re-verify before release claims.
…expect allSatisfy(\.effective) trips the macro's throwing-call detection; closure form is identical and compiles.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9d5c7ae212
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…cite canonical code in beta docs; bump pins - BenchmarkCommand now loads the runtime snapshot, then routes through the same Start.prepareServeRuntime seam as the serve path: the provider.toml Gemma projection applies before GPUEnforcement's first MLX device access, so a rollback A/B benchmark can no longer measure the default-enabled stack against an operator-disabled config (Codex P1 on #607). - docs/provider/beta-features.md gains canonical file:line references for every behavioral claim (docs/AGENTS.md code-wins rule): settings defaults, startup projection seam, projection/apply authority, the coupled weighted+R1 control (Codex docs P1). - Pins libs/mlx-swift-lm 1d46abc (four shared-tower semantics fixes + contract suite) and libs/mlx-swift cdb7bcd (C-mode MLX_API fallback for the R1 facade; Codex P1 on Layr-Labs/mlx-swift#11). make provider-test: 1666 swift-testing + 82 XCTest pass on this tree.
|
Codex review addressed in
Stack note: this push also advances the pins to @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 06442b50e1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…y follow-ups Review follow-ups on the v0.8.2 Gemma 4 optimization stack (#607): - R1 (HIGH, silent rollback defeat): the lenient ConfigManager.parse (whole-config defaults on ANY decode error) stays only as the test-facing API; production file loads route through the new strict ConfigManager.parseValidating via ConfigManager.load(from:), throwing ConfigError.parseFailed with the decode detail. A malformed [gemma_optimizations] entry (weighted_r1 = 0, prefill_layer18 = "false") previously re-armed the whole default-on stack with zero log; every snapshot-based command (start/benchmark/beta/status/...) now refuses at load. Missing files keep defaulting one layer up (loadDefault/loadRuntimeSnapshot), and missing sections/keys keep their per-key default-on decode. - R2 (HIGH, benchmark A/B integrity): darkbloom benchmark now reads the three low-level env keys before applying the projection and FAILS (stderr naming key, shell value, config value + ExitCode.failure) when a shell preset conflicts, instead of silently overwriting it and invalidating the runner.py os.environ artifact metadata; on success it prints one line with the effective controls. Benchmark A/B is explicitly config-driven (docs/provider/beta-features.md updated with the new citations/line numbers). - R3 (MED, seam placement): the ordering seam moved from Start.prepareServeRuntime into the new ServeRuntimePreparer.prepareRuntime (config projection strictly BEFORE the first MLX touch; a rejected projection aborts before engine construction). Start.prepareServeRuntime remains as a forwarding shim for compatibility (StartCommand.run + one shim test); BenchmarkCommand and StartCommandTests now target ServeRuntimePreparer directly. - R4 (LOW): darkbloom beta enable/disable no longer no-ops when the key (or its [section]) is ABSENT but the decode default matches the target: the value is materially written (BetaFeature.configAddress + tomlKeyPresent), pinning operator intent against future default flips. A key already pinned at the target is still a true no-op (no rewrite). - R5 (LOW): CHANGELOG stat fixes vs the committed-tree tree benchmark artifact: arrival TTFT speedups 1.411/1.385/1.470/1.762 -> "1.38-1.76x" (was "1.39-1.76x"); "peak memory unchanged" replaced with the retention memory control numbers (B1/B2/B4 peaks 14.0/14.64/15.05 GiB vs default 14.06/14.70/15.11; tmp/benchmarks/v0-8-2-retention-comparison.json). - R6 (LOW): the beta load->modify->save window is serialized by an exclusive flock(2) on a stable provider.toml.lock SIDECAR file, with the config re-loaded inside the lock (lost-update RMW race). Sidecar chosen over locking the config file itself: ConfigManager.save writes atomically via temp-file+rename, so the config inode changes on every save and concurrent writers holding different inodes would NOT exclude each other; the sidecar path is never renamed, giving every contender the same inode. fd close releases the lock even across a throw. R7 (note only, semantics unchanged): GemmaOptimizationEnvironment holds no armed/latched global state -- projection() is pure and apply() only setenv()s the three latches at process start. The POSIX process environment is global and getenv/setenv are not thread-safe, which stays correct because both call sites (start, benchmark) apply before MLX init and before any worker concurrency; a future call site that applies mid-run would need its own serialization. Tests: 1692 swift-testing tests + 82 XCTest, all passing (make provider-test). New: ConfigValidationTests (11), BetaCommandTests (10), RuntimeSnapshotConfigTests (2), StartCommandTests +4 (env-conflict guard + shim forwarding; pre-existing Start seam tests retargeted).
… cascade); fix benchmark stdout hygiene - mlx-swift-lm f00c9bd: completes the global-KV-head rule across the CBv2 layer-kind derivation and MTP validation, re-forges the vacuous fp16 overflow proof at norm gains, normalizes short/empty layer_types, pins rank-1 at both entries and the canonical ProportionalRoPE construction. - mlx-swift 746d7c5 (with nested 8d538a08): fail-safe sortedness check in the R1 tile builder (host re-routes to legacy on violation), shared route predicate, hit-only bias normalization, atomic armed_; mirrors/facades byte-synced; xcodeproj generator now preserves the C-mode MLX_API fallback; facade ABI pinned with _Static_asserts; ON-path test invocation documented. - SlotSizingDriftTests: track the gemma4LayerKinds signature change. - BenchmarkCommand: project the effective settings echo to stderr — benchmark subcommands emit machine-parsed JSON on stdout (a stdout line broke the gemma-contbatch harness).
|
Deep-review round complete — fixes landed as
Perf-gate caveat (important): the freshest full-matrix rerun executed during an environment-limited window — the machine had been run flat-out for hours of builds/test waves/benchmarks, and a same-minute control with the untouched pre-edit binary measured ~1.7× slower decode than its own Aug-4 baseline (61 vs 106 tok/s, no thermal warnings, healthy free RAM). Absolute numbers from that window are NOT stack attribution and are excluded; the clean evidence remains @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c8045ce633
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…widen preload-gate margin under parallel load; bump pins - UpdateCommand: only ConfigError.readFailed falls back to defaults — an existing-but-malformed provider.toml now fails loudly instead of silently resetting every setting and aiming the update check at the wrong coordinator (final root-verifier residual item; watchdog's deliberate fail-open posture unchanged). - StartupPreloadTests: wall-clock margin 2.8s -> 4.5s; the semantic signal is the .timedOut outcome + background continuation, and the tight margin flaked at 2.8-3.5s under full-suite parallelism on loaded machines (1692-test runs, twice, on this box). - Pins: libs/mlx-swift-lm 06046c3 (last k_eq_v-gated head rule gone from validateAttentionProducts), libs/mlx-swift 38eaa93 (nested 9b0d1b4c: sound adjacent-pair sortedness oracle + dedicated fallback_sortedness_retracted counter, ABI 88, generator post- condition; mirrors/facades byte-synced).
|
Final review wave (#2) complete. Every prior finding is closed; the final verifier pass signed off each repo:
Perf-gate status unchanged: attribution rerun deferred to a cooled machine (see the prior comment — the box measured ~1.7× slow during the late-night window). The remaining open perf item is the nested PR's per-hit @codex review |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…search-v0.8.2 # Conflicts: # .github/workflows/ci.yml # .github/workflows/integration.yml # coordinator/api/server.go # docs/provider/beta-features.md # docs/provider/cli-reference.md # docs/reports/2026-07-25-paged-kv-migration-plan.md # libs/mlx-swift-lm # provider-swift/Sources/ProviderBenchmark/SchedulerPrefillBenchmark.swift # provider-swift/Sources/ProviderBenchmark/ThroughputSweep.swift # provider-swift/Sources/ProviderCore/Config/BetaFeatures.swift # provider-swift/Sources/ProviderCore/Config/ProviderConfig.swift # provider-swift/Sources/ProviderCore/Inference/EngineV2Config.swift # provider-swift/Sources/ProviderCore/Process/BoundedProcess.swift # provider-swift/Sources/ProviderCore/ProviderCore.swift # provider-swift/Sources/ProviderCore/ProviderLoop+ModelLoading.swift # provider-swift/Sources/darkbloom/BetaCommand.swift # provider-swift/Tests/ProviderCoreTests/BetaFeaturesTests.swift # provider-swift/Tests/ProviderCoreTests/LaunchAgentRestartTests.swift # provider-swift/Tests/ProviderCoreTests/StartupPreloadTests.swift
|
Final dependency follow-up at |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
GitHub Actions
Failure
|
@codex review |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
GitHub Actions
Failure
|
@codex review |
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
GitHub Actions
Failure
|
Dependency follow-up at |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Final dependency-pin follow-up at |
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Status
Ready for review at
f1d63cbba. mlx #4, mlx-swift #11, and mlx-swift-lm #102 are merged. The root pins merged mlxdarkbloom-baseata4b2b4c2, mlx-swiftmainat97b3c921, and mlx-swift-lmmainated55bee. The merged mlx-swift-lm tree is identical to the reviewed feature headc5c5338. Currentorigin/master(77c99fb89) is integrated, all review threads are resolved, and final Codex review found no major issues.All code and blocking integration gates are green. Threat Model Review is externally blocked because the repository
ANTHROPIC_API_KEYreturns 401. E2E Benchmarks intentionally waits for a human approval in the protectedbenchmarksenvironment.Dependency order: merged mlx #4 -> merged mlx-swift #11 -> merged mlx-swift-lm #102 -> this PR.
What ships
.autoserving resolves contiguous at B=4, and the Gemma benchmark defaults to explicit contiguous in every phase. Paged remains an explicit opt-in with blocking safety and parity coverage.provider.tomlbefore first Metal access; malformed or unreadable existing configs fail loudly.darkbloom updatefalls back only for a genuine missing-file read error and propagates permissions or other failures from existing paths.Before / After: behavior
flowchart LR subgraph Before A1[stock provider] --> B1[contiguous serving] A2[Gemma benchmark with no flag] --> B2[explicit paged measurement] A3[unreadable update config] --> B3[silent default fallback] end subgraph After C1[stock provider] --> D1[contiguous serving at B=4] C2[Gemma benchmark with no flag] --> D2[explicit contiguous in every phase] C3[unreadable update config] --> D3[loud read failure] endBefore / After: code
flowchart TD subgraph Before S1[Start / Benchmark] --> E1[independent environment handling] R1[gemma_contbatch config] --> P1[DEFAULT_KV_BACKEND paged] U1[Update.run catch readFailed] --> F1[loadDefault for every read error] end subgraph After S2[ServeRuntimePreparer] --> E2[GemmaOptimizationEnvironment] R2[gemma_contbatch config] --> P2[DEFAULT_KV_BACKEND contiguous] U2[loadUpdateConfig] --> F2[fallback only for Cocoa/POSIX missing-file] endVerification
go test ./...passes; exact CIgolangci-lint v2.1.6reports 0 issues.f1d63cbba..auto = contiguoussmoke.Performance evidence
The clean attribution epoch measured the production CBv2 path at 2K TTFT 1715 -> 1390 ms, arrival TTFT 1.41-1.76x, arrival E2E +18-25%, and decode +1.3-2.3% on an M4 Max.
The final reviewed tree used one binary, one source-matched metallib, one model/workload, explicit contiguous KV, and typed ON/OFF provenance. Correctness passed; late timing is not promoted because the host drifted during the bracket. The bracket still found and fixed EOS-shortened cells and weighted reduction leaking into ordinary direct prefill.
Rollback
Release boundary
This PR prepares code only. It does not tag, release, register a provider release row, or deploy production. Those remain separate human-approved operations.