feat: loader-aware background prefetch (cattensors, integrated)

[GencSaracaydin]

feat: loader-aware background prefetch (cattensors, integrated)

Why

fal's registry spent June–July bolting an external page-cache prewarmer (cattensors) onto ~50 apps, several of which point it (or vendored copies of its reader) at .flashpack files. That composition is exactly what #28's gate fix made hazardous: an uncoordinated warm makes pages resident, the loader's mincore gate then legitimately picks buffered reads — but the warm itself cost a full extra pass over the file, and if the load starts mid-warm the two race each other for filesystem bandwidth with the gate frozen on a stale answer. Registry apps meanwhile keep a real need the warm was serving: overlapping the pack read with imports/setup work (e.g. qwen-image-2512 vendors a copy of the cattensors reader solely to warm the pack its LoRA manager re-reads on CPU).

This PR moves that capability inside flashpack, where it can cooperate with the loader instead of fighting it.

What

flashpack.prefetch_flashpack_file(path) -> FlashpackPrefetch — a cancellable, chunk-parallel, buffered page-cache warm to start before imports/setup work:

• residency-gated: an already-hot file returns a pre-completed handle, zero I/O;
• idempotent per path (process-wide registry; concurrent calls share one warm);
• advisory by contract: errors are recorded on the handle, never raised — the load performs its own error-checked reads regardless.

The reader settles the prefetch before choosing an I/O path (parallel_read_into_storage): finished → the mincore gate sees a stable hot cache and routes buffered at memory speed; ≥90 % done → bounded wait for the remainder; barely started → cancel (join) and take O_DIRECT cold, which is faster from there. Partial warms and post-warm evictions self-correct because the gate re-checks residency itself. A warm can no longer race a load or flip its gate mid-flight — by construction.

read_flashpack_file(..., eager_cpu=) — per-call choice of eager parallel CPU materialization (True), lazy mmap (False), or the FLASHPACK_CPU_PARALLEL_READ env opt-in (None, default; unchanged behavior). revert_from_file accepts the same knob (default stays lazy — full-pack RSS is the wrong default for streaming consumers like the CLI unpack path; pass eager_cpu=True for full-read consumers, where #28 measured the eager path 2.2× faster cold).

Measurements

H100 fal runners, JuiceFS /data, 24 GB pack, device=cuda, 10 s of simulated setup work, every load bit-verified against checksums computed by a different node (seeder boot_id 5d15f771…, bench e812b261…), cache tiers enforced and labeled with juicefs warmup --evict/--check. (Harness lives in fal's setup-time comparison tooling, not this repo — methodology: scenarios below, cache tiers enforced+verified per-scenario, page-cache eviction via fadvise, mount-cache eviction via juicefs warmup --evict.)

tier (--check before load)	scenario	load call	total incl. 10 s work
net-cold (0 % cached, DC peer holds blocks)	load-only	35.1 s	35.1 s
net-cold	prefetch-overlap	15.4 s	26.0 s
object-cold (0 % cached, no peer → GCS)	load-only	33.2–42.5 s	33.2–42.5 s
object-cold	prefetch-overlap	9.3–11.5 s	20.1–22.2 s
local-cache-warm (fadvise)	load-only	2.26 s	2.26 s
local-cache-warm	prefetch-overlap	1.91 s	12.8 s (≈ work window)
local-cache-warm	settle-cancel	2.32 s	2.67 s
local-cache-warm	load-hot	1.90 s	1.90 s

Reading it as an app would experience it (setup = 10 s of imports + the pack load):

• object-cold cold start: 47.9 s → 21.2 s (2.3×); net-cold: 45.1 s → 26.0 s (1.7×). The 10 s window only warmed ~60 % of 24 GB — the settle policy cancelled the in-flight warm (progress < 0.9) and finished O_DIRECT, and the partial warm still paid: the load call dropped 2.3–4.6×.
• Warm node: parity, not regression. After a completed prefetch the residency gate routes buffered at hot speed (1.91 s ≈ load-hot 1.90 s); a load racing a just-started warm cancels it for a noise-level 0.06–0.35 s.
• The honesty row quantifies the anti-pattern this PR retires: sequential prefetch-then-load on a warm-cache node is 9.5 s vs 2.26 s — a 4.2× worsening. That is exactly what an uncoordinated external prewarmer (cattensors / warm_up_directory) does when it isn't overlapped or gated; the integrated API makes it structurally impossible (residency no-op + settle).
• Side quest that proves the eager_cpu path: the seeding harness itself first tried to checksum the 24 GB pack via the default lazy mmap walk and blew a 55-minute budget; with eager_cpu=True the identical read took 38 s (>85×). That is the capture_original_weights shape.

Real-app validation (post-scenario-bench)

End-to-end A/B on H100 runners where the ONLY difference is one prefetch_flashpack_file() line before real cold imports (torch+transformers+diffusers, fresh interpreter per sample), pack state --evict+fadvise-verified 0% before every run:

• Synthetic 24 GB pack: prefetch won every pairing — e.g. import-symmetric pairs: totals 31.9→27.7 s and 28.1→20.0 s; loads 28.2→23.4 s and 24.4→15.6 s. In every sample the warm was only 12–30 % done when the load arrived; the settle cancelled it and the partially-warmed blocks (now in the mount's local cache) still cut the O_DIRECT load.
• qwen-image-2512's real 25 GB int8 pack: first attempt exposed the SC_AVPHYS_PAGES guard bug (fixed here — on production nodes the page cache keeps free-RAM ≈ 0, silently no-oping every warm). Post-fix: prefetch at parity-to-winning (loads 26.4–33.1 s vs 33.7–36.8 s control).
• Literal before/after on fresh nodes (released flashpack==0.4.0 control vs this PR, real pack, cold venvs, evict-verified): totals 111.4–122.0 s → 64.1 s (~-45 %); loads 56.8–58.1 s → 27.9 s from a 2-thread warm that reached only ~20 % before the settle cancelled and banked it. Imports were not degraded by the gentle warm.
• One catastrophic outlier (never reproduced across 8 further samples): an 8-thread warm overlapping a cold-venv import phase inflated imports 42→189 s — the warm's mount-cache writeback competes with venv reads on the same NVMe. This drove the default down to 2 warm threads (FLASHPACK_PREFETCH_THREADS): measured, 2 threads deliver ≥ the load benefit of 8 with far less collateral I/O.

Tests

• tests/test_prefetch.py — 22 hermetic CI tests + 1 gpu-marked: bit-identity after prefetch, settle policy (cancel-barely-started / wait-nearly-resident / residency-vs-bytes-read / bounded wait-timeout / noop), in-flight end-to-end cancel through a real eager load, prefetch-vs-load race, concurrent prefetch idempotency, fork-child settle, memory guard, symlink aliasing, error-is-advisory, hot-file no-op, eager_cpu plumbing incl. kill-switch downgrade, eager/lazy revert equality.
• Full suite green on ubuntu (py3.10–3.12) and windows (py3.12); adversarially reviewed (fork/race/timeout findings fixed pre-merge).

Registry follow-ups this unlocks (separate PRs)

• qwen_image_2512.py deletes _warm_file_parallel_chunked / _start_weights_page_cache_warm (vendored cattensors) → prefetch_flashpack_file(transformer_pack) at setup start + eager_cpu=True in capture_original_weights.
• wan_vace / editto drop their git-pinned cattensors ("flashpack-read patch") in favor of the native prefetch.
• The improve-setup-time skill's C4 row gets a supported, gate-safe answer for pack prewarming.

🤖 Generated with Claude Code

https://claude.ai/code/session_016ZBPf2mGKiK6P6WjN785SK

[GencSaracaydin]

@codex review
@claude review
@cursor review

[GencSaracaydin]

@codex review

[GencSaracaydin]

self review: should remove the benchmarking references as a seperate harness will be handling that

[GencSaracaydin]

There are windows fallback features (like parallel_read available which are completely unnecessary, remove those)

[GencSaracaydin]

Addressed across three commits:

• b326532 removed the behavioral fallbacks (preadv-only reads, no non-POSIX no-op branch, dropped the non-POSIX test)
• 9ff6549 gentle 2-thread warm default from the production A/B
• 188c7dc removes parallel_read_available entirely — no platform gating anywhere; parallel_read_supported and the eager_cpu override consult only the FLASHPACK_PARALLEL_READ kill switch

The one platform-conditional left is the hasattr(os, "register_at_fork") guard, kept strictly as import-safety: flashpack/__init__ imports the prefetch module, and the repo's packing side still runs windows-latest CI (#17) — without it import flashpack crashes there. It guards the import, not behavior; happy to drop it too if the Windows CI job is on its way out.