feat: reader-thread affinity clamp + distributed-load hardening & public API - #31
Open
AlperenKonukbay wants to merge 3 commits into
Open
feat: reader-thread affinity clamp + distributed-load hardening & public API#31AlperenKonukbay wants to merge 3 commits into
AlperenKonukbay wants to merge 3 commits into
Conversation
Two independent fleet-facing improvements, both measured on real registry apps and multi-GPU runners (receipts in the PR body): Reader-thread affinity clamp - effective_read_threads(): reader-pool sizes clamp to the process's CPU affinity mask. Prod runners execute in dedicated cpusets (measured: 12 CPUs exposed where the default requested 16 threads); oversubscribed byte-copy threads measurably hurt (-18% read wall on a 38GB pack read, 1.70s vs 2.07s on a 12-cpu H200 runner; app-level: id-lora request 4.9% faster with byte-identical output). FLASHPACK_NO_THREAD_CLAMP=1 restores the raw request. Distributed (rank0-read + broadcast) load hardening + public API - _broadcast_storage now broadcasts uint8 VIEWS of the macroblocks: the collective moves bits, and torch's NCCL dtype map lacks float8_e8m0fnu (the mxfp8 scale dtype; gloo lacks all float8s) — native-dtype broadcast of quantized packs crashes. Byte views are dtype-agnostic and copy-free (verified byte-exact on real NCCL across 8 GPUs). - read_flashpack_file_distributed(): the rank0-read + broadcast load as a public storage-level API (assign_from_file's use_distributed_loading branch now delegates to it), so low-level callers — falcon's quantized loaders are the motivating case — can opt in with one line. Measured on 8x H200: all-ranks-read 25.8s (page-cold) -> broadcast 2.44s on all eight ranks; clear errors for no-process-group and NCCL+CPU. - first tests for the distributed path (2-rank gloo, CPU-only): byte-view broadcast incl. float8 blocks, end-to-end assign_from_file, and the new public API. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
Author
|
@fal-9000 review |
Contributor
|
Review by fal-9000 (GPT-5.6 Sol, xhigh) — reviewed at head No issues found. Checked:
Checks were green at review time. |
…d reads regress when clamped below it Pre-merge regression gate (same-node interleaved A/B, 12-CPU H200, 38 GB pack, parity-checked): clamping the raw file->GPU path to the affinity mask starved the IO queue at every tier -- cold 11.3-11.9 GB/s at 12 threads vs 13.2-14.2 at 16 (R=1.18), 2x on a 6-CPU cpuset, and 1.19x even page-hot. Reader threads on that path are mostly blocked in pread: they are the IO queue depth, not CPU consumers, so the affinity budget does not apply. Oversubscribed requests still get capped (64 threads on 12 CPUs measured unstable: 1.9-4.6 s swings vs stable 3.1). effective_read_threads gains a floor kwarg: the GPU path floors the clamp at the 16-thread default (cap = max(affinity, default)); the CPU-destination eager path keeps the pure affinity clamp -- its work is minflt/memcpy-bound where oversubscription genuinely never helps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent improvements, both measured on real registry apps and multi-GPU runners.
Reader-thread cap (oversubscription only)
Reader-thread requests on the raw file->GPU path are capped at
max(cpu affinity, 16); the CPU-destination eager path keeps a pure affinity clamp.FLASHPACK_NO_THREAD_CLAMP=1restores the raw request.This section originally shipped a pure affinity clamp on all paths. A pre-merge regression gate (same-node interleaved A/B, 12-CPU H200, 38 GB pack, parity-checked) showed that clamping the raw path below the default starves the IO queue — reader threads there are mostly blocked in
pread, so they are queue depth, not CPU consumers: 12 threads read 11.3–11.9 GB/s cold vs 13.2–14.2 at 16; on a 6-CPU cpuset the gap was 2x. The clamp now floors at the default and only caps oversubscribed requests, which is the case it measurably helps: 64 threads on 12 CPUs ran unstable (1.9–4.6 s swings) while capped-to-16 was 27% faster than unclamped-64 and steady. Re-receipt on the fixed head, same node: R = 1.04/1.00/1.00 across cold-native, 6-CPU, and hot cells (pass lines 1.05/1.10).Distributed load: hardening + public API
The rank0-read + NCCL-broadcast path runs in several production apps but had no tests and two latent issues, and was unreachable for low-level callers.
_broadcast_storagebroadcastsuint8views. Torch's NCCL dtype map lacksfloat8_e8m0fnu(the mxfp8 scale dtype), so native-dtype broadcast of an mxfp8 pack cannot work (latent failure, found by inspection rather than a production crash); byte views are dtype-agnostic and copy-free. Verified byte-exact over NCCL on 8 GPUs, including the float8 dtypes.read_flashpack_file_distributed();assign_from_filedelegates to it. This lets low-level callers (falcon's quantized loaders) opt in — today every rank of a world-size-N app re-reads the full pack, since the O_DIRECT reader shares nothing through the page cache.assign_from_file.Measured on the real 38 GB pack, 8× H200: all-ranks-read 25.8 s max/rank page-cold (4.89 s page-hot) → rank0+broadcast 2.44 s on all eight ranks (2.22 s hot); at world=2 the cold win is ~−25%. Cross-rank parity verified on every run. Cold-tier magnitude is node-dependent (rank self-contention on one node's disk); the mechanism — N duplicate O_DIRECT reads collapsed to one — is scale-invariant.
Review notes