Thread sweep: tooling, symmetric --numa, tuning guide, and findings (#152) - #156
Merged
Conversation
Replaces the ad-hoc Python that has been pasted at these logs all session, and encodes the one thing that must not be got wrong. Segments are aligned by CLUSTER INDEX, never by wall time. At the same t=, a 96-thread run and a 48-thread run sit at different cluster positions, and the serial fraction collapses over a run -- so fixed wall-clock windows compare different phases. Doing exactly that produced a confidently reported conclusion with the sign of the trend inverted before it was caught. Reports four things: - run_dada per thread count, with replicate spread beside every mean, and an explicit note when an arm has one replicate that differences under ~2% are not distinguishable from run-to-run variation. - The serial block (store+shuffle+bud+p_update), which is invariant in seconds and rising in share -- 16S R1 is 208.5s at 48 threads and 207.3s at 96, so map work walks toward the Amdahl wall rather than away from it. That is the figure to quote when weighing bandwidth work (#28) against serial work (#154). - Marginal seconds per cluster segment, showing where in the run the benefit or penalty lands. - The map's screen/align split, which predicts the knee. On that last point the calibration is printed rather than a threshold, because four arms is not a rule: 50.8% screen still scales at 96, 67.7% turns over near 64, and 83-86% turns over below 48. It predicts WITHIN a pool as well as across -- 16S's two reads differ by 17 points of screen share and have different knees -- so it tracks arithmetic intensity, not the dataset. The thread numbers are specific to the EPYC 7713; the ordering is what travels. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two corrections, both to errors the 24-thread ITS2 arm exposed. ITS2's knee is AT 48 threads, not below it. From 48/64/96 alone the pool looked monotonically degrading and was reported as "already past the knee below 48". Adding 24 shows 48 is the peak, beating 24 by 12-17% on wall. A sweep that does not bracket the peak cannot locate it, and reading a trend off three points on one side of a maximum is how that happened. The screen/align split is itself THREAD-DEPENDENT: ITS2 R1 reads 80.0% screen at 24 threads and 85.8% at 48, because the bandwidth-bound half degrades faster under contention. The script took the composition from the lowest thread count in the sweep, so adding 24 to ITS2 silently made its number incomparable to 16S's (still measured at 48) -- in the same table that presents them as a calibration. Now pinned via --ref-threads (default 48), with a warning when that count is absent from a sweep. The calibration ordering survives both corrections: 50.8% screen -> best at 96, 67.7% -> 64, 83.2% and 85.8% -> 48. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The #152 sweep says every arm buys wall time at ~4x the core-seconds -- 16S R1 goes 1072.5s at 24 threads to 594.3s at 96, 45% parallel efficiency at the top -- which implies "pack jobs, don't scale threads". Nobody has measured two jobs sharing a node, so that is currently an inference and not a result. The two jobs would contend for exactly the memory bandwidth the k-mer screen is already limited by. Three arms. The interesting one is NUMA policy, because it may invert an existing finding: dev/numa_pin.sh documents that binding to one domain is the wrong choice for a single job, costing the parallel map 21% by forcing every thread through one node's controllers. With two concurrent jobs on a 2-domain node that reasoning may reverse -- each job gets private controllers and neither disturbs the other, where interleaving both has them share every controller. Not predictable from the single-job result, so both are measured. Runs the SAME read twice rather than R1+R2. Pairing 16S R1 (713.8s) with R2 (767.8s) at 48 threads lets the shorter job finish first, leaving the longer one's tail uncontended -- which flatters the pair and blurs the contention being measured. Times via epoch timestamps rather than /usr/bin/time, which is not installed on the benchmark node. Drops the split arm automatically on a single-domain machine rather than reporting a meaningless comparison, and reminds the operator that outputs must be byte-identical across arms -- concurrency and NUMA policy change timing only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ITS2 R1 came back with the split arm running each job 28% FASTER than a job running alone (141.0s vs 196.9s run_dada), giving 2.79x throughput and better per-job latency at the same time. The gain shows up in every phase, not just the map -- map -29%, store -23%, shuffle -27%, p_update -35% -- which is the signature of memory locality rather than bandwidth: under --interleave=all roughly half of every access is remote (distance 32 vs 10 on this node), and the single-threaded serial phases have nothing to hide that latency behind. But split_a was still sharing the node with split_b, so locality and packing are confounded. The sbind arm runs ONE job bound to one domain. If it lands near split, the entire gain is locality and packing is incidental; if it lands near solo, the gain needs the second job. This matters beyond the packing question: dev/numa_pin.sh recommends --interleave=all over binding, on a measurement showing binding cost the map 21%. That was 64 threads on PacBio, where 64 threads bound to a 64-core domain saturates it and 48 leaves headroom -- so the guidance may be thread-count-specific rather than wrong. Every ITS2 number published so far was measured under that policy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Answers "how many threads should I use, and does pinning memory help" from measurement rather than from "use all your cores" -- which costs 5% of wall on one benchmark pool and leaves 45% on the table on another. Keyed to the screen/align split the verbose log already prints, rather than to dataset names, so it is usable by someone whose data is neither of ours. The screen streams per-sequence vectors and is memory-bound; the aligner does dense arithmetic and is core-bound; the ratio predicted the thread knee across four arms (50.8% screen -> scales to 96, 67.7% -> 64, 83-86% -> 48). Records what moves that ratio, since it is the actionable part: amplicon and diversity dominate (16S and ITS2 here came from the SAME run and samples, so that gap is biology, not instrument), but read position moves it 17 points within one library and changes the recommended thread count. Separates the latency question from the throughput one -- every arm buys wall time at ~4x the core-seconds, so a queue wants fewer threads and more jobs while a single urgent sample wants the table. The NUMA section is marked provisional: it is soil ITS2 only, pending the 16S runs, and we expect a smaller effect there because compute-bound work has arithmetic to hide memory latency behind. Publishing a bind-instead-of- interleave rule from one pool would repeat the mistake that put the opposite rule in dev/numa_pin.sh, which came from a single PacBio measurement at a thread count that saturated one domain. Ends with an explicit list of what has not been tested, and an invitation to contradict it -- four arms on one machine is a summary, not a law. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The heading starts with a backticked `--verbose`, which slugifies to a leading single hyphen. Read the generated id out of the built HTML rather than guessing the slug a third time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nditional 16S R1 gains 25% from binding one job to one NUMA domain, against ITS2's 27-29%. We expected the compute-bound pool to gain much less, since arithmetic can hide memory latency; it did not. A 35-point gap in screen share moves the thread-count recommendation by a factor of two but barely moves this, so the tuning guide now presents NUMA binding as the one recommendation that does not depend on the data. 150/150 outputs byte-identical, and every phase gains including the single-threaded ones (16S: map -27%, store -22%, shuffle -21%, p-update -31%). Also annotates dev/numa_pin.sh, which recommends the opposite. Its 21% penalty for binding was measured at 64 threads, where 64 threads bound to a 64-core domain saturates it. At 48 threads, which leaves headroom, the result reverses on both pools. The default is left alone because whether binding is as REPRODUCIBLE as interleaving is not yet measured, and reproducibility is what that helper exists for -- but "interleave beats bind" is no longer safe to read as general, and benchmark numbers gathered under it are conservative in absolute terms at sub-domain thread counts. A/B deltas are unaffected: both arms always shared the policy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Memory placement is worth 25-30% of run_dada on this hardware -- larger than most changes this harness exists to measure -- and the harness had no handling for it at all: both stacks ran through a bare subprocess.Popen. Applied in numa_wrap, which every launch routes through (run_step and run_phase_concurrent), rather than at each call site. That is the whole design constraint: R DADA2 threads the same algorithm through multithread=, with the same k-mer screen and banded NW, so pinning dada2-rs and not R would inflate every speedup number by up to ~25% for free. Wrapping at the convenient call sites is exactly how that would happen by accident. Whether R gains as much as we do is the interesting question and is answerable only with both arms treated alike. Degrades rather than fails, and never silently: - no numactl installed -> unpinned, warning, recorded as numa=none - single NUMA domain -> unpinned, warning, recorded as numa=none - numactl --hardware errors -> unpinned, warning, recorded as numa=none - threads exceed one domain's CPUs -> falls back to interleave with a warning, because bind cannot honour it: at or above a domain's core count binding saturates that domain's controllers and loses to interleaving, which is how dev/numa_pin.sh came to recommend the opposite from a 64-thread measurement on a 64-core domain. summary.csv records the EFFECTIVE policy alongside the requested one in a header comment. A run that asked for bind on a machine without numactl must not be filed as bound -- that is the #145 failure mode, where a gate resolved silently and two full A/B runs compared a configuration against itself. Default is none, so new runs stay comparable to every archived one. Not a recommendation; docs/tuning-for-your-data.md carries that. Verified on this machine (no numactl: all three policies degrade to none and numa_wrap is a no-op) and against a stub numactl reporting 2 domains x 8 CPUs: threads=4 binds, threads=8 binds with a saturation note, threads=12 falls back to interleave. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four arms across two pools now, and the one-job binding gain sits in a 25-29% band on all of them: ITS2 -27/-29%, 16S -25/-27%. A 35-point gap in screen share moves the thread recommendation by a factor of two and barely touches this, which is what makes it the one recommendation in the guide that does not depend on the data. Also sharpens the two-unpinned-jobs observation. The spread across arms is -8% to +6%, and rather than call the 16S read 1 result unexplained, the spread IS the finding: that is the arm where placement is left to the kernel, so an unpinned run's performance depends on where its pages happen to land. A bandwidth shortage would slow both jobs reliably; it does not. Distance to memory, not throughput to it. 150/150 outputs byte-identical across all six arms. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The lab-notebook counterpart to the tuning guide on this branch: what was tested, the numbers, and what each result dictates. Two results with opposite characters. Thread count is strongly data-dependent -- the best count differs two-fold between two reads of the SAME pool -- and is predicted by the map's screen/align split, monotonic across four arms and predictive within a pool as well as across pools. Memory placement is not data-dependent: binding one job to one NUMA domain is worth 25-29% on every arm despite a 35-point gap in screen share, which falsified our prediction that compute-bound work would gain less. The section worth reading is the reconciliation with findings/measuring-on-numa.md, which concluded that binding is "the obvious fix that turns out to be wrong". Both are right, and the earlier page ALREADY CONTAINED the mechanism: its own phase table shows b_shuffle 23% and store 37% faster bound, within a point or two of what we just measured on entirely different pools. The map penalty at 64 threads outweighed them, the run-level total said binding loses, and the phase table underneath went unexamined -- by this author among others. What changes at 48 threads is the map penalty, not the serial gain. That reconciliation is explicitly flagged as confounded: the old measurement is PacBio at 64 threads and the new one NovaSeq at 48, so platform and thread count move together. The clean test is one pool bound at both 48 and 64, named as a prerequisite before numa_pin.sh's default changes. Closes with the four instrument errors from this arc, including comparing fixed wall-clock windows across thread counts -- which inverted the sign of a reported trend before it was caught -- and locating a knee from one side of it. Each produced a plausible, quotable number. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cjfields
changed the title
Member
Author
|
Rolled the #152 findings page into this branch (
Both are right, and the earlier page already contained the mechanism:
Its serial phases were already 23% and 37% faster bound — within a point or two That reconciliation is flagged as confounded, not proven: the old measurement Also closes with the four instrument errors from this arc — including comparing
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tooling and user-facing docs from the #152 thread sweep. No library code —
dev/scripts, the benchmark harness, and two docs pages.What the measurements found
Two results, with opposite characters, which is what the docs are built around:
Thread count is strongly data-dependent. Four arms, two replicates each,
across 24/48/64/96 threads:
The best thread count varies two-fold between reads of the same pool, and it
is predicted by a number the verbose log already prints — the share of map time
in the k-mer screen (memory-bound) versus alignment (compute-bound). 50.8% screen
scales to 96; 67.7% peaks at 64; 83–86% peaks at 48 and degrades past it.
Memory placement is not data-dependent. Binding one job to one NUMA domain:
A 25–29% band across a 35-point gap in screen share. Every phase gains, including
the single-threaded ones (16S: map −27%, store −22%, shuffle −21%, p-update −31%)
— the signature of memory latency, not bandwidth. Two bound jobs give 2.6–2.9×
the throughput of one default job. All outputs byte-identical.
Contents
dev/analyze_thread_sweep.py— replaces ad-hoc analysis of sweep logs.Aligns segments by cluster index, never wall time: at the same
t=, a96-thread run and a 48-thread run sit at different cluster positions in a
workload whose serial fraction collapses over the run. Doing that wrong reported
a trend with the sign inverted before it was caught, so the constraint is encoded
rather than remembered. Prints replicate spread beside every mean and warns when
a single-replicate arm cannot distinguish sub-2% differences.
dev/run_concurrency_test.sh—solo/sbind/both/splitarms.The
sbindarm exists to separate locality from packing, and settled it: onebound job captures ~92% of what two bound jobs get, so the win is locality and
packing is incidental.
bench_pooled.py --numa {none,interleave,bind}— see below.docs/tuning-for-your-data.md— the user-facing guide, keyed to workloadcharacter rather than dataset names so it is usable by someone whose data is
neither of ours. Separates the latency question from the throughput one (every
arm buys wall at ~4× the core-seconds, so a queue wants fewer threads and more
jobs). Ends with an explicit list of what has not been tested.
docs/benchmarking.md§6c —--numafor benchmark operators.The part most worth reviewing:
--numasymmetrybench_pooled.pyhad no placement handling at all — both stacks ran through abare
subprocess.Popen. Since placement is worth 25–30%, a harness that pinsone stack and not the other is not a harness. R DADA2 threads the same
algorithm through
multithread=, with the same k-mer screen and banded NW.So the policy is applied in
numa_wrap, which every launch routes through, notat each call site — wrapping the convenient call sites is exactly how an arm gets
missed and the speedup silently inflates by ~25%.
It degrades rather than fails, and never silently:
numactlnuma=nonenuma=nonenumactl --hardwarefailsnuma=noneinterleavewith a warningsummary.csvrecords the effective policy alongside the requested one. A runthat asked for
bindon a machine withoutnumactlmust not be filed as bound —that is the #145 failure mode, where a silently-resolved gate cost two full A/B
runs. Default is
none, so new runs stay comparable to every archived one.Verified with no
numactlpresent (all policies degrade,numa_wrapis a no-op)and against a stub reporting 2 domains × 8 CPUs (threads=4 binds, 8 binds with a
saturation note, 12 falls back).
dev/numa_pin.shis annotated, not changedIt recommends
--interleave=allover binding, from a 64-thread PacBiomeasurement where binding cost the map 21% — 64 threads bound to a 64-core domain
saturates it. At 48 threads the result reverses on both pools. The default is
left alone because that helper exists for reproducibility, and whether binding
replicates as tightly is unmeasured. But "interleave beats bind" is no longer
safe to read as general, and it now says so where someone would find it.
One consequence stated plainly in the annotation: benchmark numbers gathered
under it are conservative in absolute terms at sub-domain thread counts. A/B
deltas are unaffected — both arms always shared the policy.
Known limitation
filterandremove_primersfan one subprocess per sample viarun_phase_concurrent; under--numa bindthey all pile onto domain 0 whiledomain 1 idles. It does not affect the denoising comparison (
learn-errorsanddada*go throughrun_step), but those two phases' bound timings should not bequoted. Tracked in #155 with two proposed fixes.
Caveat carried into the docs
The thread table was measured entirely under interleaving, so those knees may
shift once memory is pinned. Called out in the guide's "what we have not tested"
section rather than left to be discovered.
Full test suite passes;
mkdocs build --strictclean.