docs(findings): record the #147 levers that did not pay, and the benchmark that showed it

[cjfields]

Docs and diagnostic tooling for the second half of #147, now closed. No
production code changes — examples/parallel_overhead.rs plus the findings
write-up.

#148 merged the lever that paid (e_minmax into a dense array, −20 to −21% of
run_dada). This records the two that did not, and one hypothesis that turned
out to be false — plus the benchmark that settled both, since the findings page
cites it.

What the write-up records

The allocation levers are rejected, with the mechanism confirmed. Removing
b_compare_parallel's per-call allocation — either by shrinking the element
under glibc's 32 MB mmap cap (lever A) or by reusing one buffer (lever D) —
collapses sys by 70–94% and returns nothing: wall +2.5%, user +16%, total
CPU up. parallel_overhead isolates the cost at ~5 ms/call above the cap and
finds no consistent difference below it, and the arithmetic closes against
production's sys figure. That time simply was never on the critical path.

The section says explicitly not to retry either lever on the strength of the
sys number, and leaves both branches unmerged for whoever wants to explain the
user rise with hardware counters.

Load imbalance in the map is falsified. The 86–90% parallel efficiency
looked like tail stragglers; work-matched uniform-vs-skewed arms measure ±0.1%
on both pool shapes. This retires a claim on the existing page that read that
figure as 1,166 core-seconds of idle — busy sums timers taken inside the map
closure and never measured rayon's task dispatch or the collect's stores, so the
gap is mostly unmeasured work.

The part worth reviewing

A section on three instrument errors, because all three returned clean-looking
nulls:

1. A rayon spin-wait hypothesis killed by accounting rather than node time —
   the run had 838 core-seconds of CPU outside busy against a 1,211-second rise
   to explain.
2. A skew model whose period matched the task grain, so mean task cost equalled
   max by construction. It reported "no imbalance" and would have done so
   whatever the truth was.
3. A per-round-minimum estimator applied to the one arm whose variance is
   intrinsic (allocator state) rather than contention, which made the
   allocation cost look like it shrank as the vector grew. On medians it is
   5.04 vs 5.10 ms — flat, as a page-fault cost should be.

The rule they share, and the reason the section exists: a null is evidence
only once the instrument has been shown capable of returning something else.

The benchmark

examples/parallel_overhead runs six arms as two matched pairs — fresh vs
reused destination, and work-matched uniform vs skewed cost. Defaults follow
production rather than round numbers (--skew 14 is the measured
aligner/screen ratio, --heavy-frac 0.008 is ITS2's pass rate, --base 430
puts the arms at ~2,100 core-ms/call against production's 2,086).

It is built to survive a shared node: arms interleave round-robin so contention
drift is common-mode, and a med/min line reports how quiet the node was. Both
min and median print for every comparison, with a comment explaining which to
read and why they disagree.

mkdocs build --strict clean; clippy clean.

[cjfields]

Updated with the verbose B-vs-D run (soil ITS2, two reps, 180/180 byte-identical).

Corrected: the read-for-ownership account is falsified. busy rises +16.0% on
R2, matching the +16.4% user rise measured non-verbose — so the cost is inside
the map closure. But busy times the closure body, and the write into the
destination vector happens in rayon's collect after the closure returns, so a cost
on the destination store could never have appeared there. I wrote the confirming
prediction against an instrument that could not observe the thing it predicted.

What replaces it is narrower and more general: reused memory is slower than
freshly-faulted memory for this workload, across two independent implementations
and two buffer sizes — lever A reuses 15 MB via glibc's arena for +11%, lever D
reuses 45 MB explicitly for +16%. Size modulates the penalty, it doesn't cause it.
Worth recording beyond this issue, since hoisting an allocation out of a loop is a
routine instinct and here it costs more than it saves. The LLC-contention candidate
is labelled a hypothesis with the instrument named (hardware counters, not another
timer).

Half the map-efficiency residual is now attributed. Removing the allocation
raises map parallel efficiency 87–88% → 93–94% with busy unchanged or higher, so
~6 of the missing 10–14 points was allocation and page-fault work — inside map wall,
outside the per-item timers. The remaining ~6 points is still open.

New section: the run is not uniform. Per-window progress lines (#150) show
occupancy climbing 22 → 42 effective cores of 64 over a single run, so the
end-of-run mean of ~29 describes no window of it. The ramp is the serial fraction
shrinking, not the workload changing — align falls six-fold and rises again while
occupancy climbs straight through. This matters for how the project measures:
"effective cores" and "map parallel efficiency" have both been used to rank levers,
and both are means over a run that spans nearly a factor of two.

D's verdict is unchanged: −3.9% on R1, +6.5% on R2, not merged.