fix(chaos): bound the recovery kill point by construction, not by a race

[devarismeroxa]

Tier 3 — test harness only. Confined to tests/chaos; no engine or data-path code touched, and
the holdAt seam is inert unless a test sets it. Unblocks #2832 and #2834, since
tests/chaos (race, x3) is a required context on main.

The failure

TestSIGKILL_RecoveryLoop_CrashDuringRecoveredRun failed on #2832's CI
(run 32792092690):

recovery_test.go:300: not true: committedAtKill < cfg.total

That is a vacuity guard firing correctly. The test asserts the SIGKILL landed mid-run; it hadn't,
because the child committed all 60 records before dying, so the test refused to pass while exercising
nothing. The guard is the good part — this PR fixes what made its precondition false, and does not
touch the guard.

Root cause

waitForReadCount is a lagging, parent-side observation of a free-running child. The parent's
stdout reader goroutine and its 1ms poll loop are both subject to descheduling, so "I last saw 30
READ lines" says nothing about where the child is now.

The margin is ~0.3s, not the "orders of magnitude" a sibling scenario's comment claims: at 30 READs
the child is at position 30, and 30 records x 3ms + the 10ms lcPersistDelay debounce is ~150ms to
finish the whole run. Injecting a stall between waitForReadCount and sigkill reproduces CI's
assertion verbatim at 400ms; 200ms and 120ms both pass.

The fix

1. holdAt production ceiling on recoverySourcePlugin.produceLoop. Once position holdAt (20)
   is sent the child prints LC_HELD and stops for good, so it can never reach total (60). Inert
   at 0, which every other scenario leaves it at. parseLCChildEnv rejects graceful && holdAt>0
   and holdAt >= total — both would otherwise surface only as opaque timeouts.
2. Kill gated on the durable committed watermark, read from the same on-disk marker the
   assertion reads. upstreamStore.Commit is write -> fsync -> atomic rename, so a cross-process
   reader never sees a torn value.

Why this is deterministic rather than tuned: lower bound observed, upper bound structural. The
first run provably never sends failAt (produceLoop closes the stream instead), so a watermark
past it is positive proof the recovered run is producing. The ceiling means nothing past 20 is ever
produced, so nothing past 20 can ever be committed — however long the parent is descheduled. The
kill point is bracketed in [10, 20] and every value in that interval yields the identical
verdict, so killAfterCommitted tunes in-flight depth, not pass/fail.

Rejected: enlarging total (tuning, not a bound). Rejected: gating only on committed count — fixes
the lower bound but leaves the upper bound racing, since a stalled parent can observe committed >= 5
while the child is at 60.

This is the same shape as #2534's fix already in this package: sigkill_test.go's mid-snapshot
case had its precondition made structurally true via a 600s persister debounce rather than a wider
margin. Applied here to the production side.

The test still tests what it says: the kill fires at watermark 10 while the producer is still on its
way to 20, so records are genuinely in flight. committedAtKill > cfg.failAt-1 remains meaningful
and is now guaranteed. Added committedAtKill <= cfg.holdAt so removing the cap fails
deterministically instead of flaking back to life.

Proof

	result
injected 400ms stall, pre-fix	FAIL, assertion identical to CI
injected 3s stall (7.5x), post-fix	10/10 pass
soak, 24 CPU spinners, load avg 61	100/100
same soak, pre-fix	20/20 green
full tests/chaos -race	green, run 3x

That pre-fix 20/20 is the honest number: brute-force repetition cannot measure this flake locally
— it is rare on CI and nightly has been green 8 days. Repetition counts are not the evidence; the
bound is. Hence the deterministic reproduction.

Regression test TestRecoveryChild_HoldAt_CapsProductionBelowTotal waits for the ceiling, stalls
400ms (the exact window that reproduced the flake), and asserts the child has not moved. Verified to
fail without the fix — the LC_HELD wait times out and diagnostics print READ 1 … READ 60. That
sleep is adversarial, not load-bearing: enlarging it can only make an unsound cap fail harder, the
inverse of a masking sleep. Documented at the constant.

Independently verified before merge: target test and regression test 5/5 each under -race, full
tests/chaos green under -race.

Known-unfixed sibling

tests/chaos/property2_test.go carries the same unsound pattern across all three cases and both
prune classes (6 subtests) — kill-point vacuity guards racing a free-running child, and unlike
sigkill_test.go it never got persistDelayMS. With a 1.5s stall injected, mid-handoff and
mid-position-write fail, and mid-snapshot hangs for 41.6s rather than failing fast — the
starvation hang sigkillCase's own doc comment describes. On a required check that is a 40s stall
per subtest.

Filed separately rather than folded in, per the small-PR rule. fanout_sigkill_test.go:65 and
nsource_sigkill_test.go:102 also gate on waitForReadCount, but their post-kill assertions do not
depend on the child being mid-run — they degrade to silently weaker coverage, not to red.

Caveats

• The flake was never observed naturally on the dev machine, under contention or otherwise. Every
  reproduction used an injected stall. The assertion, the margin arithmetic and the READ 1…60
  diagnostics all line up with what CI hit, but it was not caught in the wild.
• In-flight depth at the kill is typical, not guaranteed: a badly starved runner could drain all 20
  before the signal lands. The assertions still hold (40 records were never produced, so it is still
  a mid-run crash) — only the depth of the invariant-1 window degrades, never the verdict.
• Developed on darwin; CI runs Linux.

Roadmap: v0.20 WS9-B (flaky tests fixed at cause, never masked).