test(lifecycle-poc): wait for the live status on the recovery path, don't instant-read it

[devarismeroxa]

Tier 3 (test files only — no production code touched).

Fixes the TestServiceLifecycle_Recovery_TransientErrorRecovers failure seen on main at
run 32785888309
(service_test.go:778: Running != Recovering). The commit that failed was docs-only and the next
commit passed, so this is a flake, not a regression.

Root cause

Neither watermark the test waits on is ordered after the status write it then asserts.

statusRecorder.UpdateStatus appends the status and runs its hook before delegating to
pipeline.Service, which is what actually calls Instance.SetStatus. So waitForRecovered's
"Running after a Recovering" fires strictly before pl.GetStatus() can return Running, with a
whole UpdateStatus (a Get, two metrics updates, a store write) still in flight.

waitForRecordsAcked adds no delay either: runPipeline does close(registered) — releasing every
worker goroutine — before it calls UpdateStatus(StatusRunning), so the recovered run can read,
write and ack its whole record set while the status write is still pending.

The identical instant assertion is sound on the initial start (and in pkg/lifecycle) because
ls.Start returns only after runPipeline's status write completes. A recovery restart runs on
the tomb's cleanup goroutine, and nothing in the test goroutine is synchronized with it. The
assertion was passing on a ~microsecond margin that a 4-core CI runner under -race -shuffle=on
loses to a single goroutine preemption.

Ruled out: this is not a second recovery. Both dispensers are .Times(2) with no over-call
reported (a second recovery needs a third dispense), and the three missing Teardowns are the
recovered run's own plugins — run 2 was alive and healthy when is.Equal called FailNow.
Everything after line 778 is cascade.

Fix

waitForRecovered now also polls the live pl.GetStatus() until it reports Running, so the helper
means what its name says. A genuine second recovery is still caught: the helper does not latch past
it, and the AC-7 exact-sequence assertion would fail on [Running Recovering Running Recovering …].

Four sites carried the same unsound pattern; all are fixed. The fourth
(TestServiceLifecycle_Recovery_LiveEntryPublishedBeforeRunningStatus) has no waitForRecovered
call and was the most exposed, because close(release) only lets the status write proceed while the
records it then waits on were already acked during the freeze.

Perturbation proof

Could not reproduce naturally on a 16-core M-series box: 3840 runs of the race binary
(24x40 at -cpu=1,4; 64x30 at GOMAXPROCS=1) all green. So the preemption was injected directly — a
sleep in statusRecorder between recording the status and delegating (injection not committed):

injection	before	after
200ms, 4 tests, -race -count=10	40/40 FAIL (byte-identical to CI)	40/40 PASS
20ms, same 80-run matrix	46/80 FAIL	0/80 PASS

Package green at -race -count=3 -shuffle=on; golangci-lint reports 0 issues.

Relationship to prior fixes

Not a recurrence of the #2746/#2806 production bugs — those were about runningPipelines holding
a dead run, and 4ec2b5f is intact. It is the same class as the async assertion fixed in 34bf97d,
and that commit is where the fourth site was introduced: the PR that fixed one async assertion
shipped a new one of the same kind. waitForRecovered itself dates to a61d4bc (#2718).

Not fixed here

A Tier-1 production ordering finding surfaced during this work and is filed separately rather than
folded in — see the linked issue. It needs its own PR with a deterministic regression test.

Roadmap: v0.20 WS9-B (flaky tests fixed at cause, never masked).