Skip to content

test(service): assert circuit-breaker parallelism by overlap, not by stopwatch - #4106

Open
aryanmehrotra wants to merge 2 commits into
developmentfrom
fix/circuit-breaker-timing-flake
Open

test(service): assert circuit-breaker parallelism by overlap, not by stopwatch#4106
aryanmehrotra wants to merge 2 commits into
developmentfrom
fix/circuit-breaker-timing-flake

Conversation

@aryanmehrotra

Copy link
Copy Markdown
Member

TestCircuitBreaker_MixedHTTPMethods and TestCircuitBreaker_ParallelExecution both proved "these requests ran in parallel" by timing them: the handler slept one second, five requests were fired, and the test asserted the total came in under two (respectively four) seconds.

That is a race between the implementation and the machine.

It fails when it shouldn't. The bound sat one second above a one-second floor, so a loaded runner fails a circuit breaker that is behaving perfectly. Caught in the act during a parallel local sweep:

circuit_breaker_test.go:1116:
    Error:    "2.021348s" is not less than "2s"
    Messages: Different HTTP methods should execute in parallel

And it passes when it shouldn't. A serialized-but-fast implementation sails through both, because elapsed time cannot distinguish concurrent from quick. The 4s bound on ParallelExecution is loose enough to admit a lot of serialization.

What replaces it

A barrier the handler blocks on. Every request waits until all five are inside the handler at the same instant — reachable only if the client dispatched them concurrently, unreachable if it did not, at any speed. A request whose peers never arrive answers 503 and the test names it:

Request 3 did not overlap the others: requests were serialized, not parallel

No wall clock is consulted, so there is nothing left to tune per machine.

The barrier's timeout is a liveness bound, not a performance one — it exists only so serialization fails as an assertion instead of hanging until go test's own deadline. What it has to cover is five goroutines each opening a localhost connection, so 15s is roughly four orders of magnitude of headroom, and it is spent only on a run that was going to fail anyway. The first waiter to time out releases the others, which keeps a failing run at one timeout rather than five (120s → 15s) and holds it inside the 5-minute CI step budget.

Verified in both directions

A test that no longer fails is only half the job — the other half is proving it still can.

Result
Healthy, -race 10/10 pass
Healthy, runtime ~2s → ~0.26s for the pair (the two 1s sleeps are gone)
Regressed both fail in 15s, with the "serialized, not parallel" message
golangci-lint v2.12.2 on pkg/gofr/service/... 0 issues
Full pkg/gofr/service package pass ×3

The regression injected was moving cb.mu.Lock() above the f(ctx) call in executeWithCircuitBreaker — holding the lock across the request, which is the exact mistake the "try recovery without holding lock" comment there warns against. The old stopwatch assertions caught it too, but only because the handler slept a second; they would not have caught a fast one.


Found while verifying #4105 (Go 1.27), but unrelated to it — the flake reproduces on 1.26, so it is split out to keep that PR to the upgrade.

…stopwatch

TestCircuitBreaker_MixedHTTPMethods and TestCircuitBreaker_ParallelExecution
both proved "these requests ran in parallel" by timing them: the handler slept
one second, five requests were fired, and the test asserted the total came in
under two (respectively four) seconds.

That is a race between the implementation and the machine. The bound sat one
second above a one-second floor, so a loaded runner fails a circuit breaker that
is behaving perfectly -- MixedHTTPMethods returned 2.021348s against its 2s
bound during a parallel local sweep. It is also weak in the other direction: a
serialized-but-fast implementation would have passed both, because elapsed time
cannot distinguish "concurrent" from "quick".

Replaced with a barrier the handler blocks on. Every request waits until all
five are inside the handler at once, which is reachable only if they were
dispatched concurrently and unreachable if they were not, at any speed. A
request whose peers never arrive answers 503 and the test names it. No wall
clock is consulted, so there is nothing left to tune per machine.

The barrier's timeout is a liveness bound, not a performance one -- it exists
only so serialization fails as an assertion rather than hanging until go test's
deadline. The first waiter to time out releases the others, which keeps a
failing run at one timeout instead of five and holds it inside the CI step
budget.

Verified in both directions. Healthy: 10/10 passes under -race, and dropping the
two one-second sleeps takes the pair from ~2s to ~0.26s. Regressed: moving
`cb.mu.Lock()` above the `f(ctx)` call in executeWithCircuitBreaker -- the exact
mistake the comment there warns against -- fails both tests in 15s with
"requests were serialized, not parallel".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant