Bound the SQS handler drain on shutdown + de-flake the drain test - #3899
Draft
jayjanssen wants to merge 1 commit into
Draft
jayjanssen wants to merge 1 commit into
jayjanssen wants to merge 1 commit into
Conversation
…top() - Add shutdown_timeout_ms to SqsQueueConfig: an optional per-queue bound on how long doStop() waits for in-progress handlers before cancelling the queue's remaining work. Default null keeps the unbounded join. - De-flake `queues are drained drain`: the Approximate* queue depth attributes are exact in ElasticMQ but eventually consistent on real SQS, so poll until conservation holds instead of asserting on a single read taken right after stop. - Remove the public SqsJobConsumer.stop(): it bypasses the Guava service state machine. Tests use stopAsync().awaitTerminated(). - New integration test for the stuck-handler path, config resolution tests for shutdown_timeout_ms, and the regenerated API dump. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #3895 (now merged) — three small things on top of it. FYI @mateuszmrozewski since this builds directly on your merge; take what you like:
1. Optional bound on the handler drain (
shutdown_timeout_ms, default null = current behavior).doStop()joins the handling jobs with no timeout, so one stuck handler holds shutdown until the pod is SIGKILLed — and a SIGKILL mid-drain loses the acks for every other job that finished after the kill signal would have been graceful. This adds a per-queue opt-in bound in the same shape asshutdown_grace_period_ms: on expiry, log and cancel that queue's remaining work (its messages stay in the visibility window for redelivery, same as today's cancellation). Default null keeps #3895's behavior unchanged. New integration test covers the stuck-handler path (shutdown terminates in ~1s, message left unacked for redelivery).2. De-flaked
queues are drained drain.I ran #3895's branch through a real-SQS harness (against a throwaway staging account — see https://github.com/squareup/jayj-Notebook/tree/main/2026-08/real-sqs-validation): 12/14, and the shutdown logic itself validated cleanly, including the grace-period escalation. But this test's conservation assert read
visible 1 + invisible 1 + handled 999 = 1001when sampled immediately after stop —ApproximateNumberOfMessages*are exact in ElasticMQ but eventually consistent on real SQS. The test now polls the depths until conservation holds (or 20s), then asserts. Same intent, immune to counter lag — and cheap insurance against slow-CI timing too.3. Removed the public
stop().It bypasses the Guava state machine: calling it on a RUNNING service skips the STOPPING transition and listener lifecycle (and depending on Guava version
notifyStopped()may throw), leaving service state inconsistent with reality. Tests usestopAsync().awaitTerminated()instead.The real-SQS run also flagged
retrying worksas timing-tight against real AWS (redelivery vs a 10s latch) — pre-existing on master, so I left it alone here.🤖 Generated with Claude Code