drpcmanager: add soft cancel grace period - #57
Open
mniewrzal wants to merge 2 commits into
Open
Conversation
soft cancel only avoids destroying the transport if the cancel can actually be sent. SendCancel uses TryLock everywhere so that a wedged transport write can never block teardown, which means a stream that happens to be sending when the cancel arrives reports busy and the transport is torn down. for clients that send and receive concurrently that race is common and benign, and paying a dial plus a handshake for it is a poor trade. add Options.SoftCancelGrace, which bounds how long a busy soft cancel will wait for the in-flight send to finish. rather than blocking on the mutex, which would give back the property that lets a dead transport be torn down, retry SendCancel with backoff until the send lands, the grace elapses, or the manager terminates. zero or negative keeps the current behavior. also report the outcome of each soft cancel attempt (clean, busy or error) through a callback in the internal options, so the ratio of kept to destroyed transports is observable in production.
mniewrzal
force-pushed
the
soft-cancel-grace
branch
from
August 9, 2026 19:22
0bac5c0 to
21179d1
Compare
egonelbre
approved these changes
Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Options.SoftCancelexists so that canceling a stream does not cost the whole transport: the manager sends aKindCancelpacket and keeps the connection for the next stream.That only works if the cancel can actually be sent.
Stream.SendCancelneeds the stream's state and write mutexes and takes both withTryLock— deliberately, so that a wedged transport write can never block teardown. The consequence is that a stream which happens to be sending when the cancel arrives reports busy, and the manager hard cancels: transport destroyed, next call pays a dial and a handshake.For clients whose streams send and receive concurrently, that race is both common and benign. Losing the connection over it is a poor trade.
Change
Options.SoftCancelGracebounds how long a busy soft cancel will wait for the concurrent send to finish. Zero or negative preserves today's behavior.Manager.sendCancelimplements the wait by retryingSendCancelwith backoff (100µs, doubling to a 5ms cap) rather than blocking on the mutex. Blocking would give back exactly the property that lets a dead transport be torn down; polling keeps it. The loop also exits early if the manager terminates, since there is nothing left to preserve at that point.CancelClean,CancelBusyorCancelError— is reported through a callback in the internal options, following the existingstatsCBpattern. The ratio of kept to destroyed transports is the difference between a connection pool that works and one that never gets a hit, and it is not otherwise observable in production.Tests
TestSoftCancel_Graceparks a flush inside the transport, waits until the write is actually blocked, cancels the stream's context, and releases the write 50ms later. With no grace the manager reports busy; with a grace period the retry picks the stream up once the send lands and the cancel is delivered cleanly.