Skip to content

drpcmanager: add soft cancel grace period - #57

Open
mniewrzal wants to merge 2 commits into
mainfrom
soft-cancel-grace
Open

drpcmanager: add soft cancel grace period#57
mniewrzal wants to merge 2 commits into
mainfrom
soft-cancel-grace

Conversation

@mniewrzal

Copy link
Copy Markdown
Contributor

Problem

Options.SoftCancel exists so that canceling a stream does not cost the whole transport: the manager sends a KindCancel packet and keeps the connection for the next stream.

That only works if the cancel can actually be sent. Stream.SendCancel needs the stream's state and write mutexes and takes both with TryLock — deliberately, so that a wedged transport write can never block teardown. The consequence is that a stream which happens to be sending when the cancel arrives reports busy, and the manager hard cancels: transport destroyed, next call pays a dial and a handshake.

For clients whose streams send and receive concurrently, that race is both common and benign. Losing the connection over it is a poor trade.

Change

  • Options.SoftCancelGrace bounds how long a busy soft cancel will wait for the concurrent send to finish. Zero or negative preserves today's behavior.
  • Manager.sendCancel implements the wait by retrying SendCancel with backoff (100µs, doubling to a 5ms cap) rather than blocking on the mutex. Blocking would give back exactly the property that lets a dead transport be torn down; polling keeps it. The loop also exits early if the manager terminates, since there is nothing left to preserve at that point.
  • The outcome of each soft cancel attempt — CancelClean, CancelBusy or CancelError — is reported through a callback in the internal options, following the existing statsCB pattern. The ratio of kept to destroyed transports is the difference between a connection pool that works and one that never gets a hit, and it is not otherwise observable in production.

Tests

TestSoftCancel_Grace parks a flush inside the transport, waits until the write is actually blocked, cancels the stream's context, and releases the write 50ms later. With no grace the manager reports busy; with a grace period the retry picks the stream up once the send lands and the cancel is delivered cleanly.

soft cancel only avoids destroying the transport
if the cancel can actually be sent. SendCancel
uses TryLock everywhere so that a wedged transport
write can never block teardown, which means a
stream that happens to be sending when the cancel
arrives reports busy and the transport is torn
down. for clients that send and receive
concurrently that race is common and benign, and
paying a dial plus a handshake for it is a poor
trade.

add Options.SoftCancelGrace, which bounds how long
a busy soft cancel will wait for the in-flight
send to finish. rather than blocking on the mutex,
which would give back the property that lets a
dead transport be torn down, retry SendCancel with
backoff until the send lands, the grace elapses,
or the manager terminates. zero or negative keeps
the current behavior.

also report the outcome of each soft cancel
attempt (clean, busy or error) through a callback
in the internal options, so the ratio of kept to
destroyed transports is observable in production.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants