[improve][client] Log V5 segment-gone send retries at DEBUG - #26615
Merged
lhotari merged 1 commit intoSep 17, 2026
Merged
Conversation
A segment split terminates the parent segment topic before the new layout is published, so the per-segment v4 producer fails every pending message in one burst and the V5 producer retries each of them until the DAG watch delivers the new layout. Each retry was logged at INFO, one line per in-flight message, which floods the log on every split at a high publish rate. The split itself stays visible through the INFO "Closing producer for sealed segment" and "Layout applied" lines; the per-message retries are a DEBUG-level detail.
void-ptr974
approved these changes
Sep 17, 2026
lhotari
approved these changes
Sep 17, 2026
lhotari
left a comment
lhotari
left a comment
Member
There was a problem hiding this comment.
LGTM. Thanks for reducing the per-message retry noise while keeping layout changes visible at INFO.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
When a scalable topic segment is split, the broker terminates the parent segment topic before it publishes the new layout. The per-segment v4 producer is disconnected, its reconnect is rejected with
TopicTerminated, and it fails every pending message in one burst. The V5ScalableTopicProducerthen retries each message with backoff until the DAG watch delivers the new layout, and logged every one of those retries at INFO:At a high publish rate that is one line per in-flight message on every split, which floods the log. Publishing recovers on its own once the new layout arrives; the noise is the problem.
Modifications
Lower the two "Target segment gone" retry logs in
ScalableTopicProducer(the async path and its sync counterpart) from INFO to DEBUG. The split itself remains visible at INFO through the existing "Closing producer for sealed segment" and "Layout applied" lines, which are emitted once per layout change rather than once per message.