Describe the bug
A failing Kafka source client can block a Tokio runtime worker for tens of seconds. Concurrent client failures can occupy all workers and prevent unrelated work, including barrier processing, from being scheduled promptly.
The RCA reports this sequence:
- Kafka sources periodically publish consumed offsets to consumer groups. The Kafka account lacks permission to commit those offsets, triggering
GroupAuthorizationFailed.
- Following a failed offset commit, a client can block for more than 45 seconds, approximately the configured
session.timeout.ms, before exiting and being recreated.
- Multiple clients fail concurrently. On the affected 8-core compute node, the RCA reports 8 runtime workers; blocking waits occupy those workers and barriers accumulate.
Missing Kafka permissions trigger the failure, but the RisingWave bug is allowing the client failure path to block Tokio workers. Correcting permissions or disabling Kafka offset commits mitigates the incident; neither addresses that runtime-isolation defect.
Error message/log
GroupAuthorizationFailed (Broker: Group authorization failed)
Reported diagnostic evidence: 1,190 consumer-group authorization failures across multiple Kafka sources. At 05:23 UTC, affected checkpoint barriers took approximately 241–246 seconds, while unaffected databases completed barriers in tens of milliseconds.
To reproduce
Suggested reproduction based on the RCA; not independently reproduced for this issue:
- Configure Kafka sources with topic-read permissions but without permission to commit offsets to their consumer groups. Enable Kafka auto-commit (
properties.enable.auto.commit = 'true').
- Run enough source clients on one compute node to encounter concurrent authorization failures across its Tokio workers.
- Allow offset commits to fail and clients to exit/restart. Observe worker stacks, task scheduling delays, and barrier latency during the failures.
Expected behavior
Kafka client failures, retries, and cleanup must not perform long blocking waits on Tokio workers. Other tasks and barrier processing should remain schedulable even when multiple Kafka clients fail simultaneously.
Additional context / investigation
The exact blocking call still needs confirmation with a stack trace or profile. Investigate the failure-to-recreation path, including consumer close/drop and any synchronous librdkafka waits; the RCA alone does not establish whether the wait occurs in the commit itself or subsequent teardown.
In the inspected source, KafkaSplitReader owns a StreamConsumer, and its async data stream propagates Kafka errors with ?: reader.rs. This is a starting point for investigation, not confirmation of the blocking stack or affected deployment revision.
The fix should isolate any blocking client operations, including cleanup on error/cancellation, from Tokio workers. Add regression coverage that induces concurrent client failures and verifies an independent task continues to make progress while cleanup/retries occur, with integration validation of barrier progress.
Workarounds from the RCA: disable Kafka consumer-group offset commits (properties.enable.auto.commit = 'false' when creating sources), or grant the required consumer-group permissions. These Kafka-side commits are auxiliary; RisingWave maintains source progress through its own checkpoints and state management.
Source: supplied incident report RCA-0915.md. Deployment method and affected RisingWave version were not provided.
Describe the bug
A failing Kafka source client can block a Tokio runtime worker for tens of seconds. Concurrent client failures can occupy all workers and prevent unrelated work, including barrier processing, from being scheduled promptly.
The RCA reports this sequence:
GroupAuthorizationFailed.session.timeout.ms, before exiting and being recreated.Missing Kafka permissions trigger the failure, but the RisingWave bug is allowing the client failure path to block Tokio workers. Correcting permissions or disabling Kafka offset commits mitigates the incident; neither addresses that runtime-isolation defect.
Error message/log
Reported diagnostic evidence: 1,190 consumer-group authorization failures across multiple Kafka sources. At 05:23 UTC, affected checkpoint barriers took approximately 241–246 seconds, while unaffected databases completed barriers in tens of milliseconds.
To reproduce
Suggested reproduction based on the RCA; not independently reproduced for this issue:
properties.enable.auto.commit = 'true').Expected behavior
Kafka client failures, retries, and cleanup must not perform long blocking waits on Tokio workers. Other tasks and barrier processing should remain schedulable even when multiple Kafka clients fail simultaneously.
Additional context / investigation
The exact blocking call still needs confirmation with a stack trace or profile. Investigate the failure-to-recreation path, including consumer close/drop and any synchronous librdkafka waits; the RCA alone does not establish whether the wait occurs in the commit itself or subsequent teardown.
In the inspected source,
KafkaSplitReaderowns aStreamConsumer, and its async data stream propagates Kafka errors with?: reader.rs. This is a starting point for investigation, not confirmation of the blocking stack or affected deployment revision.The fix should isolate any blocking client operations, including cleanup on error/cancellation, from Tokio workers. Add regression coverage that induces concurrent client failures and verifies an independent task continues to make progress while cleanup/retries occur, with integration validation of barrier progress.
Workarounds from the RCA: disable Kafka consumer-group offset commits (
properties.enable.auto.commit = 'false'when creating sources), or grant the required consumer-group permissions. These Kafka-side commits are auxiliary; RisingWave maintains source progress through its own checkpoints and state management.Source: supplied incident report
RCA-0915.md. Deployment method and affected RisingWave version were not provided.