Describe the bug
A shared postgres-cdc source (enable_shared_source = true) panicked in the source parser while appending the CDC payload to the JSONB column builder. The panic aborted the whole compute process (exit code 133). The source offset had not advanced past the failing transaction, so every restart replayed the same WAL position and panicked again about 6 seconds after cdc connector started. This happened 13 times over roughly 45 minutes.
We run a single compute node, so all streaming jobs in the database stopped. The meta node repeatedly failed recovery with no workers to assign. Because the replication slot stayed inactive, WAL kept accumulating on the upstream primary.
Our reading of the code: JsonbArrayBuilder shares one jsonbb::Builder across every row appended to a chunk (jsonb_array.rs). For a top-level value, Builder::offset() is the absolute length of that shared buffer, and Entry::object asserts offset <= 0x1FFFFFFF (entry.rs). A single chunk therefore cannot hold more than about 512 MiB of JSONB payload in total. Inside an ongoing transaction, SourceStreamChunkBuilder does not cut the chunk at chunk_size: it keeps rows together until MAX_TRANSACTION_SIZE = 4096 rows, a row-count limit with no byte-size limit (chunk_builder.rs). A single upstream transaction with a few thousand wide rows (about 125 KiB of Debezium payload per row across 4096 rows) is enough to cross the limit.
For a shared CDC source the JSONB column is the full Debezium event. It includes every column of every table in the publication, not only the columns declared by downstream CREATE TABLE ... FROM source. Rows from tables and columns that RisingWave never materializes still count toward the limit.
As of today, main still has both the 4096-row transaction cap and the jsonbb assertion, and the v3.0.4 release notes mention neither. Related closed issues with the same panic on other code paths: #22282 (sink log store), #26662 (Iceberg list executor).
Error message/log
thread 'rw-streaming' (33) panicked at /root/.cargo/git/checkouts/jsonbb-749fd24ab2e9c735-shallow/f618cd8/src/entry.rs:77:9:
offset too large
stack backtrace:
20: jsonbb::entry::Entry::object
21: jsonbb::builder::Builder<W>::add_value
22: <risingwave_common::array::jsonb_array::JsonbArrayBuilder as risingwave_common::array::ArrayBuilder>::append_n
23: risingwave_common::array::ArrayBuilderImpl::append_n
24: risingwave_common::array::ArrayBuilderImpl::append
25: <risingwave_connector::parser::chunk_builder::InsertAction as risingwave_connector::parser::chunk_builder::RowWriterAction>::apply
32: risingwave_connector::parser::chunk_builder::SourceStreamChunkRowWriter::do_action
33: risingwave_connector::parser::chunk_builder::SourceStreamChunkRowWriter::do_insert
34: risingwave_connector::parser::plain_parser::PlainParser::parse_rows::{{closure}}
35: risingwave_connector::parser::plain_parser::PlainParser::parse_inner::{{closure}}
36: <risingwave_connector::parser::plain_parser::PlainParser as risingwave_connector::parser::ByteStreamSourceParser>::parse_one_with_txn::{{closure}}
37: risingwave_connector::parser::parse_message_stream::{{closure}}
40: risingwave_connector::source::common::into_chunk_event_stream::{{closure}}
59: risingwave_stream::executor::source::apply_rate_limit_to_source_reader_event::{{closure}}
62: risingwave_stream::executor::source::reader_stream::StreamReaderBuilder::into_retry_stream::{{closure}}
76: risingwave_stream::executor::source::source_executor::SourceExecutor<S>::execute_inner::{{closure}}
*** await tree context of current task ***
Actor 51775: `<shared source>` [12.880s]
Epoch 11293620534312960 [1.320s]
Source CA3F00002712 [1.320s] <== current
receive_barrier [1.320s]
Sequence on every restart (Debezium lines trimmed):
INFO risingwave_connector_node: WAL resume position 'LSN{…/B8E85900}' discovered
INFO risingwave_connector::source::cdc::source::reader: cdc connector started source_id=273
INFO risingwave_connector_node: Message with LSN 'LSN{…/B8E85900}' arrived, switching off the filtering
WARN risingwave_storage::hummock::event_handler::hummock_event_handler: handle multiple uploaded ssts in batch batch_size=2
<panic above, ~6s after the connector started>
Meta, meanwhile:
ERROR risingwave_meta::barrier::worker: recovery failed error=global recovery failed due to failure of databases [6]
WARN risingwave_meta::barrier::worker: failed to inject database initial barrier database_id=6 e=no workers to assign; assignment is meaningless
To Reproduce
We have not run a synthetic reproduction. The following matches our production setup and the code path above.
-
Create a shared Postgres CDC source and at least one table from it:
CREATE SOURCE pg_source WITH (
connector = 'postgres-cdc',
hostname = '…', port = '5432', username = '…', password = '…',
database.name = '…', schema.name = 'public',
slot.name = '…', publication.name = '…',
ssl.mode = '…', postgres.is.aws.rds = 'true'
);
CREATE TABLE t (id BIGINT PRIMARY KEY, name VARCHAR)
FROM pg_source TABLE 'public.wide_table';
-
Upstream, have a published table with wide rows, for example a jsonb or text column holding around 150 KiB per row. The column does not need to be declared in the RisingWave table.
-
In a single upstream transaction, update 4096 or more such rows (for example UPDATE wide_table SET updated_at = now() WHERE id <= 5000;).
-
The compute node panics with offset too large once the chunk's accumulated JSONB passes 2^29 bytes. It then crash-loops on every restart, because the offset of the failing transaction is never committed.
In our incident the triggering transaction was a bulk update of roughly 10k rows on a wide table. We did not capture the transaction itself before working around the problem.
Expected behavior
I expected to see this happen: the source splits the chunk (or the transaction) before the JSONB builder's offset limit, or at worst reports a recoverable error that pauses that source.
Instead, this happened: a const fn assertion aborted the compute process, taking down every streaming job on the node. The same WAL position crashed it again on every restart, so the cluster could not recover without manual intervention.
How did you deploy RisingWave?
Kubernetes (EKS), via risingwave-operator v0.17.2 (Helm chart risingwave-operator-0.1.39).
- Meta store: PostgreSQL. State store: S3.
- 3 meta, 2 frontend, 1 compactor, 1 compute node: Graviton
r8g.2xlarge (arm64), 7 CPU / 60 GiB limit, RW_PARALLELISM=7.
streaming.developer.chunk_size = 256, enable_shared_source = true.- Upstream: Amazon RDS for PostgreSQL 17.9 (aarch64), logical replication via a publication, Debezium
max.queue.size = 8192.
The version of RisingWave
risingwave_compute::server: > version: 3.0.3 (ec07f2eb759bd2d8a12a55b030bf581b82adb4b9)
image: risingwavelabs/risingwave:v3.0.3
digest: sha256:4697d18f2fb5626f1607dfe6db099bd00643dcd734ea68e82d9fb2f6ad620ccf
Additional context
Workaround that worked: ALTER SOURCE <source> SET source_rate_limit TO 1000;, applied while compute was in backoff. On the next start the source got past the failing transaction without panicking and has been stable since. We have not confirmed in the code that a rate limit enables transaction splitting (SourceCtrlOpts::split_txn); this is what we observed.
Possibly a separate bug: after the rate-limit ALTER, the source reader was rebuilt shortly after a restart, and two Debezium engines for the same source id overlapped. The old one logged JNI sender broken detected, stop the engine and engine#273 terminated after the new one logged engine#273 start ok: true. From then on, data flowed (~270 rows/s), but every offset commit failed, so the replication slot never advanced:
ERROR risingwave_connector::source: source#273: failed to commit cdc offset: {…}. source_id=273 source_name="<shared source>" error=Failed to commit offset to upstream for source: 273.: Null pointer in call_method obj argument
The same run also logged Producer failure … Unable to register the MBean 'debezium.postgres:type=connector-metrics,context=snapshot,server=RW_CDC_273' and Failed to send handshake message to channel. sourceId=273. A second ALTER SOURCE … SET source_rate_limit rebuilt the engine again, and commits resumed.
Describe the bug
A shared
postgres-cdcsource (enable_shared_source = true) panicked in the source parser while appending the CDC payload to the JSONB column builder. The panic aborted the whole compute process (exit code 133). The source offset had not advanced past the failing transaction, so every restart replayed the same WAL position and panicked again about 6 seconds aftercdc connector started. This happened 13 times over roughly 45 minutes.We run a single compute node, so all streaming jobs in the database stopped. The meta node repeatedly failed recovery with
no workers to assign. Because the replication slot stayed inactive, WAL kept accumulating on the upstream primary.Our reading of the code:
JsonbArrayBuildershares onejsonbb::Builderacross every row appended to a chunk (jsonb_array.rs). For a top-level value,Builder::offset()is the absolute length of that shared buffer, andEntry::objectassertsoffset <= 0x1FFFFFFF(entry.rs). A single chunk therefore cannot hold more than about 512 MiB of JSONB payload in total. Inside an ongoing transaction,SourceStreamChunkBuilderdoes not cut the chunk atchunk_size: it keeps rows together untilMAX_TRANSACTION_SIZE = 4096rows, a row-count limit with no byte-size limit (chunk_builder.rs). A single upstream transaction with a few thousand wide rows (about 125 KiB of Debezium payload per row across 4096 rows) is enough to cross the limit.For a shared CDC source the JSONB column is the full Debezium event. It includes every column of every table in the publication, not only the columns declared by downstream
CREATE TABLE ... FROM source. Rows from tables and columns that RisingWave never materializes still count toward the limit.As of today,
mainstill has both the 4096-row transaction cap and the jsonbb assertion, and the v3.0.4 release notes mention neither. Related closed issues with the same panic on other code paths: #22282 (sink log store), #26662 (Iceberg list executor).Error message/log
To Reproduce
We have not run a synthetic reproduction. The following matches our production setup and the code path above.
Create a shared Postgres CDC source and at least one table from it:
CREATE SOURCE pg_source WITH ( connector = 'postgres-cdc', hostname = '…', port = '5432', username = '…', password = '…', database.name = '…', schema.name = 'public', slot.name = '…', publication.name = '…', ssl.mode = '…', postgres.is.aws.rds = 'true' ); CREATE TABLE t (id BIGINT PRIMARY KEY, name VARCHAR) FROM pg_source TABLE 'public.wide_table';Upstream, have a published table with wide rows, for example a
jsonbortextcolumn holding around 150 KiB per row. The column does not need to be declared in the RisingWave table.In a single upstream transaction, update 4096 or more such rows (for example
UPDATE wide_table SET updated_at = now() WHERE id <= 5000;).The compute node panics with
offset too largeonce the chunk's accumulated JSONB passes 2^29 bytes. It then crash-loops on every restart, because the offset of the failing transaction is never committed.In our incident the triggering transaction was a bulk update of roughly 10k rows on a wide table. We did not capture the transaction itself before working around the problem.
Expected behavior
I expected to see this happen: the source splits the chunk (or the transaction) before the JSONB builder's offset limit, or at worst reports a recoverable error that pauses that source.
Instead, this happened: a
const fnassertion aborted the compute process, taking down every streaming job on the node. The same WAL position crashed it again on every restart, so the cluster could not recover without manual intervention.How did you deploy RisingWave?
Kubernetes (EKS), via risingwave-operator
v0.17.2(Helm chartrisingwave-operator-0.1.39).r8g.2xlarge(arm64), 7 CPU / 60 GiB limit,RW_PARALLELISM=7.streaming.developer.chunk_size = 256,enable_shared_source = true.max.queue.size = 8192.The version of RisingWave
Additional context
Workaround that worked:
ALTER SOURCE <source> SET source_rate_limit TO 1000;, applied while compute was in backoff. On the next start the source got past the failing transaction without panicking and has been stable since. We have not confirmed in the code that a rate limit enables transaction splitting (SourceCtrlOpts::split_txn); this is what we observed.Possibly a separate bug: after the rate-limit
ALTER, the source reader was rebuilt shortly after a restart, and two Debezium engines for the same source id overlapped. The old one loggedJNI sender broken detected, stop the engineandengine#273 terminatedafter the new one loggedengine#273 start ok: true. From then on, data flowed (~270 rows/s), but every offset commit failed, so the replication slot never advanced:The same run also logged
Producer failure … Unable to register the MBean 'debezium.postgres:type=connector-metrics,context=snapshot,server=RW_CDC_273'andFailed to send handshake message to channel. sourceId=273. A secondALTER SOURCE … SET source_rate_limitrebuilt the engine again, and commits resumed.