Skip to content

Add tunables bounding compaction pipeline memory - #1993

Open
KranzL wants to merge 1 commit into
apache:mainfrom
KranzL:compaction-memory-tunables
Open

Add tunables bounding compaction pipeline memory#1993
KranzL wants to merge 1 commit into
apache:mainfrom
KranzL:compaction-memory-tunables

Conversation

@KranzL

@KranzL KranzL commented Sep 5, 2026

Copy link
Copy Markdown

Fixes #1982.

The compaction read+write pipeline had fixed memory characteristics: the scanner decodes read.parquet.batch-size rows per Arrow batch (131072 by default) with no per-scan override, and each rolling data writer buffers up to 64 record batches in its input channel. On wide rows the product of those two numbers dominates a compaction worker's peak memory.

  • WithArrowBatchSize scan option caps the rows decoded per Arrow batch for one scan, overriding the table's read.parquet.batch-size without mutating caller-supplied scan options.
  • The rolling data writers' record channel capacity (default 64 batches) is now configurable via the WithRecordBatchBufferSize write option, threaded through recordWritingArgs to the writer factory. Applies to both the fanout and the clustered write paths, which share the rolling writer.
  • WithParquetRowGroupLimit write option overrides the table's write.parquet.row-group-limit per write, bounding rows buffered per row group before each flush.
  • Compaction group options forward all three into ExecuteCompactionGroup: WithCompactionReadBatchSize, WithCompactionRecordBatchBufferSize, and WithCompactionParquetRowGroupLimit.

Defaults stay exactly as they are today. Tests: table/write_read_tuning_test.go.

The compaction read+write pipeline had fixed memory characteristics:
the scanner decodes read.parquet.batch-size rows per Arrow batch
(131072 by default) with no per-scan override, and each rolling data
writer buffers up to 64 record batches in its input channel with no way
to lower or raise that bound. On wide rows the product of those two
numbers dominates a compaction worker's peak memory.

- WithArrowBatchSize scan option caps the rows decoded per Arrow batch
  for one scan, overriding the table's read.parquet.batch-size without
  mutating caller-supplied scan options.
- The rolling data writers' record channel capacity (default 64
  batches) is now configurable via the WithRecordBatchBufferSize write
  option, threaded through recordWritingArgs to the writer factory. It
  applies to both the fanout and the clustered write paths, which share
  the rolling writer.
- WithParquetRowGroupLimit write option overrides the table's
  write.parquet.row-group-limit per write, bounding rows buffered per
  row group before each flush.
- Compaction group options forward all three into
  ExecuteCompactionGroup: WithCompactionReadBatchSize,
  WithCompactionRecordBatchBufferSize, and
  WithCompactionParquetRowGroupLimit.

Signed-off-by: KranzL <50032317+KranzL@users.noreply.github.com>
@KranzL
KranzL requested a review from zeroshade as a code owner September 5, 2026 05:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No way to bound memory use when compacting tables with wide rows

1 participant