Skip to content

perf(table): compute partition residuals for local scan tasks - #1971

Open
fallintoplace wants to merge 4 commits into
apache:mainfrom
fallintoplace:perf/local-scan-partition-residuals
Open

perf(table): compute partition residuals for local scan tasks#1971
fallintoplace wants to merge 4 commits into
apache:mainfrom
fallintoplace:perf/local-scan-partition-residuals

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

What

  • Compute partition residuals during local planning.
  • Cache one evaluator per partition spec and reuse it for every data file in that spec.
  • Use strict and inclusive projections to return AlwaysTrue, AlwaysFalse, or the remaining row filter for each task.
  • Cover identity, time, truncate, bucket, null, set, boolean, and transformed filter predicates. Unsupported transforms stay conservative.
  • Read only the fields needed by each task residual, so a fully satisfied partition filter does not add filter-only columns to the file read.
  • Include the identity-filter coverage from perf(table): elide identity partition filters #1966: nested fields, decoded decimal partition values, mismatched transforms, and end-to-end reads.

Example

For ts >= ... AND ts < ... on day(ts) partitions:

  • Interior partitions get AlwaysTrue.
  • Boundary partitions keep only the needed timestamp predicate.
  • Partitions outside the range get AlwaysFalse.

Benchmark

Apple M1 Pro, Go 1.26.3, GOMAXPROCS=4. Medians of six 1-second runs on 2c72b57.

Arrow reads: 32,768 rows from in-memory Parquet, selecting only payload.

Filter Task residual ms/op MB/op allocs/op
Identity only Original filter 2.367 8.95 944
Identity only AlwaysTrue 1.479 5.55 521
Mixed Original filter 3.388 19.76 1,830
Mixed Amount predicate only 2.941 17.33 1,486
  • Identity only: 37.5% less read time and 37.9% fewer allocated bytes.
  • Mixed filter: 13.2% less read time and 12.3% fewer allocated bytes.
  • These compare the Arrow read step with and without task residuals. Planning is measured separately.

Residual evaluation for 4,096 files, reusing one evaluator:

  • Identity: 0.714 ms/op, 393,216 B/op, 18,432 allocs/op.
  • Day range: 1.153 ms/op, 553,801 B/op, 31,950 allocs/op.
GOMAXPROCS=4 go test ./table -run '^$' -bench '^(BenchmarkPartitionResidualPlanning|BenchmarkArrowScanTaskResidual)$' -benchmem -benchtime=1s -count=6

Checks

  • go test ./table/... -count=1
  • go test -race ./table -count=1
  • go test ./... -run=^$ -count=1
  • go vet ./...
  • go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2 run --timeout=10m

🔥

Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
@fallintoplace
fallintoplace force-pushed the perf/local-scan-partition-residuals branch from 635798d to a8d1268 Compare August 30, 2026 22:18
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant