Skip to content

perf(table): flatten manifest entries after parallel reads - #1976

Open
fallintoplace wants to merge 3 commits into
apache:mainfrom
fallintoplace:perf/flatten-manifest-entries
Open

perf(table): flatten manifest entries after parallel reads#1976
fallintoplace wants to merge 3 commits into
apache:mainfrom
fallintoplace:perf/flatten-manifest-entries

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Keep one classified result per manifest while reads run in parallel.
  • Flatten after all reads finish into exact-sized data and delete slices.
  • Preserve manifest-list order and keep manifestEntries.merge for existing callers.
  • Add focused tests for mixed buckets, empty result slots, and exact capacities.
  • Add a merge-versus-flatten benchmark.

Benchmark

Apple M1 Pro. 64 manifests, 10,000 entries per manifest, concurrency 16.

  • Data manifests: 18.4 ms/op and 71.9 MB/op with merge, versus 6.4 ms/op and 20.7 MB/op with flatten.
  • Delete manifests: 12.9 ms/op and 74.6 MB/op with merge, versus 6.3 ms/op and 32.4 MB/op with flatten.

Tests

  • go test ./... -count=1
  • go vet ./table
  • Targeted go test -race ./table pass.

Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
@fallintoplace
fallintoplace force-pushed the perf/flatten-manifest-entries branch from ecf6c87 to 93325d6 Compare August 30, 2026 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant