Skip to content

perf(metadata): avoid re-cloning defensive schema copies - #1990

Open
fallintoplace wants to merge 5 commits into
apache:mainfrom
fallintoplace:perf/avoid-metadata-schema-reclone
Open

perf(metadata): avoid re-cloning defensive schema copies#1990
fallintoplace wants to merge 5 commits into
apache:mainfrom
fallintoplace:perf/avoid-metadata-schema-reclone

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

This keeps the defensive copies, but avoids doing the same work more than once.

  • Schema copies: reuse the deep copy already returned by Schema.Fields().
  • Partition specs: add a cache-aware PartitionSpec.Clone() so fields are copied once and the source index is rebuilt.
  • Sort orders: deep-copy mutable bucket and truncate transforms returned by metadata getters.
  • Row-lineage and Hive paths: reuse one copied field slice instead of copying the schema again.
  • Added regression tests for nested defaults, mutable transforms, partition-spec cloning, and row-lineage fields.

Testing

  • go test ./table ./view
  • go test ./catalog/hive ./table ./view
  • go test -race ./table ./view
  • go vet ./table ./view
  • make lint with golangci-lint v2.12.2

Benchmark

  • BenchmarkCloneSchemaWithNestedDefaults: about 3.0-3.8 us/op, 4320 B/op, 63 allocs/op to 1.82-1.84 us/op, 2696 B/op, 38 allocs/op.
  • BenchmarkClonePartitionSpecs with 32 fields: 108 to 75 allocs/op and 12,968 to 10,024 B/op in local runs.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Behavior is provably equivalent (Schema.Fields() already deep-copies), but the view-side deep-copy removal is pinned by a vacuous test and the PR's new partition/transform regression assertions do not bite under mutation.

Re-review verification: 0 of 1 prior findings confirmed fixed at 5e04984 (each verified by mutating the fix and observing the suite go red, not by taking the claim on trust).

Verification performed
go build ./...; go vet . ./table ./view ./catalog/hive; go test -count=1 . ./table ./view ./catalog/hive (all ok); go test -race -count=1 . ./table ./view ./catalog/hive (ok: iceberg 2.24s, table 19.81s, view 1.72s, hive 2.48s); 5 mutation probes on view/metadata.go:425, table/metadata.go:2407, table/metadata.go:2539, partitions.go:546 (shallow fields) and partitions.go:551 (drop initialize)

This review was drafted by an AI-assisted tool and confirmed by an Apache Iceberg Go maintainer. The findings below are observations, not blockers; an Apache Iceberg Go maintainer — a real person — will take the next look at the PR. If you think a finding is mis-applied, please reply on the PR and a maintainer will weigh in.

More on how Apache Iceberg Go handles maintainer review: CONTRIBUTING.md.

Comment thread view/metadata.go
slices.Clone(schema.IdentifierFieldIDs),
cloneNestedFields(schema.Fields())...,
schema.Fields()...,
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

major — View cloneSchema deep-copy guarantee is entirely unpinned; guarding test is vacuous

This PR removes view's local cloneNestedFields/cloneSchemaType and relies on Schema.Fields() being a deep copy. No view test verifies that. TestCloneSchemaCopiesNestedValues (view/metadata_test.go:712) mutates cloned.Field(i), but Schema.Field (schema.go:279) itself returns cloneField(...), so every mutation lands on a throwaway copy. A future change to view cloneSchema or to Fields() would silently alias view metadata's internal field slice with no test failure. Fix: mirror the table-side test and read the internal slice via FieldsRef(internal.SchemaRef{}) as TestMetadataSchemaGetterCopiesNestedValues already does.

Evidence
Mutated view/metadata.go:425 `schema.Fields()...` -> `schema.FieldsRef(iceint.SchemaRef{})...` (full aliasing); `go test -count=1 ./view` => `ok github.com/apache/iceberg-go/view 0.408s`. Same mutation on table/metadata.go:2407 correctly fails: `--- FAIL: TestMetadataSchemaGetterCopiesNestedValues` with diff `InitialDefault: 01 02 03 -> 63 02 03` and `Name: "list" -> "changed"`. Restored with git checkout; git status --porcelain empty.

Comment thread partitions_test.go
}

func TestPartitionSpecCloneCopiesFields(t *testing.T) {
transform := &iceberg.BucketTransform{NumBuckets: 16}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor — TestPartitionSpecCloneCopiesFields does not test that Clone copies fields

The test mutates clone.Field(0), but PartitionSpec.Field (partitions.go:757) already returns clonePartitionField(...), so the mutation never reaches clone.fields. Only the trailing FieldsBySourceID(1) assertion is non-vacuous (it pins initialize()). To actually guard the deep copy, compare spec/clone after mutating through an accessor that returns internal state, or assert on the SourceIDs slice identity of the two specs' FieldsBySourceID results.

partitionField.SourceIDs[0] = 99
partitionField.Name = "mutated"
partitionField.Transform.(*iceberg.BucketTransform).NumBuckets = 32
require.Equal(t, []int{1}, metadata.Specs[0].Field(0).SourceIDs)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor — New transform-mutation assertions in TestMetadataGettersReturnDefensiveCopies are vacuous

The added partitionField.Transform.(*iceberg.BucketTransform).NumBuckets = 32 (line 115) and the sort-order equivalent (line 166) read through PartitionSpec.Field() and SortOrder.Fields(), both of which already deep-copy the transform (partitions.go:757, table/sorting.go:328). The assertions therefore pass regardless of whether cloneSortOrder/PartitionSpec.Clone copy transforms, so they add no regression protection for the behavior the PR description claims they cover. (The cloneSortOrder guarantee itself is still pinned by TestMetadataBuilderFromBaseCopiesBuiltinMetadata, so this is coverage theater rather than a hole.)

Comment thread table/metadata.go
}

func clonePartitionSpec(spec iceberg.PartitionSpec) iceberg.PartitionSpec {
fields := make([]iceberg.PartitionField, spec.NumFields())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit — clonePartitionSpec is now a bare one-line pass-through

After this PR clonePartitionSpec(spec) is just return spec.Clone(). Five call sites could call spec.Clone() directly and drop the wrapper, or keep it if a doc comment explains the indirection.

fields[i] = iceberg.PartitionField{
SourceIDs: []int{i + 1}, FieldID: i + 1000,
Name: "field", Transform: iceberg.IdentityTransform{},
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit — Benchmark helper builds a spec with duplicate partition field names

partitionSpecCloneBenchmarkFields assigns Name: "field" to all N fields, producing a spec UnmarshalJSON would reject and that no real table can have. Using strconv.Itoa(i) for the name (strconv is already imported) makes the benchmark measure a realistic spec, including distinct url.QueryEscape work in initialize().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants