Spec: Add collations for string types - #16972
Conversation
github-actions
Bot
added
Specification
szehon-ho
left a comment
szehon-ho
left a comment
There was a problem hiding this comment.
Thanks for working on this! A few comments here and inline:
-
Nested string fields need an explicit story. The spec says a
stringfield may carry acollation, and the JSON change addscollationtoStructField. Does that intentionally excludelist<string>,map<string, ...>, andmap<..., string>? If nested strings are in scope, the spec should define where collation is stored for list elements, map keys, and map values. If they are out of scope, that limitation should be explicit. -
Collation evolution can be better defined in terms of schema evolution. What happens when a column changes from one collation to another, or from byte-order to an ICU collation? Is this a compatible schema evolution, a type change, or disallowed? Existing files may contain bounds for older collations/versions, so readers should only use
collation_boundsentries matching the collation resolved for the read schema and otherwise treat the file as possibly matching. -
Please clarify the scope beyond file pruning. If collation affects query semantics, then equality deletes, sort orders, partition transforms, and bucket/hash semantics may also need rules. If this change is only defining schema annotation and file-level pruning metadata, it would help to say that explicitly so engines do not infer broader SQL/equality semantics from the table-format metadata alone.
Add an optional `collation` annotation on string fields (top-level and nested list/map positions) that changes comparison and ordering without changing storage. Collation-aware file bounds are stored as ordinary content_stats under schema-declared collation metric fields (`collation-metrics`: id + collation + version), so a file can carry bounds for multiple collations/versions and readers resolve them by field id at planning time. Scope is limited to the annotation and file-level pruning. Byte-order bounds, partition/bucket transforms, equality-delete matching, and identifier-field equality stay binary and collation-unaware; sort orders are not defined here. Collation change is a metadata-only schema change. The version records the provider's collation version (for `icu`, the ICU collator version) and is compared as an opaque string; a mismatch falls back to a scan, never a wrong result. Collation bounds must be tight. Update the REST OpenAPI spec (yaml + generated models) with the `collation` attribute and the CollationMetric type.
c70f434 to
6581e52
Compare
| | `collation` | `string` | Collation the bounds are produced for, e.g. `icu.en_US-ci` | | ||
| | `version` | `string` | Provider's collation version the bounds were selected under (for `icu`, the ICU collator version); compared as an opaque string on read. See [Collations](#collations) | | ||
|
|
||
| The bounds themselves are an ordinary [field stats struct](#field-statistics) in `content_stats` under the metric field's id: its `lower_bound` and `upper_bound` are `string` values holding the file's minimum and maximum **under the collation order** — the original values, not collation sort keys (sort keys are not stable across implementation versions) — and `tight_bounds` indicates whether they are exact. Collation bounds must be tight: because there is no defined successor under an arbitrary collation order, a writer that cannot store the exact minimum and maximum must omit the bound rather than approximate it. A field may declare several collation metric entries — typically one per `(collation, version)` — so a single file can serve readers pinned to different versions during an upgrade. |
There was a problem hiding this comment.
Collation bounds must be tight: because there is no defined successor under an arbitrary collation order, a writer that cannot store the exact minimum and maximum must omit the bound rather than approximate it.
How would we define tight? Does it mean we do not allow truncation? I'm worried that omitting bounds could negatively impact performance
| | `collation` | `string` | Collation the bounds are produced for, e.g. `icu.en_US-ci` | | ||
| | `version` | `string` | Provider's collation version the bounds were selected under (for `icu`, the ICU collator version); compared as an opaque string on read. See [Collations](#collations) | | ||
|
|
||
| The bounds themselves are an ordinary [field stats struct](#field-statistics) in `content_stats` under the metric field's id: its `lower_bound` and `upper_bound` are `string` values holding the file's minimum and maximum **under the collation order** — the original values, not collation sort keys (sort keys are not stable across implementation versions) — and `tight_bounds` indicates whether they are exact. Collation bounds must be tight: because there is no defined successor under an arbitrary collation order, a writer that cannot store the exact minimum and maximum must omit the bound rather than approximate it. A field may declare several collation metric entries — typically one per `(collation, version)` — so a single file can serve readers pinned to different versions during an upgrade. |
There was a problem hiding this comment.
— the original values, not collation sort keys (sort keys are not stable across implementation versions)
that is a good point 👍 unfortunately, original strings are not stable across versions, ICU does not guarantee stable ordering, either.
If I understand correctly, this means that original strings provide no pruning benefits compared to sort keys, in the general case.
The situation would be different if we had some additional metrics on the data, though. Most of the ordering changes are limited to a small range of codepoints. If we had metadata proving that a file contains none of those (e.g., because it only contains ASCII), we could probably even prune across versions, which I'd consider a point in favor of sort keys
|
|
||
| **Writers** are not required to write byte-order bounds for collated columns — as with any column, statistics are optional — but writing them is encouraged so collation-unaware readers can still prune. A writer that supports a declared `(collation, version)` should additionally write the `content_stats` struct for that metric field with the exact collation-order minimum and maximum. A writer that does not support the collation or version must not write bounds for that metric field. | ||
|
|
||
| **Readers** may prune a file using a collation metric field's bounds only when the metric's `collation` and `version` match the collation the reader resolves for the column; otherwise those bounds must be ignored. A reader must not use byte-order bounds to prune comparison, equality, or prefix predicates on a collated column. Collation bounds support comparison and equality pruning only: prefix predicates such as `STARTS_WITH` or `LIKE 'x%'` cannot be pruned by either byte-order or collation bounds, because prefix relationships are not preserved under an arbitrary collation order. When no usable collation bounds are available, the file must be scanned. A collation-unaware reader ignores the `collation` attribute and any collation metric fields, reading the column as a UTF-8 byte-order string. |
There was a problem hiding this comment.
SQL standard supports pad characteristic (NO PAD | PAD SPACE). How do engines know whether icu.en_US-ci is pad-space or no-pad?
|
|
||
| A collation may be attached to any field of `string` type, top-level or nested: as a `collation` attribute on a struct field, or as an `element-collation`, `key-collation`, or `value-collation` attribute on an enclosing `list` or `map` (see [Appendix C](#appendix-c-json-serialization)). It may be attached only to `string` types. A string field with no collation defaults to UTF-8 byte-order comparison, the behavior of all prior versions. Nesting does not change how collation bounds work: because every nested string position has its own field id, a collation on a nested string is annotated and given collation metrics exactly as a top-level string field (see [Collation Bounds](#collation-bounds)). | ||
|
|
||
| A collation is identified by a provider-qualified name of the form `<provider>.<name>`, for example `icu.en_US-ci`. The provider names the library that defines the collation (`icu` for collations defined by the [Unicode Collation Algorithm](https://unicode.org/reports/tr10/) over [CLDR](https://cldr.unicode.org/) locale data; other providers may define engine-specific collations such as case-folding variants). The name selects a locale and optional modifiers for case sensitivity (`ci`/`cs`), accent sensitivity (`ai`/`as`), trimming, and case folding. |
There was a problem hiding this comment.
In SQL standard, the collation name is a schema-qualified identifier. The suggested name format may not fit some engines.
|
|
||
| SQL-level comparison, equality, ordering, grouping, and distinctness are the engine's responsibility. Engines read the `collation` annotation and apply their own semantics; they must not infer broader behavior from the table-format metadata alone. | ||
|
|
||
| A collation may be attached to any field of `string` type, top-level or nested: as a `collation` attribute on a struct field, or as an `element-collation`, `key-collation`, or `value-collation` attribute on an enclosing `list` or `map` (see [Appendix C](#appendix-c-json-serialization)). It may be attached only to `string` types. A string field with no collation defaults to UTF-8 byte-order comparison, the behavior of all prior versions. Nesting does not change how collation bounds work: because every nested string position has its own field id, a collation on a nested string is annotated and given collation metrics exactly as a top-level string field (see [Collation Bounds](#collation-bounds)). |
There was a problem hiding this comment.
Under a case-insensitive collation, should map['ABC'] and map['abc'] resolve to the same entry? And if so, what happens when a map contains both keys? Is that an invalid state, or is the result engine-defined?
5 participants
This adds collation support for string types to the v3 spec, following the discussion in the dev-list thread. A string field can carry a collation (e.g. icu.en_US-ci) that defines case-insensitive, accent-insensitive, or locale-aware comparison and ordering, and a new data_file.collation_bounds field carries collation-aware min/max so collated columns can still be pruned
The spec changes:
Two decisions I'd flag for discussion.
Backed by a working reference implementation in iceberg-go: apache/iceberg-go#1318.
Rationale and the differences from the original proposal are written up separately (proposal doc).
See also the dev-list thread. Feedback very welcome.