Skip to content

Spec: Add collations for string types #16972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
laskoviymishka wants to merge 1 commit into
base: main
Choose a base branch
from
+134 −6
Draft

Spec: Add collations for string types #16972

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 61 additions & 6 deletions format/spec.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ The full set of changes are listed in [Appendix E](#version-3).
Version 4 of the Iceberg spec restructures metadata for improved performance and new capabilities:

* Support for [relative locations](#file-locations-in-metadata) in metadata fields
* Collations for string types

The full set of changes are listed in [Appendix E](#version-4).

Expand Down Expand Up @@ -275,7 +276,7 @@ Supported primitive types are defined in the table below. Primitive types added
| | **`timestamptz`** | Timestamp, microsecond precision, with timezone | [2] |
| [v3](#version-3) | **`timestamp_ns`** | Timestamp, nanosecond precision, without timezone | [1] |
| [v3](#version-3) | **`timestamptz_ns`** | Timestamp, nanosecond precision, with timezone | [2] |
| | **`string`** | Arbitrary-length character sequences | Encoded with UTF-8 [3] |
| | **`string`** | Arbitrary-length character sequences; may carry a [collation](#collations) | Encoded with UTF-8 [3] |
| | **`uuid`** | Universally unique identifiers | Should use 16-byte fixed |
| | **`fixed(L)`** | Fixed-length byte array of length L | |
| | **`binary`** | Arbitrary-length byte array | |
Expand Down Expand Up @@ -345,9 +346,55 @@ For example, a struct column `point` with fields `x` (default 0) and `y` (defaul

Default values are attributes of fields in schemas and serialized with fields in the JSON format. See [Appendix C](#appendix-c-json-serialization).

#### Collations

A `string` field may carry a **collation**, an attribute that changes how the field's values are compared and ordered without changing how they are stored. Collations enable case-insensitive, accent-insensitive, and locale-aware comparison and sorting. A collation only affects comparison: the stored value is returned unchanged (a value written as `'appLE'` is read back as `'appLE'`).

This change defines exactly two things: the `collation` annotation on string fields, and collation-aware file pruning through collation metric fields stored in `content_stats`. It does not give collation semantics to any other part of the table format, because collation order and equality are not stable across collation implementation versions or engines, while the rest of the format must be deterministic. Specifically:

* Byte-order `lower_bounds`/`upper_bounds`, partition transforms (including `truncate`), and `bucket`/hash all operate on the binary UTF-8 value and are collation-unaware. Collation-equal but byte-distinct values may therefore land in different partitions or buckets, and a reader must not use partition or bucket pruning to eliminate candidates for a predicate evaluated under a collation.
* Equality-delete matching and identifier-field equality are binary value equality by field id; a `collation` annotation does not make either collation-aware. An engine that needs a collation-dependent delete must resolve it to position deletes or an explicit set of values.
* This change does not define collation-aware sort orders.

SQL-level comparison, equality, ordering, grouping, and distinctness are the engine's responsibility. Engines read the `collation` annotation and apply their own semantics; they must not infer broader behavior from the table-format metadata alone.

A collation may be attached to any field of `string` type, top-level or nested: as a `collation` attribute on a struct field, or as an `element-collation`, `key-collation`, or `value-collation` attribute on an enclosing `list` or `map` (see [Appendix C](#appendix-c-json-serialization)). It may be attached only to `string` types. A string field with no collation defaults to UTF-8 byte-order comparison, the behavior of all prior versions. Nesting does not change how collation bounds work: because every nested string position has its own field id, a collation on a nested string is annotated and given collation metrics exactly as a top-level string field (see [Collation Bounds](#collation-bounds)).

@ebyhr ebyhr Aug 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Under a case-insensitive collation, should map['ABC'] and map['abc'] resolve to the same entry? And if so, what happens when a map contains both keys? Is that an invalid state, or is the result engine-defined?


A collation is identified by a provider-qualified name of the form `<provider>.<name>`, for example `icu.en_US-ci`. The provider names the library that defines the collation (`icu` for collations defined by the [Unicode Collation Algorithm](https://unicode.org/reports/tr10/) over [CLDR](https://cldr.unicode.org/) locale data; other providers may define engine-specific collations such as case-folding variants). The name selects a locale and optional modifiers for case sensitivity (`ci`/`cs`), accent sensitivity (`ai`/`as`), trimming, and case folding.

@ebyhr ebyhr Aug 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In SQL standard, the collation name is a schema-qualified identifier. The suggested name format may not fit some engines.


The bare name `utf8` is reserved for UTF-8 byte-order comparison and is the one collation name exempt from the `<provider>.<name>` form. It is exactly equivalent to omitting the `collation` attribute: readers and writers must treat a field annotated `utf8` and a field with no collation identically, and adding or removing an explicit `utf8` on an already-uncollated field is a no-op that does not change comparison semantics. A field with collation `utf8` (or no collation) uses the byte-order `lower_bounds`/`upper_bounds` directly and must not declare `collation-metrics`.

The schema stores the collation **name without a version**, so any engine that supports the collation can read the table. UCA, DUCET, CLDR, and ICU collation orders are [not stable across versions](https://unicode.org/reports/tr10/#Non-Goals), so collation-aware metrics carry the implementation version they were produced under (see below) and a reader uses them only when it can produce the same order.
Comment thread
laskoviymishka marked this conversation as resolved.

The version is the identifier the provider uses to version its collation order, and must be a value that changes only when the collation order can change, so that a byte-exact match between two engines means they produce the same order. For the `icu` provider this is the [ICU collator version](https://unicode-org.github.io/icu/userguide/collation/architecture.html#versioning) reported for the collation (which folds in the UCA, DUCET, and CLDR versions and the tailoring), formatted as ICU reports it. Two engines built against the same collation data therefore report the same version and can share bounds; a provider release that does not change any order reports the same version, so existing bounds stay usable and no new metric field is needed. The version is provider-defined and is compared as an opaque string: a reader treats bounds as usable only on an exact `(collation, version)` match and otherwise scans, so a mismatch degrades pruning but never correctness.

##### Collation Bounds

Because a collation can reorder values (for example `'a' < 'B'` under a case-insensitive collation, but `'a' > 'B'` in byte order), byte-order bounds — the v3 `lower_bounds`/`upper_bounds` maps and the byte-order bounds in `content_stats` — cannot be used to evaluate predicates on a collated column. Collation-aware bounds are stored instead as ordinary [content stats](#content-stats), under dedicated **collation metric fields** declared in the schema.

A collated field declares its collation metric fields as a `collation-metrics` list on the field (see [Appendix C](#appendix-c-json-serialization)). Each entry has:

| Key | Type | Description |
|-----|------|-------------|
| `id` | `int` | Field id whose `content_stats` struct holds the collation-aware bounds. It is allocated from the table's column id space using `last-column-id`; it is not a data column and carries only stats |
| `collation` | `string` | Collation the bounds are produced for, e.g. `icu.en_US-ci` |
| `version` | `string` | Provider's collation version the bounds were selected under (for `icu`, the ICU collator version); compared as an opaque string on read. See [Collations](#collations) |

The bounds themselves are an ordinary [field stats struct](#field-statistics) in `content_stats` under the metric field's id: its `lower_bound` and `upper_bound` are `string` values holding the file's minimum and maximum **under the collation order** — the original values, not collation sort keys (sort keys are not stable across implementation versions) — and `tight_bounds` indicates whether they are exact. Collation bounds must be tight: because there is no defined successor under an arbitrary collation order, a writer that cannot store the exact minimum and maximum must omit the bound rather than approximate it. A field may declare several collation metric entries — typically one per `(collation, version)` — so a single file can serve readers pinned to different versions during an upgrade.

@sfc-gh-aloeser sfc-gh-aloeser Aug 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Collation bounds must be tight: because there is no defined successor under an arbitrary collation order, a writer that cannot store the exact minimum and maximum must omit the bound rather than approximate it.

How would we define tight? Does it mean we do not allow truncation? I'm worried that omitting bounds could negatively impact performance

@sfc-gh-aloeser sfc-gh-aloeser Aug 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

— the original values, not collation sort keys (sort keys are not stable across implementation versions)

that is a good point 👍 unfortunately, original strings are not stable across versions, ICU does not guarantee stable ordering, either.

If I understand correctly, this means that original strings provide no pruning benefits compared to sort keys, in the general case.

The situation would be different if we had some additional metrics on the data, though. Most of the ordering changes are limited to a small range of codepoints. If we had metadata proving that a file contains none of those (e.g., because it only contains ASCII), we could probably even prune across versions, which I'd consider a point in favor of sort keys


**Writers** are not required to write byte-order bounds for collated columns — as with any column, statistics are optional — but writing them is encouraged so collation-unaware readers can still prune. A writer that supports a declared `(collation, version)` should additionally write the `content_stats` struct for that metric field with the exact collation-order minimum and maximum. A writer that does not support the collation or version must not write bounds for that metric field.

**Readers** may prune a file using a collation metric field's bounds only when the metric's `collation` and `version` match the collation the reader resolves for the column; otherwise those bounds must be ignored. A reader must not use byte-order bounds to prune comparison, equality, or prefix predicates on a collated column. Collation bounds support comparison and equality pruning only: prefix predicates such as `STARTS_WITH` or `LIKE 'x%'` cannot be pruned by either byte-order or collation bounds, because prefix relationships are not preserved under an arbitrary collation order. When no usable collation bounds are available, the file must be scanned. A collation-unaware reader ignores the `collation` attribute and any collation metric fields, reading the column as a UTF-8 byte-order string.

@ebyhr ebyhr Aug 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SQL standard supports pad characteristic (NO PAD | PAD SPACE). How do engines know whether icu.en_US-ci is pad-space or no-pad?


##### Collation Evolution

Adding, changing, or removing a field's collation is a metadata-only schema change: the field keeps its id and `string` type and no data files are rewritten (storage is always UTF-8). Because collation is an attribute of the field, it is not a type promotion — only comparison semantics change — and it is always permitted at the format level. Whether an engine exposes it as DDL, and how it handles dependent constructs such as sort orders or equality deletes, is engine policy.

Adding a collation, or a new collation version, adds a `collation-metrics` entry with a newly allocated metric field id; this is a metadata-only change like any other. Because data files are not rewritten, a file written before the change carries collation stats for the previous collation, or none if the field was previously uncollated. A reader resolves the field's collation from the current schema and, per the reader rules above, uses a collation metric field's bounds only when its `collation` and `version` match; otherwise they are ignored and the file is treated as possibly matching until new stats are written (for example by compaction). Byte-order bounds are unaffected and keep serving collation-unaware readers; removing a collation makes them directly usable again.

#### Schema Evolution

Schemas may be evolved by type promotion or adding, deleting, renaming, or reordering fields in structs (both nested structs and the top-level schema’s struct).
Schemas may be evolved by type promotion or adding, deleting, renaming, or reordering fields in structs (both nested structs and the top-level schema’s struct). A string field's [collation](#collations) may also be added, changed, or removed as a metadata-only change; see [Collation Evolution](#collation-evolution).

Evolution applies changes to the table's current schema to produce a new schema that is identified by a unique schema ID, is added to the table's list of schemas, and is set as the table's current schema.

Expand Down Expand Up @@ -1680,9 +1727,9 @@ Types are serialized according to this table:
|**`fixed(L)`**|`JSON string: "fixed[<L>]"`|`"fixed[16]"`|
|**`binary`**|`JSON string: "binary"`|`"binary"`|
|**`decimal(P, S)`**|`JSON string: "decimal(<P>,<S>)"`|`"decimal(9,2)"`,<br />`"decimal(9, 2)"`|
|**`struct`**|`JSON object: {`<br />&nbsp;&nbsp;`"type": "struct",`<br />&nbsp;&nbsp;`"fields": [ {`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"id": <field id int>,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"name": <name string>,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"required": <boolean>,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"type": <type JSON>,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"doc": <comment string>,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"initial-default": <JSON encoding of default value>,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"write-default": <JSON encoding of default value>`<br />&nbsp;&nbsp;&nbsp;&nbsp;`}, ...`<br />&nbsp;&nbsp;`] }`|`{`<br />&nbsp;&nbsp;`"type": "struct",`<br />&nbsp;&nbsp;`"fields": [ {`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"id": 1,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"name": "id",`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"required": true,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"type": "uuid",`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"initial-default": "0db3e2a8-9d1d-42b9-aa7b-74ebe558dceb",`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"write-default": "ec5911be-b0a7-458c-8438-c9a3e53cffae"`<br />&nbsp;&nbsp;`}, {`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"id": 2,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"name": "data",`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"required": false,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"type": {`<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`"type": "list",`<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`...`<br />&nbsp;&nbsp;&nbsp;&nbsp;`}`<br />&nbsp;&nbsp;`} ]`<br />`}`|
|**`list`**|`JSON object: {`<br />&nbsp;&nbsp;`"type": "list",`<br />&nbsp;&nbsp;`"element-id": <id int>,`<br />&nbsp;&nbsp;`"element-required": <bool>`<br />&nbsp;&nbsp;`"element": <type JSON>`<br />`}`|`{`<br />&nbsp;&nbsp;`"type": "list",`<br />&nbsp;&nbsp;`"element-id": 3,`<br />&nbsp;&nbsp;`"element-required": true,`<br />&nbsp;&nbsp;`"element": "string"`<br />`}`|
|**`map`**|`JSON object: {`<br />&nbsp;&nbsp;`"type": "map",`<br />&nbsp;&nbsp;`"key-id": <key id int>,`<br />&nbsp;&nbsp;`"key": <type JSON>,`<br />&nbsp;&nbsp;`"value-id": <val id int>,`<br />&nbsp;&nbsp;`"value-required": <bool>`<br />&nbsp;&nbsp;`"value": <type JSON>`<br />`}`|`{`<br />&nbsp;&nbsp;`"type": "map",`<br />&nbsp;&nbsp;`"key-id": 4,`<br />&nbsp;&nbsp;`"key": "string",`<br />&nbsp;&nbsp;`"value-id": 5,`<br />&nbsp;&nbsp;`"value-required": false,`<br />&nbsp;&nbsp;`"value": "double"`<br />`}`|
|**`struct`**|`JSON object: {`<br />&nbsp;&nbsp;`"type": "struct",`<br />&nbsp;&nbsp;`"fields": [ {`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"id": <field id int>,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"name": <name string>,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"required": <boolean>,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"type": <type JSON>,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"doc": <comment string>,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"initial-default": <JSON encoding of default value>,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"write-default": <JSON encoding of default value>,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"collation": <collation name string>,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"collation-metrics": [ { "id": <int>, "collation": <string>, "version": <string> }, ... ]`<br />&nbsp;&nbsp;&nbsp;&nbsp;`}, ...`<br />&nbsp;&nbsp;`] }`|`{`<br />&nbsp;&nbsp;`"type": "struct",`<br />&nbsp;&nbsp;`"fields": [ {`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"id": 1,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"name": "id",`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"required": true,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"type": "uuid",`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"initial-default": "0db3e2a8-9d1d-42b9-aa7b-74ebe558dceb",`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"write-default": "ec5911be-b0a7-458c-8438-c9a3e53cffae"`<br />&nbsp;&nbsp;`}, {`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"id": 2,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"name": "data",`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"required": false,`<br />&nbsp;&nbsp;&nbsp;&nbsp;`"type": {`<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`"type": "list",`<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`...`<br />&nbsp;&nbsp;&nbsp;&nbsp;`}`<br />&nbsp;&nbsp;`} ]`<br />`}`|
|**`list`**|`JSON object: {`<br />&nbsp;&nbsp;`"type": "list",`<br />&nbsp;&nbsp;`"element-id": <id int>,`<br />&nbsp;&nbsp;`"element-required": <bool>`<br />&nbsp;&nbsp;`"element": <type JSON>,`<br />&nbsp;&nbsp;`"element-collation": <collation name string>`<br />`}`|`{`<br />&nbsp;&nbsp;`"type": "list",`<br />&nbsp;&nbsp;`"element-id": 3,`<br />&nbsp;&nbsp;`"element-required": true,`<br />&nbsp;&nbsp;`"element": "string"`<br />`}`|
|**`map`**|`JSON object: {`<br />&nbsp;&nbsp;`"type": "map",`<br />&nbsp;&nbsp;`"key-id": <key id int>,`<br />&nbsp;&nbsp;`"key": <type JSON>,`<br />&nbsp;&nbsp;`"key-collation": <collation name string>,`<br />&nbsp;&nbsp;`"value-id": <val id int>,`<br />&nbsp;&nbsp;`"value-required": <bool>`<br />&nbsp;&nbsp;`"value": <type JSON>,`<br />&nbsp;&nbsp;`"value-collation": <collation name string>`<br />`}`|`{`<br />&nbsp;&nbsp;`"type": "map",`<br />&nbsp;&nbsp;`"key-id": 4,`<br />&nbsp;&nbsp;`"key": "string",`<br />&nbsp;&nbsp;`"value-id": 5,`<br />&nbsp;&nbsp;`"value-required": false,`<br />&nbsp;&nbsp;`"value": "double"`<br />`}`|
| **`variant`**| `JSON string: "variant"`|`"variant"`|
| **`geometry(C)`** |`JSON string: "geometry(<C>)"`|`"geometry(srid:4326)"`|
| **`geography(C, A)`** |`JSON string: "geography(<C>, <A>)"`|`"geography(srid:4326, spherical)"`|
Expand Down Expand Up @@ -1886,7 +1933,7 @@ The binary single-value serialization can be used to store the lower and upper b

### Version 4

Relative path support is added in v4.
Relative path support is added in v4. Collations for string types are added in v4.

Reading v3 or prior metadata for v4:

Expand All @@ -1908,6 +1955,14 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))

Collation changes:

* String fields may carry a `collation` attribute that defines case-insensitive, accent-insensitive, or locale-aware comparison and ordering (see [Collations](#collations))
* Collation-aware bounds are stored as ordinary `content_stats` under dedicated collation metric fields declared on the field (`collation-metrics`), holding the file's minimum and maximum as original values under the collation order, tagged with the collation and its implementation version
* Writers are encouraged (but not required, as statistics are always optional) to write byte-order bounds for collated columns so collation-unaware readers can still prune
* Readers must not prune comparison, equality, or prefix predicates on a collated column using byte-order bounds, and may use a collation metric field's bounds only when its collation and version match
* Collation-unaware readers ignore the `collation` attribute and collation metric fields, reading collated columns as UTF-8 byte-order strings

### Version 3

Default values are added to struct fields in v3.
Expand Down
Loading
Loading