Skip to content

docs: propose backward-compatible typed per-message fields #1752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
amacneil wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

docs: propose backward-compatible typed per-message fields #1752

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
863354e
docs: propose backward-compatible N auxiliary timestamps per message
cursoragent Jun 29, 2026
c32cee6
ci: add 'seekable' to cspell dictionary
cursoragent Jun 29, 2026
e90cded
docs: rework proposal into unified typed message fields
cursoragent Jun 29, 2026
7314361
docs: add field sentinel to signal and detect dropped fields records
cursoragent Jun 29, 2026
107db43
docs: add float16 to field encodings registry
cursoragent Jun 29, 2026
de4a98a
docs: make field indexing implicit, drop indexed flag
cursoragent Jun 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cspell.config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ words:
- rustup
- Saronic
- schemaless
- seekable
- serde
- sfixed
- srgb
Expand Down
112 changes: 100 additions & 12 deletions website/docs/spec/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ sidebar_position: 1
[message_encodings]: ./registry.md#well-known-message-encodings
[schema_encodings]: ./registry.md#well-known-schema-encodings
[profiles]: ./registry.md#well-known-profiles
[field_encodings]: ./registry.md#field-encodings
[feature_explanations]: ./notes.md#feature-explanations

## Overview
Expand Down Expand Up @@ -60,9 +61,12 @@ The following records are allowed to appear in the data section:
- [Schema](#schema-op0x03)
- [Channel](#channel-op0x04)
- [Message](#message-op0x05)
- [Field](#field-op0x10)
- [Message Fields](#message-fields-op0x11)
- [Attachment](#attachment-op0x09)
- [Chunk](#chunk-op0x06)
- [Message Index](#message-index-op0x07)
- [Field Index](#field-index-op0x12)
- [Metadata](#metadata-op0x0c)
- [Data End](#data-end-op0x0f)

Expand All @@ -82,7 +86,9 @@ The following records are allowed to appear in the summary section:

- [Schema](#schema-op0x03)
- [Channel](#channel-op0x04)
- [Field](#field-op0x10)
- [Chunk Index](#chunk-index-op0x08)
- [Field Chunk Index](#field-chunk-index-op0x13)
- [Attachment Index](#attachment-index-op0x0a)
- [Metadata Index](#metadata-index-op0x0d)
- [Statistics](#statistics-op0x0b)
Expand Down Expand Up @@ -181,9 +187,11 @@ The message encoding and schema must match that of the Channel record correspond
| 8 | publish_time | Timestamp | Time at which the message was published. If not available, must be set to the log time. |
| N | data | Bytes | Message data, to be decoded according to the schema of the channel. |

A message may carry additional named, typed values (such as extra timestamps or per-message metadata) by following it with a [Message Fields](#message-fields-op0x11) record. The Message record itself is never extended, so readers that predate the Message Fields record continue to read `log_time` and `publish_time` unchanged.

### Chunk (op=0x06)

A Chunk contains a batch of records. Readers should expect Schema, Channel, and Message records to be present in chunks, but future spec changes may include others. Private records may also be included. The batch of records contained in a chunk may be compressed or uncompressed.
A Chunk contains a batch of records. Readers should expect Schema, Channel, and Message records to be present in chunks, but future spec changes may include others. Chunks may also contain [Field](#field-op0x10) and [Message Fields](#message-fields-op0x11) records. Private records may also be included. The batch of records contained in a chunk may be compressed or uncompressed.

All messages in the chunk must reference channels recorded earlier in the file (in a previous chunk, earlier in the current chunk, or earlier in the data section).

Expand Down Expand Up @@ -293,17 +301,20 @@ A metadata index record contains the location of a metadata record within the fi

A Statistics record contains summary information about the recorded data. The statistics record is optional, but the file should contain at most one.

| Bytes | Name | Type | Description |
| ----- | ---------------------- | --------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| 8 | message_count | uint64 | Number of Message records in the file. |
| 2 | schema_count | uint16 | Number of unique schema IDs in the file, not including zero. |
| 4 | channel_count | uint32 | Number of unique channel IDs in the file. |
| 4 | attachment_count | uint32 | Number of Attachment records in the file. |
| 4 | metadata_count | uint32 | Number of Metadata records in the file. |
| 4 | chunk_count | uint32 | Number of Chunk records in the file. |
| 8 | message_start_time | Timestamp | Earliest message log_time in the file. Zero if the file has no messages. |
| 8 | message_end_time | Timestamp | Latest message log_time in the file. Zero if the file has no messages. |
| 4 + N | channel_message_counts | `Map<uint16, uint64>` | Mapping from channel ID to total message count for the channel. An empty map indicates this statistic is not available. |
| Bytes | Name | Type | Description |
| ----- | ---------------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 8 | message_count | uint64 | Number of Message records in the file. |
| 2 | schema_count | uint16 | Number of unique schema IDs in the file, not including zero. |
| 4 | channel_count | uint32 | Number of unique channel IDs in the file. |
| 4 | attachment_count | uint32 | Number of Attachment records in the file. |
| 4 | metadata_count | uint32 | Number of Metadata records in the file. |
| 4 | chunk_count | uint32 | Number of Chunk records in the file. |
| 8 | message_start_time | Timestamp | Earliest message log_time in the file. Zero if the file has no messages. |
| 8 | message_end_time | Timestamp | Latest message log_time in the file. Zero if the file has no messages. |
| 4 + N | channel_message_counts | `Map<uint16, uint64>` | Mapping from channel ID to total message count for the channel. An empty map indicates this statistic is not available. |
| 4 + N | field_value_bounds | `Map<uint16, Tuple<uint64, uint64>>` | Optional. Mapping from indexed field ID to the (minimum, maximum) value of that field across all messages in the file. Each bound is the field's 8-byte value, interpreted according to the field's encoding. An empty map indicates this statistic is not available. |

The `field_value_bounds` field is an optional backward-compatible extension appended after `channel_message_counts`. Readers that predate the Field record read the preceding fields and ignore the trailing bytes. Writers that do not record indexed fields may omit the field entirely. Only fields with a fixed-width 64-bit encoding may appear here (see [Field](#field-op0x10)).

When using a Statistics record with a non-empty channel_message_counts, the Summary Data section MUST contain a copy of all Channel records. The Channel records MUST occur prior to the statistics record.

Expand All @@ -319,6 +330,83 @@ A Summary Offset record contains the location of records within the summary sect
| 8 | group_start | uint64 | Byte offset from the start of the file of the first record in the group. |
| 8 | group_length | uint64 | Total byte length of all records in the group. |

### Field (op=0x10)

A Field record declares a named, typed value that messages may carry in addition to the fields built into the [Message](#message-op0x05) record. Fields are used both for additional timestamps (e.g. an indexable `publish_time`) and for arbitrary per-message metadata (e.g. middleware attributes carried out of band from the message payload).

Fields are uniquely identified within a file by their field ID. A Field record must occur at least once in the file prior to any [Message Fields](#message-fields-op0x11), [Field Index](#field-index-op0x12), or [Field Chunk Index](#field-chunk-index-op0x13) record referring to its ID. Any two Field records sharing a common ID must be identical.

| Bytes | Name | Type | Description |
| ----- | -------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 2 | id | uint16 | A unique identifier for this field within the file. Must not be zero. |
| 4 + N | name | String | A human-readable name for the field, e.g. `publish_time`, `sensor_time`, `attachment`. |
| 4 + N | encoding | String | The value encoding. One of the [well-known field encodings][field_encodings] (e.g. `timestamp`, `uint32`, `float64`, `string`, `bytes`). |
| 1 | length | uint8 | The wire length of the value. If the high bit (`0x80`) is set, the value is variable-length (see below); otherwise the low 7 bits are a fixed byte length (0–127). |
| 1 | flags | uint8 | Bit flags. Bit 0 (`0x01`) set indicates the field is _indexed_ (see [Field Index](#field-index-op0x12)). All other bits are reserved and must be zero. |

The `length` byte is the physical descriptor used to parse a value; the `encoding` is the logical descriptor used to interpret it. A reader can therefore skip a field whose `encoding` it does not recognize, as long as it has the Field record (which gives the length). For well-known encodings, `length` must be consistent with the encoding (e.g. `timestamp` and `uint64` must use `length = 8`).

The `indexed` flag may only be set for fixed-width 64-bit orderable encodings (`timestamp`, `uint64`, `int64`, `float64`); i.e. `length` must equal 8.

Field records may be duplicated in the summary section. A Field record with an id of zero is invalid and should be ignored by readers. Readers that do not support fields will skip this record.

### Message Fields (op=0x11)

A Message Fields record provides additional named, typed values for a single [Message](#message-op0x05) record.

It MUST appear immediately after the Message record it annotates, with no intervening records, in the same record stream (the data section or within the same chunk). At most one Message Fields record may be associated with a given Message.

> Because this record uses a new opcode, readers that predate it skip it and continue reading the `log_time` and `publish_time` of the preceding Message unchanged. This makes message fields a backward-compatible addition: existing readers and writers are unaffected, and only readers that wish to access the additional values need to be updated. Messages that carry no fields incur no overhead.

| Bytes | Name | Type | Description |
| ----- | ---------- | ---------------------------------- | ------------------------------------------------------------------------------------ |
| 2 | channel_id | uint16 | Channel ID. Must match the `channel_id` of the immediately preceding Message record. |
| 4 + N | fields | Array of field entries (see below) | The field values carried by this message. |

Each entry in `fields` is serialized as:

<uint16 field_id><value>

`field_id` must be registered by a Field record and must not appear more than once within the record. `value` is serialized according to that field's `length`:

- **Fixed-width** (`length` high bit clear): exactly `length & 0x7F` bytes, little-endian for numeric encodings.
- **Variable-length** (`length` high bit set): a `uint32` byte length followed by that many bytes.

A Message Fields record that is not immediately preceded by a Message record, or whose `channel_id` does not match the preceding Message, is invalid and should be ignored by readers.

### Field Index (op=0x12)

A Field Index record allows readers to locate individual Message records within a chunk by the value of an indexed field, analogous to the [Message Index](#message-index-op0x07) record for `log_time`.

A sequence of Field Index records may occur immediately after a chunk, interleaved with that chunk's Message Index records. At most one Field Index record exists per (channel, field ID) combination for which messages in the chunk carry that field. Only fields whose Field record has the `indexed` flag set are indexed.

| Bytes | Name | Type | Description |
| ----- | ---------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2 | channel_id | uint16 | Channel ID. |
| 2 | field_id | uint16 | The field ID being indexed. Must be registered by a Field record with the `indexed` flag set. |
| 4 + N | records | `Array<Tuple<uint64, uint64>>` | Array of (field value, offset) pairs. The value is the field's 8-byte value, interpreted per its encoding. Offset is the position of the Message record relative to the start of the uncompressed chunk data. |

After seeking to a Message via its offset, the message's fields can be read from the Message Fields record that immediately follows it.

Messages outside of chunks cannot be indexed.

### Field Chunk Index (op=0x13)

A Field Chunk Index record describes the range of an indexed field's value within a chunk and locates that chunk's Field Index records, analogous to the [Chunk Index](#chunk-index-op0x08) record for `log_time`. Readers use it to prune chunks when seeking by a field value.

One Field Chunk Index record exists for every (chunk, indexed field ID) combination that is indexed.

| Bytes | Name | Type | Description |
| ----- | --------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 2 | field_id | uint16 | The indexed field ID. Must be registered by a Field record with the `indexed` flag set. |
| 8 | chunk_start_offset | uint64 | Offset to the Chunk record from the start of the file. Matches the `chunk_start_offset` of the corresponding [Chunk Index](#chunk-index-op0x08) record. |
| 8 | min_value | uint64 | Minimum value of this field among messages in the chunk, interpreted per its encoding. Zero if no messages in the chunk carry it. |
| 8 | max_value | uint64 | Maximum value of this field among messages in the chunk, interpreted per its encoding. Zero if no messages in the chunk carry it. |
Comment on lines +407 to +408

@claude claude Bot Jun 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

min_value/max_value are typed uint64, but indexed encodings now include int64 and float64 — two things bake in here:

  1. Ordering. Pruning compares the query against [min_value, max_value]. Treating these as raw uint64 (as the type column says) gives wrong results for int64 (negative two's-complement sorts above positives) and float64 (sign bit + bit-pattern order ≠ numeric order). The Field Index records value column and the Statistics field_value_bounds share the trap. Either state explicitly that min/max and all comparisons use the field's logical type ordering — and define what float64 NaN / ±0 do — or restrict indexing to uint64/timestamp.

  2. Zero sentinel. "Zero if no messages in the chunk carry it" collides with a legitimate value of 0 (common for int/float, unlike log_time). A chunk with no field-carrying messages is then indistinguishable from one whose values are all 0, so a value == 0 query can't prune it. If a Field Chunk Index is emitted per (chunk, indexed field) regardless of presence, add an explicit presence/count signal rather than overloading 0.

| 4 + N | message_index_offsets | `Map<uint16, uint64>` | Mapping from channel ID to the offset of the Field Index record for this field ID and channel, from the start of the file. An empty map indicates no field indexing is available for this chunk. |
| 8 | message_index_length | uint64 | Total length in bytes of the Field Index records for this field ID after the chunk. |

> Fields such as `publish_time` are not necessarily monotonic across a recording. When they are not, the `[min_value, max_value]` ranges of adjacent chunks may overlap, reducing the effectiveness of chunk pruning compared to `log_time`.

## Serialization

### Fixed-width types
Expand Down
Loading
Loading