docs: propose backward-compatible typed per-message fields

[amacneil]

Goals

This is a docs-only draft spec proposal to let an MCAP Message (and Attachment) carry an arbitrary number of additional named, typed values — without breaking existing readers or writers. It unifies two long-standing asks under one mechanism:

1. N timestamps per message. Today a Message has exactly log_time + publish_time, and only log_time is indexable, so seeking/playing back by publish time requires a full scan (see discussion #1542, and the closed PR #1196). We want extra timestamps (e.g. publish_time, sensor_time) that are optionally indexed and seekable.
2. Arbitrary per-message metadata. Some pub/sub systems carry attributes out of band from the payload — e.g. a Zenoh Sample has attachment, priority, congestion_control, reliability, express, source_info (discussion #1369). Storing per-message-varying values today forces either a channel explosion (channel metadata) or re-wrapping an already-encoded payload (schema change + extra copy). We want a place for per-message metadata that avoids both.

Additional goals that shaped the design:

• Backward compatibility (hard requirement): existing readers and writers keep working unchanged; only readers that want the new values need updating.
• Zero cost when unused: messages without fields add no bytes (addresses the per-message-overhead concern raised in Support Message metadata field (like Channel) #1369).
• Detectable data loss: if a file is round-tripped through a writer that drops the new records, readers can detect and warn rather than silently losing data.
• Ecosystem-consistent types: scalar types align with the broader data ecosystem (Apache Arrow's scalar taxonomy), using Rust-style width-explicit names (float64, not double).
• Forward compatibility: readers can skip values whose type they don't recognize.

Why this can be backward compatible

The Message record is frozen — its trailing data field has no length prefix, so nothing can be added before or after it within the record. Instead, all new data lives in records with new opcodes, which existing readers already skip (default: continue // skip unrecognized opcodes in Go; parseUnknown in TypeScript). Old readers keep reading log_time/publish_time; new readers pick up the fields.

Design

Five additive records (opcodes 0x10–0x14) plus one optional Statistics field:

Opcode	Record	Purpose
0x10	Field	File-global declaration: id (uint16), name, encoding (logical type), length (physical width). Written like Schema/Channel; duplicatable in the summary.
0x11	Message Fields	Carries (field_id, value) pairs for the message it immediately follows (positional adjacency).
0x12	Field Index	Per-chunk index of an indexed field's value → message offset (mirrors Message Index).
0x13	Field Chunk Index	Per-chunk [min,max] value bounds + index locations for an indexed field (mirrors Chunk Index), for chunk pruning.
0x14	Attachment Fields	Same as Message Fields, but for an Attachment (reuses the same Field declarations; not indexed).

Statistics gains an optional trailing field_value_bounds map (file-global per-indexed-field bounds).

Field sentinel (signal + integrity check)

When a Message has a Message Fields record, the writer MUST set the message's publish_time to the reserved sentinel 0xFFFFFFFFFFFFFFFF (and an Attachment with an Attachment Fields record sets create_time likewise). This rides in a field that existing writers preserve, while the fields record itself rides in a record they may drop. So:

• In-band signal: readers only look for a following fields record when the sentinel is set.
• Integrity check: a sentinel without a following fields record means the fields were almost certainly dropped during a round-trip through a writer that predates this feature — readers SHOULD warn instead of silently losing data.
• The true publish_time/create_time, if any, is carried as a field; otherwise readers fall back to log_time as today. Old readers see the sentinel as a literal (year-2554) timestamp.

Type system: physical length + logical encoding

Following the physical/logical split common to Parquet/Avro/Arrow:

• length byte = physical: high bit set ⇒ variable-length (uint32 length prefix per value); otherwise the low 7 bits are a fixed byte width. This alone lets a reader parse/skip any value, even one whose encoding it has never seen (forward-compatible to future types).
• encoding string = logical: a concise, Arrow-aligned scalar set with Rust-style names — bool, int8/16/32/64, uint8/16/32/64, float16/32/64, timestamp (uint64 ns, matches log_time), string, bytes. Deliberately scalar-only: composite data belongs in the payload + schema. Structured bytes may reuse MCAP's existing well-known message encodings (json, protobuf, …).

A field entry is <uint16 field_id><value>; values are little-endian.

Indexing (implicit, like log_time)

There is no indexed flag. A field is indexed if and only if Field Index / Field Chunk Index records are present for it — exactly how log_time indexing is signaled by the presence of Message Index / Chunk Index records. Indexing is thus an optional, per-field choice the writer makes by emitting (or not emitting) the index records. Only fixed-width 64-bit orderable encodings (timestamp, uint64, int64, float64) may be indexed.

Key choices

• Adjacency, not a foreign key (placed after the record so an index-based seek naturally reads it next), with the sentinel as the authoritative presence signal.
• 2-byte field IDs referencing a global registry, consistent with schema_id/channel_id.
• Non-monotonicity is called out — unlike log_time, fields like publish_time may not be monotonic, so chunk ranges can overlap and prune less effectively.

Mapping the use cases

• Extra indexed timestamp: Field{name:"publish_time", encoding:"timestamp", length:8} + write Field Index / Field Chunk Index records → seekable.
• Zenoh per-message metadata: attachment → bytes; per-message source_sn → uint32; constant-per-publisher QoS (priority, reliability, …) → channel metadata.

Changes

• website/docs/spec/index.md: five new record definitions; the field-sentinel section; data/summary allowed-records lists; Message/Attachment/Chunk/Statistics notes.
• website/docs/spec/registry.md: new "Field encodings" section (scalar type registry + recommended field names).
• website/docs/spec/notes.md: implementation notes for reading, seeking, and the sentinel.
• website/docs/spec/mcap.ksy: Kaitai definitions for the new records (the Message Fields/Attachment Fields value layout is registry-dependent, so its fields blob is left unexpanded in the Kaitai model, with a doc note).
• cspell.config.yaml: add seekable.

This is docs-only — no library code is implemented yet. It captures the design for discussion.

Testing

• yarn workspace website build — Docusaurus build succeeds with onBrokenLinks: "throw"; verified generated anchors (field-op0x10, message-fields-op0x11, field-index-op0x12, field-chunk-index-op0x13, attachment-fields-op0x14, field-sentinel, field-encodings) match all cross-links.
• prettier --check — passes on all edited spec files.
• cspell — passes on all edited spec files.
• python3 -c "import yaml; yaml.safe_load(...)" — mcap.ksy parses as valid YAML (no Kaitai compiler available in the environment).

  

[claude]

Draft proposal — flagging design gaps now so they don't bake in. Cross-cutting items to resolve before this is implementable:

• Index-record layout (inline): message_index_length semantics break down under "interleaved" placement, for both the standard Chunk Index and the Auxiliary Chunk Index. Needs a deterministic byte layout.
• Summary self-sufficiency (inline ×2): seeking by name from the summary alone needs the Timestamp Name registry to be in the summary, and needs names to be unique — both are currently optional/unspecified.

Minor, on the PR description: the overhead math looks light. Each timestamp entry is 10 bytes (uint16 id + uint64 value), not 8, plus a one-time 4-byte array length prefix — so a one-aux-timestamp message is ~25 raw bytes, not ~17–19. Doesn't change the "compresses to nearly nothing" conclusion, but worth correcting.

[claude]

interleaved with that chunk's Message Index records makes message_index_length ambiguous in two places.

The standard Chunk Index defines message_index_length as the total byte length of the message index records after the chunk, and some readers read that whole span in one shot (e.g. the Rust CLI's remote range reads). If 0x12 records are interleaved, does that length now include them or not?

And the Auxiliary Chunk Index's own message_index_length is "total length ... for this timestamp ID" — with multiple timestamp IDs interleaved, the records for a given ID are no longer contiguous, so a single length can't describe them.

Either mandate a deterministic layout (e.g. all standard Message Index records contiguous, then each timestamp ID's aux index records contiguous) and define each length over its own block, or spell out exactly how a reader walks the interleaving. As written it's underspecified.

[claude]

Timestamp Name in the summary is optional here ("may be duplicated"), but indexed seeking needs it. Compare the existing Chunk Index rule: "A Schema and Channel record MUST exist in the summary section for all messages in chunks that are indexed by Chunk Index records." Same logic applies — if Auxiliary Chunk Index records (and auxiliary_message_time_bounds in Statistics, which is keyed by ID) are present in the summary, a reader doing a summary-only seek can't resolve a name → ID without falling back to a data-section scan, defeating the point of the index. Should this be a MUST-duplicate-in-summary when aux indexes exist?

[claude]

map the desired name to its ID assumes names are unique, but the spec only requires IDs to be unique (registry.md: "Names are free-form"). Two Timestamp Name records with different IDs and the same name are legal today, which makes name → ID a one-to-many lookup and this step ambiguous. Either require names to be unique within a file, or define how a reader resolves a name that maps to multiple IDs.

[claude]

Given strict adjacency to a Message that already carries channel_id, what does repeating it here buy beyond a redundant validation check? The PR description calls this an "optional channel_id", but the table marks it required with a MUST-match rule — reconcile the two. If it's purely a consistency guard, say so and own the 2 bytes/message; if it's meant to be omittable, the layout and parsing need to reflect that.

[claude]

keeps merge operations cheap — does it, really? On merge, timestamp IDs collide across inputs (file A's ID 1 ≠ file B's ID 1), exactly like schema/channel IDs do today. Resolving that means remapping IDs inside every Message Auxiliary Timestamps record, plus both aux index records and the Statistics bounds — which is at odds with the "tools that copy records through verbatim preserve this pairing" point right above. Worth calling out the remap cost instead of implying merge is free.

[claude]

Confirmed the Go reader bounds its read by the channel_message_counts length prefix and ignores trailing bytes (ParseStatistics in go/mcap/parse.go), so this is compatible there. Worth confirming the same for TS/Python/Rust/C++/Swift before relying on it — a stricter reader that validates record length against parsed fields would reject the extra bytes. Also worth a one-line note on why Statistics is extended in place here, when the rest of the proposal deliberately avoids touching frozen records and adds new opcodes instead.

[claude]

This rework renamed the records (Timestamp Name → Field, Message Auxiliary Timestamps → Message Fields, Auxiliary Message/Chunk Index → Field Index / Field Chunk Index) but kept the structure identical, so every concern from my prior review still applies to the renamed records. Those threads now show as "outdated" but remain unaddressed: the message_index_length ambiguity when Field Index records interleave with Message Index records after a chunk; Field records being only optionally duplicated in the summary (an indexed name→ID seek needs them in the summary, cf. the existing Schema/Channel MUST rule for Chunk Index); name→ID resolution assuming unique names while the registry still says names are free-form; the redundant channel_id in Message Fields; recommending a publish_time field name that collides with the Message's built-in publish_time; the unstated ID-remap cost on merge; and extending the frozen Statistics record in place rather than via a new opcode. Worth working back through those before this moves past draft.

New for the typed design: see the inline note on Field Chunk Index min_value/max_value.

[claude]

min_value/max_value are typed uint64, but indexed encodings now include int64 and float64 — two things bake in here:

1. Ordering. Pruning compares the query against [min_value, max_value]. Treating these as raw uint64 (as the type column says) gives wrong results for int64 (negative two's-complement sorts above positives) and float64 (sign bit + bit-pattern order ≠ numeric order). The Field Index records value column and the Statistics field_value_bounds share the trap. Either state explicitly that min/max and all comparisons use the field's logical type ordering — and define what float64 NaN / ±0 do — or restrict indexing to uint64/timestamp.

2. Zero sentinel. "Zero if no messages in the chunk carry it" collides with a legitimate value of 0 (common for int/float, unlike log_time). A chunk with no field-carrying messages is then indistinguishable from one whose values are all 0, so a value == 0 query can't prune it. If a Field Chunk Index is emitted per (chunk, indexed field) regardless of presence, add an explicit presence/count signal rather than overloading 0.

[claude]

Reviewing the field-sentinel commit (7314361) on top of my last pass.

The sentinel is a clever presence/integrity signal, but it flips the feature from purely additive to mutating a populated built-in field (publish_time/create_time). That tradeoff is the root of both inline gaps: a real publish_time can be lost the moment you attach any unrelated field, and the Attachment Index's duplicated create_time now has undefined meaning under the sentinel. Worth deciding whether mutating the built-in is acceptable vs. signaling presence out of band before this moves past draft.

The structural concerns from my prior passes (index interleaving layout, summary self-sufficiency for name→ID, free-form name uniqueness, merge ID-remap cost, extending the frozen Statistics record in place, and the int64/float64 min/max ordering + zero-sentinel) remain open in their threads — the rename to typed fields kept the underlying structure, so they still apply.

[claude]

The sentinel turns Message Fields from purely additive into something that mutates publish_time — and the salvage is only "if any," not a MUST. So a writer that attaches unrelated metadata (say a Zenoh source_sn) to a message whose publish_time ≠ log_time now MUST also re-encode the original publish_time as a field, or it's silently downgraded to log_time. The previous design didn't have this footgun: fields were extra, publish_time stayed put.

Two ways out: require it ("a writer that sets the sentinel and had a real, non-sentinel publish_time MUST carry it as a publish_time field"), or reconsider signaling presence out of band (e.g. a Message flag bit / separate signal) instead of clobbering a populated field. As written, the act of adding any field can lose a real timestamp. 🤔

[claude]

Extending the sentinel to attachments collides with the Attachment Index (op=0x0A), which duplicates create_time (line 290). When an attachment carries fields, its record create_time is the sentinel — what does the index copy hold?

• If it mirrors the record (sentinel), a summary-only attachment listing shows year-2554 for every attachment that carries fields, and the real create_time lives in the Attachment Fields record, which isn't in the summary — so it's unrecoverable without scanning the data section.
• If it holds the real value, it diverges from the record and breaks the "sentinel ⇒ fields follow" contract that holds everywhere else.

Messages dodge this because Message Index keys only on log_time. Attachments don't, so the spec needs to say which value the index carries (and ideally how a summary reader recovers the true create_time).

[claude]

Only delta since my last pass (107db43) is dropping the indexed flag — see the inline note on why making indexing implicit needs an explicit per-chunk completeness invariant or field seeks go unsound. The structural threads from earlier passes are still open and unaddressed; worth working through them before this leaves draft.

[claude]

Dropping the flag removes the only file-global "field F is indexed" signal — indexed-ness is now defined purely by the presence of index records, which is per-(chunk, field) and circular (line 401: "every (chunk, indexed field ID) combination that is indexed"). Nothing then requires that every chunk carrying F has a Field Chunk Index for it.

A summary-only seek (notes.md "Seeking by a field value") collects the Field Chunk Index records for F and prunes by them — so any chunk that carries F but lacks a Field Chunk Index for it is invisible to the seek, and its messages are silently missed. log_time can't hit this because Chunk Index is mandatory for every chunk in an indexed file; with the flag gone, F has no equivalent guarantee.

State the invariant explicitly: if any Field Chunk Index exists for F, then every chunk containing a message that carries F MUST have one (and the corresponding Field Index). Otherwise index-based field seeks are unsound. 🤔