Skip to content

feat: support reading _rw_timestamp in streaming queries #27115

Description

@chenzl25

Is your feature request related to a problem? Please describe.

The hidden _rw_timestamp system column is currently available in batch queries, but selecting it in a streaming query is explicitly rejected:

Not supported: selecting `_rw_timestamp` in a streaming query is not allowed

It would be useful to expose this column to streaming SQL for debugging and observing when rows in an upstream RisingWave table or materialized view were last updated, and to retain that timestamp in downstream materialized views or sinks.

Describe the solution you'd like

Support explicitly selecting _rw_timestamp from a RisingWave table or materialized view in streaming queries. For example:

CREATE TABLE events (id BIGINT PRIMARY KEY, payload VARCHAR);

CREATE MATERIALIZED VIEW events_with_write_time AS
SELECT id, payload, _rw_timestamp AS source_rw_timestamp
FROM events;

Preserve the existing batch semantics: a timestamptz derived from the upstream row version's storage epoch. Reading or backfilling a row should not replace its timestamp with the current processing time.

Correct retractions are essential for mutable inputs. If a row is inserted at T1 and updated at T2, the stream should retract the old row with T1 and insert the new row with T2:

+ (id=1, payload=A, source_rw_timestamp=T1)
- (id=1, payload=A, source_rw_timestamp=T1)
+ (id=1, payload=B, source_rw_timestamp=T2)

Stamping both sides of an update with T2 would break downstream queries that filter, group, or join on this column. Deletes must likewise retract the previously emitted timestamp.

Expected coverage:

  • Consistent timestamps across snapshot reads, changelog catch-up, and live upstream consumption.
  • Correct insert/update/delete behavior, including multiple changes within an epoch and primary-key changes.
  • Correct behavior after recovery and rescaling, including backfill-to-live transitions.
  • Explicit rejection of unsupported scan/backfill modes if support is introduced incrementally.

Describe alternatives you've considered

  • Use batch queries to inspect _rw_timestamp; this works for ad hoc debugging but does not expose it to downstream streaming computations.
  • Carry an application or connector timestamp as a regular column; this records a different event and does not generally represent the upstream RisingWave row's storage version.

Additional context

Related original system-column request: #11629. The documentation currently describes _rw_timestamp as batch-only.

Initial source inspection at 70b0ab709cfc2c6346d61334d0700b07ce72f489 suggests this needs more than removing the frontend validator:

  • Snapshot backfill: ordinary BatchTable snapshot reads already synthesize _rw_timestamp from the storage key. Changelog catch-up and live streaming need additional support. The changelog row decoder currently projects row values without synthesizing this system column. Catch-up may span several epochs, so assigning the range's end epoch to every row would be incorrect.
  • Arrangement backfill: its replicated state-table read path needs separate handling to preserve and expose row-version timestamps.
  • Cross-database backfill: use the upstream database's version timestamps across both snapshots and changelog reads.
  • Planning and live input: the virtual system column is absent from normal upstream row output, so scan schemas/column mappings and old-row timestamp handling need a design. Possible approaches include retaining timestamps per primary key or exposing version information at storage/materialization boundaries; neither approach is prescribed by this request.

This does not inherently require changing every downstream relational operator or every external CDC/source backfill implementation: the requested scope is reading the system column from already-materialized RisingWave relations.

A possible first phase is append-only relations with snapshot backfill, followed by mutable inputs and additional scan modes. Even that first phase needs consistent snapshot, changelog, and live-stream timestamps. The final rollout scope remains open for discussion.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions