Is your feature request related to a problem? Please describe.
The hidden _rw_timestamp system column is currently available in batch queries, but selecting it in a streaming query is explicitly rejected:
Not supported: selecting `_rw_timestamp` in a streaming query is not allowed
It would be useful to expose this column to streaming SQL for debugging and observing when rows in an upstream RisingWave table or materialized view were last updated, and to retain that timestamp in downstream materialized views or sinks.
Describe the solution you'd like
Support explicitly selecting _rw_timestamp from a RisingWave table or materialized view in streaming queries. For example:
CREATE TABLE events (id BIGINT PRIMARY KEY, payload VARCHAR);
CREATE MATERIALIZED VIEW events_with_write_time AS
SELECT id, payload, _rw_timestamp AS source_rw_timestamp
FROM events;
Preserve the existing batch semantics: a timestamptz derived from the upstream row version's storage epoch. Reading or backfilling a row should not replace its timestamp with the current processing time.
Correct retractions are essential for mutable inputs. If a row is inserted at T1 and updated at T2, the stream should retract the old row with T1 and insert the new row with T2:
+ (id=1, payload=A, source_rw_timestamp=T1)
- (id=1, payload=A, source_rw_timestamp=T1)
+ (id=1, payload=B, source_rw_timestamp=T2)
Stamping both sides of an update with T2 would break downstream queries that filter, group, or join on this column. Deletes must likewise retract the previously emitted timestamp.
Expected coverage:
- Consistent timestamps across snapshot reads, changelog catch-up, and live upstream consumption.
- Correct insert/update/delete behavior, including multiple changes within an epoch and primary-key changes.
- Correct behavior after recovery and rescaling, including backfill-to-live transitions.
- Explicit rejection of unsupported scan/backfill modes if support is introduced incrementally.
Describe alternatives you've considered
- Use batch queries to inspect
_rw_timestamp; this works for ad hoc debugging but does not expose it to downstream streaming computations.
- Carry an application or connector timestamp as a regular column; this records a different event and does not generally represent the upstream RisingWave row's storage version.
Additional context
Related original system-column request: #11629. The documentation currently describes _rw_timestamp as batch-only.
Initial source inspection at 70b0ab709cfc2c6346d61334d0700b07ce72f489 suggests this needs more than removing the frontend validator:
- Snapshot backfill: ordinary
BatchTable snapshot reads already synthesize _rw_timestamp from the storage key. Changelog catch-up and live streaming need additional support. The changelog row decoder currently projects row values without synthesizing this system column. Catch-up may span several epochs, so assigning the range's end epoch to every row would be incorrect.
- Arrangement backfill: its replicated state-table read path needs separate handling to preserve and expose row-version timestamps.
- Cross-database backfill: use the upstream database's version timestamps across both snapshots and changelog reads.
- Planning and live input: the virtual system column is absent from normal upstream row output, so scan schemas/column mappings and old-row timestamp handling need a design. Possible approaches include retaining timestamps per primary key or exposing version information at storage/materialization boundaries; neither approach is prescribed by this request.
This does not inherently require changing every downstream relational operator or every external CDC/source backfill implementation: the requested scope is reading the system column from already-materialized RisingWave relations.
A possible first phase is append-only relations with snapshot backfill, followed by mutable inputs and additional scan modes. Even that first phase needs consistent snapshot, changelog, and live-stream timestamps. The final rollout scope remains open for discussion.
Is your feature request related to a problem? Please describe.
The hidden
_rw_timestampsystem column is currently available in batch queries, but selecting it in a streaming query is explicitly rejected:It would be useful to expose this column to streaming SQL for debugging and observing when rows in an upstream RisingWave table or materialized view were last updated, and to retain that timestamp in downstream materialized views or sinks.
Describe the solution you'd like
Support explicitly selecting
_rw_timestampfrom a RisingWave table or materialized view in streaming queries. For example:Preserve the existing batch semantics: a
timestamptzderived from the upstream row version's storage epoch. Reading or backfilling a row should not replace its timestamp with the current processing time.Correct retractions are essential for mutable inputs. If a row is inserted at T1 and updated at T2, the stream should retract the old row with T1 and insert the new row with T2:
Stamping both sides of an update with T2 would break downstream queries that filter, group, or join on this column. Deletes must likewise retract the previously emitted timestamp.
Expected coverage:
Describe alternatives you've considered
_rw_timestamp; this works for ad hoc debugging but does not expose it to downstream streaming computations.Additional context
Related original system-column request: #11629. The documentation currently describes
_rw_timestampas batch-only.Initial source inspection at
70b0ab709cfc2c6346d61334d0700b07ce72f489suggests this needs more than removing the frontend validator:BatchTablesnapshot reads already synthesize_rw_timestampfrom the storage key. Changelog catch-up and live streaming need additional support. The changelog row decoder currently projects row values without synthesizing this system column. Catch-up may span several epochs, so assigning the range's end epoch to every row would be incorrect.This does not inherently require changing every downstream relational operator or every external CDC/source backfill implementation: the requested scope is reading the system column from already-materialized RisingWave relations.
A possible first phase is append-only relations with snapshot backfill, followed by mutable inputs and additional scan modes. Even that first phase needs consistent snapshot, changelog, and live-stream timestamps. The final rollout scope remains open for discussion.