Motivation
PR #27065 fixes the correctness issue in #27035 without introducing shared subplan execution for batch queries:
- Correlated scalar aggregates that can be decorrelated without a correlation domain, such as
SUM, are rewritten directly into an aggregate + join plan.
- When general decorrelation still requires evaluating the outer relation twice, batch queries containing impure expressions such as
random() are rejected instead of returning incorrect results.
- Streaming execution can already use
LogicalShare to ensure that the outer relation is evaluated once and consumed by multiple branches.
A more general solution for batch execution would be to support query-scoped shared subplans, e.g. a spool/materialization operator or DAG execution, so that plans such as
shared outer
/ \
final output correlation domain
can consume the same evaluation of the outer relation.
Why not implement this now
Supporting batch sharing is a relatively large piece of execution infrastructure. A complete implementation may need to consider:
- multiple consumers of the same batch subplan;
- memory management and possible spilling;
- scheduling and backpressure between producer and consumers;
- distribution/exchange placement;
- lifetime and cancellation handling;
- optimizer and physical-plan support for DAG-shaped plans.
The workload is currently uncommon for RisingWave compared with typical streaming and analytical workloads. It provides a safe behavior today: optimize common correlated aggregate cases directly, and reject the remaining unsafe batch cases rather than returning incorrect results.
Therefore, we do not plan to introduce batch sharing solely for #27035.
Possible future scope
If there is sufficient user demand, or if batch shared subplans become useful for multiple features such as:
- general correlated-subquery decorrelation;
- common-subexpression elimination;
- CTE/subplan reuse;
- avoiding duplicate expensive scans or aggregates;
we can revisit a query-scoped batch share/spool implementation.
Until then, unsupported correlated batch queries that require shared evaluation can be materialized explicitly by the user before running the query.
Motivation
PR #27065 fixes the correctness issue in #27035 without introducing shared subplan execution for batch queries:
SUM, are rewritten directly into an aggregate + join plan.random()are rejected instead of returning incorrect results.LogicalShareto ensure that the outer relation is evaluated once and consumed by multiple branches.A more general solution for batch execution would be to support query-scoped shared subplans, e.g. a spool/materialization operator or DAG execution, so that plans such as
can consume the same evaluation of the outer relation.
Why not implement this now
Supporting batch sharing is a relatively large piece of execution infrastructure. A complete implementation may need to consider:
The workload is currently uncommon for RisingWave compared with typical streaming and analytical workloads. It provides a safe behavior today: optimize common correlated aggregate cases directly, and reject the remaining unsafe batch cases rather than returning incorrect results.
Therefore, we do not plan to introduce batch sharing solely for #27035.
Possible future scope
If there is sufficient user demand, or if batch shared subplans become useful for multiple features such as:
we can revisit a query-scoped batch share/spool implementation.
Until then, unsupported correlated batch queries that require shared evaluation can be materialized explicitly by the user before running the query.