Skip to content

fix: preserve scalar aggregate cardinality in EXISTS unnest - #3683

Open
vishnujayvel wants to merge 1 commit into
dolthub:mainfrom
vishnujayvel:fix/gms-exists-scalar-agg
Open

fix: preserve scalar aggregate cardinality in EXISTS unnest#3683
vishnujayvel wants to merge 1 commit into
dolthub:mainfrom
vishnujayvel:fix/gms-exists-scalar-agg

Conversation

@vishnujayvel

Copy link
Copy Markdown
Contributor

Summary

Correlated EXISTS whose body is a scalar aggregate (implicit grouping — no GROUP BY keys) should be TRUE for every outer row, including rows with an empty match set. SQL aggregates emit exactly one row over empty input (SUMNULL, COUNT(*)0), so existence never fails for that shape.

On current main, those queries return no outer rows:

CREATE TABLE t(id INT PRIMARY KEY, a INT);
CREATE TABLE u(x INT);
INSERT INTO t VALUES (1,10),(2,20),(3,30);
INSERT INTO u VALUES (10),(10);

-- MySQL: (1),(2),(3) — GMS main: empty
SELECT id FROM t WHERE EXISTS (SELECT SUM(u.x) FROM u WHERE u.x = t.a) ORDER BY id;

-- MySQL: empty — GMS main: (1),(2),(3)  (NOT EXISTS inverted the same bug)
SELECT id FROM t WHERE NOT EXISTS (SELECT SUM(u.x) FROM u WHERE u.x = t.a) ORDER BY id;

The same bug also hit uncorrelated scalar-aggregate EXISTS in FILTER position when the match set is empty (e.g. WHERE EXISTS (SELECT SUM(u.x) FROM u WHERE u.x = 999) or EXISTS (SELECT SUM(e.y) FROM e) on an empty table). Those go through the len(joinFilters)==0 Limit-CrossJoin path after the same GroupBy strip and were wrong on main; this patch fixes them. Projection-position uncorrelated forms (SELECT EXISTS(...)) were already correct and are unchanged.

Root cause

simplifyPartialJoinParents in sql/analyzer/unnest_exists_subqueries.go strips *plan.GroupBy when building the right side of a decorrelated EXISTS semi-join. For len(GroupByExprs) == 0 (scalar/implicit grouping) that changes empty-input cardinality from 1 → 0, so the semi-join sees no right-side rows and drops every outer row.

Fix

When the GroupBy grouping-key list is empty, refuse unnest (return nil, false) so the ExistsSubquery evaluation path keeps the aggregate and preserves the one empty-input row. Explicit GROUP BY columns still strip as before (empty groups produce no rows, so bare-match EXISTS semantics are correct).

This is the same strip list that has bitten related shapes (see dolthub/dolt#10493 for Limit, and historical empty-input existence issues around dolthub/dolt#6898). Window stripping is intentionally out of scope here (row-preserving; handled separately).

Performance

The fix converts SemiJoin unnest into one ExistsSubquery execution per outer row for scalar-aggregate bodies. Correctness outranks the wrong fast plan: an unnested semi-join that strips the scalar aggregate is not a free win when it changes empty-input cardinality.

A strictly faster follow-up would constant-fold EXISTS(scalar aggregate with no HAVING/LIMIT) to TRUE (and NOT EXISTS(...) to FALSE) without executing the subquery at all. That was deliberately not done here — minimal refusal keeps the change small, reviewable, and confined to the strip that causes the bug, rather than adding a new optimizer rewrite with its own edge cases.

Related / rebase notes

  • dolt#10493 (LIMIT 0): this patch incidentally fixes the LIMIT 0 problem for the scalar-aggregate arm only. Bare-column EXISTS (... LIMIT 0) remains wrong before and after this patch; that is partial overlap, not a full fix for #10493.
  • gms PR 3682: same two files, same hunk anchors (join_queries.go after var JoinScriptTests and unnest_exists_subqueries.go after case plan.Having). Textual conflict is certain on rebase; merged semantics are compatible (Window strips then scalar GroupBy refuses).
  • Nested EXISTS: nested EXISTS inside a scalar-aggregate body is no longer unnested (perf-only side effect). Results were verified correct.

Test plan

  • go test ./sql/analyzer/ -count=1
  • go test ./enginetest/ -run TestJoinQueries -count=1
  • go test ./enginetest/... -count=1
  • New JoinScriptTests script covering:
    • correlated bare-column control (match-only)
    • correlated EXISTS (SELECT SUM(...)) empty-match → all outer rows
    • correlated NOT EXISTS (SELECT SUM(...)) → none
    • correlated EXISTS (SELECT COUNT(*)) same
    • correlated EXISTS (SELECT SUM(...) GROUP BY col) empty-match → match-only
    • uncorrelated FILTER-position scalar-agg with empty match (WHERE u.x = 999) → all / none
    • uncorrelated FILTER-position scalar-agg over empty table e → all outer rows

Correlated EXISTS whose body is a scalar aggregate (implicit grouping,
no GROUP BY keys) must be TRUE even when the match set is empty:
aggregates emit exactly one row over empty input. simplifyPartialJoinParents
stripped *plan.GroupBy unconditionally, turning empty-input cardinality
from 1 to 0 and dropping every outer row (NOT EXISTS kept every row).

Refuse unnest when GroupByExprs is empty so ExistsSubquery evaluates the
aggregate correctly. Explicit GROUP BY keys still strip as before.

Add enginetest coverage for SUM/COUNT(*) EXISTS and NOT EXISTS, a bare-
column control, explicit GROUP BY empty-match, and uncorrelated guards.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants