[SPARK-59030][SQL] Support DSv2 Join pushdown for DB2 connector - #58321
Open
shrirangmhalgi wants to merge 1 commit into
Open
[SPARK-59030][SQL] Support DSv2 Join pushdown for DB2 connector#58321shrirangmhalgi wants to merge 1 commit into
shrirangmhalgi wants to merge 1 commit into
Conversation
Fix DB2SQLQueryBuilder.build() to use the tableOrQuery method instead of directly accessing options.tableOrQuery. The tableOrQuery method returns the join subquery when joinQuery is set by withJoin(), which is required for join pushdown to produce correct SQL. Without this fix, enabling supportsJoin would generate queries against the raw table name, ignoring the pushed join entirely. Also enables supportsJoin = true for DB2Dialect, completing the join pushdown support that was added for Oracle (SPARK-52823), PostgreSQL (SPARK-52906), and MySQL/MSSQL (SPARK-52929). DB2 uses ANSI standard join syntax, so no dialect-specific SQL generation override is needed -- the shared JdbcSQLQueryBuilder.withJoin() framework handles it correctly. Tests: - Added unit test in JDBCSuite verifying DB2 join query builder output - Added DB2JoinPushdownIntegrationSuite (Docker integration test) - All 317 existing JDBC tests pass with zero regressions
shrirangmhalgi
commented
Aug 26, 2026
Contributor
Author
There was a problem hiding this comment.
cc: @PetarVasiljevic-DB / @cloud-fan. Could you PTAL. This PR extends the join pushdown series built (SPARK-52187 / SPARK-52823 / SPARK-52906 / SPARK-52929) to DB2 - also fixes a latent bug in DB2SQLQueryBuilder.build() that used options.tableOrQuery instead of the tableOrQuery method.
Thank you.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR enables DSv2 Join pushdown for the DB2 JDBC connector by:
DB2SQLQueryBuilder.build()to use thetableOrQuerymethod instead of directly accessingoptions.tableOrQuery. ThetableOrQuerymethod returns the join subquery (when set bywithJoin()), which is required for join pushdown to produce correct SQL.override def supportsJoin: Boolean = trueto DB2Dialect.Why are the changes needed?
Join pushdown was introduced in SPARK-52187 and enabled for Oracle (SPARK-52823), PostgreSQL (SPARK-52906), and MySQL/MSSQL (SPARK-52929). DB2 was not included in that batch despite supporting ANSI standard join syntax natively.
Additionally,
DB2SQLQueryBuilder.build()had a latent bug - it used${options.tableOrQuery}(the raw table name) instead of$tableOrQuery(the method that returns the join subquery when one is pushed). Without this fix, enabling supportsJoin would generate incorrect SQL that ignores the pushed join entirely. All other dialects with custom build() overrides (Oracle, MySQL, MSSQL) correctly use the$tableOrQuerymethod.Does this PR introduce any user-facing change?
No. Join pushdown is an internal optimization that produces the same query results. Users reading from DB2 via JDBC with joins will see improved performance as the join is now executed on the DB2 server instead of in Spark.
How was this patch tested?
Was this patch authored or co-authored using generative AI tooling?
Yes. Co-Authored using Claude Opus 4.8.