[SQL] Fix JDBC V2 join pushdown for partitioned reads and partial AVG - #58958
Draft
urosstan-db wants to merge 2 commits into
Draft
urosstan-db wants to merge 2 commits into
urosstan-db wants to merge 2 commits into
Conversation
urosstan-db
commented
Sep 22, 2026
| val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "false") { | ||
| sql(sqlQuery).collect().toSeq | ||
| } | ||
| assert(rows.nonEmpty) |
Contributor
Author
There was a problem hiding this comment.
Let's use only join pushdown = true path, assert join is not pushed down, and assert on entire result
urosstan-db
commented
Sep 22, 2026
|
|
||
| test("Join pushdown preserves aliases in partially pushed averages") { | ||
| assume(supportsAggregatePushdown, "Aggregate pushdown is not supported") | ||
| val id = caseConvert("id") |
Contributor
Author
There was a problem hiding this comment.
Move id to inner function, name it idCol
urosstan-db
commented
Sep 22, 2026
| |""".stripMargin | ||
| } | ||
|
|
||
| val rows = withSQLConf(SQLConf.DATA_SOURCE_V2_JOIN_PUSHDOWN.key -> "true") { |
Contributor
Author
There was a problem hiding this comment.
Use better naming, e.g. rowsWithJoinPushdown
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fix two JDBC Data Source V2 join pushdown bugs:
partitionColumnconfigured as Spark-side joins, preserving partitioned JDBC reads. Log the reason at debug level.AVGintoSUMandCOUNT.Add SQL regressions to
JDBCV2JoinPushdownIntegrationSuiteBase.Why are the changes needed?
Join pushdown creates a JDBC
query, which cannot be combined withpartitionColumn. Partitioned inputs currently fail withOptions 'query' and 'partitionColumn' can not be specified together.Partial
AVGnormalization uses the original relation output instead of the pushed-join aliases. This can silently aggregate the wrong column: the H2 self-join regression returns5.413043instead of4.413043without the fix.Does this PR introduce any user-facing change?
Yes. With JDBC V2 join pushdown enabled, partitioned input queries execute successfully using Spark-side joins, and partially pushed averages over joins return the correct result.
How was this patch tested?
The shared SQL tests cover partitioned reads with one and two partitions, and compare AVG results with one partition, two partitions, and join pushdown disabled. The AVG test is skipped when aggregate pushdown is unsupported.
Confirmed both original regressions fail before their fixes. Also reran the revised three-way AVG test without its fix and confirmed the incorrect result above.
Validation on the committed changes with JDK 21 and H2: all 24 tests in
JDBCV2JoinPushdownSuiteand all 3 selected partial-pushdown tests inJDBCV2Suitepassed.Other databases' integration suites were not run.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex CLI 0.154.0