perf: skip DecimalPrecision.promote's rewrite when there is no decimal arithmetic - #5216
perf: skip DecimalPrecision.promote's rewrite when there is no decimal arithmetic#5216andygrove wants to merge 4 commits into
Conversation
…l arithmetic QueryPlanSerde.exprToProto runs DecimalPrecision.promote over every expression it converts. The rule is a transformUp, so it walks and rebuilds every node even when nothing matches, which is the common case. Guard it with a cheap traversal that looks for arithmetic producing a decimal, and only run the rewrite when one exists.
0lai0
left a comment
There was a problem hiding this comment.
Thanks @andygrove looks good to me
| object DecimalPrecision { | ||
| def promote(expr: Expression): Expression = { | ||
| // `transformUp` walks and rebuilds every node even when no case matches, and the serde calls | ||
| // this once per expression it converts, so skip it when there is nothing to rewrite. |
There was a problem hiding this comment.
Nit: Could you reword this comment slightly?
transformUp does call mapChildren at every node, but specialized mapChildren (UnaryLike / BinaryLike / etc.) skip the copy via fastEquals when nothing changed , so it doesn't really "rebuild every node".
The cost is the traversal + allocations (children.map / mapProductIterator).
The PR body's wording already captures this well; matching that here would help future profilers.
There was a problem hiding this comment.
Reworded in 4e0502e — you're right, "rebuilds every node" overstates it and points a future profiler at the wrong thing.
// `transformUp` walks the whole tree and calls `mapChildren` at every node, which allocates
// through `children.map` / `mapProductIterator` even when no case matches (the specialized
// `mapChildren` overrides do skip the node copy via `fastEquals`, so the cost is the traversal
// and its allocations, not a rebuild). The serde calls this once per expression it converts,
// so skip it when there is nothing to rewrite.That now says the same thing as the PR body, and naming fastEquals explicitly should stop the next reader concluding there is a copy to eliminate.
Which issue does this PR close?
Part of #5199 (item 4:
DecimalPrecision.promoteis a second full traversal of every expression tree).Rationale for this change
QueryPlanSerde.exprToProtocallsDecimalPrecision.promoteon every expression it converts, andpromoteis aTreeNode.transformUp.transformUpwalks the whole tree and callsmapChildrenatevery node, which allocates through
mapProductIteratorregardless of whether any case matches. Theresult is a full extra traversal of every expression tree before the serde walks it again, paid on
the driver for every query and again for every query stage under AQE.
The rule only ever rewrites arithmetic that produces a decimal, so on the overwhelming majority of
expressions the traversal produces the identical tree back.
Measured with a throwaway probe over decimal-free expression trees of varying size (Spark 4.1 /
JDK 17, 20k iterations per measurement, best of 5 after warm-up):
exprToProtobeforeTrees that do contain decimal arithmetic pay the guard traversal on top of the rewrite, but that
traversal short-circuits at the first match and does not allocate.
What changes are included in this PR?
Adds a
containsDecimalArithmeticguard and only runs thetransformUpwhen it holds.The guard is deliberately a superset of the rule: it matches
Add/Subtract/Multiply/Divide/RemainderwhosedataTypeis aDecimalType, ignoring the operands that the rule'sown patterns check. It is sound to decide this up front because the rewrite only wraps nodes in
CheckOverflow, andCheckOverflow's data type is its child's, so no node's data type changespart way through the
transformUpand a tree without decimal arithmetic cannot grow any.Behavior is unchanged:
transformUpalready returns the input tree by identity when no casematches, so the guard removes work rather than changing the result.
How are these changes tested?
Existing coverage:
CometExpressionSuite(140 tests) and the Spark 4.1CometDecimalArithmeticViewSuiteregression tests for #4124 and #5075 all pass.Added
DecimalPrecisionSuite, which pins the rule's behavior directly and is version-agnostic(the existing unit coverage was Spark 4.1 only). It asserts that trees without decimal arithmetic
come back by identity, that each of the five arithmetic operators is wrapped in
CheckOverflowwith the operator's own
dataTypeas the target, that a decimal operator buried under nodes therule does not rewrite is still promoted, and that a mixed tree has only its decimal arithmetic
rewritten. These pass both with and without this change, which is the point: they document that
the guard is not observable.
The probe used for the numbers above was throwaway and is not included.