Skip to content

feat(isthmus)!: preserve aggregate output types and semantics through Calcite#1017

Open
rkondakov wants to merge 2 commits into
substrait-io:mainfrom
rkondakov:pr-1016-preserve-declared-aggregate-output-types
Open

feat(isthmus)!: preserve aggregate output types and semantics through Calcite#1017
rkondakov wants to merge 2 commits into
substrait-io:mainfrom
rkondakov:pr-1016-preserve-declared-aggregate-output-types

Conversation

@rkondakov

@rkondakov rkondakov commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Fixes #1016.

What

A Substrait aggregate used to lose part of itself on the way to Calcite, and could not be
recovered on the way back. This PR gives an aggregate a resolved binding that travels with it, so
the plan's declared output type, its function options, its aggregation phase and the exact
extension declaration all survive a Substrait → Calcite → Substrait round trip.

Why — what was lost

Before After
Declared output type Passed to AggregateCall.create, but RelBuilder re-infers a call's type from its operator, so Calcite's inference won. sum(DECIMAL(10,2)), which sum:dec declares as DECIMAL(38,2), came back with the argument's own width. The plan's type is preserved.
Two measures, same shape, different types AggregateCall.equals ignores the stored type and RelBuilder deduplicates, so they collapsed into one column — the output arity changed. Both survive; deduplication is switched off for exactly those aggregates.
Function options No place for them in a Calcite aggregate call, so count(a) with overflow: ERROR came back without the option, and two counts differing only by an option were indistinguishable. Carried on the binding and restored.
Aggregation phase Converting back always produced INITIAL_TO_RESULT. A distributed plan with partial aggregates was silently turned into a plan of full aggregations. The phase is carried and restored.
Which declaration Re-matched against the extension catalog; among equally-shaped declarations it could only guess. A phase consuming an intermediate state could not be matched at all, because the call's operands are then accumulator state rather than the declaration's arguments. The exact declaration the plan used is taken from the binding.
Global aggregate with no groupings Built with zero grouping sets, so measures were re-inferred as if there were no empty group. Built as one empty grouping set — the correct global aggregation.

How

Each converted measure resolves a ResolvedAggregateBinding, which captures the semantic identity
of the invocation: anchor, kind-aware arguments (value / type / enum, including the selected enum
option), options, phase and invocation semantics. Where Calcite cannot hold what the binding
carries — the chosen output type differs from its inference, or the invocation has options or a
phase other than INITIAL_TO_RESULT — the aggregate operator is wrapped so that both the binding
and the type travel with it. AggregateFunctionConverter then rebuilds the invocation from that
binding instead of re-matching the operator.

The binding records the plan as it was converted, so it is dropped again when a planner rule has
since changed the call's arguments, and DISTINCT is always read off the Calcite call, because a
rule may legitimately have removed a redundant one.

Supporting this needed TypeExpressionEvaluator to actually work: it used to throw
UnsupportedOperationException("NYI") for anything but an already concrete return type. It now
binds numbered wildcards (the any1 of min(any1) -> any1) and integer type parameters (the P
and S of DECIMAL<P,S>) from the actual argument types and substitutes them, honouring variadic
parameter consistency and the MIRROR nullability policy. Unsupported derivations — arithmetic
derivations, if/then, return programs — stay fail-closed and never fall back to a caller-supplied
type, which is what makes a derived type trustworthy enough to validate a plan against.

Commits

  1. feat(core) — the resolved-binding model (ResolvedArgument, ResolvedFunctionBinding,
    ResolvedAggregateBinding), FunctionBindingResolver (resolve vs. opt-in validate), and a
    working TypeExpressionEvaluator. Self-contained; :core:build and :isthmus:build are green
    at this commit alone.
  2. feat(isthmus)! — conversion changes: AggregateConversion configuration, the transport
    wrapper in AggregateFunctions, and the two conversion fixes (deduplication, global aggregate).

New configuration

AggregateConversion has two independent settings:

  • OutputTypeSourcePLAN_OUTPUT (new default) preserves the plan's declared type;
    CALCITE_INFERENCE reproduces the previous behavior.
  • FunctionBindingValidationNONE (default) does not check the plan against the extension
    declaration; EXTENSION_DECLARATION requires the declared output type to match the derived one
    and rejects the plan otherwise.

The default is PLAN_OUTPUT + NONE: conversion never silently changes a type, but it also does
not assert that the plan is spec-compliant. Strict validation is deliberately opt-in, because
type derivation is fail-closed and would reject any function whose return expression it does not
yet support.

Breaking changes / migration

  • A converted aggregate now carries the plan's declared output type instead of Calcite's inferred
    one. Pass AggregateConversion with OutputTypeSource.CALCITE_INFERENCE to SubstraitToCalcite
    to restore the previous behavior.
  • The resulting AggregateCall may carry a wrapper operator rather than the plain SqlAggFunction.
    Consumers comparing the operator by identity should call AggregateFunctions.unwrapBound first;
    AggregateFunctions.boundBinding exposes the binding.
  • Rollup and splitting are disabled for a wrapped call: both re-infer the transformed call's type
    from the underlying function and would discard the preserved one. Rules that match on SqlKind
    instead (e.g. AGGREGATE_REDUCE_FUNCTIONS) have no such hook and may still rewrite the call and
    drop the type.
  • A ConverterProvider subclass overriding getSubstraitRelNodeConverter(RelBuilder) should
    override the new getSubstraitRelNodeConverter(RelBuilder, AggregateConversion) as well;
    otherwise its customization is skipped for a non-default configuration.
  • Converting a plan whose aggregate has no groupings back to Substrait now yields a single empty
    grouping instead of none — the same global aggregation, spelled the way Calcite spells it.

Open question for maintainers

Should preserving the plan's declared output type be the default, or opt-in?

This PR makes PLAN_OUTPUT the default, so existing callers get the new behavior without changing
a line. The reasoning: a converter silently changing a plan's declared type is a correctness
problem — DECIMAL(38,2) becoming DECIMAL(10,2) changes results, not just metadata — and a
producer that has to know about a configuration flag to get its own types back is a sharp edge.

The cost is real, though: the new default is a behavior change for every existing consumer, and
where the declared type diverges from Calcite's inference the operator is wrapped, which disables
rollup and aggregate splitting for that call and makes call.getAggregation() == SUM-style
comparisons fail.

The alternative is to default to CALCITE_INFERENCE — byte-for-byte the previous behavior — and
let callers opt into PLAN_OUTPUT. That keeps every existing consumer untouched and makes the
wrapper strictly opt-in, at the price of leaving the type-loss bug on by default.

Happy to flip the default either way; the configuration supports both.

Testing

  • :coreFunctionBindingResolverTest, ResolvedAggregateBindingTest,
    TypeExpressionEvaluatorTest, plus a small binding_extensions.yaml for wildcard cases.
  • :isthmus — new cases in SubstraitRelNodeConverterTest.Aggregate covering declared decimal
    width, global aggregation, operator identity, enum arguments, options, phases (including a
    parameterized intermediate state), duplicate measures, rollup and split rules, both validation
    modes, unwrapping, and converter-factory dispatch.
  • ./gradlew :core:build :isthmus:build is green at each of the two commits.

🤖 Generated with Claude Code

@rkondakov
rkondakov marked this pull request as draft July 18, 2026 12:53
rkondakov and others added 2 commits July 26, 2026 16:50
…ut types

Adds a resolved-binding model for extension function invocations, and makes
TypeExpressionEvaluator actually evaluate parameterized return types instead of
throwing.

ResolvedArgument keeps an argument's kind (value, type or enum) and its selected
enum option, so two invocations differing only by an enum argument — e.g.
std_dev(POPULATION, fp32) vs std_dev(SAMPLE, fp32) — are not conflated, and an
enum the plan left unspecified stays distinct from any specified option.
ResolvedFunctionBinding captures the semantic identity of an invocation: anchor,
ordered arguments and options. ResolvedAggregateBinding adds the aggregate phase
and invocation semantics, and selects the declaration's intermediate type for a
phase that stops at the intermediate state rather than its return type.

FunctionBindingResolver keeps the two concerns apart. resolve() only captures
identity and performs no validation, so an invocation that merely differs from
its declaration still resolves. validate() opts into checking arity, argument
kinds and types, enum options, function options and the plan-declared output
type against the declaration.

TypeExpressionEvaluator previously threw UnsupportedOperationException("NYI") for
anything but an already concrete return type, so SimpleExtension.Function's
resolveType could not evaluate a parameterized declaration at all. It now binds
numbered wildcards (the any1 of min(any1) -> any1) and integer type parameters
(the P and S of DECIMAL<P,S>) from the actual argument types and substitutes
them, taking variadic parameter consistency and the declaration's MIRROR
nullability policy into account. Occurrences of one numbered wildcard must agree
on a single type, while each plain any binds independently.

Derivations that are not supported — arithmetic derivations, if/then, return
programs — stay fail-closed: they raise rather than fall back to a
caller-supplied type, which is what makes a derived type trustworthy enough to
validate a plan against.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… Calcite

A Substrait aggregate used to lose part of itself on the way to Calcite. The
output type the plan declared was handed to AggregateCall.create, but RelBuilder
re-infers a call's type from its operator, so the plan's type was replaced by
Calcite's inference — sum(DECIMAL(10,2)), which the standard sum:dec declaration
types as DECIMAL(38,2), came back as the argument's own width. Function options
and the aggregation phase have no place in a Calcite aggregate call at all, so
they were dropped, and converting back re-matched the operator against the
extension catalog: among equally-shaped declarations it could only guess, it
always produced a full INITIAL_TO_RESULT aggregation without options, and for a
phase consuming an intermediate state it could not match at all, because the
call's operands are then accumulator state rather than the declaration's
arguments.

Each converted measure now resolves a ResolvedAggregateBinding. Where Calcite
cannot hold what the binding carries — the chosen output type differs from its
inference, or the invocation has options or a phase other than INITIAL_TO_RESULT
— the operator is wrapped so that both the binding and the type travel with it,
and AggregateFunctionConverter rebuilds the invocation from that binding instead
of re-matching. The binding records the plan as it was converted, so it is
dropped again when a planner rule has since changed the call's arguments, and
DISTINCT is always read off the Calcite call because a rule may legitimately
have removed a redundant one.

Two further conversion fixes come with it. Measures that are equal to Calcite
but carry different types no longer collapse into a single column: AggregateCall
equality ignores the stored type and RelBuilder deduplicates, so deduplication
is now switched off for exactly those aggregates. And an aggregate with no
groupings is built as one empty grouping set rather than none at all, so its
measures are no longer re-inferred as if there were no empty group; converting
such a plan back therefore spells the same global aggregation the way Calcite
spells it, as a single empty grouping.

BREAKING CHANGE: converting a Substrait aggregate to Calcite now preserves the
plan's declared output type instead of Calcite's inferred one, and the resulting
AggregateCall may carry a wrapper operator rather than the plain SqlAggFunction.
Consumers that compare the operator by identity should call
AggregateFunctions.unwrapBound first; rollup and splitting are disabled for a
wrapped call, since both re-infer the type from the underlying function and
would discard the preserved one. Pass AggregateConversion with
OutputTypeSource.CALCITE_INFERENCE to SubstraitToCalcite to restore the previous
behavior. A ConverterProvider subclass that overrides
getSubstraitRelNodeConverter(RelBuilder) should override the new
getSubstraitRelNodeConverter(RelBuilder, AggregateConversion) as well, otherwise
its customization is skipped for a non-default configuration.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rkondakov
rkondakov force-pushed the pr-1016-preserve-declared-aggregate-output-types branch from db81ad0 to 4d7aa8f Compare July 26, 2026 10:08
@rkondakov rkondakov changed the title fix(isthmus): preserve declared aggregate output types feat(isthmus)!: preserve aggregate output types and semantics through Calcite Jul 26, 2026
@rkondakov
rkondakov marked this pull request as ready for review July 26, 2026 10:12
@rkondakov

Copy link
Copy Markdown
Contributor Author

@nielspardon This PR lets aggregate output types come either from the plan or from Calcite’s inference.

I defaulted to the plan’s type because re-inference can change results, e.g. DECIMAL(38,2) to DECIMAL(10,2). The downside is that mismatches require a wrapper operator, which can break operator identity checks and disable rollup or aggregate splitting.

Keeping CALCITE_INFERENCE preserves current behavior but leaves type loss enabled by default.

Which default do you prefer?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[isthmus][SubstraitToCalcite] Preserve declared aggregate output types

1 participant