Skip to content

feat(isthmus): observe window function return types - #1059

Open
alexandrefimov wants to merge 3 commits into
substrait-io:mainfrom
alexandrefimov:issue-379-window-type-observer
Open

feat(isthmus): observe window function return types#1059
alexandrefimov wants to merge 3 commits into
substrait-io:mainfrom
alexandrefimov:issue-379-window-type-observer

Conversation

@alexandrefimov

Copy link
Copy Markdown
Contributor

Summary

  • report Calcite-inferred return types through TypeObserver during window-function conversion
  • preserve the Substrait-supplied return type and skip extra inference for TypeObserver.NOOP
  • allow callers to configure the observer through ConverterProvider.Builder

Motivation

Scalar-function type observation was introduced in #1015. Issue #379 also identifies window functions as a conversion point where Substrait-supplied and Calcite-inferred types can diverge. This change extends the same observation side channel to window functions and makes it available without subclassing ConverterProvider.

Validation

  • ./gradlew :isthmus:spotlessCheck :isthmus:test --tests 'io.substrait.isthmus.ConverterProviderBuilderTest' --tests 'io.substrait.isthmus.SubstraitExpressionConverterTest' :isthmus:javadoc
  • ./gradlew build --rerun-tasks

Related to #379.

Report Calcite-inferred window return types through the existing observer while preserving the Substrait-supplied RexOver type. Keep the NOOP fast path and convert inference failures into observations.
Allow callers to install a type observer without subclassing ConverterProvider.
@alexandrefimov
alexandrefimov force-pushed the issue-379-window-type-observer branch from 3f70f98 to fa97d02 Compare August 4, 2026 03:32
@alexandrefimov
alexandrefimov marked this pull request as ready for review August 4, 2026 03:41
Assisted-by: gpt-5.6-sol (OpenAI Codex)
@alexandrefimov

Copy link
Copy Markdown
Contributor Author

@vbarua this follows up on #1015 — it extends type observation to window functions (the second conversion point named in #379) and wires the observer into ConverterProvider.Builder, which is what you suggested after the merge. Would you have a chance to take a look?

@vbarua vbarua left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, the changes look reasonable.

There is one potential issue around nullability in the Calcite inferred output types. From a cursory look, I don't know how painful it would be to have the observation path follow the true Calcite inference path more closely. If it's not too bad, it would be worth doing in this PR. However, if it would require a fair bit of poking into Calcite internals to achieve, I would be happy with documenting the limitation for now and merging.

Let me know what you think @alexandrefimov

observeType(
expr,
TypeObservation.Source.WINDOW_FUNCTION,
() -> rexBuilder.deriveReturnType(operator, args));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude did identify one potential bit of weirdness around nullability differences in certain cases. I still think this is useful overall for cases where the types are fully different, but it might not be reliable as-is to detect nullability deviations.

Claude Summary

rexBuilder.deriveReturnType(operator, args) wraps args in a plain RexCallBinding where hasEmptyGroup() always returns false.

Calcite's actual window inference path (via SqlOverOperator) sets hasEmptyGroup() based on whether the window bounds guarantee non-emptiness. For SUM, MIN/MAX, AVG, and several other aggregate window functions, this flag controls result nullability.

Concrete example: SUM(x BIGINT NOT NULL) OVER (...) — the real Calcite inference returns BIGINT nullable; deriveReturnType returns BIGINT NOT NULL. The observation records the wrong inferred type, defeating the diagnostic purpose of the observer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants