feat(isthmus): observe window function return types - #1059
feat(isthmus): observe window function return types#1059alexandrefimov wants to merge 3 commits into
Conversation
Report Calcite-inferred window return types through the existing observer while preserving the Substrait-supplied RexOver type. Keep the NOOP fast path and convert inference failures into observations.
Allow callers to install a type observer without subclassing ConverterProvider.
3f70f98 to
fa97d02
Compare
Assisted-by: gpt-5.6-sol (OpenAI Codex)
vbarua
left a comment
There was a problem hiding this comment.
Overall, the changes look reasonable.
There is one potential issue around nullability in the Calcite inferred output types. From a cursory look, I don't know how painful it would be to have the observation path follow the true Calcite inference path more closely. If it's not too bad, it would be worth doing in this PR. However, if it would require a fair bit of poking into Calcite internals to achieve, I would be happy with documenting the limitation for now and merging.
Let me know what you think @alexandrefimov
| observeType( | ||
| expr, | ||
| TypeObservation.Source.WINDOW_FUNCTION, | ||
| () -> rexBuilder.deriveReturnType(operator, args)); |
There was a problem hiding this comment.
Claude did identify one potential bit of weirdness around nullability differences in certain cases. I still think this is useful overall for cases where the types are fully different, but it might not be reliable as-is to detect nullability deviations.
Claude Summary
rexBuilder.deriveReturnType(operator, args) wraps args in a plain RexCallBinding where hasEmptyGroup() always returns false.
Calcite's actual window inference path (via SqlOverOperator) sets hasEmptyGroup() based on whether the window bounds guarantee non-emptiness. For SUM, MIN/MAX, AVG, and several other aggregate window functions, this flag controls result nullability.
Concrete example: SUM(x BIGINT NOT NULL) OVER (...) — the real Calcite inference returns BIGINT nullable; deriveReturnType returns BIGINT NOT NULL. The observation records the wrong inferred type, defeating the diagnostic purpose of the observer.
Summary
TypeObserverduring window-function conversionTypeObserver.NOOPConverterProvider.BuilderMotivation
Scalar-function type observation was introduced in #1015. Issue #379 also identifies window functions as a conversion point where Substrait-supplied and Calcite-inferred types can diverge. This change extends the same observation side channel to window functions and makes it available without subclassing
ConverterProvider.Validation
./gradlew :isthmus:spotlessCheck :isthmus:test --tests 'io.substrait.isthmus.ConverterProviderBuilderTest' --tests 'io.substrait.isthmus.SubstraitExpressionConverterTest' :isthmus:javadoc./gradlew build --rerun-tasksRelated to #379.