Skip to content

fix(isthmus): use schema types for virtual table literals - #1065

Open
alexandrefimov wants to merge 1 commit into
substrait-io:mainfrom
alexandrefimov:codex/isthmus-values-schema-literals
Open

fix(isthmus): use schema types for virtual table literals#1065
alexandrefimov wants to merge 1 commit into
substrait-io:mainfrom
alexandrefimov:codex/isthmus-values-schema-literals

Conversation

@alexandrefimov

Copy link
Copy Markdown
Contributor

Summary

  • convert LogicalValues tuple literals using the complete row-schema field type;
  • preserve the existing literal-based behavior for other expression conversions;
  • add a regression test for a TINYINT tuple literal in an INTEGER field.

VirtualTableScan requires row field types to exactly match its schema. Calcite
may infer a narrower type for a tuple literal than for the corresponding
LogicalValues row field. Previously Isthmus copied only schema nullability, so
the conversion could construct an I8 row under an I32 schema and fail its own
validation.

The new LiteralConverter overload accepts the schema RelDataType. Existing
callers continue to derive the result type from the literal itself.

Fixes #1064

Testing

./gradlew :isthmus:spotlessCheck :isthmus:test --tests io.substrait.isthmus.VirtualTableScanTest
./gradlew :isthmus:test
./gradlew :isthmus:check
./gradlew build --rerun-tasks

All checks pass on JDK 17.

Use the complete LogicalValues row-schema field type when converting tuple literals so virtual table rows match their schema. Preserve literal-derived behavior for other conversions and cover TINYINT-to-INTEGER widening.

Fixes substrait-io#1064
@alexandrefimov

Copy link
Copy Markdown
Contributor Author

@nielspardon would you mind taking a look at this one? Small isthmus fix for #1064: VirtualTableScan requires row field types to match its schema exactly, but Calcite can infer a narrower type for a tuple literal than for the corresponding LogicalValues row field — so the conversion could build an I8 row under an I32 schema and then fail its own validation.

@@ -108,14 +107,34 @@ public Expression.Literal convert(RexLiteral literal) {
* @return the converted Substrait Literal
*/
public Expression.Literal convert(RexLiteral literal, boolean nullable) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would advocate for getting rid of this method entirely actually.

It's not used internally after your changes, and any user that wants to set the nullability explicitly on the Calcite type can use the RelDataTypeFactory#createTypeWithNullability to set it, and then pass it to the new method below.

return convert(literal, resultType, type.nullable());
}

private Expression.Literal convert(RexLiteral literal, RelDataType resultType, boolean nullable) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting rid of the boolean nullable would let us get rid of the nullable flag here and just rely on the resultType as being the actual desired result type with the correct nullability.

@vbarua

vbarua commented Aug 6, 2026

Copy link
Copy Markdown
Member

To confirm two things.

In your test, you construct a LogicalValue containing literals with different but compatible(?) types than the row type of the LogicalValue relation, and that ends up being valid because Calcite checks if the type can be assigned to value?
https://github.com/apache/calcite/blob/052a35015c56db2927727afd7e08ed7a18a26ace/core/src/main/java/org/apache/calcite/rel/core/Values.java#L169-L190. The issue that you've identified is that Calcite isn't as strict as Substrait, so when we convert from Calcite we can't rely on the literal types and need to override them with the row type?

In Substrait, we assume that all literals in VirtualTables have the same types as given in the schema. When we convert from Substrait to Calcite, do the tuple values in the Calcite LogicalValues relation this, or is that conversion also potentially lossy?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LogicalValues conversion fails when tuple literal type is narrower than row schema

2 participants