fix(isthmus): use schema types for virtual table literals - #1065
fix(isthmus): use schema types for virtual table literals#1065alexandrefimov wants to merge 1 commit into
Conversation
Use the complete LogicalValues row-schema field type when converting tuple literals so virtual table rows match their schema. Preserve literal-derived behavior for other conversions and cover TINYINT-to-INTEGER widening. Fixes substrait-io#1064
|
@nielspardon would you mind taking a look at this one? Small isthmus fix for #1064: |
| @@ -108,14 +107,34 @@ public Expression.Literal convert(RexLiteral literal) { | |||
| * @return the converted Substrait Literal | |||
| */ | |||
| public Expression.Literal convert(RexLiteral literal, boolean nullable) { | |||
There was a problem hiding this comment.
I would advocate for getting rid of this method entirely actually.
It's not used internally after your changes, and any user that wants to set the nullability explicitly on the Calcite type can use the RelDataTypeFactory#createTypeWithNullability to set it, and then pass it to the new method below.
| return convert(literal, resultType, type.nullable()); | ||
| } | ||
|
|
||
| private Expression.Literal convert(RexLiteral literal, RelDataType resultType, boolean nullable) { |
There was a problem hiding this comment.
Getting rid of the boolean nullable would let us get rid of the nullable flag here and just rely on the resultType as being the actual desired result type with the correct nullability.
|
To confirm two things. In your test, you construct a LogicalValue containing literals with different but compatible(?) types than the row type of the LogicalValue relation, and that ends up being valid because Calcite checks if the type can be assigned to value? In Substrait, we assume that all literals in VirtualTables have the same types as given in the schema. When we convert from Substrait to Calcite, do the tuple values in the Calcite LogicalValues relation this, or is that conversion also potentially lossy? |
Summary
LogicalValuestuple literals using the complete row-schema field type;TINYINTtuple literal in anINTEGERfield.VirtualTableScanrequires row field types to exactly match its schema. Calcitemay infer a narrower type for a tuple literal than for the corresponding
LogicalValuesrow field. Previously Isthmus copied only schema nullability, sothe conversion could construct an
I8row under anI32schema and fail its ownvalidation.
The new
LiteralConverteroverload accepts the schemaRelDataType. Existingcallers continue to derive the result type from the literal itself.
Fixes #1064
Testing
All checks pass on JDK 17.