Repository navigation
Lower DuckDB integer // to plain division for the DataFusion target - #487
Closed
eddietejeda wants to merge 1 commit into
Closed
eddietejeda wants to merge 1 commit into
eddietejeda wants to merge 1 commit into
Conversation
DataFusion's / has the same type-dependent semantics as DuckDB's //: integer operands truncate toward zero and a floating operand makes it float division. So `a / NULLIF(b, 0)` is exact, keeps NULL on a zero divisor, and needs no operand-type resolution. Two cases need a DOUBLE cast on the dividend: a DECIMAL operand on either side (DuckDB gives DOUBLE, DataFusion would keep decimal arithmetic), and a source `/` inside a // operand (always DOUBLE in DuckDB, but integer division in DataFusion when both sides are integers). A lowered integer // stays integer-typed and is classified that way when nested.
Owner
|
Superseded by #489, closing. Thanks! |
eddietejeda
added a commit
to hotdata-dev/polyglot
that referenced
this pull request
Oct 9, 2026
Upstream merged its own versions of the MOD grouping (tobilg#488) and DataFusion integer-division (tobilg#489) work, built on tobilg#486/tobilg#487. Upstream's implementation and tests are taken for every conflicting hunk. Upstream now rejects unresolved operand types for DataFusion // as it does for other targets, so untyped `col // 2` needs a schema (see transpile_with_schema).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #484. Since 0.13.2, DuckDB
//with integer operands gives an error for the DataFusion target. This adds a lowering for it.DataFusion's
/works like DuckDB's//. Two integers give an integer, rounded toward zero. If one side is a float, the result is a float. Soa / NULLIF(b, 0)gives the same value as DuckDB, and NULL when the divisor is zero. DataFusion picks the type at run time, so this lowering does not need to know the operand types first.Two cases need a cast to DOUBLE:
/inside the//. DuckDB's/always returns DOUBLE. DataFusion's/on two integers does not. Without the cast,(7 / 2) // 2would give 1 instead of 1.75.Untyped operands
For other targets,
x // ygives an error when the column types are unknown. For DataFusion this PR accepts it, because the value is correct for every type. One small gap: if a column is DECIMAL, DataFusion returns DECIMAL where DuckDB returns DOUBLE. The value is the same; only the type differs. This is noted in the code. If you prefer to keep the error for DataFusion too (in strict mode or always), I can change it.SELECT 7 / 2on its own (DuckDB 3.5, DataFusion 3) is an older, separate gap. I can send a follow-up for it.Verification
Tests cover: integer literals, negative numbers, a literal zero, untyped columns, nested
//, DECIMAL in each position, and a nested/, in default and strict mode. DataFusion is removed from the "no lowering" test; MySQL and Snowflake stay in it. I ran every generated query indatafusion-cliand compared it with DuckDB running the source query:Execution results (datafusion-cli 55.2.0 vs DuckDB 1.5.5) — 20/20 match
SELECT 7 // 2 AS vSELECT 7 / nullif(2, 0) AS vSELECT -7 // 2 AS vSELECT -7 / nullif(2, 0) AS vSELECT 7 // 0 AS vSELECT 7 / nullif(0, 0) AS vSELECT 7 // -0 AS vSELECT 7 / nullif(-0, 0) AS vSELECT 7.0 // 2 AS vSELECT CAST(7.0 AS DOUBLE) / nullif(2, 0) AS vSELECT 7 // 2.0 AS vSELECT CAST(7 AS DOUBLE) / nullif(2.0, 0) AS vSELECT CAST(7 AS DOUBLE) // 2 AS vSELECT CAST(7 AS DOUBLE) / nullif(2, 0) AS vSELECT CAST(7 AS DECIMAL(10, 1)) // 2 AS vSELECT CAST(CAST(7 AS DECIMAL(10, 1)) AS DOUBLE) / nullif(2, 0) AS vSELECT 7 // CAST(2 AS DECIMAL(10, 1)) AS vSELECT CAST(7 AS DOUBLE) / nullif(CAST(2 AS DECIMAL(10, 1)), 0) AS vSELECT 8 // 2 // 2 AS vSELECT 8 / nullif(2, 0) / nullif(2, 0) AS vSELECT 8 // (4 // 2) AS vSELECT 8 / nullif((4 / nullif(2, 0)), 0) AS vSELECT 7 // 2 // CAST(3 AS DECIMAL(10, 1)) AS vSELECT CAST(7 / nullif(2, 0) AS DOUBLE) / nullif(CAST(3 AS DECIMAL(10, 1)), 0) AS vSELECT (7 / 2) // 2 AS vSELECT (CAST(7 AS DOUBLE) / 2) / nullif(2, 0) AS vSELECT 7 // (4 / 2) AS vSELECT 7 / nullif((CAST(4 AS DOUBLE) / 2), 0) AS vSELECT 7.5 / 2 // 2 AS vSELECT 7.5 / 2 / nullif(2, 0) AS vSELECT 9007199254740995 // 2 AS vSELECT 9007199254740995 / nullif(2, 0) AS vSELECT -9007199254740995 // 2 AS vSELECT -9007199254740995 / nullif(2, 0) AS vSELECT 7 // (1 - 1) AS vSELECT 7 / nullif((1 - 1), 0) AS vSELECT x // y AS v FROM (VALUES (7, 2), (-7, 2), (7, 0)) AS t(x, y)SELECT x / nullif(y, 0) AS v FROM (VALUES (7, 2), (-7, 2), (7, 0)) AS t(x, y)SELECT x // 2 AS v FROM (VALUES (7.5), (9.0)) AS t(x)SELECT x / nullif(2, 0) AS v FROM (VALUES (7.5), (9.0)) AS t(x)We run DuckDB SQL on DataFusion in production. That is how we found this.