Skip to content

Lower DuckDB integer // to plain division for the DataFusion target - #487

Closed
eddietejeda wants to merge 1 commit into
tobilg:mainfrom
hotdata-dev:fix/datafusion-integer-division
Closed

eddietejeda wants to merge 1 commit into
tobilg:mainfrom
hotdata-dev:fix/datafusion-integer-division

Conversation

@eddietejeda

Copy link
Copy Markdown
Contributor

Follow-up to #484. Since 0.13.2, DuckDB // with integer operands gives an error for the DataFusion target. This adds a lowering for it.

DataFusion's / works like DuckDB's //. Two integers give an integer, rounded toward zero. If one side is a float, the result is a float. So a / NULLIF(b, 0) gives the same value as DuckDB, and NULL when the divisor is zero. DataFusion picks the type at run time, so this lowering does not need to know the operand types first.

Two cases need a cast to DOUBLE:

  • A DECIMAL operand. DuckDB returns DOUBLE. DataFusion would keep DECIMAL.
  • A DuckDB / inside the //. DuckDB's / always returns DOUBLE. DataFusion's / on two integers does not. Without the cast, (7 / 2) // 2 would give 1 instead of 1.75.
SELECT 7 // 2 AS v        ->  SELECT 7 / NULLIF(2, 0) AS v
SELECT x // y FROM t      ->  SELECT x / NULLIF(y, 0) FROM t
SELECT 7 // 0 AS v        ->  SELECT 7 / NULLIF(0, 0) AS v    -- NULL, as in DuckDB
SELECT (7 / 2) // 2 AS v  ->  SELECT (CAST(7 AS DOUBLE) / 2) / NULLIF(2, 0) AS v

Untyped operands

For other targets, x // y gives an error when the column types are unknown. For DataFusion this PR accepts it, because the value is correct for every type. One small gap: if a column is DECIMAL, DataFusion returns DECIMAL where DuckDB returns DOUBLE. The value is the same; only the type differs. This is noted in the code. If you prefer to keep the error for DataFusion too (in strict mode or always), I can change it.

SELECT 7 / 2 on its own (DuckDB 3.5, DataFusion 3) is an older, separate gap. I can send a follow-up for it.

Verification

Tests cover: integer literals, negative numbers, a literal zero, untyped columns, nested //, DECIMAL in each position, and a nested /, in default and strict mode. DataFusion is removed from the "no lowering" test; MySQL and Snowflake stay in it. I ran every generated query in datafusion-cli and compared it with DuckDB running the source query:

Execution results (datafusion-cli 55.2.0 vs DuckDB 1.5.5) — 20/20 match

Source Generated (DataFusion) Result (DataFusion = DuckDB)
SELECT 7 // 2 AS v SELECT 7 / nullif(2, 0) AS v 3
SELECT -7 // 2 AS v SELECT -7 / nullif(2, 0) AS v -3
SELECT 7 // 0 AS v SELECT 7 / nullif(0, 0) AS v NULL
SELECT 7 // -0 AS v SELECT 7 / nullif(-0, 0) AS v NULL
SELECT 7.0 // 2 AS v SELECT CAST(7.0 AS DOUBLE) / nullif(2, 0) AS v 3.5
SELECT 7 // 2.0 AS v SELECT CAST(7 AS DOUBLE) / nullif(2.0, 0) AS v 3.5
SELECT CAST(7 AS DOUBLE) // 2 AS v SELECT CAST(7 AS DOUBLE) / nullif(2, 0) AS v 3.5
SELECT CAST(7 AS DECIMAL(10, 1)) // 2 AS v SELECT CAST(CAST(7 AS DECIMAL(10, 1)) AS DOUBLE) / nullif(2, 0) AS v 3.5
SELECT 7 // CAST(2 AS DECIMAL(10, 1)) AS v SELECT CAST(7 AS DOUBLE) / nullif(CAST(2 AS DECIMAL(10, 1)), 0) AS v 3.5
SELECT 8 // 2 // 2 AS v SELECT 8 / nullif(2, 0) / nullif(2, 0) AS v 2
SELECT 8 // (4 // 2) AS v SELECT 8 / nullif((4 / nullif(2, 0)), 0) AS v 4
SELECT 7 // 2 // CAST(3 AS DECIMAL(10, 1)) AS v SELECT CAST(7 / nullif(2, 0) AS DOUBLE) / nullif(CAST(3 AS DECIMAL(10, 1)), 0) AS v 1.0
SELECT (7 / 2) // 2 AS v SELECT (CAST(7 AS DOUBLE) / 2) / nullif(2, 0) AS v 1.75
SELECT 7 // (4 / 2) AS v SELECT 7 / nullif((CAST(4 AS DOUBLE) / 2), 0) AS v 3.5
SELECT 7.5 / 2 // 2 AS v SELECT 7.5 / 2 / nullif(2, 0) AS v 1.875
SELECT 9007199254740995 // 2 AS v SELECT 9007199254740995 / nullif(2, 0) AS v 4503599627370497
SELECT -9007199254740995 // 2 AS v SELECT -9007199254740995 / nullif(2, 0) AS v -4503599627370497
SELECT 7 // (1 - 1) AS v SELECT 7 / nullif((1 - 1), 0) AS v NULL
SELECT x // y AS v FROM (VALUES (7, 2), (-7, 2), (7, 0)) AS t(x, y) SELECT x / nullif(y, 0) AS v FROM (VALUES (7, 2), (-7, 2), (7, 0)) AS t(x, y) 3|-3|NULL
SELECT x // 2 AS v FROM (VALUES (7.5), (9.0)) AS t(x) SELECT x / nullif(2, 0) AS v FROM (VALUES (7.5), (9.0)) AS t(x) 3.75|4.5

We run DuckDB SQL on DataFusion in production. That is how we found this.

DataFusion's / has the same type-dependent semantics as DuckDB's //:
integer operands truncate toward zero and a floating operand makes it
float division. So `a / NULLIF(b, 0)` is exact, keeps NULL on a zero
divisor, and needs no operand-type resolution.

Two cases need a DOUBLE cast on the dividend: a DECIMAL operand on
either side (DuckDB gives DOUBLE, DataFusion would keep decimal
arithmetic), and a source `/` inside a // operand (always DOUBLE in
DuckDB, but integer division in DataFusion when both sides are
integers). A lowered integer // stays integer-typed and is classified
that way when nested.
@tobilg

tobilg commented Oct 9, 2026

Copy link
Copy Markdown
Owner

Superseded by #489, closing. Thanks!

@tobilg tobilg closed this Oct 9, 2026
eddietejeda added a commit to hotdata-dev/polyglot that referenced this pull request Oct 9, 2026
Upstream merged its own versions of the MOD grouping (tobilg#488) and DataFusion
integer-division (tobilg#489) work, built on tobilg#486/tobilg#487. Upstream's implementation
and tests are taken for every conflicting hunk. Upstream now rejects
unresolved operand types for DataFusion // as it does for other targets,
so untyped `col // 2` needs a schema (see transpile_with_schema).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants