Summary
A column of Arrow type Timestamp -- the same type as DuckDB's own
TIMESTAMP -- only coerces successfully under SCALE <col> VIA date when
its table lives natively in the reader's DuckDB catalog. The identical
column type, when the table is instead provided via Reader::register()
(the API for handing a reader an externally-produced table), fails to
coerce with:
Validation error: Cannot coerce column '__ggsql_aes_pos1__' of type Timestamp(Microsecond, None) to date
Both cases below produce a column that is unambiguously Timestamp(Microsecond, None) -- confirmed independently for each. Only the table's origin differs.
Case 1: native DuckDB table -- succeeds
CREATE TABLE t AS
SELECT DATE_TRUNC('month', d) AS month, v
FROM (VALUES (DATE '2024-01-15', 10), (DATE '2024-02-20', 20)) AS x(d, v);
DESCRIBE t reports month as TIMESTAMP; .arrow().schema confirms timestamp[us].
VISUALISE month AS x, v AS y FROM t DRAW bar SCALE x VIA date
This succeeds.
Case 2: registered table -- fails
Calling the core crate directly, with no other bindings involved:
use std::sync::Arc;
use arrow::array::{Int64Array, TimestampMicrosecondArray};
use ggsql::reader::{DuckDBReader, Reader};
use ggsql::DataFrame;
fn main() {
let reader = DuckDBReader::from_connection_string("duckdb://memory").unwrap();
let month = TimestampMicrosecondArray::from(vec![
1704067200000000i64, // 2024-01-01T00:00:00
1706745600000000i64, // 2024-02-01T00:00:00
]);
let v = Int64Array::from(vec![10i64, 20i64]);
let df = DataFrame::new(vec![
("month", Arc::new(month) as _),
("v", Arc::new(v) as _),
]).unwrap();
println!("registered column dtype: {:?}", df.column_dtype("month"));
reader.register("t", df, false).unwrap();
match reader.execute("VISUALISE month AS x, v AS y FROM t DRAW bar SCALE x VIA date") {
Ok(_) => println!("SUCCEEDED (unexpected)"),
Err(e) => println!("FAILED: {}", e),
}
}
Output:
registered column dtype: Ok(Timestamp(Microsecond, None))
FAILED: Validation error: Cannot coerce column '__ggsql_aes_pos1__' of type Timestamp(Microsecond, None) to date
Environment: posit-dev/ggsql main @ 983e38e3 (v0.5.2).
Root cause
src/execute/scale.rs, coerce_column_to_type, the ArrayElementType::Date
match arm (~line 754) only handles a DataType::Utf8 source (parsing
YYYY-MM-DD strings); every other source dtype -- including
Timestamp(Microsecond, None) -- falls into the catch-all _ branch and
hard-errors:
ArrayElementType::Date => match dtype {
DataType::Utf8 => {
// ... parses date strings into a Date32 array ...
}
_ => {
return Err(GgsqlError::ValidationError(format!(
"Cannot coerce column '{}' of type {:?} to date",
column_name, dtype
)));
}
},
Arrow's cast_array (already imported and used a few lines above for
ArrayElementType::Number) supports a safe Timestamp -> Date32 cast that
just truncates the time-of-day component, but this arm never reaches it for
non-Utf8 inputs. Since Case 1 and Case 2 produce the identical Arrow
Timestamp(Microsecond, None) type, something upstream of this function
must be normalizing/pre-casting the column differently depending on whether
it came from register() or from the reader's own catalog -- worth tracing
that path too, since it's the reason this bug is inconsistent rather than
a hard failure every time a Timestamp reaches a date scale.
Note the neighboring ArrayElementType::DateTime arm has the identical
Utf8-only pattern, so a native Date32 column being coerced to a datetime
target may have the mirror-image problem -- I haven't verified that case,
but it's worth checking alongside this fix.
Suggested fix
Add a DataType::Timestamp(_, _) => cast_array(column, &DataType::Date32)?
branch to the ArrayElementType::Date arm in coerce_column_to_type,
mirroring the existing Number arm's use of cast_array. That would make
Case 2 succeed unconditionally rather than only when the upstream
normalization happens to have already run.
Workaround
Explicitly CAST(... AS DATE) the derived expression in SQL before it
reaches ggsql, e.g. CAST(DATE_TRUNC('month', d) AS DATE). That produces a
genuine Date32-typed column, which coerces fine regardless of table origin.
Summary
A column of Arrow type
Timestamp-- the same type as DuckDB's ownTIMESTAMP-- only coerces successfully underSCALE <col> VIA datewhenits table lives natively in the reader's DuckDB catalog. The identical
column type, when the table is instead provided via
Reader::register()(the API for handing a reader an externally-produced table), fails to
coerce with:
Both cases below produce a column that is unambiguously
Timestamp(Microsecond, None)-- confirmed independently for each. Only the table's origin differs.Case 1: native DuckDB table -- succeeds
DESCRIBE treportsmonthasTIMESTAMP;.arrow().schemaconfirmstimestamp[us].This succeeds.
Case 2: registered table -- fails
Calling the core crate directly, with no other bindings involved:
Output:
Environment:
posit-dev/ggsqlmain @983e38e3(v0.5.2).Root cause
src/execute/scale.rs,coerce_column_to_type, theArrayElementType::Datematch arm (~line 754) only handles a
DataType::Utf8source (parsingYYYY-MM-DDstrings); every other source dtype -- includingTimestamp(Microsecond, None)-- falls into the catch-all_branch andhard-errors:
Arrow's
cast_array(already imported and used a few lines above forArrayElementType::Number) supports a safeTimestamp -> Date32cast thatjust truncates the time-of-day component, but this arm never reaches it for
non-
Utf8inputs. Since Case 1 and Case 2 produce the identical ArrowTimestamp(Microsecond, None)type, something upstream of this functionmust be normalizing/pre-casting the column differently depending on whether
it came from
register()or from the reader's own catalog -- worth tracingthat path too, since it's the reason this bug is inconsistent rather than
a hard failure every time a
Timestampreaches adatescale.Note the neighboring
ArrayElementType::DateTimearm has the identicalUtf8-only pattern, so a native
Date32column being coerced to adatetimetarget may have the mirror-image problem -- I haven't verified that case,
but it's worth checking alongside this fix.
Suggested fix
Add a
DataType::Timestamp(_, _) => cast_array(column, &DataType::Date32)?branch to the
ArrayElementType::Datearm incoerce_column_to_type,mirroring the existing
Numberarm's use ofcast_array. That would makeCase 2 succeed unconditionally rather than only when the upstream
normalization happens to have already run.
Workaround
Explicitly
CAST(... AS DATE)the derived expression in SQL before itreaches ggsql, e.g.
CAST(DATE_TRUNC('month', d) AS DATE). That produces agenuine
Date32-typed column, which coerces fine regardless of table origin.