Skip to content

SCALE ... VIA date fails to coerce Timestamp columns from registered tables #559

Description

@cpsievert

Summary

A column of Arrow type Timestamp -- the same type as DuckDB's own
TIMESTAMP -- only coerces successfully under SCALE <col> VIA date when
its table lives natively in the reader's DuckDB catalog. The identical
column type, when the table is instead provided via Reader::register()
(the API for handing a reader an externally-produced table), fails to
coerce with:

Validation error: Cannot coerce column '__ggsql_aes_pos1__' of type Timestamp(Microsecond, None) to date

Both cases below produce a column that is unambiguously Timestamp(Microsecond, None) -- confirmed independently for each. Only the table's origin differs.

Case 1: native DuckDB table -- succeeds

CREATE TABLE t AS
SELECT DATE_TRUNC('month', d) AS month, v
FROM (VALUES (DATE '2024-01-15', 10), (DATE '2024-02-20', 20)) AS x(d, v);

DESCRIBE t reports month as TIMESTAMP; .arrow().schema confirms timestamp[us].

VISUALISE month AS x, v AS y FROM t DRAW bar SCALE x VIA date

This succeeds.

Case 2: registered table -- fails

Calling the core crate directly, with no other bindings involved:

use std::sync::Arc;
use arrow::array::{Int64Array, TimestampMicrosecondArray};
use ggsql::reader::{DuckDBReader, Reader};
use ggsql::DataFrame;

fn main() {
    let reader = DuckDBReader::from_connection_string("duckdb://memory").unwrap();

    let month = TimestampMicrosecondArray::from(vec![
        1704067200000000i64, // 2024-01-01T00:00:00
        1706745600000000i64, // 2024-02-01T00:00:00
    ]);
    let v = Int64Array::from(vec![10i64, 20i64]);

    let df = DataFrame::new(vec![
        ("month", Arc::new(month) as _),
        ("v", Arc::new(v) as _),
    ]).unwrap();

    println!("registered column dtype: {:?}", df.column_dtype("month"));

    reader.register("t", df, false).unwrap();

    match reader.execute("VISUALISE month AS x, v AS y FROM t DRAW bar SCALE x VIA date") {
        Ok(_) => println!("SUCCEEDED (unexpected)"),
        Err(e) => println!("FAILED: {}", e),
    }
}

Output:

registered column dtype: Ok(Timestamp(Microsecond, None))
FAILED: Validation error: Cannot coerce column '__ggsql_aes_pos1__' of type Timestamp(Microsecond, None) to date

Environment: posit-dev/ggsql main @ 983e38e3 (v0.5.2).

Root cause

src/execute/scale.rs, coerce_column_to_type, the ArrayElementType::Date
match arm (~line 754) only handles a DataType::Utf8 source (parsing
YYYY-MM-DD strings); every other source dtype -- including
Timestamp(Microsecond, None) -- falls into the catch-all _ branch and
hard-errors:

ArrayElementType::Date => match dtype {
    DataType::Utf8 => {
        // ... parses date strings into a Date32 array ...
    }
    _ => {
        return Err(GgsqlError::ValidationError(format!(
            "Cannot coerce column '{}' of type {:?} to date",
            column_name, dtype
        )));
    }
},

Arrow's cast_array (already imported and used a few lines above for
ArrayElementType::Number) supports a safe Timestamp -> Date32 cast that
just truncates the time-of-day component, but this arm never reaches it for
non-Utf8 inputs. Since Case 1 and Case 2 produce the identical Arrow
Timestamp(Microsecond, None) type, something upstream of this function
must be normalizing/pre-casting the column differently depending on whether
it came from register() or from the reader's own catalog -- worth tracing
that path too, since it's the reason this bug is inconsistent rather than
a hard failure every time a Timestamp reaches a date scale.

Note the neighboring ArrayElementType::DateTime arm has the identical
Utf8-only pattern, so a native Date32 column being coerced to a datetime
target may have the mirror-image problem -- I haven't verified that case,
but it's worth checking alongside this fix.

Suggested fix

Add a DataType::Timestamp(_, _) => cast_array(column, &DataType::Date32)?
branch to the ArrayElementType::Date arm in coerce_column_to_type,
mirroring the existing Number arm's use of cast_array. That would make
Case 2 succeed unconditionally rather than only when the upstream
normalization happens to have already run.

Workaround

Explicitly CAST(... AS DATE) the derived expression in SQL before it
reaches ggsql, e.g. CAST(DATE_TRUNC('month', d) AS DATE). That produces a
genuine Date32-typed column, which coerces fine regardless of table origin.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions