[SPARK-59273][SQL] Complete CHAR/VARCHAR support at core execution boundaries

[srielau]

What changes were proposed in this pull request?

When spark.sql.charVarchar.standardSemantics.enabled is true, CharType and VarcharType are first-class StringType subtypes. Several execution-boundary matchers still used exact StringType (or an explicit CHAR/VARCHAR reject), so those paths failed or skipped constrained string columns.

This patch treats CHAR/VARCHAR as the string family at:

• JDBC getters/setters and JDBC array element conversion
• JDBC schema inference (CHAR/VARCHAR keep first-class types when standard semantics is on, even if charVarcharAsString is also set)
• File partition value decoding
• RowToColumnConverter
• DataFrame.na.fill for string replacement values
• ANALYZE TABLE ... FOR COLUMNS (string-family stats)

Read-side CHAR padding and VARCHAR overflow still come from existing CAST / ApplyCharTypePadding paths rather than being reimplemented in each converter.

Why are the changes needed?

With first-class CHAR/VARCHAR, JDBC scans/writes, file-only partition discovery, columnar conversion, na.fill("..."), and column stats currently throw or silently ignore those columns. That blocks enabling standard semantics.

JIRA: https://issues.apache.org/jira/browse/SPARK-59273 (subtask of SPARK-58794)

Does this PR introduce any user-facing change?

Yes, when spark.sql.charVarchar.standardSemantics.enabled is true (still default false):

• JDBC read/write of CHAR/VARCHAR (including arrays) no longer fails with an unsupported JDBC type.
• File partition columns declared as CHAR/VARCHAR can be decoded; CHAR is padded on scan and oversize VARCHAR fails with EXCEED_LIMIT_LENGTH.
• Columnar row-to-column conversion accepts CHAR/VARCHAR.
• df.na.fill("x") fills null CHAR/VARCHAR columns (CHAR values are padded by CAST).
• ANALYZE TABLE ... FOR COLUMNS collects string-family stats on CHAR/VARCHAR instead of rejecting them.

How was this patch tested?

Added/extended unit tests:

• JDBCSuite: read CHAR/VARCHAR and arrays; write CHAR/VARCHAR; standard semantics wins over charVarcharAsString in schema inference
• ParquetV1PartitionDiscoverySuite / ParquetV2PartitionDiscoverySuite: CHAR/VARCHAR partition values and oversize VARCHAR
• RowToColumnConverterSuite: CHAR/VARCHAR and nested CHAR arrays
• DataFrameNaFunctionsSuite: na.fill on CHAR/VARCHAR
• StatisticsCollectionSuite: ANALYZE TABLE ... FOR COLUMNS on CHAR/VARCHAR

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor Grok 4.6

[cloud-fan]

Review summary

The execution-boundary changes and their direct tests are otherwise consistent, including the corrected JDBC configuration precedence. Before this is ready, the CBO consumers of column statistics need to handle CHAR/VARCHAR as members of the string family so enabling the new ANALYZE path cannot turn valid query planning into a MatchError.

Findings

1 total: 0 P0, 1 P1, 0 P2, 0 P3.

Blocking (P1)

• Handle constrained strings in CBO after collecting their statistics — sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeColumnCommand.scala:145 — see inline.

[cloud-fan]

Review summary

The previously reported CBO MatchError after collecting CHAR/VARCHAR statistics is fixed, and the execution-boundary changes are consistent with the shared first-class string-family contract. Two non-blocking follow-ups remain: add CHAR/VARCHAR to the owning join-estimation type matrix so the widened interval and size paths cannot regress untested, and correct one newly added comment that uses MatchError as a verb.

Findings

2 total: 0 P0, 0 P1, 1 P2, 1 P3.

Non-blocking (P2)

• Exercise CHAR/VARCHAR join estimation — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/ValueInterval.scala:56 — see inline.

Nit (P3)

• Use MatchError as an exception, not a verb — sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala:189 — see inline.

Re-review status

Prior AI findings: 1 addressed, 0 still present; additional unresolved findings in this review: 2.

New attribution: 2 newly introduced, 0 late catch, 0 previously raised, 0 unattributed.

Remaining prior AI findings

No prior AI findings remain.

[dtenedor]

LGTM, merging to master + 4.x

[dtenedor]

Merge Summary:

• merged into master 51f54b0
• merged into branch-4.x a9fd0a1

Posted by merge_spark_pr.py