[SPARK-58814][SQL] Cover CHAR/VARCHAR round-trips and stop ORC truncating scans - #58317
[SPARK-58814][SQL] Cover CHAR/VARCHAR round-trips and stop ORC truncating scans#58317srielau wants to merge 9 commits into
Conversation
cloud-fan
left a comment
cloud-fan
left a comment
There was a problem hiding this comment.
Please address the non-vectorized ORC decoder incompatibility and preserve-only VARCHAR enforcement, then make the nested ORC regression test materialize the collection fields it introduces.
This mapping is unconditional, so it also runs when preserveCharVarcharTypeInfo=true is the only reason the constrained type is retained and standard semantics is disabled. That branch of addPaddingForScan installs no VARCHAR read-side check. Requesting ORC string therefore removes native varchar(n) enforcement and can expose abcdef under a VARCHAR(4) schema. Please preserve the native constrained schema in preserve-only mode or add equivalent validation, with a value-level regression test for this flag combination.
cloud-fan
left a comment
cloud-fan
left a comment
There was a problem hiding this comment.
Review summary
Two changes are needed before this is safe to merge: bind ORC reader schema selection to the persisted/analyzed CHAR/VARCHAR semantics so view results cannot silently change with the caller configuration, and make the Avro regression test observe its array and map values.
Findings
2 total: 0 P0, 1 P1, 1 P2, 0 P3.
Blocking (P1)
- Keep the view-bound CHAR/VARCHAR mode for ORC scans —
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:443— see inline.
Non-blocking (P2)
- Assert the Avro array and map values that the test creates —
connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala:3742— see inline.
cloud-fan
left a comment
cloud-fan
left a comment
There was a problem hiding this comment.
Review summary
The ordinary standard-semantics paths and the previously raised ORC/Avro issues are addressed, but the scan-binding mechanism puts trusted analysis state into caller-controlled schemas and connector options. The blocking request is therefore one owner-level carrier redesign, with option isolation and both configuration transitions serving as its validation, plus one separate preserve-only writer regression test.
Findings
2 total: 0 P0, 1 P1, 1 P2, 0 P3.
Blocking (P1)
- Carry the analyzed ORC semantics mode in trusted relation state —
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ApplyCharTypePadding.scala:91
Could we model the CHAR/VARCHAR mode captured during analysis as explicit engine-private, relation-scoped state and pass it directly to ORC schema construction? The current carrier uses caller-owned namespaces: V1 trustsStructFieldmetadata, while V2 stores an ORC-only marker in the public, case-insensitive options map that is forwarded to everySupportsReadconnector. This lets a caller forge or clobber the mode, can make Catalyst's length checks disagree with the physical ORC schema, and can break unrelated connectors that reject or collide with the undocumented option. It also represents preserve-only by absence, sofalseis indistinguishable from unbound or lost state. Please bind both standard and preserve-only modes explicitly through a trusted internal carrier, leave non-ORCr.optionsunchanged, and consume the state only when deriving the ORC request schema. The regression coverage should include caller-supplied marker attempts, a strict non-ORC connector whose options remain unchanged, and both analysis/execution configuration directions across V1/V2 and row/vector readers.
Non-blocking (P2)
- Restore preserve-only ORC writer coverage —
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:451— see inline.
Re-review status
2 addressed, 0 remaining, 2 new to this AI review.
New attribution: 2 newly introduced, 0 late catch, 0 previously raised, 0 unattributed.
Remaining findings
No prior AI findings remain.
Verification
- The carrier defect spans both V1 and V2: caller schema metadata and read options can select the physical ORC schema independently of the mode that installed Catalyst length checks.
- A correct owner-level repair must preserve unrelated V2 connector options and bind standard and preserve-only modes across both analysis/execution configuration directions.
|
Addressed the blocking carrier finding in 5117aaf. The analyzed mode is now stored as an explicit private The regression matrix now covers both analysis/execution configuration directions in V1/V2 and row/vector modes, caller attempts to forge the old metadata/option markers, and exact preservation of a non-ORC V2 relation's options. Verification: the focused SPARK-58814 test and SQL main/test scalastyle checks pass. |
|
Follow-up in 5f0006d removes the remaining unbound fallback: first-class CHAR/VARCHAR relations now bind an explicit Boolean even when read-side padding is disabled, and neither V1 nor V2 ORC readers consult execution-time SQLConf when no binding is present. The preserve-only cross-configuration tests now disable read-side padding to cover this path. The focused SPARK-58814 test and SQL main/test scalastyle checks pass. |
cloud-fan
left a comment
cloud-fan
left a comment
There was a problem hiding this comment.
Review summary
This revision addresses the earlier review feedback: ORC now handles all StringType subtypes in row readers, covers nested collections, preserves analyzed view semantics, avoids caller-controlled metadata/options as the carrier, and explicitly binds preserve-only mode. The expanded V1/V2 and row/vectorized regression coverage is strong.
One blocking correctness issue remains. The bound mode affects ORC results but is not represented in Spark's semantic plan identity, allowing cache or plan reuse across incompatible modes.
Findings
1 total: 0 P0, 1 P1, 0 P2, 0 P3.
Blocking (P1)
-
Make the bound ORC semantics mode part of plan identity —
ApplyCharTypePadding.scala:43TreeNodeTagis excluded from canonicalization andsameResult. A preserve-only cached relation that truncated"abcdef"to"abcd"can therefore replace the relation beneath a standard-semantics length check, causing the expectedEXCEED_LIMIT_LENGTHerror to be missed.V2 has the same identity gap because
OrcScan.equalsdoes not comparecharVarcharStandardSemantics. Please carry the mode in semantically compared relation/scan state and add V1/V2 cache-reuse coverage.
Re-review status
All previously reported issues are addressed. This is one newly identified issue in the replacement carrier design.
Verification
Reviewed head 5f0006d949ed876e788b9fefb6ae5ca21728700e. All current GitHub CI checks pass, including SQL, Avro, extended tests, and linters. No local tests were run.
|
Addressed in 57a9c67.
|
cloud-fan
left a comment
cloud-fan
left a comment
There was a problem hiding this comment.
Review summary
Two blocking correctness issues remain: V1 writes can miss bound-mode cache entries and leave stale rows, and bound ORC scans can bypass reader overrides on OrcFileFormat subclasses. The new ORC mode parameter also needs its missing Scaladoc entry. Previously reported row-decoder, nested-value, persisted-view, carrier, preserve-only, and cross-mode cache-reuse issues are resolved at the pinned head.
Findings
3 total: 0 P0, 2 P1, 0 P2, 1 P3.
Blocking (P1)
- Invalidate every bound-mode cache after a V1 save —
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SaveIntoDataSourceCommand.scala:74— see inline. - Preserve OrcFileFormat subclass reader dispatch —
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:763— see inline.
Nit (P3)
- Document the ORC semantics parameter —
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:552— see inline.
Re-review status
2 addressed, 0 remaining, 3 new to this AI review.
New attribution: 3 newly introduced, 0 late catch, 0 previously raised, 0 unattributed.
Remaining findings
No prior AI findings remain.
…ting scans Keep Spark responsible for CHAR/VARCHAR assignment and scan checks: ORC was mapping an explicit CHAR/VARCHAR read schema to native ORC types, which truncated oversize values before Spark could reject them. Add Parquet/ORC/CSV/Avro coverage for nested inference, user schemas, and INSERT assignment under standardSemantics.
…cement Decode all string-family types in the row reader, while retaining native ORC constraints in preserve-only mode. Materialize nested collections and verify nested overflow checks across V1/V2 and vectorized/row readers.
Carry analyzed semantics through private plan state so callers and unrelated connectors cannot alter the ORC request schema.
Bind preserve-only mode even when read-side padding is disabled and remove execution-session fallbacks from ORC readers.
TreeNodeTags are dropped by canonicalization, so a preserve-only cached ORC scan could be reused under standard semantics. Carry the bound mode on relation/scan case-class state that sameResult compares.
…ate ORC reader Invalidate cached V1 relations by matching the written BaseRelation while ignoring CHAR/VARCHAR scan-mode identity, so a successful save/insert refreshes None, Some(false), and Some(true) cache entries without weakening ordinary cross-mode cache identity. Restrict the mode-aware private ORC reader overload to the exact built-in OrcFileFormat so subclasses keep public buildReaderWithPartitionValues dispatch. Document the orcResultSchemaString charVarcharStandardSemantics param.
4b6e0e5 to
2ad887c
Compare
There was a problem hiding this comment.
Review summary
The earlier row-decoder, nested-value, persisted-view, carrier, plan-identity, cache-invalidation, and subclass-dispatch findings are addressed. One blocking correctness gap remains in the V1 compatibility path: the exact-class guard restores the existing subclass override, but a delegating override loses the analysis-bound standard mode when it calls the public OrcFileFormat super method, which defaults to preserve-native behavior.
Please remove this ORC-specific dispatch split and make the semantic state explicit end to end.
Requested design
Define the mode in Catalyst, where both V1/V2 logical relations can use it:
private[sql] sealed trait CharVarcharScanMode extends Product with Serializable
private[sql] object CharVarcharScanMode {
case object PreserveNative extends CharVarcharScanMode
case object SparkStandard extends CharVarcharScanMode
}Store Option[CharVarcharScanMode] in LogicalRelation, DataSourceV2Relation,
HiveTableRelation, and their physical scans. None means that first-class CHAR/VARCHAR
semantics do not apply; preserve-only and standard modes are both explicit values. Analysis binds
the value once and does not replace a value already captured for a view. Because it is case-class
state, result identity remains mode-sensitive; the existing BaseRelation-specific cache
invalidation remains deliberately mode-insensitive.
Add this overload to FileFormat:
def buildReaderWithPartitionValues(
sparkSession: SparkSession,
dataSchema: StructType,
partitionSchema: StructType,
requiredSchema: StructType,
filters: Seq[Filter],
options: Map[String, String],
hadoopConf: Configuration,
charVarcharScanMode: CharVarcharScanMode)
: PartitionedFile => Iterator[InternalRow]Its default implementation should clone hadoopConf, overwrite an engine-private mode entry,
and invoke the existing seven-argument method virtually. FileSourceScanExec selects the old or
new overload based only on whether the bound mode is present; it does not inspect the format class.
The existing public ORC method reads the compatibility entry and calls a non-virtual internal
builder. Thus an existing subclass still receives its old override, and super recovers the
bound mode. Direct callers of the old overload retain preserve-native behavior.
Use the same typed mode in the V2 builder, OrcScan, reader factory, and
OrcUtils.orcResultSchemaString. Only ORC maps SparkStandard to physical STRING and
PreserveNative to native constrained types.
Findings
2 total: 0 P0, 1 P1, 0 P2, 1 P3.
Blocking (P1)
- Define and preserve CHAR/VARCHAR scan mode through the generic FileFormat overload —
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:762—
see inline.
Nit (P3)
- Describe TreeNodeTag equality rather than tag removal —
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ApplyCharTypePadding.scala:56—
see inline.
Re-review status
The three findings from the prior AI review are addressed. The blocking finding below is introduced
by that dispatch fix; the comment correction was identified during the broader re-review.
…eFormat overload Introduce a typed `CharVarcharScanMode` (`PreserveNative` / `SparkStandard`) in Catalyst and carry `Option[CharVarcharScanMode]` on the relation carriers (`LogicalRelation`, `DataSourceV2Relation`, `HiveTableRelation`) and the V1 scan node (`FileSourceScanExec`), replacing the previous `Option[Boolean]` field. Add a mode-aware overload to `FileFormat.buildReaderWithPartitionValues`. Its default implementation clones the per-call Hadoop configuration, writes an engine-private entry with the explicit mode, and invokes the existing seven-argument method virtually. `FileSourceScanExec` calls this overload whenever the relation has a bound mode, regardless of the concrete file format, so the ORC-specific format match is removed. `OrcFileFormat`'s public seven-argument override reads the bridged entry and passes the mode to its non-virtual reader builder. Existing subclasses therefore keep their override, and a delegating `super` call retains the analyzed mode instead of defaulting to preserve-native (which could bypass standard-semantics length checks). The Hadoop entry is only a bridge across the legacy signature; the authoritative state remains the typed plan field and overload parameter. Extend the `OrcFileFormat` subclass regression with an over-length value: `SparkStandard` must raise `EXCEED_LIMIT_LENGTH`, while `PreserveNative` retains native ORC truncation, covering both the row and vectorized readers. Also reword the `ApplyCharTypePadding` comment: a `TreeNodeTag` survives canonicalization (`makeCopy` calls `copyTagsFrom`) but does not participate in structural plan equality / `sameResult`, which is why a case-class field is used instead.
cloud-fan
left a comment
cloud-fan
left a comment
There was a problem hiding this comment.
Review summary
The typed scan-mode redesign resolves the earlier carrier and dispatch problems, but one blocking gap remains: catalog-less V2 writes recache through an unbound relation and can leave mode-bound cached reads stale after a successful append or overwrite. The ORC compatibility key also needs isolation from caller-controlled options. Finally, update the canonicalization rationale and the V1 cache-test comment to match the actual implementation. No local tests were run.
Findings
4 total: 0 P0, 1 P1, 1 P2, 2 P3.
Blocking (P1)
- Invalidate mode-bound cache entries after catalog-less V2 writes —
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:122— see inline.
Non-blocking (P2)
- Keep the ORC mode bridge out of caller options —
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala:314— see inline.
Nit (P3)
- Do not claim all relation tags survive canonicalization —
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ApplyCharTypePadding.scala:58— see inline. - Name the typed scan modes in the V1 cache test —
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SaveIntoDataSourceCommandSuite.scala:76— see inline.
Re-review status
Prior AI findings: 3 addressed, 0 still present; additional unresolved findings in this review: 4.
New attribution: 3 newly introduced, 1 late catch, 0 previously raised, 0 unattributed.
Remaining prior AI findings
No prior AI findings remain.
Existing discussions
- existing discussion — The revised comment correctly identifies structural equality as a problem, but its new blanket canonicalization-survival claim is false for LogicalRelation and HiveTableRelation direct canonicalization copies.
cloud-fan
left a comment
cloud-fan
left a comment
There was a problem hiding this comment.
Review summary
The remaining blocking issue is at the shared relation-cache boundary: analysis-bound charVarcharScanMode is preserved when a cached V2 table relation is reused by a fresh analysis, so the query can inherit another session's semantics and silently truncate an over-length ORC value instead of raising EXCEED_LIMIT_LENGTH. The normal cache key should remain mode-sensitive, but a SharedState table-relation cache hit must be rebound to the current analysis. Non-blocking follow-ups are to keep the FileFormat bridge internal, preserve V2 cache storage across rename, cover the InsertIntoDataSourceCommand and positional ORC branches under bound modes, correct the documented configuration key, and remove three discarded relation copies.
Findings
7 total: 0 P0, 1 P1, 4 P2, 2 P3.
Blocking (P1)
- Rebind a shared V2 relation to the current analysis mode —
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:122— see inline.
Non-blocking (P2)
- Keep the scan-mode bridge internal to Spark SQL —
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala:182— see inline. - Exercise InsertIntoDataSourceCommand recaching in both bound modes —
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoDataSourceCommand.scala:48— see inline. - Preserve a V2 table cache across rename in bound modes —
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala:75— see inline. - Cover CHAR/VARCHAR checks through positional ORC evolution —
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:559— see inline.
Nit (P3)
- Name the registered standard-semantics configuration —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharScanMode.scala:50— see inline. - Remove discarded relation copies from the padding rule —
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ApplyCharTypePadding.scala:99— see inline.
Re-review status
Prior AI findings: 4 addressed, 0 still present; additional unresolved findings in this review: 7.
New attribution: 2 newly introduced, 5 late catch, 0 previously raised, 0 unattributed.
Remaining prior AI findings
No prior AI findings remain.
| timeTravelSpec: Option[TimeTravelSpec] = None, | ||
| // Bound at analysis so sameResult / cache reuse distinguish preserve-only vs standard | ||
| // CHAR/VARCHAR scans. None means the relation was not analyzed under first-class types. | ||
| charVarcharScanMode: Option[CharVarcharScanMode] = None) |
There was a problem hiding this comment.
Blocking (P1): Because this mode is analysis-bound, retaining it on a relation returned by the SharedState relation cache lets a fresh analysis inherit the mode that populated the cache. For example, a relation cached under PreserveNative can remain preserve-bound when a later session analyzes it under SparkStandard, allowing an over-length ORC value to appear truncated rather than raising EXCEED_LIMIT_LENGTH. Please clear the mode specifically when adapting a shared cached table relation so the current analysis rebinds it, while preserving the mode stored in persisted view plans, and add a same-SharedState mode-switch regression.
Recommended change: Rebind relations obtained from the shared table-relation cache to the current analysis mode.
Why this works: Clear charVarcharScanMode only on RelationResolution's adaptation of a SharedState cache hit, before ApplyCharTypePadding binds the current SQLConf mode.
Scope: The shared V2 table-relation cache adaptation and a focused cross-session or cross-configuration cache regression.
Compatibility: Keep ordinary cached-plan equality mode-sensitive and preserve analysis-bound mode state embedded in persisted view plans.
Risks: Clearing the field on persisted view plans would regress their caller-independent semantics. Making normal cache equality mode-insensitive would permit reuse of incompatible cached results.
Constraints: Limit rebinding to relations obtained from the SharedState table-relation cache. Exercise two analyses sharing SharedState with different modes and an over-length ORC value.
Success: A fresh SparkStandard analysis cannot inherit PreserveNative from an earlier shared cache entry and raises EXCEED_LIMIT_LENGTH, while persisted views retain their analyzed mode.
| def fromName(name: String): CharVarcharScanMode = name match { | ||
| case "PreserveNative" => PreserveNative | ||
| case "SparkStandard" => SparkStandard | ||
| case other => throw new IllegalArgumentException(s"Unknown CharVarcharScanMode: $other") |
There was a problem hiding this comment.
Nit (P3): This names a configuration that does not exist. The registered key is spark.sql.charVarchar.standardSemantics.enabled; please use that exact name here.
| * and a call to `super` retains the bound mode. The Hadoop entry is only a transport across the | ||
| * legacy signature; the authoritative state remains the typed plan field and this parameter. | ||
| */ | ||
| def buildReaderWithPartitionValues( |
There was a problem hiding this comment.
Non-blocking (P2): This overload is an internal planner bridge, but leaving it public enlarges the supported custom FileFormat surface and can make an existing untyped eta-expansion of buildReaderWithPartitionValues ambiguous. Please make this overload, CHAR_VARCHAR_SCAN_MODE, and CharVarcharScanMode package-private to Spark SQL. Internal callers retain access, and the default implementation can still dispatch through a third-party subclass's existing seven-argument override.
| // Re-cache all cached plans(including this relation itself, if it's cached) that refer to this | ||
| // data source relation. | ||
| sparkSession.sharedState.cacheManager.recacheByPlan(sparkSession, logicalRelation) | ||
| sparkSession.sharedState.cacheManager.recacheByV1Relation( |
There was a problem hiding this comment.
Non-blocking (P2): Please exercise this changed caller with a materialized cache under both PreserveNative and SparkStandard. InsertSuite currently tests this command only with an unbound relation, while the new two-mode regression covers the sibling SaveIntoDataSourceCommand. The focused test should perform the overwrite, verify that the query remains cached, and verify that it returns only the replacement rows; that will fail if this path regresses to mode-sensitive plan matching.
| if (conf.readSideCharPadding || standardSemantics) { | ||
| val newPlan = boundPlan.resolveOperatorsUpWithNewOutput { | ||
| case r: LogicalRelation => | ||
| bindStandardSemantics(r) |
There was a problem hiding this comment.
Nit (P3): The result of this pure helper is discarded, so an unbound relation is copied and has its tags copied without affecting the plan. The same applies to the standalone calls in the V2 and Hive cases below. Please remove all three; the initial binding traversal and the cleaned-relation thunks already retain the returned nodes that are needed.
| cacheManager.recacheTableOrView(session, nameParts, includeTimeTravel = false) | ||
| case _ => | ||
| cacheManager.recacheByPlan(session, r) | ||
| cacheManager.recacheByV2Relation(session, r) |
There was a problem hiding this comment.
Non-blocking (P2): This rename path still discovers the old cache entry with normal sameResult semantics. Under either first-class mode the cached relation is bound to Some(mode), while invalidateTableCache constructs an unbound probe, so the lookup returns no storage level and the renamed table is left uncached. Please use the mutation-specific comparison that ignores only charVarcharScanMode here, retain the existing table/time-travel restrictions, and run the rename-cache regression under both bound modes.
Recommended change: Use mode-insensitive relation matching only for V2 rename cache discovery.
Why this works: Route invalidateTableCache's lookup through a mutation-specific relation comparison that ignores charVarcharScanMode while retaining table identity and time-travel exclusions.
Scope: DataSourceV2Strategy's rename invalidation path and the existing V2 rename-cache regression under both bound modes.
Compatibility: Preserve ordinary mode-sensitive cache equality and preserve the cached table's prior storage level across rename.
Risks: A matcher that ignores more than the scan mode could select an unrelated or time-travel cache entry. Changing normal lookupCachedData semantics would allow incompatible read caches to match.
Constraints: Keep the relaxed comparison local to mutation-specific rename discovery. Retain current table identity and time-travel filtering.
Success: Renaming a cached V2 table under PreserveNative or SparkStandard restores it under the new identifier at the prior storage level.
| OrcUtils.getOrcSchemaString(resultSchema, charVarcharStandardSemantics) | ||
| } else { | ||
| OrcUtils.getOrcSchemaString(StructType(dataSchema.fields ++ partitionSchema.fields)) | ||
| OrcUtils.getOrcSchemaString( |
There was a problem hiding this comment.
Non-blocking (P2): The new CHAR/VARCHAR matrix reaches only the named-field (canPruneCols=true) path, so this changed full-schema branch has no standard-semantics regression. Please add an OrcSourceSuite case using forced positional evolution or an all-_col physical schema. It should prove that an in-range CHAR is padded and an over-length positional STRING raises EXCEED_LIMIT_LENGTH rather than being truncated natively by ORC.
2 participants
What changes were proposed in this pull request?
Parent: SPARK-58794 (first-class CHAR/VARCHAR under
spark.sql.charVarchar.standardSemantics.enabled).standardSemantics, keep Spark responsible for CHAR/VARCHAR assignment and scan checks on ORC by requesting physical ORC STRING. The row decoder accepts allStringTypesubtypes recursively. Preserve-only mode retains native ORC CHAR/VARCHAR enforcement.standardSemantics:standardSemantics=falsecollapses to STRING;preserveCharVarcharTypeInfokeeps typesParquet already embeds Spark schema JSON in footer metadata, so nested CHAR/VARCHAR inference needed tests rather than a writer change.
Why are the changes needed?
Without this, Spark-written files and user-specified schemas do not keep first-class CHAR/VARCHAR through round-trips in the remaining format gaps. For ORC specifically, an explicit
schema("c CHAR(4)")read sent native ORCchar(n)viaMAPRED_INPUT_SCHEMA, so ORC truncated"abcdef"to"abcd"beforecharTypeReadSideCheckandEXCEED_LIMIT_LENGTHnever fired.Does this PR introduce any user-facing change?
Yes, when
spark.sql.charVarchar.standardSemantics.enabledis true (unreleased / master):EXCEED_LIMIT_LENGTH.How was this patch tested?
New tests in
BasicCharVarcharTestSuite(SPARK-58814: major formats preserve CHAR/VARCHAR schemas and values) andAvroSuite(V1 and V2). Also ran existing ORC query suites and scalastyle:Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Grok 4.6