[SPARK-59278][SQL] Fix CHAR comparison rewrite edge cases

[srielau]

What changes were proposed in this pull request?

Fix two CHAR comparison padding edge cases:

• Preserve each foldable IN list expression in its original position while computing padding lengths, including NULL entries.
• Recognize analyzer-inserted CAST(attribute AS STRING) nodes used to coerce untyped NULL, without treating user-specified casts as CHAR comparisons.
• Accumulate padding requirements across every struct field instead of retaining only the last field's result.
• Preserve nullable parent structs when rebuilding nested values for comparison.

JIRA: https://issues.apache.org/jira/browse/SPARK-59278

Why are the changes needed?

The previous IN rewrite partitioned null values away from non-null values and then zipped the original expression list with only the non-null lengths. If NULL preceded a matching literal, the matching literal could be dropped and a true result became null.

Nested struct padding also overwrote its needPadding state for each field. Padding required by an earlier field was therefore lost when a later field needed no rewrite. Activating that rewrite additionally required preserving a null parent struct rather than rebuilding it as a non-null struct of null fields.

Does this PR introduce any user-facing change?

Yes. CHAR comparisons now preserve SQL three-valued IN/NOT IN semantics when NULL precedes a matching literal, and nested struct comparisons apply all required CHAR padding while preserving parent nulls.

How was this patch tested?

Added regression tests covering:

• IN and NOT IN with NULL before and between matching and non-matching literals.
• Implicit analyzer casts versus user-specified CAST(... AS STRING).
• Multi-field nested struct padding.
• Null parent structs with regular and null-safe equality.

Ran:

sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \
  -Dsbt.override.build.repos=true \
  'sql/testOnly org.apache.spark.sql.FileSourceCharVarcharTestSuite org.apache.spark.sql.DSV2CharVarcharTestSuite -- -z SPARK-59278'

sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \
  -Dsbt.override.build.repos=true \
  'sql/testOnly org.apache.spark.sql.FileSourceCharVarcharTestSuite org.apache.spark.sql.DSV2CharVarcharTestSuite -- -z SPARK-34833' \
  'sql/testOnly org.apache.spark.sql.FileSourceCharVarcharTestSuite org.apache.spark.sql.DSV2CharVarcharTestSuite -- -z SPARK-50847' \
  'sql/testOnly org.apache.spark.sql.FileSourceCharVarcharTestSuite org.apache.spark.sql.DSV2CharVarcharTestSuite -- -z SPARK-51732'

dev/scalastyle sql

All selected tests and Scala style checks passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor Auto

[srielau]

Closing as a duplicate of #58553, which was opened first and already has committer approval.

That PR fixes the same IN NULL-order and nested-struct padding bugs, and also addresses the redundant InTypeCoercion Cast and non-orderable struct error-message case. I left a note there about the extra InSet conversion coverage from this PR.