Fix BigDiffy NPE on parquet schema mismatch in array fields - #902
Fix BigDiffy NPE on parquet schema mismatch in array fields#902monzalo14 wants to merge 4 commits into
Conversation
When diffing parquet files with different schemas, getCompatibleSchemaForFiles picks one schema as the reader schema. Fields present in the reader but absent in the writer get null-filled by the parquet reader, regardless of Avro nullability. For required array/map fields, this causes a NullPointerException in GenericDatumWriter.getArraySize during Avro coder encoding. Fix: after selecting the compatible schema, wrap non-nullable fields that are missing from the writer schema in a union with null (original type first to preserve existing defaults). This lets the Avro coder serialize null values without NPE. Add unit tests for makeNullableForMissingFields (arrays, maps, already-nullable fields, identical schemas) and an integration test verifying AvroCoder can encode records with null array fields under the transformed schema.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #902 +/- ##
==========================================
+ Coverage 70.75% 71.12% +0.36%
==========================================
Files 44 44
Lines 1939 1967 +28
Branches 335 308 -27
==========================================
+ Hits 1372 1399 +27
- Misses 567 568 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The initial fix only wrapped fields missing from the writer schema. Reproduction against the actual user data (InteractionFact.v2) revealed that the NPE also occurs when the field exists in both schemas but the parquet data contains nulls in non-nullable array columns (e.g. the 'path' field). Replace makeNullableForMissingFields with makeCollectionFieldsNullable, which wraps ALL non-nullable array/map fields in a union with null -- regardless of whether they differ between schemas. Also recurses into nested records. Verified: the NPE is gone when running BigDiffy on the actual user endpoints (part-00000 of InteractionFact.v2.sampled, partition 2026-07-20T10).
These four jobs (diffy_212, diffy_213, sampling_212, sampling_213) intermittently get OOM-killed or timeout on medium+ (6GB). They run the heaviest test suites (parquet I/O, Scio pipelines) with coverage instrumentation enabled. Bumping to large (8GB) gives them headroom. Other jobs (common, cli, scalacheck, shapeless, examples) remain on medium+.
| simulatedRhsRecord.put("id", 1) | ||
| simulatedRhsRecord.put("tags", null) | ||
|
|
||
| val encoded = CoderUtils.encodeToByteArray(beamCoder, simulatedRhsRecord) |
There was a problem hiding this comment.
Blocking: this proves the transformed schema can encode a hand-built null record, but it never runs BigDiffy. I extended this fixture through diffParquet and sc.run(), and it still fails with an NPE while Kryo decodes the array stored in a Delta. Could we make this a full-pipeline regression and fix that downstream serialization path?
| field.doc(), | ||
| field.defaultVal() | ||
| ) | ||
| } else if (fieldType == Schema.Type.RECORD) { |
There was a problem hiding this comment.
Blocking: recursion only handles bare RECORD fields. Arrays inside nullable record unions or array/map child records stay non-nullable; focused tests for both shapes still returned ARRAY. Could we recursively visit union branches, array elements, and map values, with cycle handling for named recursive records?
| // null second so the coder tolerates nulls from the parquet reader. | ||
| val nullableType = | ||
| Schema.createUnion(field.schema(), Schema.create(Schema.Type.NULL)) | ||
| new Schema.Field( |
There was a problem hiding this comment.
Blocking: these constructors drop field order, aliases, and custom properties, and the new record also loses its aliases and properties. A focused aliased-schema test lost the record alias immediately. Could we use the Avro field copy constructor and explicitly preserve the record metadata?
2 participants
Problem
When running BigDiffy in parquet mode, parquet can produce null values for non-nullable array/map fields. This happens both when:
In either case, Avro's
GenericDatumWriter.getArraySizecalls((Collection) null).size()with no null guard, causing aNullPointerException.--nullableCodersdoes not help -- that flag operates at the Scio coder boundary, not inside Avro record serialization.Solution
After selecting the compatible schema in
getCompatibleSchemaForFiles, wrap ALL non-nullable array and map fields in a union with null viamakeCollectionFieldsNullable. The original type is placed first in the union to preserve existing default values (e.g.,[]for arrays), and null is added second so the Avro coder can serialize null values without NPE. The transformation recurses into nested record fields.Also adds an early return when both schemas are identical and no collection fields need fixing.
Test plan
makeCollectionFieldsNullableinParquetIOTest: arrays+maps, already-nullable fields, no-collection-fieldsBigDiffyTest: writes parquet files with mismatched schemas, verifies the compatible schema wraps collection fields in nullable unions, and confirms AvroCoder can encode/decode records with null array fields2026-07-20T10) using a local ratatool build; verified the NPE is gone with this fix