[yaml] - mongodb read normalization by derrickaw · Pull Request #38772 · apache/beam

derrickaw · 2026-06-02T14:27:23Z

Fixes [yaml] Normalize MongoDBIO #28690
Adds the ReadFromMongoDb transform for yaml

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Squashed draft MongoDB read connector changes from PR apache#35802.

…, and tests - Fully implemented MongoDB read configuration and Provider with JSON schema parsing - Enhanced MongoDbUtils with a deep BSON-to-Beam row conversion supporting all primitives, arrays, maps, and nested rows - Added comprehensive Java unit tests for MongoDbUtils and MongoDbReadSchemaTransformProvider - Mapped WriteToMongoDB and ReadFromMongoDB in standard_io.yaml - Implemented end-to-end integration test verifying write/read pipeline against containerized MongoDB

…ansforms - Standardized standard_io mappings to snake_case (error_handling, batch_size) - Extended integration test to verify error-handling queues are empty for clean runs

…ix lint issues

derrickaw · 2026-06-09T19:05:58Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for reading from MongoDB as a SchemaTransform in the Java SDK, along with exposing this capability in Python YAML pipelines. It includes configuration classes, conversion utilities between BSON Documents and Beam Rows, and corresponding unit tests. Key feedback from the review highlights a potential serialization issue in the Java read transform due to an anonymous inner class, performance optimization opportunities in the BSON-to-Row conversion by using Map instead of Document to avoid copying nested structures, and a robustness improvement in the Python YAML reader to support string-based filters.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

derrickaw · 2026-06-10T19:51:12Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a MongoDB Read SchemaTransform to Apache Beam, enabling schema-based reads from MongoDB in both Java and YAML pipelines. Key additions include configuration and provider classes in Java, utility methods for converting BSON documents to Beam Rows, and integration with Python YAML IO. Feedback on the changes highlights a compilation error in MongoDbReadSchemaTransformProvider due to an incorrect .iterator() call, potential robustness issues in the error-handling path if doc.toJson() fails, missing defensive type checks in MongoDbUtils that could lead to ClassCastExceptions, and a suggestion to parse the projection parameter in Python YAML for consistency.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

codecov · 2026-06-18T02:17:10Z

Codecov Report

❌ Patch coverage is 29.41176% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.25%. Comparing base (6ed0635) to head (bf628ad).
⚠️ Report is 26 commits behind head on master.

Files with missing lines	Patch %	Lines
sdks/python/apache_beam/yaml/yaml_io.py	29.41%	12 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #38772   +/-   ##
=========================================
  Coverage     60.25%   60.25%           
  Complexity    20572    20572           
=========================================
  Files          3318     3318           
  Lines        315452   315508   +56     
  Branches      17142    17142           
=========================================
+ Hits         190073   190112   +39     
- Misses       116830   116847   +17     
  Partials       8549     8549

Flag	Coverage Δ
python	`79.26% <29.41%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

gemini-code-assist · 2026-06-24T14:13:07Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces the ReadFromMongoDB transform for the Apache Beam YAML SDK, addressing issue #28690. It provides a structured way to read data from MongoDB by defining a schema, supporting optional filtering, and including error handling for malformed or incompatible documents. The changes span across Java IO utilities, Python YAML definitions, and necessary build configuration updates.

Highlights

MongoDB Read Transform: Added the ReadFromMongoDB transform to the YAML SDK, enabling users to read data from MongoDB collections using a schema-based approach.
Schema Normalization: Implemented robust conversion utilities to map BSON documents to Beam Rows, ensuring data integrity and type safety during the read process.
Error Handling: Integrated error handling capabilities into the MongoDB read transform, allowing for the redirection of failed rows to a separate output collection.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a MongoDB Read SchemaTransform provider (MongoDbReadSchemaTransformProvider) and its configuration (MongoDbReadSchemaTransformConfiguration) in Java, enabling users to read from MongoDB using Beam schemas. It also updates MongoDbUtils to support converting BSON documents to Beam Rows, registers ReadFromMongoDB in the YAML standard IO, and implements read_from_mongodb in Python's yaml_io.py. Key feedback includes: using BsonDocument.parse instead of Document.parse to avoid serialization and type mismatch issues; explicitly handling ObjectId and Number types in MongoDbUtils conversions; removing redundant null checks in the configuration class; correcting the validation tests to assert on empty strings instead of nulls; and normalizing native BSON types in Python before converting them to Beam Rows.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

derrickaw · 2026-06-24T14:32:03Z

assign set of reviewers

github-actions · 2026-06-24T14:33:21Z

Assigning reviewers:

R: @tvalentyn for label python.
R: @kennknowles for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

tvalentyn · 2026-06-29T15:03:31Z

Assigning reviewers:

R: @tvalentyn for label python. R: @kennknowles for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling

remind me after tests pass - tag the comment author after tests pass

waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

python bits LGTM

github-actions · 2026-06-29T15:04:34Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

derrickaw · 2026-06-29T15:07:57Z

R: @ahmedabu98

ahmedabu98

Overall looks good! Just left a few comments.

Had a higher level question:
IIUC this PR is adding two ReadFromMongoDB implementations to YAML: One from normalized Java and one from Python
Is it okay that their configurations don't overlap entirely? I know YAML switches tries to use the same implementation that matches the SDK of neighboring transforms. Does it also take configuration compatibility into account?

ahmedabu98 · 2026-06-29T17:24:58Z

+        if (value instanceof Map) {
+          return toRow((Map<?, ?>) value, rowSchema);
+        } else {
+          throw new IllegalArgumentException("Cannot convert value to Row: " + value);


Maybe instead just mention the value type (not the value itself)

ahmedabu98 · 2026-06-29T17:27:10Z

+
+    PAssert.that(errorRows)


Add a PAssert that MongoDbReadSchemaTransformProvider.OUTPUT_TAG pcollection is empty

derrickaw · 2026-06-29T20:57:41Z

Had a higher level question:
IIUC this PR is adding two ReadFromMongoDB implementations to YAML: One from normalized Java and one from Python
Is it okay that their configurations don't overlap entirely? I know YAML switches tries to use the same implementation that matches the SDK of neighboring transforms. Does it also take configuration compatibility into account?

I believe it does take into consider the configuration compatibility, but you bring up a good point and I have reverted those changes to where they match up better now.

github-actions Bot added python java io mongodb yaml labels Jun 2, 2026

arnavarora2004 and others added 10 commits June 9, 2026 13:21

feat(yaml): Add MongoDB read connector draft

43bc9f4

Squashed draft MongoDB read connector changes from PR apache#35802.

test(yaml): Add error-handling coverage for MongoDB Write and Read tr…

eb55f98

…ansforms - Standardized standard_io mappings to snake_case (error_handling, batch_size) - Extended integration test to verify error-handling queues are empty for clean runs

remove old edits to other transforms from original PR

205d95a

add python support

8095180

revert changes to integration tests not needed from original PR and f…

73aad3a

…ix lint issues

fix more lint

0a09db3

fix generate external transforms

5808923

pin read to python provider

9983d7a

comment out read part

3bec200

derrickaw force-pushed the 20260602_readFromMongoDb branch from 9df8c17 to 3bec200 Compare June 9, 2026 13:21

retry readfrommongodb

e288769

gemini-code-assist Bot reviewed Jun 9, 2026

View reviewed changes

fix gemini comments

79f4ff1

gemini-code-assist Bot reviewed Jun 10, 2026

View reviewed changes

derrickaw added 3 commits June 17, 2026 18:42

address gemini

bb2b75b

Merge remote-tracking branch 'upstream' into 20260602_readFromMongoDb

5060088

fix spotless

bf628ad

derrickaw added 3 commits June 18, 2026 13:00

fix bad return type

0c28651

Merge remote-tracking branch 'upstream' into 20260602_readFromMongoDb

e5970b9

fix flakey test

9ef2324

derrickaw marked this pull request as ready for review June 24, 2026 14:12

gemini-code-assist Bot reviewed Jun 24, 2026

View reviewed changes

github-actions Bot added the Next Action: Reviewers label Jun 24, 2026

ahmedabu98 reviewed Jun 29, 2026

View reviewed changes

derrickaw added 2 commits June 29, 2026 18:44

address comments and switch to uri

fddf784

remove non-overlapping parameters

737b504

derrickaw requested a review from ahmedabu98 June 30, 2026 02:09

Uh oh!

Conversation

derrickaw commented Jun 2, 2026

GitHub Actions Tests Status (on master branch)

Uh oh!

derrickaw commented Jun 9, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

derrickaw commented Jun 10, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jun 18, 2026

Codecov Report

Uh oh!

gemini-code-assist Bot commented Jun 24, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

derrickaw commented Jun 24, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

tvalentyn commented Jun 29, 2026

Uh oh!

github-actions Bot commented Jun 29, 2026

Uh oh!

derrickaw commented Jun 29, 2026

Uh oh!

ahmedabu98 left a comment

Choose a reason for hiding this comment

Uh oh!

ahmedabu98 Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

derrickaw Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

ahmedabu98 Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

derrickaw Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

derrickaw commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

derrickaw commented Jun 29, 2026 •

edited

Loading