Skip to content

HDDS-15766. Make Recon OM DB large tarball transfer reliable#10696

Draft
devmadhuu wants to merge 8 commits into
apache:masterfrom
devmadhuu:HDDS-15766
Draft

HDDS-15766. Make Recon OM DB large tarball transfer reliable#10696
devmadhuu wants to merge 8 commits into
apache:masterfrom
devmadhuu:HDDS-15766

Conversation

@devmadhuu

@devmadhuu devmadhuu commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Problem:
Recon keeps its own copy of the OM DB and syncs it two ways: incremental delta updates (getDBUpdates) for the normal case, and a full OM DB snapshot download as the fallback when delta sync cannot be used (Recon fell too far behind, OM no longer has the requested updates, etc.).

The full-snapshot fallback was the weak part:

  • It used a single one-shot GET /dbCheckpoint call that streamed the entire OM DB as one tar.
  • Any mid-stream failure restarted the download from byte 0.

On large clusters the OM DB is big, so this approach was slow and fragile.
Approach: Instead of maintaining Recon's own one-shot download logic, Recon now reuses the same bootstrap mechanism an OM follower uses to catch up from the OM leader:

  • A chunked POST /v2/dbCheckpoint transfer. Each subsequent chunk carries a toExcludeList[] of the parts already received, so an interrupted transfer resumes instead of restarting from the beginning.
  • The transfer is chunked and ends with a completion marker; the Recon client keeps the parts it already received and only requests the rest, so an interrupted transfer resumes instead of restarting.
  • Resume/retry behavior comes from the shared, proven OM code path (RDBSnapshotProvider, the same base OmRatisSnapshotProvider uses) rather than a Recon-specific re-implementation.
  • ReconRDBSnapshotProvider overrides only the download step and the step that assembles/promotes the finished DB.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15766

How was this patch tested?

Testing:

  • New unit tests TestReconRDBSnapshotProvider
  • Updated TestOzoneManagerServiceProviderImpl— drives the new provider path for success and failure cases.
  • Existing TestReconWithOzoneManager.testOmDBSyncWithSeqNumberMismatch provides end-to-end coverage of the full-snapshot fallback against a mini-cluster
  • TestReconWithOzoneManagerHA updated to drop the stale v1-URL assertion, keeping its end-to-end leader-sync verification.

@devmadhuu devmadhuu changed the title HDDS-15766 - Recon - Make Recon OM DB large tar ball transfer reliable. HDDS-15766. Recon - Make Recon OM DB large tar ball transfer reliable. Jul 9, 2026
- TestReconRDBSnapshotProvider: guard File.list() results with requireNonNull.
- OzoneManagerServiceProviderImpl: make omSnapshotDBParentDir a local variable.
@devmadhuu devmadhuu marked this pull request as ready for review July 9, 2026 11:53
@adoroszlai adoroszlai changed the title HDDS-15766. Recon - Make Recon OM DB large tar ball transfer reliable. HDDS-15766. Make Recon OM DB large tar ball transfer reliable. Jul 9, 2026
@adoroszlai adoroszlai changed the title HDDS-15766. Make Recon OM DB large tar ball transfer reliable. HDDS-15766. Make Recon OM DB large tarball transfer reliable Jul 9, 2026
@devmadhuu devmadhuu marked this pull request as draft July 10, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant