Accelerate exhaustive Azure AI Search backups with adaptive parallel export

[paullizer]

Summary

Make Azure AI Search backup export exhaustive and fast through bounded concurrency, stable pagination, adaptive throttling, durable checkpoints, and per-document progress metadata.

User Value

Administrators need all Search documents backed up predictably at production scale. The current serial index iterator does not expose page-level progress, throughput, retries, or a defensible resume position when Azure AI Search throttles or a worker is interrupted.

Proposed Behavior

• Export supported indexes concurrently within a configurable bound while keeping each index's pagination deterministic and resumable.
• Define an exhaustive traversal strategy that remains correct beyond 100,000 documents and does not rely on deep skip paging. Pin a stable source cutoff/version strategy where the service permits it and document concurrent-write semantics.
• Use bounded pages/batches and durable checkpoints with source index, cursor/keyset position, counts, bytes, and latest-item backup state.
• Honor 429/503 service guidance, including Retry-After, with exponential backoff, jitter, retry budgets, and adaptive concurrency reduction/recovery.
• Report per-index and aggregate documents, bytes, pages, retries, throttle time, failed/skipped documents, elapsed time, documents/sec, and bytes/sec.
• Preserve index schema artifacts and validate that backed-up document fields are compatible with the captured schema needed by Add Data Management backup restore capability #1091 restore.
• Continue independent indexes/documents when safe, persist failures/skips, and fail only when exhaustive-manifest integrity cannot be established.

Acceptance Criteria

• All supported Search indexes can export concurrently within configured bounds while each index remains deterministic and resumable.
• The traversal strategy is tested above 100,000 logical results or with an equivalent boundary harness and cannot silently truncate documents.
• 429/503 responses honor service retry guidance and adjust pressure without infinite retry or busy looping.
• Job progress exposes current index/page, documents and bytes processed, retries/throttle duration, skip/failure counts, and throughput.
• Interrupted exports resume from verified checkpoints without duplicating or omitting documents in the manifest.
• Latest-only backup item state is recorded using the shared contract and does not require retaining per-item attempt history.
• Schema plus document artifacts are internally validated for restore, and unsafe partial indexes are clearly marked unavailable.
• Focused tests cover high-cardinality pagination, concurrent index export, throttling, checkpoint resume, skip behavior, schema compatibility, and secret-safe telemetry.

Notes

• The current backup path calls search('*') and streams each supported index serially into one JSONL artifact with no durable page checkpoint.
• Coordinate the keyset/cutoff design with the incremental Search work remaining in Make Data Management migrations resilient for large blob transfers with cancel, resume, and incremental sync #1043.
• Depends on Make Data Management backup jobs durable, cancellable, and overlap-safe #1092 for the shared durable backup lifecycle. Supports restore capability in Add Data Management backup restore capability #1091.

[paullizer]

Implemented in #1115, which targets Development and includes the exhaustive adaptive Azure AI Search backup exporter and focused regression coverage.