fix(prepare): keep the concurrency window full and retry transient failures

[bryantbiggs]

Problem

Two defects, found from a v0.8.0 run whose prepare phase was unchanged from before #123 made it concurrent.

The concurrency window collapsed. resolve_all used an ordered buffered queue, which frees a slot only when the head yields. One mapping listing a deep repository pinned its slot, every mapping that finished behind it kept its own, and the window sat at 2 or 3 of 16 while the config waited:

done=74 total=95 in_flight=3  slowest=docker.io/postgres slowest_secs=98  elapsed_secs=255
done=75 total=95 in_flight=2  slowest=docker.io/postgres slowest_secs=358 elapsed_secs=515
done=84 total=95 in_flight=7  slowest=docker.io/rsyslog-fips                elapsed_secs=520

Nine mappings did not complete during that final tick. They had finished minutes earlier and were waiting to be released. 625 seconds to do 442 seconds of work.

Nothing in the prepare phase retried. The engine retries the work it drives through with_retry, but the prepare phase runs before the engine exists. A single 429 failed a whole mapping.

Why the benchmark missed it

Two blind spots, both needed:

1. slow_tag_registry(mappings, delay) gives every mapping the same delay. Under uniform latency an ordered queue and a window that refills on completion are indistinguishable, because the head is never the straggler.
2. resolve_all_never_exceeds_its_concurrency_limit asserts a ceiling. buffered reaches 16 in the first instant, so that assertion passes while the window collapses behind it.

Both are fixed. skewed_tag_registry places deep repositories deliberately, and resolve_all_keeps_its_window_full_while_mappings_wait asserts the floor: while any mapping is unstarted, the window must be full. It fails on the old code with Some((1, 8)), three slots idle and eight mappings blocked.

The fixture also warms the AIMD window first. Cold, the window opens at 1, so the first mapping holds the only permit and nothing else can issue regardless of the queue above it. That hid the bug on the first attempt at this test.

Changes

Mappings and both analyze walks use buffer_unordered, carrying each outcome's index and sorting once every result is in, so watch_log, the counters, and config-order bookkeeping are unchanged. Blob aggregation in analyze now runs in completion order, which the report does not need: every figure it prints is a count, a sum, or a sorted container.

The target-listing fan-out at synchronize.rs is deliberately left as buffered. Its concurrency equals its item count, so no queue can form.

Retry in the prepare phase, through one shape in a new ocync_distribution::retry:

• list_tags, including the body read, so a connection reset part way through a page is re-sent rather than failing the listing.
• token_exchange::exchange, both the /v2/ ping and the realm token request.
• acr::exchange_post.
• analyze's manifest walk, at its call site via ocync_sync::retry::with_retry, because manifest_pull is shared with the engine and a retry further down would multiply the two.

Three details are load-bearing:

• Backoff is jittered. Up to 16 mappings hit one registry at once, so an unjittered schedule sends every throttled retry back in lockstep and produces the next burst.
• The auth paths return the throttled response rather than raising it. An earlier revision raised a 429 as a typed error, which gave it a status_code() and let the engine's with_retry classify an auth failure as retryable, multiplying this retry by the engine's own and aiming four times the requests at a registry already asking for less.
• A tag listing draws from one budget for the whole walk. Four retries per page across up to MAX_TAG_PAGES pages is a bound only on paper.

is_transient_transport is now the single transient-transport predicate for the workspace, and ocync_sync::retry::should_retry_transport delegates to it. The previous two copies disagreed: one listed is_timeout() || is_connect(), which is_request() already covers on the async hyper path, and omitted is_body()/is_decode(), which it does not. with_retry moved from engine.rs to retry.rs beside RetryConfig and the should_retry* predicates, and is now pub so analyze can use it.

Measurements

Skewed benchmark, 95 mappings, one deep repository per window among shallow ones, same run both ways:

Concurrency	ordered buffered	buffer_unordered
1	6.92s	6.92s
8	4.97s	1.21s
16	4.84s	1.09s
32	2.49s	1.09s

The first column is the reported defect: serial to 16-way bought 1.4x.

Live, 95 mappings against public.ecr.aws, --dry-run:

	before	after
exit code	1	0
failed mappings	5	0
AIMD halvings	5	2
retries fired and recovered	n/a	2

Live, 21 mappings, before and after the concurrency change: byte-identical dry-run output, md5 451e274b429a43d41ea3270e5dbc968a both.

Registry behavior

Measured 2026-08-28, and recorded in docs/src/content/registries/ecr-public.md per the rule in CLAUDE.md: ECR Public draws repeated 429s on TagList while ocync paces it at 8 requests per second with credentials loaded, halving the AIMD window 11 to 6 to 3. AWS publishes no TPS quota for tag listing at all; the 8 is derived from the 10 TPS authenticated pull quota, and the read window groups tag listing in with pulls.

The value is documented as measured-insufficient rather than replaced with a guess. Lowering it wants its own measurement, and the retry makes the current pacing survivable in the meantime.

Not changed, deliberately

• The docker credential helper has no retry. Its timeout path is documented as a possible wait for interactive input, and build_clients is a serial loop, so retrying would turn a 30s hang into a 150s stall of the whole prepare phase. A helper that fails for real already falls back to anonymous.
• send_with_aimd and manifest_pull still do not retry. They are on the engine's path, which already does. get_429_retries_and_succeeds pins that layering.

Tests

15 new, 1488 passing, 0 failing.

• resolve_all_keeps_its_window_full_while_mappings_wait, plus the skewed_tag_registry fixture and prepare_phase_benchmark_skewed_resolution
• list_tags_retries_after_a_429, list_tags_gives_up_after_persistent_429s (pinning the exact attempt count, so an unbounded retry fails it)
• exchange_retries_a_throttled_token_endpoint, exchange_gives_up_on_a_persistently_throttled_endpoint
• retrying_gives_up_and_surfaces_the_last_error, retrying_does_not_re_run_a_permanent_failure, only_a_throttle_is_transient_among_statuses, backoff_grows_and_is_capped, backoff_is_decorrelated, a_refused_connection_is_transient, a_status_error_is_not_a_transport_failure

a_refused_connection_is_transient provokes a real error from a closed port rather than asserting against a hand-built one, because the predicate has to match the failure that actually shows up: the run that motivated the token-exchange fix died on "error sending request for url (.../token/...)".

Every new test was mutation-checked. Reverting buffer_unordered, zeroing the retry bound, and inverting the transport predicate each fail the test that covers them.

Gate: cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace, cargo deny check, all green.