Skip to content

Queue image rebuilds behind in-flight builds instead of dropping them - #326

Open
chruffins wants to merge 2 commits into
mainfrom
hypeship/image-queue-rebuild-race
Open

Queue image rebuilds behind in-flight builds instead of dropping them#326
chruffins wants to merge 2 commits into
mainfrom
hypeship/image-queue-rebuild-race

Conversation

@chruffins

Copy link
Copy Markdown

Summary

Fixes a race in lib/images where an image deleted and re-created in the window between a build's ready signal and its queue-slot release got stuck in pending forever — the rebuild was silently swallowed and readiness waiters hung until timeout.

Root cause

buildImage calls notifyReady before it returns, but BuildQueue.MarkComplete only runs in the queue wrapper's defer after buildImage exits. A caller that unblocks on the ready event, deletes the image, and calls CreateImage again inside that window hits Enqueue while the digest is still in active — which returned "already building" without enqueueing anything. When the in-flight build then completed, the slot freed with nothing queued, so the fresh pending metadata never built.

The same window exists on the failure path: a retry of a failed build requested between the failed notification and MarkComplete was dropped the same way.

Fix

When the digest is still marked active, Enqueue now queues the fresh build behind the in-flight one instead of dropping it; MarkComplete starts it when the slot frees. This is safe for genuine duplicate callers because the manager dedups on image metadata before ever reaching Enqueue — an Enqueue that finds the digest active always represents a new build request (delete + recreate, or failed-build retry). GetPosition now reports the pending position when an image is active and queued at once.

Tests

  • TestEnqueueWhileActiveQueuesBehindInFlightBuild — deterministic queue-level repro: a request landing while the build holds its slot must queue, dedup, and run after completion.
  • TestEnqueueStartsImmediatelyAfterSlotFrees — normal path unchanged once the slot frees.
  • TestDeleteAndRecreateDuringBuildTail — end-to-end ImportLocalImage → ready → slot held → DeleteImage + recreate → image becomes ready again. Network-free (synthetic image seeded into the OCI cache, cpio exporter).

Both regression tests fail on the old code and pass with the fix. Full lib/images suite passes with -race.

Enqueue returned "already building" for any digest still marked active,
but a build signals readiness before it releases its queue slot. A
delete + recreate landing in that window was dropped, and when the
in-flight build completed nothing started the queued request, leaving
the recreated image pending forever.

Queue the fresh build behind the in-flight one so MarkComplete starts
it when the slot frees. GetPosition now reports the pending position
when an image is active and queued at once.
The ready signal fires before the tag symlink exists, so a one-shot
GetImage by tag races the build tail and can return not-found (seen on
darwin CI).
@chruffins
chruffins marked this pull request as ready for review July 30, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant