Queue image rebuilds behind in-flight builds instead of dropping them - #326
Open
chruffins wants to merge 2 commits into
Open
Queue image rebuilds behind in-flight builds instead of dropping them#326chruffins wants to merge 2 commits into
chruffins wants to merge 2 commits into
Conversation
Enqueue returned "already building" for any digest still marked active, but a build signals readiness before it releases its queue slot. A delete + recreate landing in that window was dropped, and when the in-flight build completed nothing started the queued request, leaving the recreated image pending forever. Queue the fresh build behind the in-flight one so MarkComplete starts it when the slot frees. GetPosition now reports the pending position when an image is active and queued at once.
The ready signal fires before the tag symlink exists, so a one-shot GetImage by tag races the build tail and can return not-found (seen on darwin CI).
chruffins
marked this pull request as ready for review
July 30, 2026 15:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a race in
lib/imageswhere an image deleted and re-created in the window between a build's ready signal and its queue-slot release got stuck inpendingforever — the rebuild was silently swallowed and readiness waiters hung until timeout.Root cause
buildImagecallsnotifyReadybefore it returns, butBuildQueue.MarkCompleteonly runs in the queue wrapper'sdeferafterbuildImageexits. A caller that unblocks on the ready event, deletes the image, and callsCreateImageagain inside that window hitsEnqueuewhile the digest is still inactive— which returned "already building" without enqueueing anything. When the in-flight build then completed, the slot freed with nothing queued, so the freshpendingmetadata never built.The same window exists on the failure path: a retry of a failed build requested between the failed notification and
MarkCompletewas dropped the same way.Fix
When the digest is still marked active,
Enqueuenow queues the fresh build behind the in-flight one instead of dropping it;MarkCompletestarts it when the slot frees. This is safe for genuine duplicate callers because the manager dedups on image metadata before ever reachingEnqueue— anEnqueuethat finds the digest active always represents a new build request (delete + recreate, or failed-build retry).GetPositionnow reports the pending position when an image is active and queued at once.Tests
TestEnqueueWhileActiveQueuesBehindInFlightBuild— deterministic queue-level repro: a request landing while the build holds its slot must queue, dedup, and run after completion.TestEnqueueStartsImmediatelyAfterSlotFrees— normal path unchanged once the slot frees.TestDeleteAndRecreateDuringBuildTail— end-to-endImportLocalImage→ ready → slot held →DeleteImage+ recreate → image becomes ready again. Network-free (synthetic image seeded into the OCI cache, cpio exporter).Both regression tests fail on the old code and pass with the fix. Full
lib/imagessuite passes with-race.