fix(mc): fall back to the archived minio/mc GitHub release assets and verify every download against a pinned sha256 (dl.min.io answers 410) - #3
Open
AminDhouib wants to merge 3 commits into
Conversation
… verify every download against a pinned sha256 (dl.min.io answers 410) dl.min.io has returned "410 Gone -- the MinIO Client project is archived" for every mc release since 2026-09-11/12, and the org mirror at storage.devino.ca was never populated (404), so both existing sources are dead and every consumer job fails at "Install mc" (postify run 34680054913, lighthouse job). Add the release assets of the archived github.com/minio/mc repo as a third source after the two existing ones, and pin the per-platform sha256 of the binaries so any source can be checked before install; a mismatch is recorded and skipped rather than installed. The failure now names all three URLs with their own errors instead of only the last. Verified live on all five platforms.
…ge.devino.ca is misrouted The first commit said the mirror was never populated because it 404s. That is wrong: this repo's own CI installed mc from storage.devino.ca/tools/mc/<version>/<key>/ at 2026-09-12T00:37Z (run 34662229884), and uploaded 130 MiB through the same host in that run. storage.devino.ca now answers 404 on every path including /minio/health/live, and the response carries a Next.js content-security-policy, so the hostname is routed to the wrong backend rather than MinIO being empty or down. That also explains the "MinIO STS AssumeRoleWithWebIdentity failed: HTTP 404" this branch hits in CI after mc installs successfully. The code change is unaffected; only the comment, the README and the follow-up are re-pointed at restoring the route.
…estart policy Correcting the second commit. The 404 from storage.devino.ca was a symptom, not the cause. The Docker daemon on the storage host restarted around 05:00Z on 2026-09-12 and the shared MinIO compose has no `restart:` policy, so its container stayed exited (255) while every other app on the host came back. With no container there was no Traefik router for the hostname, so requests fell through to another app and 404d every MinIO path. The same failure appears twice in that service's deploy history as "Redeploy shared MinIO - was returning 404". Starting the container restored it at 08:10Z. The TODO is now the actual fix: add `restart: unless-stopped` to the shared-minio compose. Re-verified with the mirror back up -- all five mirrored binaries are byte-identical to the pinned digests, so the checksum gate does not disturb the primary source.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
Since 2026-09-12 every job in the org that uses
DevinoSolutions/artifact/upload@v1failed in the "Install mc" step:
(postify run
34680054913,lighthousejob, 07:21:21Z.)That error names the wrong culprit.
ensure_mc()tried two sources andreported only the last exception, so the 410 from the secondary hid what the
primary said. Both were failing that morning, for unrelated reasons:
storage.devino.ca/tools/mc/<MC_VERSION>/<key>/<binname>(org mirror)404dl.min.io/client/mc/release/<key>/archive/mc.<MC_VERSION>410 GoneThe mirror was down, not empty
The Docker daemon on the storage host restarted around 05:00Z. The shared MinIO
compose has no
restart:policy, so its container stayed exited (255) whileevery other app on that host came back. With no container there was no Traefik
router for
storage.devino.ca, so requests fell through to another application,which 404s every MinIO path — including
/minio/health/live— and answers withits own headers. The same failure appears twice in that service's deploy history
as "Redeploy shared MinIO - was returning 404".
The mirror had been serving normally seven hours earlier: this repository's own
CI installed
mcfrom it at 2026-09-12T00:37Z (run34662229884, green) andpushed 130 MiB through the same host in that run. Starting the container restored
service at 08:10Z;
/minio/health/livenow returns200withx-amzheaders andall five mirrored binaries are back.
So the mirror outage was recoverable and has been recovered. The 410 is not: that
source is gone for good, which left the action with no fallback at all the
moment the primary blinked. That is what this PR fixes.
What changes
MC_VERSION(
RELEASE.2025-08-13T08-35-41Z) still exist as release assets on the archivedgithub.com/minio/mcrepo, one per platform:https://github.com/minio/mc/releases/download/<MC_VERSION>/mc.<key>.<MC_VERSION>(
.exesuffix on Windows). It is appended after the two existing URLs, sothe mirror stays primary and nothing changes on a normal day.
MC_SHA256table next toMC_VERSION, one digest perplatform, taken from the
.sha256sumasset published beside each binary.Every payload, from any of the three sources, is hashed before
chmodandos.replace. A mismatch is never installed: it is recorded as that source'serror and the loop moves to the next URL. If no digest is pinned for a
platform the action fails rather than installing something unverified. This is
what makes a third-party archived host acceptable as a fallback at all.
URL with its own reason, one per line, instead of just the last exception —
the reporting gap that sent everyone after dl.min.io this morning.
RUNNER_TOOL_CACHE/devino-mc/<version>/<key>/is still returned untouched.Verification
lib/main.pyparses, and the existing suite is green plus 12 new tests intests/test_ensure_mc.py:(34 before this branch, 46 after. Also green on Python 3.12.) The new tests patch
main.httpand touch no network: they cover the source order, the.exenamingon Windows, the cache short-circuit, a tampered payload never reaching disk, no
leftover
.tmpafter a failure, and the one-URL-per-line error.The pin was checked against both the fallback and the restored primary.
ensure_mc()was driven against the live sources once per platform, first whilethe mirror was down (falling through
404→410→ GitHub) and again after itcame back (served by the mirror on the first URL). Every binary from either
source hashes to the pinned value, so the checksum gate does not disturb the
primary path:
The installed Windows binary runs and self-reports the pinned build:
CI on this branch
Run
34682225523was started while the mirror was still down, and is the clearestevidence the fix works: all three
roundtripjobs got past the step that wasbreaking the fleet, on real runners, on three operating systems —
— and then failed one step later on the dead host, not on anything in this diff
(
MinIO STS AssumeRoleWithWebIdentity failed: HTTP 404: Not Found). That is thewhole point: with the old code those jobs never reached STS at all. See the latest
run for the end-to-end result now that storage is back.
Still standard library only; no new dependencies.
Owner follow-up
restart: unless-stoppedto the shared-minio compose. It is the onlyservice on that host without one, which is why a routine daemon restart took
it out while everything else recovered, and why this is the third occurrence.
A
# TODO:marking this sits at theurls = [...]list.mcis nolonger maintained: there will be no future
MC_VERSIONto pin, no securityfixes, and the GitHub assets are a third-party host that can be pruned at any
time. This action uses
mcfor exactly two verbs,cpandls --json. Bothare reachable with plain SigV4-signed HTTPS from the standard library, which
is already how the STS exchange is done in this file, so dropping the binary
is a contained change worth scheduling.
/minio/health/livedirectly. Themirror and the STS endpoint share one hostname, so whatever takes one down
takes the tool and the storage with it, and today nothing alerted.