Skip to content

fix(mc): fall back to the archived minio/mc GitHub release assets and verify every download against a pinned sha256 (dl.min.io answers 410) - #3

Open
AminDhouib wants to merge 3 commits into
mainfrom
fix/mc-from-github-releases-with-sha256
Open

fix(mc): fall back to the archived minio/mc GitHub release assets and verify every download against a pinned sha256 (dl.min.io answers 410)#3
AminDhouib wants to merge 3 commits into
mainfrom
fix/mc-from-github-releases-with-sha256

Conversation

@AminDhouib

@AminDhouib AminDhouib commented Sep 12, 2026

Copy link
Copy Markdown
Member

What broke

Since 2026-09-12 every job in the org that uses DevinoSolutions/artifact/upload@v1
failed in the "Install mc" step:

##[error]Could not download the MinIO client: HTTP 410: 410 Gone
The open-source MinIO Server, MinIO Client (mc) and MinIO KES projects are archived
and no longer maintained. ... These files are no longer served from this site.

(postify run 34680054913, lighthouse job, 07:21:21Z.)

That error names the wrong culprit. ensure_mc() tried two sources and
reported only the last exception, so the 410 from the secondary hid what the
primary said. Both were failing that morning, for unrelated reasons:

source status during the incident why
storage.devino.ca/tools/mc/<MC_VERSION>/<key>/<binname> (org mirror) 404 the MinIO container was down, see below
dl.min.io/client/mc/release/<key>/archive/mc.<MC_VERSION> 410 Gone MinIO archived the client, permanently

The mirror was down, not empty

The Docker daemon on the storage host restarted around 05:00Z. The shared MinIO
compose has no restart: policy, so its container stayed exited (255) while
every other app on that host came back. With no container there was no Traefik
router for storage.devino.ca, so requests fell through to another application,
which 404s every MinIO path — including /minio/health/live — and answers with
its own headers. The same failure appears twice in that service's deploy history
as "Redeploy shared MinIO - was returning 404".

The mirror had been serving normally seven hours earlier: this repository's own
CI installed mc from it at 2026-09-12T00:37Z (run 34662229884, green) and
pushed 130 MiB through the same host in that run. Starting the container restored
service at 08:10Z; /minio/health/live now returns 200 with x-amz headers and
all five mirrored binaries are back.

So the mirror outage was recoverable and has been recovered. The 410 is not: that
source is gone for good, which left the action with no fallback at all the
moment the primary blinked. That is what this PR fixes.

What changes

  1. A third source. The binaries for the pinned MC_VERSION
    (RELEASE.2025-08-13T08-35-41Z) still exist as release assets on the archived
    github.com/minio/mc repo, one per platform:
    https://github.com/minio/mc/releases/download/<MC_VERSION>/mc.<key>.<MC_VERSION>
    (.exe suffix on Windows). It is appended after the two existing URLs, so
    the mirror stays primary and nothing changes on a normal day.
  2. A checksum pin. New MC_SHA256 table next to MC_VERSION, one digest per
    platform, taken from the .sha256sum asset published beside each binary.
    Every payload, from any of the three sources, is hashed before chmod and
    os.replace. A mismatch is never installed: it is recorded as that source's
    error and the loop moves to the next URL. If no digest is pinned for a
    platform the action fails rather than installing something unverified. This is
    what makes a third-party archived host acceptable as a fallback at all.
  3. A readable failure. After all three sources fail, the error names every
    URL with its own reason, one per line, instead of just the last exception —
    the reporting gap that sent everyone after dl.min.io this morning.
  4. The tool-cache short-circuit is unchanged: a binary already in
    RUNNER_TOOL_CACHE/devino-mc/<version>/<key>/ is still returned untouched.
  5. README documents the third source, the checksum pin, and the 410.

Verification

lib/main.py parses, and the existing suite is green plus 12 new tests in
tests/test_ensure_mc.py:

$ python -m unittest discover -s tests -p "test_*.py"
Ran 46 tests in 0.026s

OK

(34 before this branch, 46 after. Also green on Python 3.12.) The new tests patch
main.http and touch no network: they cover the source order, the .exe naming
on Windows, the cache short-circuit, a tampered payload never reaching disk, no
leftover .tmp after a failure, and the one-URL-per-line error.

The pin was checked against both the fallback and the restored primary.
ensure_mc() was driven against the live sources once per platform, first while
the mirror was down (falling through 404410 → GitHub) and again after it
came back (served by the mirror on the first URL). Every binary from either
source hashes to the pinned value, so the checksum gate does not disturb the
primary path:

platform bytes GitHub asset mirror object
linux-amd64 30535864 matches pin matches pin
linux-arm64 28704952 matches pin matches pin
darwin-amd64 31376080 matches pin matches pin
darwin-arm64 29692306 matches pin matches pin
windows-amd64 31468032 matches pin matches pin

The installed Windows binary runs and self-reports the pinned build:

mc version RELEASE.2025-08-13T08-35-41Z (commit-id=7394ce0dd2a80935aded936b09fa12cbb3cb8096)
Runtime: go1.24.6 windows/amd64

CI on this branch

Run 34682225523 was started while the mirror was still down, and is the clearest
evidence the fix works: all three roundtrip jobs got past the step that was
breaking the fleet, on real runners, on three operating systems —

Installed mc ... from https://github.com/minio/mc/releases/download/.../mc.linux-amd64....    (sha256 01f866e9... verified)
Installed mc ... from https://github.com/minio/mc/releases/download/.../mc.darwin-arm64....   (sha256 a877fd0c... verified)
Installed mc ... from https://github.com/minio/mc/releases/download/.../mc.windows-amd64....exe (sha256 c8db13eb... verified)

— and then failed one step later on the dead host, not on anything in this diff
(MinIO STS AssumeRoleWithWebIdentity failed: HTTP 404: Not Found). That is the
whole point: with the old code those jobs never reached STS at all. See the latest
run for the end-to-end result now that storage is back.

Still standard library only; no new dependencies.

Owner follow-up

  1. Add restart: unless-stopped to the shared-minio compose. It is the only
    service on that host without one, which is why a routine daemon restart took
    it out while everything else recovered, and why this is the third occurrence.
    A # TODO: marking this sits at the urls = [...] list.
  2. Decide whether to keep depending on an archived client at all. mc is no
    longer maintained: there will be no future MC_VERSION to pin, no security
    fixes, and the GitHub assets are a third-party host that can be pruned at any
    time. This action uses mc for exactly two verbs, cp and ls --json. Both
    are reachable with plain SigV4-signed HTTPS from the standard library, which
    is already how the STS exchange is done in this file, so dropping the binary
    is a contained change worth scheduling.
  3. Consider monitoring the mirror objects or /minio/health/live directly. The
    mirror and the STS endpoint share one hostname, so whatever takes one down
    takes the tool and the storage with it, and today nothing alerted.

Note on this branch's history: the first commit said the mirror had "never been
populated" and the second said the host was "misrouted". Both were wrong, and
both are corrected in the third commit. The container-with-no-restart-policy
account above is the accurate one.

… verify every download against a pinned sha256 (dl.min.io answers 410)

dl.min.io has returned "410 Gone -- the MinIO Client project is archived" for
every mc release since 2026-09-11/12, and the org mirror at storage.devino.ca
was never populated (404), so both existing sources are dead and every consumer
job fails at "Install mc" (postify run 34680054913, lighthouse job).

Add the release assets of the archived github.com/minio/mc repo as a third
source after the two existing ones, and pin the per-platform sha256 of the
binaries so any source can be checked before install; a mismatch is recorded and
skipped rather than installed. The failure now names all three URLs with their
own errors instead of only the last. Verified live on all five platforms.
…ge.devino.ca is misrouted

The first commit said the mirror was never populated because it 404s. That is
wrong: this repo's own CI installed mc from
storage.devino.ca/tools/mc/<version>/<key>/ at 2026-09-12T00:37Z (run
34662229884), and uploaded 130 MiB through the same host in that run.

storage.devino.ca now answers 404 on every path including /minio/health/live,
and the response carries a Next.js content-security-policy, so the hostname is
routed to the wrong backend rather than MinIO being empty or down. That also
explains the "MinIO STS AssumeRoleWithWebIdentity failed: HTTP 404" this branch
hits in CI after mc installs successfully.

The code change is unaffected; only the comment, the README and the follow-up
are re-pointed at restoring the route.
…estart policy

Correcting the second commit. The 404 from storage.devino.ca was a symptom, not
the cause. The Docker daemon on the storage host restarted around 05:00Z on
2026-09-12 and the shared MinIO compose has no `restart:` policy, so its
container stayed exited (255) while every other app on the host came back. With
no container there was no Traefik router for the hostname, so requests fell
through to another app and 404d every MinIO path. The same failure appears twice
in that service's deploy history as "Redeploy shared MinIO - was returning 404".
Starting the container restored it at 08:10Z.

The TODO is now the actual fix: add `restart: unless-stopped` to the
shared-minio compose. Re-verified with the mirror back up -- all five mirrored
binaries are byte-identical to the pinned digests, so the checksum gate does not
disturb the primary source.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant