Skip to content

ci(eng-prod): mirror the sd shard redis images to ghcr [CLK-1591713] - #7

Merged
xnap merged 1 commit into
mainfrom
xpan/clk-1591713-mirror-shard-redis-images
Sep 30, 2026
Merged

xnap merged 1 commit into
mainfrom
xpan/clk-1591713-mirror-shard-redis-images

Conversation

@xnap

@xnap xnap commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Since https://github.com/time-loop/sd/pull/43863 made every sd test shard start its own Redis, about 14% of shards can't pull the Redis images and fall back to the shared host Redis. In the first day after the merge, 461 of ~3,300 shards fell back, across 135 of 416 runs. One cause is Docker Hub's anonymous pull limit, toomanyrequests: You have reached your unauthenticated pull rate limit: the whole runner fleet pulls anonymously through one NAT IP. We have no Docker Hub credential, so logging in isn't an option.

This PR copies the three images to GHCR once, so runners pull from there instead. We do it by reusing the pattern this repo already uses for ci-runner, ci-storage and ci-scaler:

  • log into GHCR with the built-in GITHUB_TOKEN (packages: write), no PAT
  • push only on main; on PRs, check only
  • make each package public once, so sd pulls anonymously with no login step

No new secret, no AWS change, and no new way of working with images.

What it does

  • docker/mirror-images.txt lists each image as <path>:<tag>@<digest>: bitnamilegacy/redis-cluster:6.2.16, bitnamilegacy/redis:6.2.16, library/alpine:3.21.
  • .github/workflows/mirror-images.yml runs skopeo copy --all --preserve-digests from the pinned digest to ghcr.io/time-loop/mirror/<path>:<tag>, then confirms the mirror has the same digest. It runs when either file changes on main, and on demand. On PRs it only checks that each pinned digest still exists, and warns if a tag has moved upstream.
  • PUBLISH.md documents the mirror and lists the three new packages for the one-time switch to public.

Rollout

  1. Merge. The first run creates the three packages (private by default).
  2. Make them public, the same one-time step as the existing images.
  3. Check an anonymous pull: docker logout ghcr.io && docker pull ghcr.io/time-loop/mirror/bitnamilegacy/redis:6.2.16.
  4. Then sd sets SHARD_REDIS_IMAGE_PREFIX: ghcr.io/time-loop/mirror in a separate PR. It must come after step 2: pointing sd at private packages would make every shard fall back.

Test plan

  • This PR's Mirror Images run checks all three pinned digests without pushing.
  • After merge, the run on main logs mirrored … @sha256:… for each image, and the packages appear under the org.

@xnap
xnap requested a review from a team as a code owner September 30, 2026 19:17
@upwind-code-us

upwind-code-us Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Upwind Upwind Code Scan - ✅ Passed

0 newly introduced vulnerabilities · 0 resolved · 0 total in this PR vs main

View full analysis in Upwind Console

Scan completed in 12s

Scan history (1 scan)
Commit Scanned at New Resolved Net
36d003a < 2026-09-30 19:17 UTC 0 0 0

Last scanned: 36d003a · 2026-09-30 19:17 UTC

@upwind-code-us

upwind-code-us Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Upwind Upwind IaC Scan - ⚠️ Warn

8 misconfigurations detected

Total breakdown: 🔴 2 Critical | 🔶 3 High | 🟡 2 Medium | 🟢 3 Low

No default-branch baseline yet — showing all findings.


🔴 Critical · 2 findings
Rule Resource File
Secrets passed via build-args or envs or copied secret files — docker/ci-scaler/Dockerfile
Secrets passed via build-args or envs or copied secret files — docker/ci-runner/Dockerfile

🔶 High · 3 findings
Rule Resource File
Image user should not be 'root' — docker/ci-runner/Dockerfile
Image user should not be 'root' — docker/ci-scaler/Dockerfile
Image user should not be 'root' — docker/ci-storage/Dockerfile

🟡 Medium · 2 findings
Rule Resource File
'RUN cd ...' to change directory — docker/ci-runner/Dockerfile
Port 22 exposed — docker/ci-storage/Dockerfile

🟢 Low · 3 findings
Rule Resource File
No HEALTHCHECK defined — docker/ci-storage/Dockerfile
No HEALTHCHECK defined — docker/ci-runner/Dockerfile
No HEALTHCHECK defined — docker/ci-scaler/Dockerfile

View full analysis in Upwind Console →

Scan completed in 2s

Scan history (1 scan)
Commit Scanned at New Resolved Net
36d003a < 2026-09-30 19:17 UTC +10 0 +10

Last scanned: 36d003a · 2026-09-30 19:17 UTC

Comment thread .github/workflows/mirror-images.yml
@xnap
xnap merged commit 98ca154 into main Sep 30, 2026
12 of 13 checks passed
@xnap
xnap deleted the xpan/clk-1591713-mirror-shard-redis-images branch September 30, 2026 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants