Problem
Deploying the Keycloak theme to the staging server was fully manual: build a jar locally, scp it to the box, drop it in ~/keycloak/, and rebuild the Docker image by hand.
On 2026-08-27 that process was found to have installed a seven-month-stale theme jar — a105d96a…, built 2025-07-14 — copied from a retired server, while production was running 88df0d80… built 2026-02-21.
Nothing logged an error. The theme name matched, the jar was in providers/, and Keycloak started cleanly. The only symptom was the login page looking wrong, and diagnosing it took comparing sha256 sums across three servers.
What was done
CI now builds the image and the server only pulls it:
push to dev
-> docker build (node+maven builds the jar, bakes it into keycloak, runs kc.sh build)
-> push to ghcr.io/civicdatalab/dataspacekeycloaktheme
-> ssh staging -> docker compose pull && up -d
-> health gate -> auto-rollback to the previous image on failure
Dockerfile — multi-stage build. The jar is copied by name; Keycloakify emits one jar per Keycloak version range, and a glob would install several themes and let Keycloak pick one..github/workflows/deploy-keycloak-staging.yml — digest-pinned image, health gate, automatic rollback to the previously-running image. Mirrors deploy-backend.yml in DataSpaceBackend..dockerignore.
Every image carries the source commit and the theme jar's sha256 as Docker labels, so "which theme is this box running, and from which commit" is now answered by one docker inspect instead of a cross-server hash hunt.
Verification
- Deployed to staging from
dev and confirmed the running container's commit label matches dev HEAD.
- Rollback proven, not assumed: a deliberately broken build was pushed, the health gate failed it for 3 minutes, and staging was automatically restored to the previous image with health back at 200. The run was correctly marked failed.
~/keycloak/ on the server no longer contains a theme jar or a Dockerfile.
References
PR #6 · commits 55da1ed, 398b545, 4074eac, 63c0683
Problem
Deploying the Keycloak theme to the staging server was fully manual: build a jar locally,
scpit to the box, drop it in~/keycloak/, and rebuild the Docker image by hand.On 2026-08-27 that process was found to have installed a seven-month-stale theme jar —
a105d96a…, built 2025-07-14 — copied from a retired server, while production was running88df0d80…built 2026-02-21.Nothing logged an error. The theme name matched, the jar was in
providers/, and Keycloak started cleanly. The only symptom was the login page looking wrong, and diagnosing it took comparing sha256 sums across three servers.What was done
CI now builds the image and the server only pulls it:
Dockerfile— multi-stage build. The jar is copied by name; Keycloakify emits one jar per Keycloak version range, and a glob would install several themes and let Keycloak pick one..github/workflows/deploy-keycloak-staging.yml— digest-pinned image, health gate, automatic rollback to the previously-running image. Mirrorsdeploy-backend.ymlin DataSpaceBackend..dockerignore.Every image carries the source commit and the theme jar's sha256 as Docker labels, so "which theme is this box running, and from which commit" is now answered by one
docker inspectinstead of a cross-server hash hunt.Verification
devand confirmed the running container's commit label matchesdevHEAD.~/keycloak/on the server no longer contains a theme jar or a Dockerfile.References
PR #6 · commits
55da1ed,398b545,4074eac,63c0683