Add a workflow to reset the deployed catalog to fixtures - #59
Merged
Merged
Conversation
Participants create collections as they work through the notebooks, and nothing removed them. The deployed catalog currently holds 31 collections, of which 2 are fixtures and 29 are leftovers. Until now the only options were loading more data or destroying the whole stack. Add a `Reset Workshop Data` workflow that deletes every STAC collection that is not a fixture, plus a DEPLOYMENT.md section covering it. The keep-list is derived rather than maintained by hand: scripts/fixture_collections.py discovers collection ids from data/*.json and data/*.ndjson, so adding a fixture file is enough. Only fixtures fetched from a remote STAC API need listing, in REMOTE_FIXTURES. Verified that it already picks up data/tenant-collections.ndjson from the stac-auth-proxy branch, so it will keep working once that merges. Guards, because this is irreversible: mode defaults to dry-run, and applying requires typing the project name. Rehearsed read-only against the deployed database, which caught three bugs worth recording: - psql needs -X. A developer's ~/.psqlrc with `\timing on` put "Time: 35.712 ms" lines into the id list, which would have been passed to DELETE. - The temp-table join failed on the second run with `relation "keep" already exists`. The database is reached through pgbouncer in transaction pooling mode, where temp tables leak between pooled backends. The subtraction is done client-side instead. - Those two together miscounted 32 strays against 31 total collections. After the fixes the rehearsal reports 31 collections, 29 to delete, and asserts both fixtures survive. Items are deleted explicitly before each collection rather than relying on a cascade, since pgstac.delete_collection is only a DELETE of the collection row, and the run reports any orphaned items afterwards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P1CCcrx5fDKuAGsNh8DUFG
The reset authenticates to the config Lambda with WORKSHOP_TOKEN. Rotating that variable without redeploying leaves the variable and the Lambda disagreeing, so the config endpoint answers 401 (confirmed against the deployed endpoint) and the run dies on a bare `curl -sf` exit code with nothing explaining why. Catch the failure and name the likely cause, and say in DEPLOYMENT.md that rotation means updating the variable and running CDK Deploy *before* resetting. The reset itself still has nothing to do with the token: it only deletes STAC collections. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P1CCcrx5fDKuAGsNh8DUFG
DEPLOYMENT.md advertised WORKSHOP_TOKEN as "optional, auto-generated if not provided". config.py does generate one, but the generated value never reaches anybody who needs it: the deploy's own Load Workshop Data step and the Reset Workshop Data workflow both authenticate with the variable, so leaving it unset means sending an empty bearer token to a config Lambda holding a generated one. It is also regenerated on every synth, so the token changes on each deploy. Mark it required and explain why. That failure was also invisible. The data-load step used `curl -s` without `-f`, so a 401 produced a null jq result and surfaced several lines later as a psql error, after the stack had already deployed. Use `curl -sf` and name the likely cause, matching the guard added to reset-data.yml. Verified against the deployed config endpoint: a wrong bearer token returns 401 and `curl -sf` exits non-zero, while the correct token exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P1CCcrx5fDKuAGsNh8DUFG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a way to reset the deployed catalog to just the fixture data. Until now the only options were loading more data or destroying the whole stack — there was nothing in between.
The drift is real. The deployed catalog currently holds 31 collections: 2 fixtures and 29 participant leftovers (
aged-glade-7956-sentinel-2-c1-l2a,ancient-mouse-8070-…, and so on).What's here
Reset Workshop Dataworkflow — deletes every STAC collection that is not a fixture. Defaults todry-run; applying additionally requires typing thePROJECTname. Needs no AWS credentials, since the config Lambda already vends database credentials to a bearer token.scripts/fixture_collections.py— derives the keep-list instead of maintaining it by hand. It discovers collection ids fromdata/*.jsonanddata/*.ndjson, so adding a fixture file is enough; only fixtures fetched from a remote STAC API need listing, inREMOTE_FIXTURES. I checked it already picks updata/tenant-collections.ndjsonfrom thefeat/stac-auth-proxy-lambdabranch, so it keeps working once #49 merges.DEPLOYMENT.md — a
Resetting the Workshop Databasesection.Nothing reloads fixtures: they're upserted on every CDK Deploy and preserved by the reset, so the two workflows compose — reset returns you to fixtures-only, deploy restores anything missing.
Rehearsed against the live database
I ran the dry-run path read-only against the deployed database rather than trusting it by inspection. That caught three bugs, all now fixed:
psqlneeded-X. A developer's~/.psqlrcwith\timing oninjectedTime: 35.712 mslines into the id list — which would have been fed straight toDELETE. It wouldn't reproduce on a CI runner, which is exactly what makes it worth pinning down.relation "keep" already exists. The database sits behind pgbouncer in transaction pooling mode, whereTEMPtables leak between pooled backends. The subtraction is done client-side now.After the fixes: 31 collections, 29 to delete, and the rehearsal asserts both fixtures survive.
Deliberate choices
Items are deleted explicitly before each collection rather than relying on a cascade —
pgstac.delete_collectionin 0.9.8 is onlyDELETE FROM collections WHERE id = _id, and I could not confirm from the pgstac SQL that item partitions cascade with it. The run reports any orphaned items afterwards and warns if the count is non-zero.The reset leaves the
features.ecoregionstable alone; it's a fixture the deploy rebuilds.One sharp edge, documented
The keep-list comes from the checked-out
data/directory, so the workflow must be run with the samerefyou deployed. Running it frommaintoday would deleteprivate-alice-demoandprivate-bob-demo, since those fixture files only exist on the #49 branch. That's called out in a warning callout in DEPLOYMENT.md.Not done
The
applypath has not been executed — only the read-only dry-run. The first real run should be a dry-run from the Actions tab.🤖 Generated with Claude Code
https://claude.ai/code/session_01P1CCcrx5fDKuAGsNh8DUFG