Repository navigation
feat(npmjs): keep known versions in a DynamoDB table - #1941
Merged
Merged
Conversation
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF Scorecard
Scanned Files
|
cdklabs-automation
enabled auto-merge
October 6, 2026 13:27
mrgrain
disabled auto-merge
October 6, 2026 13:38
mrgrain
force-pushed
the
mrgrain/feat/npmjs/known-versions-table
branch
from
October 6, 2026 13:48
8addb71 to
79af0f3
Compare
rix0rrr
reviewed
Oct 7, 2026
| billing: this.props.knownVersionsTableBilling ?? Billing.onDemand(), | ||
| encryption: TableEncryptionV2.dynamoOwnedKey(), | ||
| pointInTimeRecoverySpecification: { pointInTimeRecoveryEnabled: true }, | ||
| removalPolicy: RemovalPolicy.RETAIN, |
Collaborator
There was a problem hiding this comment.
But a new deployment is going to recreate a fresh table under a different name, no? Do we mean to give this table a fixed name as well?
Contributor
Author
There was a problem hiding this comment.
I have no strong feelings. In practice this only happens with migrations and you'll need a plan to sort it out either way:
- both random or fixed name can be solved with imports
- a random name allows to create a new table and than copy the data somehow
- a fixed name would prevent replacement, which might be desired
Comment on lines
+227
to
+228
| partitionKey: { | ||
| name: KnownVersionsAttribute.NAME, |
Collaborator
There was a problem hiding this comment.
We are probably going to want another filter in the future:
fresh
┌───────────┐ sequence ┌─────────┐ ┌───────────┐ fresh
feed │ Feed │ numbers │ │ │ Packument │ versions Doc
────────▶│ Follower │───────────▶│ Queue │────────▶│ Retriever │───────────▶ processing
│ │ │ │ │ │ ...
└───────────┘ └─────────┘ └───────────┘
▲ ▲
│
│ │ RIGHT NOW:
│ filter packument
│ └────── versions against
seen versions here
│
FUTURE:
└ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ filter sequence
numbers here
When that happens, we might as well use the same DynamoDB table for that scratch space, by leveraging a single table design.
What that effectively means is:
- Mix the entity type into the primary key. So instead of
"aws-cdk-lib","@cdklabs/construct-hub"we will store something like"version:aws-cdk-lib"and"version:@cdklabs/construct-hub"and also"seq: 134844620"in that table. - Let go of the semantic meanings of key fields. The
namefield is not necessarily a "name" anymore, it's just a "primary key". Theversionfield is not necessarily a "version" anymore, it's just a sort key.
The table looks like this:
| pk | sk | modified |
|---|---|---|
| version:aws-cdk-lib | 2.153.0 | 1791370179355 |
| version:aws-cdk-lib | 2.154.0 | 1791370178389 |
| version:@cdklabs/construct-hub | 1.3.0 | 1791370179383 |
| seq:134844620 | ||
| seq:134847364 | ||
| version:@cdklabs/construct-hub | 1.3.1 | 1791370203858 |
The table will just be "the" Npm follower deduplication table.
Alternative table schema
Or perhaps we will do the following schema for sequence numbers:
| pk | sk | modified |
|---|---|---|
| seq | 134844620 | |
| seq | 134847364 |
Which would have the advantage that we can use Query to retrieve a whole bunch of sequence number records at the same time; otherwise we would do BatchGetItem. Both are feasible, we need to figure a bit which one is better.
Contributor
Author
There was a problem hiding this comment.
I don't see any relationship between versions and sequence numbers, which is the typical case for a single table design (as opposed to your relational database normalized tables). So yes, I was planning on two tables here.
We could treat this as a generic work-log of sorts, but I am not sure what we are gaining from it.
mrgrain
force-pushed
the
mrgrain/feat/npmjs/known-versions-table
branch
2 times, most recently
from
October 7, 2026 13:10
378c7ca to
1ff5fc3
Compare
The follower kept the package versions it has processed in a single JSON file in S3. Every run loaded all of it and wrote all of it back, so only one worker could ever use it. Known versions now live in a DynamoDB table, keyed by package name and version. Writes are conditional, so the stored modified timestamp only moves forward. The table billing mode is configurable through the new knownVersionsTableBilling prop and defaults to on-demand. Existing deployments migrate automatically: when the migration marker object is missing, the first run copies the known versions file (or the legacy marker file) into the table and then writes the marker. The old file is left in place and no longer updated, so a rollback only re-stages versions modified since the migration.
mrgrain
force-pushed
the
mrgrain/feat/npmjs/known-versions-table
branch
from
October 7, 2026 13:19
eafb0bf to
bb4c939
Compare
rix0rrr
approved these changes
Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The npm follower records every package version it has processed, so it doesn't stage the same version twice. Until now this list lived in a single JSON file in the staging bucket, about 20 MB and 285k versions in prod. Every run loaded the whole file and wrote the whole file back. That only works only with exactly one worker: two runs at once would each overwrite the other's changes. This is the first step towards letting several workers share the work, which we need to really fix the problems with laggy packuments.
Known versions now live in a DynamoDB table, keyed by
package nameandversion,with themodifiedtimestamp as the value. The follower only queries a package when at least one of its versions is a construct library, since nothing else is ever recorded. Writes are conditional, so the stored timestamp only moves forward, no matter which (of future multiple) writers saves first. Known versions are saved before the follower state after every chunk, so a crash can at worst re-stage a version, never skip one. The table is kept on stack deletion and has point-in-time recovery, because losing it means re-processing every construct version ever published. Billing defaults to on-demand and can be changed with the newknownVersionsTableBillingprop.Migration
Existing deployments migrate on their own on the first run after the upgrade. If the marker object
couchdb-known-versions.migratedis missing, the follower copies the known versions file (or, for older deployments, the legacy marker file) into the table, and writes the marker last. If the copy fails part-way, the next run simply does it again; nothing else happens in a run until the migration is done. A new deployment has no file to copy, so it writes the marker right away and fills the table during the normal backfill.The old file is left in place and is no longer written. We keep it to mitigate the risk of a rollback. Deleting it would make the rolled-back code start from nothing and re-stage every version of every construct package that changes.
Keeping it means the old code just reads a slightly stale file and only re-stages the versions modified since the migration. The file can be deleted by hand once we are happy with the table.
Review guide
The main new logic is in
known-versions.lambda-shared.ts(the table access) andknown-versions-migration.lambda-shared.ts.staging-bucket.lambda-shared.tsonly holds two S3 helpers moved out of the follower, so the follower and the migration can share them.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license