feat(npmjs): re-check laggy packuments through a queue

[mrgrain]

When the npm registry serves an older revision of a package than the changes feed announced, the follower processes what it got and checks the package again later. Until now it kept these "laggy packuments" in its state file and re-checked the 50 oldest on every run. Packages that never caught up held those slots for a day, so newer ones were not checked at all.

Laggy packuments now go to an SQS queue, and a new packument processor function works through it. It checks each package again with growing waits, from 5 minutes up to 11 hours. After about 27 hours it gives up, and the message moves to a dead-letter queue. The processor can run next to the follower now that known versions live in DynamoDB (#1941).

A 404 for a change entry used to be treated as a deleted package and ignored. Most of the time it is a new package that has not replicated to the registry yet, and prod sees a thousand or more of these a day. They now go through the same queue, but get their own metrics. Change entries marked as deleted are skipped completely.

Alarms

The dead-letter queue is graphed but doesn't get an alarm: with this many 404s, it seems useless. Instead we have low severity alarms aiming to detect npm outages and processing failures. Thresholds are based on a week of prod logs.

• First one fires when more than 1,500 change entries get a 404 within 3 hours (usually about 170).
• Second one fires when laggy packuments take longer than 6 hours to catch up (p90 over 3 hours).
• Third alarm fires when the processor itself fails.

What's Next

This change deliberately only addresses laggy and missing packuments. I want to make sure the processing works as intended and observe for a bit. Once we are happy, the plan is to change the follower to do nothing more than reading the change stream and putting entries on the queue for the packument processor to pick up. This final step will make the architecture more resilient to outages: We have a single follower function that does the absolute minimum of work (read change stream, add to queue). Everything else happens detached in a queue processor and work can be fanned out. In theory this should also speed up ingestion from zero.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None