Import seventeen thousand contacts in minutes, not eighty-five - #64
Merged
Conversation
Measured from the three attempts sitting in production: 2,500 rows in 12 minutes, 4,500 in 18. About 200 rows a minute, so 17,000 rows is roughly 85 minutes and the browser tab gives up long before that. Two of the three imports are still `open` because nothing ever finished them. The cause was chattiness, not volume. Each row cost four sequential round trips to Turso — a dedupe SELECT, then three inserts — and at ~75ms each that is 300ms per row of almost pure waiting. The work was always trivial; the latency was the whole cost. A chunk is now one query and one batch. Existing mailboxes are fetched for the whole chunk in a single `IN` lookup, and every insert for the chunk is sent as one transactional batch. `db.batch` is all-or-nothing, so a unique violation would lose the other four hundred and ninety-nine rows — the old row-at-a-time path is kept as a fallback for exactly that, reached only when two imports genuinely race on one address. Finishing an import no longer enqueues anything. It used to write one `enrich_contact` job per person: seventeen thousand inserts inside one HTTP request, which is a second reason it never returned, and a queue that drains twenty-five a tick would then have taken eleven hours with every crawl waiting behind it. The set of people needing enrichment is derivable — everyone with an imported address and no `contact_enriched_at` — so the worker sweeps it. Nothing has to be written for that to begin, it resumes by itself after a crash, and it cannot drift from the people who actually exist. The same argument `metering.ts` makes for deriving usage rather than incrementing it. The sweep runs its lookups concurrently, ten at a time, because a Gravatar lookup is a network round trip and nothing else — doing them one after another wastes the entire interval. Two hundred a tick clears seventeen thousand in about ninety minutes. A timestamp rather than a flag, so "never looked up" and "looked up, found nothing" stay distinguishable. Most addresses have no published profile, and a boolean would retry every one of them forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measured from your three attempts, still in prod
~200 rows/minute → 17,000 rows ≈ 85 minutes. The browser gave up long before. 4,657 people did land.
Cause: chattiness, not volume
Each row cost four sequential Turso round trips — a dedupe
SELECT, then three inserts. At ~75ms each that is 300ms per row of almost pure waiting. The work was always trivial; the latency was the entire cost.A chunk is now one query and one batch: existing mailboxes fetched for the whole chunk in a single
INlookup, every insert sent as one transactional batch.db.batchis all-or-nothing, so a unique violation would lose the other 499 rows — the old row-at-a-time path is kept as a fallback, reached only when two imports genuinely race on one address.The second reason it never returned
finishenqueued oneenrich_contactjob per person — 17,000 inserts inside one HTTP request. And the queue drains 25/tick, so even if it had returned, enrichment would have taken ~11 hours with every crawl queued behind it.It now enqueues nothing. The set of people needing enrichment is derivable (imported address, no
contact_enriched_at), so the worker sweeps it: nothing has to be written for it to begin, it resumes by itself after a crash, and it cannot drift from the people who actually exist — the same argumentmetering.tsmakes for deriving usage rather than incrementing it.The sweep runs 10 lookups concurrently, because a Gravatar lookup is a network round trip and nothing else. 200/tick clears 17,000 in about 90 minutes, in the background, with no queue rows.
A timestamp rather than a flag, so "never looked up" and "looked up, found nothing" stay distinguishable — most addresses have no profile, and a boolean would retry them forever.
On BullMQ
You suggested BullMQ on Railway. I did not add it, and the numbers are why: the bottleneck was Turso round trips and Gravatar latency, and Redis fixes neither.
REDIS_URLis also in the vault but not set on the service, so it would be new infra for a problem that batching and concurrency already solve. Happy to add it if you want the throughput headroom — it is just not what was broken here.Verification
bun test— 1370 pass, 0 fail across 90 filestypecheck(root +apps/web) andformat:checkclean🤖 Generated with Claude Code