Read a company's site once, however many people it names

[ralyodio]

The actual runaway behind #61 and #62.

A crawl of accenture.com returns 57 people. Each becomes a refresh_research card. Each card queued another crawl of the same page, which returned the same 57.

Production before it was stopped:

226 × https://accenture.com     ← identical, all pending
 81 × https://toptal.com
 35 × https://trailofbits.com
 33 × https://bairesdev.com

Why #62 did not catch it

The per-person cooldown was the right mechanism on the wrong axis. It bounds how often one person is re-researched; the amplification is per site. Every one of those 57 people was brand new, so none had any research history to be cooled down against.

The samples said so plainly — cooldowns flat at zero while pending crawls climbed:

sample   pendCrawl  research  approvals  cooldowns
1        300        246       356        0
2        426        129       467        0
...
12       424        240       467        0

The fix

enqueue has supported a dedupe key since migration 0007, and the index behind it covers only pending and running jobs, so a key frees itself the moment the job finishes — it suppresses duplicates, never future work.

POST /prospects/by-url has passed one from the start. Neither approval path ever did. That is why one route could not queue a host twice while another queued it 226 times.

Both approval paths now pass the same key that route uses — crawl:<host>, without www. — because two paths deduping under different keys do not deduplicate against each other, which is the entire point. Keyed on host rather than URL so example.com and https://www.example.com/ are the one crawl they actually are.

The cards still all clear. Emptying the queue was never the problem.

Verification

• bun test — 1369 pass, 0 fail across 90 files
• typecheck and format:check clean
• 2 new tests: eight people at one company produce one crawl; and the key frees itself once that crawl is done, so a later re-read still happens

Prod state while this lands

• auto_approve_internal paused on wsp_04wz5p9… — the loop is stopped
• 407 duplicate pending crawls purged (423 → 16)
• Re-enable after this deploys

🤖 Generated with Claude Code