Stop re-researching the same person every tick - #62
Merged
Conversation
Automating the approval in #61 turned a visible backlog into an invisible loop. A `refresh_research` card is proposed *because* we have nothing to say about someone; approving it re-reads their company's site; if that still yields nothing, the next pass proposes the same card again. While a human had to click, those cards piled up — production held 179 — and at least the waste was in plain sight. Closing the loop turned it into crawl traffic nobody watches. Measured in prod within minutes of the deploy: 211 auto-approvals, one person already holding three cards, and 195 crawl jobs queued. A person researched inside the last 24 hours now has their card closed rather than approved, and no crawl is spent. `skipped` rather than `approved`, because approving would claim we went and looked; the status already means "over without having been acted on". Counted from `actions` rather than from crawl jobs, so a card a human approved an hour ago suppresses the automatic re-run too, and so a person whose company has no domain — and whose card therefore never produced a job — is still covered. Twenty-four hours because that is the shortest interval over which a company website plausibly changes. Shorter re-reads the same bytes; much longer delays picking up a real change. Two existing tests had to change and both were wrong rather than the code: one inserted an `actions` row without the NOT NULL `recommendation_id`, and one asserted five cards for a single person all approve — which is now one approval and four cooldowns, correctly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ralyodio
added a commit
that referenced
this pull request
Aug 19, 2026
The runaway behind #61 and #62. A crawl of accenture.com returns 57 people; each becomes a `refresh_research` card; each card queued another crawl of the same page, which returned the same 57. Production reached 226 pending crawls of that one URL and 81 of toptal.com before it was stopped. The per-person cooldown in #62 could not catch this, and the samples said so plainly: cooldowns stayed at zero while pending crawls climbed 227 to 426. Every one of those 57 people was new, so none of them had any research history to be cooled down against. The cooldown was the right mechanism for the wrong axis — it bounds how often one person is re-researched, and the amplification is per *site*. `enqueue` has supported a dedupe key since 0007, and the index behind it deliberately covers only pending and running jobs so a key frees itself when the job finishes. `POST /prospects/by-url` has passed one from the start. Neither approval path ever did, which is why one route could not queue a host twice and another could queue it 226 times. Both approval paths now pass the same key that route uses — `crawl:<host>` without `www.` — because two paths deduping under different keys do not deduplicate against each other, which is the whole point. Keyed on the host rather than the URL, so `example.com` and `https://www.example.com/` are the one crawl they actually are. The cards still all clear; only the duplicate crawls collapse. Emptying the queue was never the problem. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#61 had a defect I only saw in production. Automating the approval turned a visible backlog into an invisible loop.
A
refresh_researchcard is proposed because we have nothing to say about someone. Approving it re-reads their company's site. If that still yields nothing, the next pass proposes the same card again.While a human had to click, those cards piled up — production held 179 — and at least the waste was in plain sight. Closing the loop turned it into crawl traffic nobody watches.
Measured in prod, minutes after the deploy
The backlog dropped 152 → 7, then climbed 7 → 15 → 25 → 35 on successive samples. That is the loop.
The fix
A person researched inside the last 24 hours has their card closed rather than approved, and no crawl is spent.
skipped, notapproved— approving would claim we went and looked, and the status already means "over without having been acted on".Counted from
actionsrather than crawl jobs, which matters twice: a card a human approved an hour ago suppresses the automatic re-run too, and a person whose company has no domain — whose card therefore never produced a job — is still covered.24 hours because that is the shortest interval over which a company website plausibly changes. Shorter re-reads the same bytes; much longer delays picking up a real change.
Two existing tests changed, and both were wrong rather than the code
actionsrow without the NOT NULLrecommendation_idVerification
bun test— 1367 pass, 0 fail across 90 filestypecheckandformat:checkclean🤖 Generated with Claude Code