Skip to content

Stop re-researching the same person every tick - #62

Merged
ralyodio merged 1 commit into
mainfrom
fix/research-cooldown
Aug 19, 2026
Merged

Stop re-researching the same person every tick#62
ralyodio merged 1 commit into
mainfrom
fix/research-cooldown

Conversation

@ralyodio

Copy link
Copy Markdown
Contributor

#61 had a defect I only saw in production. Automating the approval turned a visible backlog into an invisible loop.

A refresh_research card is proposed because we have nothing to say about someone. Approving it re-reads their company's site. If that still yields nothing, the next pass proposes the same card again.

While a human had to click, those cards piled up — production held 179 — and at least the waste was in plain sight. Closing the loop turned it into crawl traffic nobody watches.

Measured in prod, minutes after the deploy

auto-approvals 211
max cards for one person 3
crawl jobs queued 195

The backlog dropped 152 → 7, then climbed 7 → 15 → 25 → 35 on successive samples. That is the loop.

The fix

A person researched inside the last 24 hours has their card closed rather than approved, and no crawl is spent.

skipped, not approved — approving would claim we went and looked, and the status already means "over without having been acted on".

Counted from actions rather than crawl jobs, which matters twice: a card a human approved an hour ago suppresses the automatic re-run too, and a person whose company has no domain — whose card therefore never produced a job — is still covered.

24 hours because that is the shortest interval over which a company website plausibly changes. Shorter re-reads the same bytes; much longer delays picking up a real change.

Two existing tests changed, and both were wrong rather than the code

  • one inserted an actions row without the NOT NULL recommendation_id
  • one asserted five cards for a single person all approve — now one approval and four cooldowns, which is the correct new behaviour

Verification

  • bun test1367 pass, 0 fail across 90 files
  • typecheck and format:check clean
  • 4 new tests, including the loop itself: approve → card returns → second pass spends no crawl and closes it

🤖 Generated with Claude Code

Automating the approval in #61 turned a visible backlog into an invisible
loop. A `refresh_research` card is proposed *because* we have nothing to
say about someone; approving it re-reads their company's site; if that
still yields nothing, the next pass proposes the same card again.

While a human had to click, those cards piled up — production held 179 —
and at least the waste was in plain sight. Closing the loop turned it into
crawl traffic nobody watches. Measured in prod within minutes of the
deploy: 211 auto-approvals, one person already holding three cards, and
195 crawl jobs queued.

A person researched inside the last 24 hours now has their card closed
rather than approved, and no crawl is spent. `skipped` rather than
`approved`, because approving would claim we went and looked; the status
already means "over without having been acted on".

Counted from `actions` rather than from crawl jobs, so a card a human
approved an hour ago suppresses the automatic re-run too, and so a person
whose company has no domain — and whose card therefore never produced a
job — is still covered.

Twenty-four hours because that is the shortest interval over which a
company website plausibly changes. Shorter re-reads the same bytes; much
longer delays picking up a real change.

Two existing tests had to change and both were wrong rather than the code:
one inserted an `actions` row without the NOT NULL `recommendation_id`,
and one asserted five cards for a single person all approve — which is now
one approval and four cooldowns, correctly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ralyodio
ralyodio merged commit 42b3be5 into main Aug 19, 2026
4 checks passed
@ralyodio
ralyodio deleted the fix/research-cooldown branch August 19, 2026 15:47
ralyodio added a commit that referenced this pull request Aug 19, 2026
The runaway behind #61 and #62. A crawl of accenture.com returns 57
people; each becomes a `refresh_research` card; each card queued another
crawl of the same page, which returned the same 57. Production reached
226 pending crawls of that one URL and 81 of toptal.com before it was
stopped.

The per-person cooldown in #62 could not catch this, and the samples said
so plainly: cooldowns stayed at zero while pending crawls climbed 227 to
426. Every one of those 57 people was new, so none of them had any
research history to be cooled down against. The cooldown was the right
mechanism for the wrong axis — it bounds how often one person is
re-researched, and the amplification is per *site*.

`enqueue` has supported a dedupe key since 0007, and the index behind it
deliberately covers only pending and running jobs so a key frees itself
when the job finishes. `POST /prospects/by-url` has passed one from the
start. Neither approval path ever did, which is why one route could not
queue a host twice and another could queue it 226 times.

Both approval paths now pass the same key that route uses —
`crawl:<host>` without `www.` — because two paths deduping under
different keys do not deduplicate against each other, which is the whole
point. Keyed on the host rather than the URL, so `example.com` and
`https://www.example.com/` are the one crawl they actually are.

The cards still all clear; only the duplicate crawls collapse. Emptying
the queue was never the problem.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant