Read the page an address points at, and know when it points at none - #65
Merged
Conversation
Gravatar answers for about one person in a hundred: 18 identities from the
first 1,800 of a real list. That is not enough to call an address enriched,
and the reason is that Gravatar is opt-in twice over — the person must have
an account and must have attached their other accounts to it.
An address usually points somewhere anyway. Measured against the 16,268-row
list now in production:
36% a publication handle — 5,934 substack.com addresses, where the
local part *is* the subdomain, so 0xshah@substack.com is the
account behind 0xshah.substack.com
31% a company domain — 4,235 distinct sites
33% nothing at all — gmail, yahoo, icloud and friends
Those first two are pages the crawler already knows how to read: it pulls a
name, a title and published social links out of a site, which is exactly
what is missing from an imported row.
This is derivation, not guessing, and the distinction is load-bearing. Every
rule turns an address into a page that address demonstrably belongs to. None
invents a handle on a network the person never mentioned — the codebase
already refuses to do that for email addresses it could at least verify by
sending to, and an unverifiable social identity is worse.
Knowing when to do nothing is the more valuable half. Sending every Gmail
address to google.com would be five thousand fetches of a page about nobody,
and it would look like the feature working. `qq.com` was missing from the
freemail list entirely — 127 addresses that would have been read as company
domains and sent the crawler to Tencent — which is what the test caught and
why the international consumer providers are now listed.
Crawls are enqueued under the same dedupe key as everything else, so four
hundred people at one company read that company's site once. That lesson
cost 226 identical crawls of accenture.com.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why Gravatar was not enough
18 identities from the first 1,800 people — about 1%. Gravatar is opt-in twice over: you need an account and you need to have attached your other accounts to it.
What the list actually is
I looked at the 16,268 rows now in prod before building anything, and they are not what the domain mix suggested:
0xshah@substack.com→0xshah.substack.com— the local part is the subdomainThe first two are pages the crawler already reads — it pulls a name, a title and published social links out of a site, which is precisely what an imported row is missing.
Derivation, not guessing
Every rule turns an address into a page it demonstrably belongs to. None invents a handle on a network the person never mentioned. This codebase already refuses to do that for email addresses — which it could at least verify by sending to — and an unverifiable social identity is worse.
Knowing when to do nothing is the valuable half
Sending every Gmail address to
google.comwould be 5,000 fetches of a page about nobody, and it would look like the feature working.The tests caught a real one:
qq.comwas missing from the freemail list — 127 addresses that would have been read as company domains and sent the crawler to Tencent's homepage. The international consumer providers are now listed (163, naver, seznam, orange, comcast, …).first.last@substack.comalso correctly resolves to nothing: that is notfirst.last.substack.com, and building a hostname from it would fetch something that does not exist.Cost control
Crawls are enqueued under the same dedupe key as everything else, so four hundred people at one company read that site once. That lesson cost 226 identical crawls of accenture.com in #63.
Verification
bun test— 1380 pass, 0 fail across 91 filestypecheckandformat:checkcleanWhat this will cost when it runs
Roughly 10,000 distinct pages (5,934 publications + 4,235 domains). At 25 jobs/tick that is several hours of crawling, and each crawl makes a model call for extraction. Worth knowing before merging — I can add a per-tick cap if you would rather meter it.
🤖 Generated with Claude Code