Keep the static mirror out of search results - #123
Merged
Conversation
Google was generating sitelinks to /about/, /team/, and portfolio pages under the root.vc result, which undercuts the terminal being the front door. Emits `noindex, follow` on every mirror page and on welcome.htm, and trims sitemap.xml to the homepage alone — listing a noindex URL in a sitemap asks Google to index something the page itself forbids, and that contradiction is what surfaced the sitelinks. Deliberately not a robots.txt Disallow: noindex is a search-indexing directive, so crawlers may still fetch and read these pages. GPTBot, ClaudeBot, PerplexityBot and llms.txt consumers keep getting the full content, while root.vc stays the only result Google shows. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Google is showing sitelinks to
/about/,/team/,/jobs/, and individual portfolio pages underneath the root.vc result. That undercuts the point of the terminal being the front door — part of the site's appeal is that you have to explore it.Approach
Uses
noindex, not arobots.txtDisallow. The distinction matters:Disallowsays don't fetch this — it would hide the mirror from the AI crawlers it exists to serve.noindexsays you may read this, just don't list it in search results — Google and Bing honor it for indexing, while GPTBot, ClaudeBot, PerplexityBot andllms.txtconsumers keep getting the full content.Changes:
noindex, followon every generated mirror page and onwelcome.htm.followkeeps link equity flowing back to the homepage.sitemap.xmltrimmed tohttps://root.vc/alone. Listing a noindex URL in a sitemap asks Google to index something the page itself forbids — that contradiction is what generated the sitelinks in the first place.robots.txtunchanged: still explicitly allows all AI crawlers, still no Disallow on the mirror.Net effect: root.vc is the only result Google shows, and LLM discovery is unaffected.
Test plan
npm test— 312/312 passing, including a newsearch invisibilityblock that asserts every mirror page is noindex, the homepage is not,welcome.htmis, robots.txt has no Disallow on the mirror, andllms.txtstill carries every portfolio companynpm run build→ verifieddist/sitemap.xmlcontains onlyhttps://root.vc/, and 76 files indist/carry the noindex tagNote on timing
The existing sitelinks won't vanish the moment this deploys. Google has to recrawl each page and see the new tag, which typically takes days to a couple of weeks. To speed it up, use Search Console's Removals tool for a temporary block on the
/about/,/team/,/portfolio/,/jobs/prefixes while the recrawl happens.Worth knowing: keeping the pages crawlable is what lets Google see the noindex tag. Blocking them in robots.txt instead would leave the already-indexed URLs stuck in the index, since Google could no longer fetch them to discover they should be dropped.
🤖 Generated with Claude Code