fix(llms): serve canonical HTML to crawlers

[nibzard]

Summary

• serve canonical, indexable HTML to autonomous AI crawlers by default
• preserve Markdown negotiation for user-directed fetchers, coding agents, explicit Markdown Accept headers, and .md URLs
• keep /overview HTML-only because it has no equivalent MDX source
• route non-Markdown / requests to /overview and preserve Vary: Accept, User-Agent
• add unit, middleware, and runtime regression coverage for the crawler/client matrix

Why

Autonomous crawler user agents were grouped with Markdown clients. Canonical requests were therefore rewritten to Markdown responses carrying X-Robots-Tag: noindex; /overview was worse because it has no Fumadocs source, so the rewrite returned 404. The root route also treated every request without Sec-Fetch-Dest as Markdown, including crawlers and generic HTTP clients.

Impact

Search and training crawlers now receive the canonical HTML representation without noindex. User-directed clients such as ChatGPT-User, Claude-User, and Perplexity-User continue receiving Markdown on regular docs pages. Generic curl / now follows the HTML route unless it explicitly requests Markdown or uses /AGENTS.md or a .md URL.

Validation

• bun run check --error-on-warnings
• bun run typecheck
• bun test — 291 passed
• bun run validate-links
• bun run build
• git diff --check

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs	Ready	Preview	Jul 30, 2026 4:06pm