Make bunit.dev agent ready (robots.txt + sitemap via docfx)

[Copilot]

Pull request description

The agent-readiness audit flagged a missing/invalid robots.txt, no sitemap.xml, and no AI-crawler/content-usage signals. bunit.dev is a static docfx 2.74.1 site deployed to GitHub Pages (docs/site → bUnit-dev.github.io), so only static-file and build-time mechanisms apply.

• docs/site/docfx.json — enable docfx's built-in sitemap (baseUrl: https://bunit.dev) so /sitemap.xml with canonical URLs is regenerated on every publish; register robots.txt as a build resource so it lands at the site root.
• docs/site/robots.txt (new) — explicit User-agent: * allow rules, a Content-Signal: search=yes, ai-input=yes, ai-train=yes directive, explicit allow rules for AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, Claude-Web, ClaudeBot, Google-Extended, PerplexityBot, Applebot-Extended), and a Sitemap: reference.

// docs/site/docfx.json
"sitemap": {
  "baseUrl": "https://bunit.dev",
  "changefreq": "weekly",
  "priority": 0.5
}

Not addressed — require capabilities GitHub Pages static hosting lacks, or are N/A for a docs site with no protected APIs/MCP server/tools: Link response headers (RFC 8288), DNS-AID, Accept: text/markdown negotiation, OAuth/OIDC & protected-resource discovery, auth.md, MCP Server Card, WebMCP, API catalog, agent-skills index. Pursuing the server-dependent items would mean moving off GitHub Pages (e.g. Cloudflare Pages/Workers).

Note: ai-train=yes is a policy choice assuming maximum reach for the MIT-licensed docs — flip to no if preferred.

PR meta checklist

• Pull request is targeted at main branch for code
  or targeted at stable branch for documentation that is live on bunit.dev.
• Pull request is linked to all related issues, if any.
• I have read the CONTRIBUTING.md document.

Code PR specific checklist

• My code follows the code style of this project and AspNetCore coding guidelines.
• My change requires a change to the documentation.
  • I have updated the documentation accordingly.
• I have updated the appropriate sub section in the CHANGELOG.md.
• I have added, updated or removed tests to according to my changes.
  • All tests passed.