fix(robots): test shared user-agent groups

[nibzard]

What changed

• make the test-local robots parser apply directives to consecutive User-agent records
• combine directives when the same agent appears in a later group
• add a characterization fixture covering stacked, repeated, and non-leaking group behavior
• remove the deprecated Host: docs.steel.dev record

Why

The parser replaced its active directive array for every User-agent line. That meant stacked agents did not share following directives, and a repeated agent lost directives from its earlier group. The current robots policy happened not to use those forms, but the test helper could silently mis-evaluate a future compacted file.

Impact

All ten crawler groups retain the same Content-Signal values and Allow: / policy. The sitemap remains unchanged; only the deprecated Host extension is removed.

Validation

• bun test — 298 passed
• bun run typecheck
• bun run check --error-on-warnings
• bun run validate-links
• confirmed 10 Content-Signal and 10 Allow: / records
• confirmed no Host: record
• git diff --check

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs	Ready	Preview	Jul 30, 2026 6:56pm