feat(agents): publish a machine-readable surface for AI agents

[naveen-bandela-au2]

Why

gofr.dev scores 53/100 (grade C) on orank's agent-readiness scan. Agents arriving from web search get a React shell of navigation chrome, and there is no way for them to discover what the site offers or how to read it.

This adds the machine surface. A companion PR in gofr-dev/gofr carries the half that lives there — it must land first, because it bumps zopdev/static-server to v1.0.0, which is what actually serves any of this.

What changed

Markdown twins. A .md twin for every content route, generated at build time from the same page.md the HTML renders from. Markdoc tags are rendered down to plain Markdown — fence-aware, so code samples containing {% survive — internal links are made absolute, and each twin opens with a top-level heading. Discoverable via <link rel="alternate"> in the page head, and served on Accept: text/markdown once static-server is bumped.

Scoped indexes. /docs/llms.txt and /docs/{section}/llms.txt, so an agent working on one area can skip the 660 KB whole-site dump.

/openapi.json. Describes the read surface that actually exists — the indexes, the twins, the sitemap, the changelog feed. The page parameter is enumerated from the generated twin list, so an agent doing function-calling cannot ask for a path that 404s. Validates clean under redocly lint.

Access model. /auth.md and /.well-known/oauth-protected-resource state the real one: anonymous, no registration, no token. Every URI they advertise resolves (verified below). No fabricated endpoints.

Agent discovery. /.well-known/{ard.json, agent-card.json, agent-skills/index.json, mcp/server-card.json, api-catalog}, /agent/identity, and a Markdown homepage at /index.md for cold arrivals.

NLWeb Schema Feeds. /schemamap.xml + /feeds/docs.jsonl, referenced from robots.txt.

public/_headers. Site-wide Link, nosniff, Referrer-Policy, and the media types the extensionless .well-known documents need. This replaces what would otherwise have been nginx config — see "One correction" below.

Two pages that are overdue independently of any scan

• /privacy — src/app/layout.jsx:114 loads Google Tag Manager (GTM-5G6KD5VJ). Every visitor, including EU visitors, is tracked with no disclosure and no policy page anywhere on the site. That is a GDPR exposure, not a scoring gap.
• /contact — routes bug reports, security disclosures, and questions to the channels that already exist. It is also what makes the new Organization.contactPoint JSON-LD truthful.

No /about: orank already verified one (it matched /why-gofr), and a second would compete with it in search.

Structured data

Organization gains contactPoint; a WebSite node is added. Two things are deliberately omitted:

• address — GoFr has no public postal address, and inventing a PostalAddress to satisfy a validator would be worse than leaving it out.
• potentialAction: SearchAction — site search is a client-side FlexSearch index with no /search?q= route, so it would point at a URL that does not resolve.

FAQPage was not added to the home page, despite it being worth points: the home page has no visible FAQ, and marking up invisible content violates Google's structured-data policy and risks a manual action.

One correction worth flagging

The obvious plan was to put content negotiation in nginix.conf. That file never reaches a user — it only backs the source-bundle image, which prod uses as a --target builder base and discards. Production is zopdev/static-server, a Go binary built with GoFr. Hence _headers rather than nginx, and hence the dependency on the companion PR.

Verification

Built with the docs overlay from gofr-dev/gofr (a standalone build of this repo has empty doc routes and produces 2 twins, which it does gracefully — no crash), then served with zopdev/static-server:v1.0.0:

106 twins, 6 scoped llms.txt, 106 schema feed entries, openapi lint clean

PASS  Accept: text/markdown → text/markdown + Vary: Accept
PASS  body starts with '# '
PASS  browser Accept: text/html still gets HTML
PASS  direct .md → text/markdown
PASS  /no-such-path → real 404; markdown 404 body pointing at /sitemap.xml
PASS  Link: sitemap / service-desc / api-catalog
PASS  /.well-known/api-catalog → application/linkset+json
PASS  /agent/identity → application/json
PASS  all 24 URIs advertised in auth.md + PRM + ard.json resolve (GET 200, OPTIONS 200)
PASS  all 106 enumerated openapi pages exist
PASS  twins absent from sitemap.xml; /privacy and /contact present

Twins and generated indexes are gitignored: they derive from docs that gofr-dev/gofr overlays at build time, so a copy committed here would be permanently stale.

What this does NOT close

Gap	Why
Wikipedia / Wikidata entity (4 pts)	Needs third-party press for notability. Not a code change.
ChatGPT app directory (2 pts)	External submission.
npm / PyPI SDKs (4 pts)	GoFr is a Go framework; an npm "SDK" would be a fake package. The honest version is publishing the gofr CLI to npm/Homebrew.
Hosted MCP, NLWeb /ask, JSON error responses, sandbox (~13 pts)	All need a running service; the site is a static export.
Bot-UA → markdown, ?mode=agent (3 pts)	static-server matches on path only and deliberately refuses UA switching (Vary: User-Agent is a cache-fragmentation anti-pattern). Would need an upstream PR.
WWW-Authenticate: Bearer 401 hint (1 pt)	Would mean faking a 401 on a genuinely public path, contradicting /auth.md.

Realistic outcome once both PRs land and a v* tag deploys: ~72–80/100, not the ~90 a naive reading of the gap list suggests.