seo(landing): Organization/WebSite entity JSON-LD, AI-crawler robots allow-list, md-twin alternates, HSTS + cf-visitor https redirect

[BSalaeddin]

What / why

The 2026-09-11 fleet SEO pull says the site is found but not understood:

• The brand query "upup" drew 663 impressions at average position 6.1 and 2 clicks. Being on page one for your own name and getting a 0.3% CTR is an entity problem, not a ranking problem: nothing on the site ever told a search engine that "upup" is a thing with a repo, an npm scope, a chat server and a parent company. There was no Organization and no WebSite JSON-LD on any of the 73 pages, and the 64 docs pages — the bulk of the indexable surface — carried zero structured data of any kind.
• http://useupup.com/ serves the full page: 200, byte-identical to https, no redirect, no HSTS. Plaintext is a second live origin for the whole site.
• Both sitemap URLs still registered in Search Console are dead: /sitemap-landing.xml 404s outright, /documentation/sitemap.xml takes two hops into a 404. GSC has been reporting "couldn't fetch" for both.
• llms.txt is linked from nowhere — 0 <a href> to it in any page's HTML. Same for the 65 /docs-md/ markdown twins: reachable only from a client-side fetch in the Copy button, and served with no canonical back-link, so if discovered they are duplicates of every docs page under a second URL.
• robots.txt has no AI-agent group — a single * rule, no GPTBot/ClaudeBot/PerplexityBot/Google-Extended anywhere in the repo.
• Sitemap lastmod was a single build timestamp on all 73 URLs — a fake "everything changed" signal Google's documentation says it learns to ignore.
• /mobile-demo/ is a 200 HTML page with the homepage's title, no canonical and no robots directive; Disallow in robots.txt blocks crawling, not indexing.

The AI-crawler probe is the one thing that came back clean: 57/57 pass at Cloudflare — nothing is being blocked at the edge, so the only reason agents don't cite us is that we never gave them anything to cite.

Changes (all apps/landing unless noted)

Entity graph. New src/components/StructuredData/EntityStructuredData.tsx emits an Organization (@id /#organization, alternateName for the three spellings the brand is searched under, logo, parentOrganization: Devino) plus a WebSite (@id /#website, publisher → the org). It is mounted from the root layout, so it reaches every page including all 64 docs pages. sameAs carries only URLs verified live on 2026-09-12: the GitHub repo, the npm package (@useupup/react, latest 3.3.3 on the registry), and the README's Discord invite (301s to discord.com/invite/…; the invite API returns a live, non-expiring guild). The existing SoftwareApplication gains @id /#software and author/publisher pointing at the org by @id. No AggregateRating, no Review — pinned by a negative assertion. FAQPage is unchanged.

Docs structured data + markdown-twin discoverability. docs/[[...slug]]/page.tsx now declares alternates.types['text/markdown'], rendering <link rel="alternate" type="text/markdown">. New DocsStructuredData emits BreadcrumbList (from the same findTrail(tree, url) the visual breadcrumb uses, so they cannot disagree) and TechArticle (isPartOf the WebSite, publisher the Organization). No dates — content/docs has none and inventing one is worse than omitting it. A crawlable <a href="/docs-md/<slug>/">View as Markdown</a> sits next to the Copy-page button, and docs-md/[[...slug]]/route.ts answers with Link: <canonical html url>; rel="canonical" so the twin is an alternate representation, not a duplicate.

llms.txt discoverability. Real <a href="/llms.txt"> links in the footer (every page) and on the docs hub.

robots.txt AI allow-list. New src/lib/seo/ai-crawlers.ts exports the exact 18 agent names; robots.ts adds a second rules entry for them alongside *, both sharing ONE disallow constant so a named agent can never get a laxer list than the wildcard (that would be a side door into /api/).

Scheme, HSTS, stale paths (next.config.mjs). Production now sends Strict-Transport-Security: max-age=63072000; includeSubDomains; preload; non-prod keeps only X-Robots-Tag and gets no HSTS. An http→https redirect keyed only on Cloudflare's cf-visitor header sits first in redirects() — never x-forwarded-proto, which Traefik rewrites and which would loop. It is two rules, not one: Next strips the trailing slash before matching a source and does not re-append it to an absolute destination, so a single ${SITE_BASE}/:path* would send /react/ to …/react and cost a second 308. The first rule matches extensionless paths and restores the slash; file paths and the bare root fall through to the second, which must not gain one. Added single-hop redirects for the two stale GSC sitemaps, and host-scoped /robots.txt + /sitemap.xml rules for the docs alias (they previously chained into a 404).

Sitemap. lastModified removed from all 73 entries. Priorities, changeFrequency and the fumadocs enumeration are untouched.

/mobile-demo/ noindex. New mobile-demo/layout.tsx carries robots: { index: false, follow: false } (the page is 'use client' and cannot export metadata). The robots.txt Disallow stays — they cover different things.

Verification

Production build (turbo run build --filter=@useupup/landing --force, 4/4 tasks) served by next start -p 4466:

# http -> https, triggered ONLY by cf-visitor, slash preserved for pages and
# correctly absent for file paths
/react/               => 308 https://useupup.com/react/
/docs/getting-started/=> 308 https://useupup.com/docs/getting-started/
/robots.txt           => 308 https://useupup.com/robots.txt
/                     => 308 https://useupup.com/
# the same requests WITHOUT the header are served normally
/react/    => 200
/robots.txt=> 200
/          => 200

# stale Search Console sitemaps, one hop each
/sitemap-landing.xml        => 308 http://localhost:4466/sitemap.xml
/documentation/sitemap.xml  => 308 http://localhost:4466/sitemap.xml

# docs alias host (Host: docs.useupup.com) — previously 308 -> 308 -> 404
/robots.txt  => 308 https://useupup.com/robots.txt
/sitemap.xml => 308 https://useupup.com/sitemap.xml

# HSTS
GET / -> 200, strict-transport-security: max-age=63072000; includeSubDomains; preload

# markdown twin
GET /docs-md/getting-started/ -> 200
  content-type: text/markdown; charset=utf-8
  link: <https://useupup.com/docs/getting-started/>; rel="canonical"

# sitemap
loc count = 73, lastmod count = 0, mobile-demo = 0, docs-md = 0

# /mobile-demo/
<meta name="robots" content="noindex, nofollow"/>

# robots.txt body
User-Agent: *
Allow: /
Disallow: /api/
Disallow: /mobile-demo/

User-Agent: GPTBot
User-Agent: OAI-SearchBot
User-Agent: ChatGPT-User
User-Agent: ClaudeBot
User-Agent: Claude-User
User-Agent: Claude-SearchBot
User-Agent: anthropic-ai
User-Agent: PerplexityBot
User-Agent: Perplexity-User
User-Agent: Google-Extended
User-Agent: Googlebot
User-Agent: Bingbot
User-Agent: Applebot
User-Agent: Applebot-Extended
User-Agent: CCBot
User-Agent: Amazonbot
User-Agent: Bytespider
User-Agent: meta-externalagent
Allow: /
Disallow: /api/
Disallow: /mobile-demo/

Sitemap: https://useupup.com/sitemap.xml

# /docs/getting-started/ HTML
<link rel="canonical" href="https://useupup.com/docs/getting-started/"/>
<link rel="alternate" type="text/markdown" href="https://useupup.com/docs-md/getting-started/"/>
<a href="/docs-md/getting-started/" data-testid="docs-view-markdown" …>
JSON-LD block 1 @graph: Organization, WebSite
JSON-LD block 2 @graph: BreadcrumbList, TechArticle
  BreadcrumbList: [1 Docs -> https://useupup.com/docs/, 2 Getting Started -> https://useupup.com/docs/getting-started/]
  TechArticle: isPartOf {@id: …/#website}, publisher {@id: …/#organization}, no dates

# homepage
JSON-LD: [Organization, WebSite], SoftwareApplication, FAQPage
SoftwareApplication @id=https://useupup.com/#software
  author=https://useupup.com/#organization  publisher=https://useupup.com/#organization
AggregateRating occurrences: 0   "Review" occurrences: 0
footer: <a href="/llms.txt" …>

Gates, all from rtk proxy with the raw exit code:

Gate	Result
turbo run lint typecheck test --filter=@useupup/landing --force	PASS — 15/15 tasks, 81 tests in 9 files
pnpm run test:quality	PASS — 400 test files + 5 workflows clean, 0 exceptions
pnpm run vocab:check	PASS — 1415 tracked files clean
prettier --check on all 15 touched files	PASS
turbo run build --filter=@useupup/landing --force	PASS — 4/4
Playwright --project docs	PASS — 21/21 (15 pre-existing + 6 new)

What CI covers

New vitest apps/landing/src/__tests__/seo-surfaces.test.ts (14 tests, runs in main.yml › Test):

• sitemap has exactly 1 + 6 + 2 + 64 entries, every URL is https://useupup.com/… with a trailing slash, none contains /mobile-demo, /api/, /docs-md or /llms, and no entry carries lastModified — the fake-freshness signal cannot come back silently.
• prod robots: the * rule allows / and disallows /api/ + /mobile-demo/; a second rule lists exactly AI_CRAWLER_USER_AGENTS with the same disallow set; the sitemap URL is the production one. Non-prod (NEXT_PUBLIC_BASE_URL stubbed to the dev host with vi.stubEnv + vi.resetModules, because clientEnv is parsed once at module load) disallows /.
• EntityStructuredData and StructuredData are rendered with react-dom/server and every application/ld+json block is JSON-parsed: Organization has its @id, ≥2 https sameAs URLs, alternateName, and Devino as parentOrganization; WebSite's publisher and SoftwareApplication's author/publisher both resolve to the org @id; AggregateRating and Review appear nowhere.
• The 18-name crawler list is pinned literally — spelling is load-bearing, since robots.txt matches on these exact tokens.

New Playwright cases in apps/e2e-test/landing/docs.spec.ts (the Docs-E2E job, which runs on every landing PR): canonical + rel=alternate markdown link on /docs/getting-started/ and the twin's 200 / text/markdown / rel="canonical" Link header; the cf-visitor 308 to https://useupup.com/react/ and the plain request still 200; /sitemap-landing.xml → 308; the HSTS header on /; User-Agent: GPTBot in the robots body; the footer /llms.txt link. The dev server the project boots leaves NEXT_PUBLIC_BASE_URL unset, so next.config.mjs computes the production SITE_BASE and takes the production branch of headers()/redirects() — confirmed live, the HSTS assertion holds under next dev.

The existing docs-llms.test.ts, docs-source.test.ts (64 pages), site-url.test.ts and the docs.spec.ts redirect cases are unchanged and green.

Owner-only items (not in this PR)

1. Cloudflare zone, useupup.com: SSL/TLS → Edge Certificates → Always Use HTTPS = ON, then HSTS = ON. The cf-visitor rule here is the in-repo braces; the zone toggle is the belt and stops the plaintext request before it reaches the container.
2. Search Console: delete the two stale sitemap submissions (/sitemap-landing.xml, /documentation/sitemap.xml) and keep only https://useupup.com/sitemap.xml. They now 308 instead of 404, but the submissions themselves should go.
3. app.useupup.com: the DNS record resolves through Cloudflare and lands on the BioFlow container (404 text/plain with BioFlow's CSP and HSTS). Delete the app record, or add a Traefik router that 308s it to https://useupup.com/.
4. HSTS preload: submit at hstspreload.org — but only after this header has been live on production for a while. preload in the value is a prerequisite for submitting, not the submission.

[codesandbox]

Review or Edit in CodeSandbox

Open the branch in Web Editor • VS Code • Insiders

Open Preview