seo(robots): block external tracking params from being crawled

[vfanucci]

What

Adds three Disallow rules to robots.txt.ts for tracking parameters appended by inbound links (not by our own markup):

Disallow: /*?clid=
Disallow: /*?from_theconsensus=
Disallow: /*?__hstc=

Why

Follow-up to #5195. The GSC Coverage → "Alternate page with proper canonical tag" drilldown exported on 17/08 shows 6 850 affected pages. The bulk of that (~5 800, discovered 11/07) is the /blueprints?tags= facet trap already fixed by #5195 — the last crawl date on every one of those URLs is 28/07, the day that PR merged, so that stock is frozen and draining.

What is still growing is a tail of external tracking params, crawled through 15/08:

Param	URLs in sample	Source
?clid=	94	third-party link tracker (base64 {i,h,p,t})
?from_theconsensus=1	27	backlinks from theconsensus
__hstc / __hssc / __hsfp	15	HubSpot

These all canonicalise correctly, so there is no ranking impact — this is purely crawl budget. It's worth fixing because the params land mostly on /docs/* and /plugins/*, the pages we most want crawled efficiently.

Notes on the rules

• Only the ?param= form is needed. Google's * wildcard doesn't span the ?, so /*?utm_ (existing) matches only when the param is first. In the sample, all 94 clid, 27 from_theconsensus and 15 __hstc occurrences are the first query parameter, and __hssc/__hsfp always follow __hstc — so one rule covers the whole HubSpot set.
• No over-blocking. /*?clid= requires the literal ?clid=, so ?fbclid= and ?gclid= are unaffected.
• No user impact. robots.txt only governs crawlers; visitors arriving on these links are unaffected, and the canonical target is already indexed.

Verification

This route is DEV/PREVIEW-gated (disabled → Disallow: *), so the preview deploy can't exercise the change. Rendered the production branch of the template directly to confirm output and formatting:

# Block tracking parameters
Disallow: /*?q=
Disallow: /*?search=
Disallow: /*?ref=
Disallow: /*?utm_
# External trackers appended by inbound links. …
Disallow: /*?clid=
Disallow: /*?from_theconsensus=
Disallow: /*?__hstc=
Sitemap: https://kestra.io/sitemap/index.xml

Caveat on the numbers: GSC caps the drilldown table at 1 000 rows, so the per-param counts are a sample of the 6 850 — indicative proportions, not exact totals.

🤖 Generated with Claude Code