feat: Adds observability domain with Sentry, Tempo, telemetry cost control skills

domains/observability/knowledge/span-sub-sampling.md

@@ -0,0 +1,72 @@
+---
+name: span-sub-sampling
+domain: observability
+description: Deterministic per-trace sub-sampling for high-frequency custom spans — global tracesSampleRate × span sub-rate, traceId-hash bucketed
+---
+
+# Span Sub-Sampling
+
+Durable fix for a custom span that fans out and eats the span budget. Layer a per-trace sub-rate **under** the global `tracesSampleRate`, keyed on the trace id so every span in a trace is kept-or-dropped together. Source: [PR #39891](https://github.com/MetaMask/metamask-extension/pull/39891) (`shared/lib/wrapper-sampling.ts`).
+
+## Rate Math
+
+```
+effective rate = global tracesSampleRate × span sub-rate
+```
+
+- Global `tracesSampleRate` is already small (extension prod: 0.75%).
+- The sub-rate cuts the custom span on top: `0.75% × 1% = 0.0075%`.
+- PR #39891 ships a sub-rate of 0.5% (`WRAPPER_SAMPLE_RATE = 0.005`) — a conservative pilot — and names 5% as the step-up once the denylist is confirmed effective in production.
+
+Pick the sub-rate from how many sampled traces the metric needs to stay useful — not from the quota alone. Too low and the metric goes dark.
+
+## Pattern
+
+```ts
+const WRAPPER_SAMPLE_RATE = 0.005;
+
+// Deterministic: same answer for the same traceId, so all spans in a trace
+// are kept or dropped together — clean waterfalls, no partial gaps.
+export function shouldSampleWrappers(traceId: string | undefined): boolean {
+  if (!traceId || traceId.length < 8) {
+    return false;
+  }
+  const hashBucket = parseInt(traceId.slice(0, 8), 16) % 10000;
+  return hashBucket < WRAPPER_SAMPLE_RATE * 10000;
+}
+```
+
+**Why deterministic, not `Math.random()` per call:** independent per-span sampling shreds a trace into partial waterfalls (some spans present, siblings missing) — useless for attribution. Hashing the trace id makes keep/drop a property of the whole trace.
+
+## Gate Order (cheapest check first)
+
+```ts
+const traceId = sentryGetActiveSpan()?.spanContext().traceId;
+if (!traceId || isReadOnlyAction(action) || !shouldSampleWrappers(traceId)) {
+  return doWorkWithoutSpan();
+}
+return trace({ name, op, data }, doWorkWithSpan);
+```
+
+1. No active trace → no span.
+2. Denylist → skip noise (below).
+3. Sub-sample miss → skip this trace's spans.
+
+## Denylist: cut before you sample
+
+Drop spans with no timing/attribution signal before sub-sampling. In PR #39891, read-only verbs are ~90% of `messenger.call` volume:
+
+```ts
+const READ_ONLY_VERB = /^(?:get|has|find|is|peek)(?:[A-Z]|$)/u;
+```
+
+Removing ~90% of volume before the sample multiplies headroom — a higher sub-rate then yields the same span budget, so kept traces are denser and more useful.
+
+## Where the Gate Goes
+
+- **Consumer (extension):** spans go through `trace()`. Gate at the call site, or for a whole span family inside the wrapper. `traceId` from `sentryGetActiveSpan()?.spanContext().traceId`.
+- **Controller package (core):** controllers call an injected `trace` callback. Gate in the package's trace util or the callback so every consumer inherits the cap. Pull the trace id from the controller's tracing context, not a fresh Sentry import.
+
+## Kill Switch
+
+Ship every always-on span family with an env disable flag (PR #39891: `SENTRY_DISTRIBUTED_TRACING_DISABLED` returns the messenger un-wrapped). It turns a future emergency cut into a config flip instead of a cherry-pick.

domains/observability/skills/grafana-tempo-queries/skill.md

@@ -0,0 +1,141 @@
+---
+name: grafana-tempo-queries
+description: Query backend traces in Grafana Tempo with TraceQL — find traces by service or span attribute, fetch a trace by id, inspect its span tree, and enumerate tag values. Covers the datasource-proxy access path, the credential-expiry failure that returns empty results indistinguishable from "no data", the negative control that proves a filter actually applied, and the id/kind/base64 decoding quirks in the response. Use when investigating backend latency, checking what the backend recorded for a request, or establishing which infrastructure tiers a trace reaches. Triggers on Tempo, TraceQL, Grafana traces, backend span inspection, "does the backend have this trace", or tracing a request past the API boundary.
+maturity: experimental
+---
+
+# grafana-tempo-queries
+
+Tempo holds **backend** spans. Client spans from the extension and mobile go to Sentry via the SDK's own transport and never appear here — so a Tempo trace normally starts at an inbound server span, and a missing root is expected rather than broken. To join the two halves, see `sentry-grafana-correlation`.
+
+## Setup
+
+Everything goes through Grafana's datasource proxy. Authenticate with a service account token. Keep the host, datasource uid, org id, and token in your environment — this repository is public, so never commit them.
+
+```bash
+# Set these once per shell, from your own Grafana instance:
+#   GRAFANA_HOST   e.g. https://grafana.<your-org-domain>
+#   TEMPO_UID      the Tempo datasource uid (see discovery below)
+#   GRAFANA_ORG    the numeric org id the datasource belongs to
+#   GRAFANA_TOKEN  a service account token, Viewer role, scoped to the Tempo datasource
+BASE="$GRAFANA_HOST/api/datasources/proxy/uid/$TEMPO_UID"
+AUTH=(-H "Authorization: Bearer $GRAFANA_TOKEN" -H "X-Grafana-Org-Id: $GRAFANA_ORG")
+```
+
+Create the token under Administration, Service accounts. A Viewer-role account is
+sufficient for every query in this skill, and the token is revocable on its own without
+disturbing anything else you have open.
+
+If your Grafana disallows service accounts and there is genuinely no token path, a browser
+`grafana_session` cookie works in the same header slot:
+
+```bash
+AUTH=(-H "Cookie: grafana_session=$GRAFANA_SESSION" -H "X-Grafana-Org-Id: $GRAFANA_ORG")
+```
+
+Reach for that only after confirming a token cannot be issued. A session cookie carries
+your whole Grafana authority rather than one datasource's read access, expires on a
+schedule you do not control, and cannot be revoked without ending your own session. It is
+also indistinguishable from you in an audit log.
+
+Discover the datasource uid rather than guessing it:
+
+```bash
+curl -s "$GRAFANA_HOST/api/datasources" "${AUTH[@]}" \
+  | node -e 'JSON.parse(require("fs").readFileSync(0)).filter(d=>d.type==="tempo").forEach(d=>console.log(d.uid,d.name))'
+```
+
+## Check the instrument before believing a result
+
+**A stale session returns HTTP 401 with an empty body, and a naive parser reports that as zero results** — indistinguishable from "this data does not exist". This is the single most expensive failure mode here: it produces confident negative conclusions about instrumentation coverage.
+
+```bash
+# 1. Prove you are authenticated. Do this first, every session.
+curl -s -o /dev/null -w 'grafana auth: HTTP %{http_code}\n' "$GRAFANA_HOST/api/user" "${AUTH[@]}"
+
+# 2. Prove the filter is actually being applied, with a query that must match nothing.
+curl -s -G "$BASE/api/search" "${AUTH[@]}" \
+  --data-urlencode 'q={span.db.system = "not-a-real-db-xyz"}' \
+  --data-urlencode "start=$START" --data-urlencode "end=$NOW" \
+  | node -e 'const j=JSON.parse(require("fs").readFileSync(0));console.log("control traces:",(j.traces||[]).length,"(must be 0)")'
+```
+
+If several different filters all return exactly your `limit`, the filter is not being applied — treat the results as unfiltered until the negative control returns 0.
+
+## Core queries
+
+Every endpoint wants an explicit epoch-seconds window. Omitting it on a by-id lookup makes the request hunt across all blocks and hit a context deadline.
+
+```bash
+NOW=$(date +%s); START=$((NOW-3600))
+```
+
+**Search by TraceQL.** Returns trace summaries plus the spans that matched.
+
+```bash
+curl -s -G "$BASE/api/search" "${AUTH[@]}" \
+  --data-urlencode 'q={resource.service.name="my-service"}' \
+  --data-urlencode "start=$START" --data-urlencode "end=$NOW" \
+  --data-urlencode "limit=20"
+```
+
+**Fetch one trace in full** (OTLP JSON: resource batches → scope spans → spans).
+
+```bash
+curl -s "$BASE/api/traces/$TRACE_ID?start=$START&end=$NOW" "${AUTH[@]}"
+```
+
+**Enumerate values for a tag** — useful for inventorying what a fleet emits. Expect a `502` on high-cardinality tags; fall back to inspecting individual traces rather than concluding the tag is unused.
+
+```bash
+curl -s -G "$BASE/api/v2/search/tag/span.db.system/values" "${AUTH[@]}" \
+  --data-urlencode "start=$START" --data-urlencode "end=$NOW"
+```
+
+## TraceQL patterns worth knowing
+
+| Goal | Query |
+| --- | --- |
+| One service | `{resource.service.name="svc-name"}` |
+| Several services | `{resource.service.name=~"(svc-a|svc-b)-prd"}` |
+| Attribute present at all | `{span.db.system != nil}` |
+| Span kind | `{kind=server}`, `{kind=client}` |
+| Slow spans | `{duration > 1s}` |
+| **Two conditions anywhere in the same trace** | `{resource.service.name="svc-a"} && {span.db.system != nil}` |
+
+The last one is the important one: `&&` between two brace groups is a **trace-level** conjunction, not a single-span filter. It answers "does a request into this service reach a database at all", which is how you map how deep a trace goes without reading traces one at a time.
+
+## Reading the response
+
+- **Span and trace ids are base64**, not hex. Decode before comparing them to anything from a header or from Sentry: `Buffer.from(id,"base64").toString("hex")`.
+- **`kind` is a string** (`SPAN_KIND_SERVER`, `SPAN_KIND_CLIENT`, `SPAN_KIND_INTERNAL`), not the numeric enum. Filtering on `sp.kind === 2` silently matches nothing.
+- **Search results drop leading zeros from trace ids.** A 31-character id is a 32-character id with a leading zero; zero-pad before using it anywhere else, or the lookup fails for a reason that looks like absence.
+- **`rootServiceName: "<root span not yet received>"`** means the trace's root is not in Tempo. For client-originated requests that is the normal case — the root is a client span living in Sentry — and it is the marker for finding them.
+- Resource attributes carry deployment context (`service.name`, kubernetes pod/namespace/cluster, region); span attributes carry the request (`http.*`, `net.*`, `db.*`).
+
+## Deep links for sharing
+
+A link is more useful than a pasted id. Build a Grafana Explore URL with the query pre-filled:
+
+```bash
+node -e '
+const left={datasource:process.env.TEMPO_UID,
+  queries:[{refId:"A",datasource:{type:"tempo",uid:process.env.TEMPO_UID},queryType:"traceql",query:process.argv[1]}],
+  range:{from:"now-6h",to:"now"}};
+console.log(`${process.env.GRAFANA_HOST}/explore?orgId=${process.env.GRAFANA_ORG}&left=${encodeURIComponent(JSON.stringify(left))}`);
+' '<trace-id-or-traceql>'
+```
+
+Prefer an absolute `from`/`to` when the link needs to outlive the event; a relative window slides off it and the reader opens an empty result.
+
+## Failure modes
+
+| Symptom | Cause | Response |
+| --- | --- | --- |
+| All queries return 0 | Session expired (401, empty body) | Check `/api/user` first |
+| Every filter returns exactly `limit` | Filter not applied | Run the negative control |
+| By-id lookup times out | No time window | Pass `start`/`end` |
+| Tag-values returns 502 | High cardinality | Inspect traces directly |
+| Id from search not found elsewhere | Leading zeros stripped | Zero-pad to 32 chars |
+| Kind filter matches nothing | Comparing to a number | Compare to `SPAN_KIND_*` |
+| Trace has no root | Root is a client span | Expected; see `sentry-grafana-correlation` |

domains/observability/skills/instrumentation/repos/metamask-extension.md

@@ -0,0 +1,61 @@
+---
+repo: metamask-extension
+parent: instrumentation
+---
+
+## Key Files
+
+| Content | Path |
+|---------|------|
+| Sentry trace wrapper | `shared/lib/trace.ts` |
+| Trace name enum | `shared/lib/trace.ts` → `TraceName` |
+| MetaMetrics controller | `app/scripts/controllers/metametrics-controller.ts` |
+| Event enum | `shared/constants/metametrics.ts` → `MetaMetricsEventName` |
+| Sentry setup + sample rate | `app/scripts/lib/setupSentry.js` → `getTracesSampleRate()` |
+| Segment tracking plan | `Consensys/segment-schema` → `tracking-plans/metamask-extension.yaml` |
+
+## Cross-Process Context (UI → Background)
+
+The extension has two Sentry hubs — one in the UI process and one in the background service worker. A trace starting in UI and continuing in background requires explicit context propagation across the RPC boundary:
+
+```typescript
+// Serialize at UI call site
+const context: SerializedTraceContext = {
+  _name: TraceName.MyOperation,
+  _traceId: span.spanContext().traceId,
+  _spanId: span.spanContext().spanId,
+}
+
+// Background receives context, creates child span
+trace({ name: TraceName.MyOperation, parentContext: context }, async () => { ... })
+```
+
+Without propagation: Sentry shows two disconnected operations. With propagation: complete tree from user action to RPC call.
+
+## Sentry Sample Rate
+
+```bash
+grep -n "tracesSampleRate" app/scripts/lib/setupSentry.js
+# Verify current value before calculating — it has changed between releases
+```
+
+## Sentry Traces Explorer Query (Volume Estimation)
+
+```
+Environment: production | Time range: 30 days | Mode: aggregate
+Query: span.op:http.client span.description:*{endpoint}*
+Group by: span.description, transaction
+Sort: -count(span.duration)
+```
+
+## Detect `isOptIn` Misuse
+
+```bash
+grep -rn "isOptIn: true" app/scripts/ ui/ --include="*.ts" --include="*.tsx"
+# Any occurrence outside the onboarding opt-in flow is suspect
+```
+
+## Data Council Contact
+
+- Slack: `#metamask-metametrics`
+- Team: `@consensys/data-council`

domains/observability/skills/instrumentation/skill.md

@@ -0,0 +1,89 @@
+---
+maturity: experimental
+name: instrumentation
+description: Create and update Sentry spans, MetaMetrics events, and Segment events — methodology, policies, common pitfalls
+---
+
+# Analytics Instrumentation
+
+## When To Use
+
+- Adding or modifying a MetaMetrics (Segment) event
+- Adding or modifying a Sentry performance span
+- Estimating event or span volume from production data
+- Auditing existing instrumentation for correctness
+
+---
+
+## Do Not Use When
+
+- Adding local debug logging with no telemetry destination
+- Investigating an existing Sentry error report (use `sentry-mcp-queries`)
+- Internal feature flag evaluation not surfaced as an analytics event
+
+---
+
+## Sentry Spans
+
+### Creating a Span
+
+1. **Register a named trace entry** in the repo's trace name enum before writing any span code. Unnamed spans are invisible in Sentry filters.
+2. **Use the repo's `trace()` wrapper**, not raw `Sentry.startSpan()`. Wrappers handle cross-process context propagation, active-span inheritance, and consistent tag injection.
+3. **Inherit parent automatically** — when no `parentContext` is provided, the wrapper inherits from `Sentry.getActiveSpan()`, making the new span a child of the active parent (e.g., a `pageload` span).
+
+### Updating a Span
+
+- Adding a tag: no governance required
+- Renaming a trace name enum entry: grep all callsites; update enum and references atomically
+- Changing an `op` value: breaks saved queries and dashboards — coordinate with whoever owns them
+
+---
+
+## MetaMetrics / Segment Events
+
+### Creating an Event
+
+1. **Check the event name enum** — event may already exist under a different phrasing.
+2. **Check the segment tracking plan** — event may be registered under a different name than the enum key.
+3. **Add to the enum**, then implement the `trackEvent` call.
+4. **Do NOT use `isOptIn: true` outside the onboarding opt-in flow.** It strips user identity unconditionally for all users, not just non-opted-in ones (see Reference Knowledge: metrametrics-identity).
+5. **Open a data governance review** before merging. There is usually no CI enforcement on schema registration — this step is easy to skip (see Reference Knowledge: segment-governance).
+6. **Register in the team's segment tracking plan** before shipping.
+
+### Updating an Event
+
+- Adding a property: requires governance review and schema update
+- Renaming an event: deprecate old + add new in tracking plan; coordinate on migration window
+- Removing an event: confirm no active dashboards depend on it before removing
+
+---
+
+## Volume Estimation via Sentry
+
+When direct Segment access is unavailable, estimate from Sentry production span data:
+
+1. **Find a correlated HTTP endpoint** — one that fires 1:1 with the event.
+2. **Query Sentry Traces Explorer** (aggregate mode):
+   ```
+   span.op:http.client span.description:*{endpoint}*
+   ```
+3. **Extrapolate:**
+   ```
+   estimated_actual = sampled_count × (1 / tracesSampleRate)
+   ```
+4. **Interpret as upper bound** — endpoint may have callers outside the event path.
+
+Caveats: sample population is MetaMetrics opted-in users only; verify the current `tracesSampleRate` before calculating (it changes between releases). For longer-range (30D+) or release-over-release queries, the sampled count is **not** comparable at face value — older releases are downsampled / retention-truncated and `.0` releases are sample-thin; see `sentry-mcp-queries` (Longer-Range Queries and Percentile Fidelity) and the `performance-attribution` skill.
+
+---
+
+## Common Pitfalls
+
+| Mistake | Correct Approach |
+|---------|-----------------|
+| `isOptIn: true` on post-onboarding events | Strips user identity for all users; only valid in onboarding flow |
+| Ship event without tracking-plan registration | No CI gate — add governance review explicitly to PR checklist |
+| Raw `Sentry.startSpan()` instead of the repo's `trace()` wrapper | Use the wrapper — handles cross-process context and active-span inheritance |
+| New span with no trace name enum entry | Register enum entry first; unnamed spans are invisible in Sentry filters |
+| Multiply sampled count by `tracesSampleRate` | Multiply by inverse: `sampled × (1 / rate)` |
+| Treat Sentry estimates as exact counts | Probabilistic sample — state sample size and confidence |