Skip to content

feat(telemetry): one outcome event per command, with no channel for a path (backend#1907) - #527

Merged
LukasWodka merged 1 commit into
developfrom
feat/1907-cli-command-outcome-telemetry
Aug 18, 2026
Merged

feat(telemetry): one outcome event per command, with no channel for a path (backend#1907)#527
LukasWodka merged 1 commit into
developfrom
feat/1907-cli-command-outcome-telemetry

Conversation

@LukasWodka

@LukasWodka LukasWodka commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Parent epic: tracebloc/backend#1872 · Ticket: tracebloc/backend#1907 · Contract: the telemetry contract

The CLI emits nothing today. That is why the backend#736 class — the binary
landing on a PATH the shell does not read, cluster info reading a kubeconfig
context nobody meant — is only ever visible when a customer happens to mention
it. This wires the #1897 helper up to one terminal event per invocation.

What is emitted

One of cli.command.succeeded / .failed / .cancelled, from
cmd/tracebloc/main.go — the single point every command path converges on, so
§6.5's "terminal event on every path" is true by construction rather than true
for the handlers somebody remembered.

layer attribute value
record event.name one of the three above
record tracebloc.cli.command data ingest — the contract's own name for this field (§7.1)
record tracebloc.cli.exit_code int
record tracebloc.cli.duration_ms int, clamped at 0
record error.type closed vocabulary, failures only (§8.4)
resource service.name / tracebloc.component cli / cli
resource service.version ldflags
resource os.type / host.arch runtime.GOOS / runtime.GOARCH
resource deployment.environment signed-in env, else $CLIENT_ENV, else prod
resource service.instance.id fresh random per process

Ctrl-C (130) is cancelled, not failed, and carries no error.type. A
cancel counted as a failure moves the rate D9's alerts are written against every
time somebody changes their mind.

"No arguments, no paths, no data" is a structure, not a rule

A rule is a thing every future call site has to remember. There is no
redaction regex anywhere in this PR
, because the record has no free-text
channel to redact:

  • the command is a lookup, not a sanitiser. internal/cli/commandPaths
    enumerates the paths from the live cobra tree; anything not in that set is
    reported as unregistered. A sanitiser has to anticipate what it strips; a
    closed set only ever admits what was enumerated. A recorder built with no
    registered commands reports unregistered for everything — forgetting to
    register cannot silently turn the lookup into a pass-through.
  • the error class is keyed on an int — the CLI's own frozen exit-code
    contract (internal/cli/exitcodes.go), which is already a reviewed,
    documented classification of every way the CLI can fail. The classifier is
    never handed an error message, so there is nothing for a path or a cell value
    to travel down. An unmapped code is unclassified, not the number
    stringified into a value that appears on its own.
  • everything else is an int.

service.instance.id is a fresh random value per process, deliberately not the
hostname (field hostnames are overwhelmingly <firstname>-macbook, which §7.3
forbids outright) and deliberately not persisted (a durable identifier we would
then have to answer erasure requests about).

The guard is derived, and mutation-proven

TestEveryEmittedStringComesFromAClosedSet does not hold a list of forbidden
keys — a list like that agrees with itself and says nothing about the twentieth
attribute somebody adds. It walks what the code actually emits over a
cross-product of adversarial command paths and exit codes, and requires every
value to be an int or a member of a set assembled from the producer's own
declarations. A free-text channel of any kind fails it whether or not anyone
thought to forbid the thing travelling down it. It carries an attribute-count
anchor, because an inert loop over an empty payload reads exactly like a clean
sweep in a log.

TestNoFlagValueOrArgumentCanReachTheRecord drives the real tree: every
command, every flag set to a canary, canary args, canary environment, then the
whole payload searched.

13 mutations run, each reddened, each anchor asserted to have applied:

mutation reddened
commandValue stops looking the path up (pass-through) 4 tests, both packages
ClassifyExit invents a value outside the vocabulary 3 tests
Record smuggles the raw command into a second attribute 3 tests
the cancel path folds into failed 2 tests
durationMS stops clamping 1 test
telemetryEnabled ignores the opt-out 1 test
telemetryEnabled always says no (feature dead) 7 tests
commandPaths stops recursing into subcommands 1 test
telemetryEnv repairs an unknown environment 2 tests
the bare root reports an empty command 2 tests
os.type drops out of resourceScope 1 test
New stops stamping host.arch 2 tests
doc/code opt-out variable drift (both directions) 1 test

One of these is worth calling out: "smuggle the raw command into a second
attribute" was caught by the telemetry-side test and NOT by the cli-side one
,
which was asserting on a single key while the record carried the path verbatim.
That test now sweeps the whole payload. Without asserting the anchor applied,
that gap would have looked exactly like coverage.

What remains to be connected

The transport, and only the transport. The ticket says "rides the gateway
and token" — that wording is stale. The 17 Aug decision replaced the Collector
gateway with an ingest endpoint on the backend (RFC amendment
rfcs#28), which is
tracebloc/backend#1905, in flight and not yet available. So:

  • internal/cli.pendingSink() returns nil — every event is validated and
    dropped
    (SetSink's documented contract).
  • Validation runs on every build regardless, so a malformed event fails in CI
    wherever the binary was built.
  • When #1905 lands, pendingSink() returns the client that posts to it and
    nothing else in this PR changes.

The ticket body has been corrected so it no longer says "blocked by the gateway".

Would the three named field failures now be visible?

#736 case visible?
CLI on ~/.local/bin, PATH advice only printed No — this is the installer's, and it is covered in the sibling client PR (tracebloc.install.cli_on_path). Nothing the CLI can emit sees it: a binary the shell never finds never runs.
cluster info reading the wrong kubeconfig context Yes, where it failstracebloc.cli.command=cluster info with error.type no_secure_environment (exit 4) or local_environment (exit 3) becomes a countable rate instead of a support thread. Honest limit: a run that succeeds against a wrong-but-valid context still looks like a success, and no attribute in this closed set can tell them apart without emitting the context name, which is a path.
apt-get hung on the dpkg lock No — installer, covered in the sibling PR via per-phase durations.

The CLI half of the DoD is the second row; the other two are the client PR.

Also here

  • internal/telemetry joins scripts/coverage-floor.sh at 95% (measures
    100.0%). It stopped being an unwired helper and became the thing that decides
    what leaves a customer's machine, so a rotting test there is the guard going
    quiet, not an ordinary coverage regression.
  • docs/troubleshooting.md gains a "Usage reporting" section naming every field
    and the opt-out. That document's claim about which variables work is itself a
    test — a user who exports a stale name believes they have opted out, and
    nothing else would ever tell them.

Test plan

make check green; make deadcode clean (the whole package entered the build
graph with this PR, so every function in it is now reachability-checked);
./scripts/coverage-floor.sh green including the new entry; 13 mutations above.

🤖 Generated with Claude Code


Note

Medium Risk
Touches default-on telemetry and the privacy boundary for what leaves customer machines; transport is still nil, but mis-wiring or enumeration drift could misreport commands or leak data if guards regress.

Overview
Adds one terminal telemetry event per CLI invocation from cmd/tracebloc/main.go after ExecuteContextC returns, recording command path (from the live cobra tree), exit code, duration, version, OS/arch, environment, and a per-process random instance id—without affecting exit behavior.

Privacy and classification are structural: internal/telemetry/outcome.go maps commands via an enumerated path set (unregistered otherwise) and error.type via frozen exit codes only; Ctrl-C (130) emits cancelled with no error.type. internal/cli/telemetry.go handles opt-out (TRACEBLOC_NO_TELEMETRY, DO_NOT_TRACK), environment resolution, and pendingSink() returning nil so events are validated and dropped until backend#1905.

Emitter changes stamp os.type / host.arch on the resource layer. Docs add a usage-reporting section in docs/troubleshooting.md. CI adds a 95% coverage floor for internal/telemetry. Broad tests exercise the live command tree, canary args/flags, and doc/code opt-out parity.

Reviewed by Cursor Bugbot for commit efff633. Bugbot is set up for automated code reviews on this repo. Configure here.

… path (backend#1907)

The CLI emits nothing today, so the backend#736 class — the binary landing on a
PATH the shell does not read, `cluster info` reading a kubeconfig context nobody
meant — is only ever visible when a customer mentions it. This wires the #1897
helper to a single terminal event per invocation: command, duration, exit code,
OS/arch, version, error class.

The ticket's "no arguments, no paths, no data" is built as a structure rather
than a rule, because a rule is a thing every future call site has to remember:

  * the command is a LOOKUP into the set of paths enumerated from the live cobra
    tree, so a value that is not a command the CLI dispatches cannot be emitted
    at all — it reports `unregistered`, which stays countable;
  * the error class is keyed on an INT, the CLI's own frozen exit-code contract.
    The classifier is never handed an error message, so there is nothing for a
    path or a cell value to travel down;
  * everything else is an int.

There is no redaction regex anywhere in the change. A sanitiser has to
anticipate what it strips; a closed set only admits what was enumerated.

os.type / host.arch go in the RESOURCE layer under OpenTelemetry's own names
(§1.1 forbids re-inventing them as tracebloc.os): they are compile-time
constants of the binary, so they describe the process, not the occurrence, and
adding them to resourceScope means a call site still cannot set them.

The guard is derived, not restated. TestEveryEmittedStringComesFromAClosedSet
walks what the code ACTUALLY emits and requires every value to be an int or a
member of a set assembled from the producer's own declarations — so a free-text
channel fails it whether or not anyone thought to forbid the thing travelling
down it. Thirteen mutations were run against it; each reddened, and each anchor
was asserted to have applied. One of them (smuggling the raw command into a
second attribute) was caught only by the telemetry-side test and NOT by the
cli-side one, which was inspecting a single key — that test now sweeps the whole
payload.

WHAT IS NOT CONNECTED. The transport. The ticket says "rides the gateway and
token"; the 17 Aug decision (rfcs#28) replaced the Collector gateway with an
ingest endpoint on the backend, which is backend#1905 and does not exist yet.
pendingSink() returns nil, so every event is validated and dropped. Validation
runs regardless, so a malformed event fails in CI wherever the binary was built,
and connecting #1905 is one function body.

Opt-out (default on) via TRACEBLOC_NO_TELEMETRY or DO_NOT_TRACK, documented in
docs/troubleshooting.md — and the document's claim about which variables work is
itself a test, because a user who exports a stale name believes they have opted
out and nothing else would ever tell them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

@LukasWodka

Copy link
Copy Markdown
Contributor Author

Sibling PR for the installer half of backend#1907: tracebloc/client#747. Independent — neither depends on the other, and the transport for both is tracebloc/backend#1905.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit efff633. Configure here.

@saqlainsyed007 saqlainsyed007 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review — approve.

Reviewed the full diff against the surrounding code it leans on (internal/telemetry/telemetry.go, internal/cli/exitcodes.go, internal/api env helpers, internal/config loader, cmd/tracebloc/main.go).

What I checked and why it holds:

  • Event grammar & vocabularycli.command.{succeeded,failed,cancelled} all pass the emitter's <domain>.<object>.<outcome> grammar and closed outcome set.
  • Exit-code → error classexitClasses (1–9) matches the frozen meanings in exitcodes.go; the shared codes (2/5/7) collapse to one bucket as intended; unmapped → unclassified (fail-closed).
  • Cancel path — the interrupted *exitError is non-nil-outer / nil-inner, so ExitCodeFromError yields 130, Record maps it to cancelled, and no error.type is attached (cancel isn't a failure outcome, so Emit doesn't require one).
  • Privacy boundary — command value is a set-membership lookup against paths enumerated after ExecuteContextC returns (so help/completion are enumerated on the same tree that dispatched, and commandPathOf(executed) can't be spuriously unregistered); the class is keyed on an int. No free-text channel, matching the PR's structural claim.
  • Env resolution — an unknown/staging value falls back and then fails closed at the emitter's Exports() gate rather than being guessed.
  • No new side effect — routing telemetry through config.Load() is read-only (v1 migration is in-memory; only a later Save writes), so recording on every invocation doesn't touch disk.
  • Duration — monotonic time.Since, clamped ≥0, zero preserved.

Tests are mutation-anchored, all 27 CI checks are green (Test, Lint, golangci-lint, govulncheck, house-rules, Bugbot), and I found no correctness defect. Nice, careful work — the "closed set, not a sanitiser" framing makes the privacy guarantee auditable.

@LukasWodka
LukasWodka merged commit 885c5c3 into develop Aug 18, 2026
28 checks passed
@LukasWodka
LukasWodka deleted the feat/1907-cli-command-outcome-telemetry branch August 18, 2026 10:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants