#39 Add OpenTelemetry

[matthewdevenny]

Summary

Adds first-class metrics and tracing to NatsCache via System.Diagnostics — a named Meter with hit/miss, latency and error instruments, and a named ActivitySource for spans around KV operations. Closes #39.

No OpenTelemetry package dependency. Consumers opt in by registering the documented names. Also bumps NATS.Net to 3.0.1 and the integration-test server to 2.14.3.

This is genuinely additive rather than duplicating the platform: neither IDistributedCache nor HybridCache exposes backend cache metrics. Verified by disassembly — Microsoft.Extensions.Caching.Hybrid 10.7.0 reports through EventCounters on HybridCacheEventSource, not a Meter, so AddMeter("Microsoft.Extensions.Caching.Hybrid") yields nothing. Even a future Meter there could not carry NATS-specific L2 latency, error types, or miss reasons.

What's new

• NatsCacheTelemetryNames — public const names for the meter, activity source, and both instruments, so consumers never hardcode strings:

  builder.Services.AddOpenTelemetry()
      .WithMetrics(metrics => metrics.AddMeter(NatsCacheTelemetryNames.MeterName))
      .WithTracing(tracing => tracing.AddSource(NatsCacheTelemetryNames.ActivitySourceName));

• NatsCacheOptions.Telemetry (NatsCacheTelemetryOptions) — currently one knob, RecordCacheKeys (default false). Get-only so it can never be null.

Instruments

Name	Type	Unit	Tags
nats.cache.operation.duration	Histogram<double>	s	nats.cache.operation, nats.cache.bucket, nats.cache.result, error.type (errors only)
nats.cache.misses	Counter<long>	{miss}	nats.cache.operation, nats.cache.bucket, nats.cache.miss.reason

nats.cache.result is hit | miss | ok | error | cancelled. nats.cache.miss.reason is a closed set of not_found | expired | undeserializable | revision_conflict — one per return null site, never derived from data.

The histogram's count gives operation rate, hit ratio and error rate, so there is no companion operations counter — one that duplicates a histogram's count is an OTel anti-pattern (the HTTP conventions deleted exactly such a counter). Conversely miss.reason lives only on the counter, so a 4-valued dimension never multiplies the histogram's bucket arrays.

Spans are ActivityKind.Client, named {operation} {bucket}. A miss leaves the status Unset; only genuine failures set Error.

Behavior / design notes

• No double counting. Only three methods are instrumented — SetAsync(byte[]), GetAndRefreshAsync, RemoveAsync — each the innermost layer every caller funnels through. RemoveCoreAsync is deliberately not instrumented: the absolute-expiry eviction path calls it, so instrumenting the core would emit a phantom operation=remove on every expired read and nest a bogus remove span inside a get. Set(ReadOnlySequence) is three delegations deep and TryGetAsync wraps the read core — the two chains HybridCache drives for L2 — and both record exactly once.
• Instrument naming. No stable OTel semantic convention for caches exists, so nats.cache.* is a library-scoped prefix. db.client.* was rejected: NATS KV has no registered db.system.name value, the DB conventions cannot express a hit/miss, and emitting it would merge cache traffic into every consumer's database dashboards. A future cache.* convention can be adopted alongside these names without breaking dashboards.
• Meter creation. IMeterFactory when DI provides it, a process-wide static meter of the same name as fallback, injected via init like the existing TimeProvider. This costs no new dependency — IMeterFactory ships in System.Diagnostics.DiagnosticSource, already in the lock files; the explicit PackageReference added here only flips it Transitive → Direct and adds zero graph nodes. It buys per-container meter isolation, without which concurrent tests and multi-host processes cross-contaminate.
• Errors follow the existing exception policy. LogException / LogSwallowedException are untouched. A TryGet failure that is swallowed to false still records result=error, never miss — the scope closes inside the read core before the exception reaches TryGetAsync's catch — so an outage cannot masquerade as a cold cache.
• Cancellation. Only the caller's token cancelling reports result=cancelled. An OperationCanceledException from anything else — a NATS request timeout — is result=error, matching how TryGetAsync already classifies the same event with when (token.IsCancellationRequested) before logging it at Warning. The inverse would hide real outages behind the cancellation filter.
• Cache keys are never recorded on metrics under any setting. RecordCacheKeys adds them to spans only, defaulting off because keys routinely embed user or tenant identifiers.
• Telemetry cannot break a cache operation. The scope completes in a finally, and Meter/ActivitySource do not guard listener callbacks, so a throwing exporter would otherwise replace the in-flight exception or fail a successful operation. Each stage is isolated, and Activity.Current is restored explicitly — Activity.Stop notifies subscribers before restoring it, so a throwing subscriber would otherwise leave every later span misparented.
• Opt-in cost. No measurement recorded, no duration timed, and no per-operation allocation until a listener subscribes. Each cache instance does build its meter, two instruments and four span-name strings once on first use — a fixed setup cost, stated in the docs.

NATS 3.0.1 / server 2.14.3

The client bump needed zero source changes. Two cleanups it enabled: 3.0 moved JetStreamContext and Bucket onto INatsKVStore, so the (NatsKVStore) cast in SetAsync is gone, and both stale #852 todos are removed.

NatsExtensions.cs stays. Verified by reflecting over the released 3.0.1 assembly: native KV TTL still covers only CreateAsync / TryCreateAsync and PurgeAsync — there is no PutAsync-with-TTL (upsert, needed by Set) and no UpdateAsync-with-TTL (by-revision, needed by the sliding-expiration refresh). The #852 work stayed backed out through 3.0. That rationale now lives in the NatsExtensions class doc so it isn't re-derived on the next upgrade.

Docs

New README "Telemetry" section — wiring snippet, instrument and tag tables, miss-reason meanings, example PromQL for hit ratio and p99, and tuning via AddView(..., MetricStreamConfiguration.Drop) (the misses counter keeps recording when the histogram is dropped).

It also documents the non-obvious gotchas, each pinned by a test: keys never on metrics, TryGet failures as error not miss, TryGet reporting as operation=get, caller-cancellation vs timeouts, invalid-expiration throws not being counted, and the naming rationale. A HybridCache interaction note explains that nats.cache.* measures only the L2 layer — an L1 hit produces no measurement — and that HybridCache's own telemetry is EventCounters, not a Meter.

util/ReadmeExample gets a commented pointer; no ServiceDefaults project, which would pull OpenTelemetry packages into a repo that has none and imply the dependency this PR exists to avoid.

Testing

109 unit (net8.0 + net10.0) + 78 integration, all green, 0 build warnings; BOM, dotnet format, and lock-file drift checks clean. CI green on Linux including the nats:2.14.3 container.

Beyond happy paths, the suite pins what would otherwise regress silently:

• Unit — exactly-one-measurement across the sync and ReadOnlySequence chains; error.type tagging; refresh distinguishable from get; cancelled vs error classification for caller-cancellation vs a timeout-shaped TaskCanceledException; misses still recording while the histogram is disabled; keys absent from metrics with RecordCacheKeys both on and off; instrument units (guards seconds-vs-milliseconds, invisible until it reaches a dashboard); public name stability (this repo has no PublicAPI.Shipped.txt, so it is the only thing turning a rename into a visible failure); a throwing exporter breaking neither the operation nor Activity.Current; and an exact zero-byte allocation delta over 1,000 unsubscribed operations.
• Integration (real NATS via Aspire) — full set/hit/ok/miss result sequence; each miss reason including expired via a FakeTimeProvider clock jump (the entry keeps its real-time TTL while the cache's clock advances past it); no phantom remove on an expired read; HybridCache L2 read and write through the two riskiest overloads; per-ServiceProvider meter isolation; and end-to-end caller cancellation.

Not included

Deliberately deferred to follow-ups: entry-size histogram, connection / bucket-creation counters, per-key-prefix tags, and histogram bucket-boundary advice (InstrumentAdvice is .NET 9+ and would force an #if for the net8.0 target; the OTel SDK defaults match anyway).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds first-class cache telemetry (metrics + tracing) to NatsCache using System.Diagnostics primitives (Meter/ActivitySource) with OTel-friendly naming/tags, plus tests and documentation so consumers can opt-in via AddMeter(...) / AddSource(...).

Changes:

• Introduces NatsCacheTelemetry + NatsCacheOperationScope to emit a duration histogram, miss counter, and client spans with stable names/tags.
• Wires IMeterFactory from DI (when available) to avoid cross-test/process meter contamination; falls back to a shared process-wide Meter when absent.
• Adds unit + integration coverage for tagging, miss reasons, cancellation/error classification, and “no double counting” across overloads; documents usage and example queries.

Reviewed changes

Copilot reviewed 40 out of 40 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
util/ReadmeExample/DistributedCache.cs	Updates example to show how to subscribe to the cache’s Meter/ActivitySource via OpenTelemetry.
util/Benchmarks/packages.win-x64.lock.json	Updates lockfile for added DiagnosticSource dependency.
util/Benchmarks/packages.osx-arm64.lock.json	Updates lockfile for added DiagnosticSource dependency.
util/Benchmarks/packages.linux-x64.lock.json	Updates lockfile for added DiagnosticSource dependency.
util/Benchmarks/packages.linux-arm64.lock.json	Updates lockfile for added DiagnosticSource dependency.
test/UnitTests/UnitTests.csproj	Adds diagnostics testing packages for metric collection in unit tests.
test/UnitTests/packages.win-x64.lock.json	Updates lockfile for new test dependencies.
test/UnitTests/packages.osx-arm64.lock.json	Updates lockfile for new test dependencies.
test/UnitTests/packages.linux-x64.lock.json	Updates lockfile for new test dependencies.
test/UnitTests/packages.linux-arm64.lock.json	Updates lockfile for new test dependencies.
test/UnitTests/Cache/TelemetryUnitTests.cs	New unit tests asserting telemetry names, units, tagging, and disabled-path behavior without a NATS server.
test/TestUtils/Services/Diagnostics/RecordingActivityListener.cs	Adds a test helper to capture completed activities for span assertions.
test/TestUtils/packages.win-x64.lock.json	Updates lockfile for added DiagnosticSource dependency.
test/TestUtils/packages.osx-arm64.lock.json	Updates lockfile for added DiagnosticSource dependency.
test/TestUtils/packages.linux-x64.lock.json	Updates lockfile for added DiagnosticSource dependency.
test/TestUtils/packages.linux-arm64.lock.json	Updates lockfile for added DiagnosticSource dependency.
test/IntegrationTests/TestBase.cs	Registers AddMetrics() so integration tests get isolated meters per ServiceProvider.
test/IntegrationTests/packages.win-x64.lock.json	Updates lockfile for new testing dependencies.
test/IntegrationTests/packages.osx-arm64.lock.json	Updates lockfile for new testing dependencies.
test/IntegrationTests/packages.linux-x64.lock.json	Updates lockfile for new testing dependencies.
test/IntegrationTests/packages.linux-arm64.lock.json	Updates lockfile for new testing dependencies.
test/IntegrationTests/IntegrationTests.csproj	Adds diagnostics/time-provider testing packages for integration tests.
test/IntegrationTests/Cache/TelemetryTests.cs	New end-to-end telemetry integration tests (metrics + spans + HybridCache interactions).
src/NatsHybridCacheExtensions/packages.win-x64.lock.json	Updates lockfile for added DiagnosticSource dependency.
src/NatsHybridCacheExtensions/packages.osx-arm64.lock.json	Updates lockfile for added DiagnosticSource dependency.
src/NatsHybridCacheExtensions/packages.linux-x64.lock.json	Updates lockfile for added DiagnosticSource dependency.
src/NatsHybridCacheExtensions/packages.linux-arm64.lock.json	Updates lockfile for added DiagnosticSource dependency.
src/NatsDistributedCache/packages.win-x64.lock.json	Updates lockfile to add DiagnosticSource as a direct dependency.
src/NatsDistributedCache/packages.osx-arm64.lock.json	Updates lockfile to add DiagnosticSource as a direct dependency.
src/NatsDistributedCache/packages.linux-x64.lock.json	Updates lockfile to add DiagnosticSource as a direct dependency.
src/NatsDistributedCache/packages.linux-arm64.lock.json	Updates lockfile to add DiagnosticSource as a direct dependency.
src/NatsDistributedCache/NatsDistributedCacheExtensions.cs	Resolves IMeterFactory from DI (optional) and assigns it onto NatsCache.
src/NatsDistributedCache/NatsDistributedCache.csproj	Adds explicit System.Diagnostics.DiagnosticSource package reference.
src/NatsDistributedCache/NatsCacheTelemetryOptions.cs	Adds public telemetry options (currently RecordCacheKeys).
src/NatsDistributedCache/NatsCacheTelemetryNames.cs	Adds stable public Meter/ActivitySource/instrument name constants + docs.
src/NatsDistributedCache/NatsCacheTelemetry.cs	Implements per-cache telemetry bundle (Meter instruments + ActivitySource + span naming).
src/NatsDistributedCache/NatsCacheOptions.cs	Adds non-null Telemetry options property to configure span key recording.
src/NatsDistributedCache/NatsCacheOperationScope.cs	Adds the core scope type that records histogram/miss counter + tags and stops activities safely.
src/NatsDistributedCache/NatsCache.cs	Wires telemetry into Set/Get/Refresh/Remove paths with correct hit/miss/error/cancel classification and “no double count” design.
README.md	Documents telemetry names, instruments, tags, miss reasons, queries, and tuning guidance.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 61 out of 61 changed files in this pull request and generated 1 comment.

[mtmk]

LGTM. Seams are correct and well tested. One nit before merge: README PromQL uses nats_cache_operation_duration_count/_bucket, but the default OTel->Prometheus unit suffix makes the real series ..._seconds_count/_seconds_bucket (and misses -> nats_cache_misses_total), so the examples return nothing on a stock exporter. Fix names or note add_metric_suffixes.

[matthewdevenny]

Good catch — fixed in d3953be.

The default OTel Prometheus exporter (add_metric_suffixes = true) appends the unit and a _total suffix, so the real series are nats_cache_operation_duration_seconds_count / _seconds_bucket and nats_cache_misses_total. The examples now use those names, note the toggle for anyone running with suffixes disabled, and I added a miss-rate-by-reason query while there.