traefik_otel: aggregate counter metrics with the ES|QL TS command and scope cumulativetodelta to histograms

[giorgi-imerlishvili-elastic]

Proposed commit message

WHY

traefik_otel 0.3.1 and its documented collector configuration disagree about the aggregation
temporality of Traefik's Prometheus counters, and ES|QL will not let both be right at once:

Field mapping	Works	Rejected
time_series_metric: counter	TS + RATE() / INCREASE()	plain SUM() / MAX()
time_series_metric: gauge	plain SUM() / MAX()	RATE() / INCREASE()

The mapping is decided by the collector. The Elasticsearch exporter maps a monotonic sum as a
counter only while its temporality is still cumulative:

// exporter/elasticsearchexporter/internal/datapoints/number.go
isCounter := sum.IsMonotonic() && sum.AggregationTemporality() == pmetric.AggregationTemporalityCumulative

The README applied cumulativetodelta with metrics: ['traefik_.*', 'go_.*', 'process_.*'], which
is wide enough to catch every counter and convert it to a delta sum, so it lands as a gauge. The
dashboards' FROM + SUM() queries only work in that state. Any user who omits or narrows the
processor gets counter_double fields and 27 assets that fail on load with:

line 1:48: argument of [SUM(traefik_entrypoint_requests_total)] must be
[aggregate_metric_double, exponential_histogram, tdigest or numeric except unsigned_long
or counter types], found value [traefik_entrypoint_requests_total] type [counter_double]

That the counter contract was the original intent is visible in the shipped alert rules, which carry
this comment directly above a query doing the opposite:

// Limit to entrypoint-level request counters (counter requires TS + INCREASE)
| WHERE traefik_entrypoint_requests_total IS NOT NULL
| STATS total = SUM(traefik_entrypoint_requests_total), ...

WHAT

This standardises on the counter contract, which is the idiomatic representation for Prometheus
counters and handles counter resets correctly.

1. Rewrite 27 ES|QL assets (24 dashboard panels, 3 alerting rule templates) to the TS source
   command with counter-aware functions: RATE() where the panel reported a per-second rate (it
   previously divided a SUM() by a DATE_DIFF window), INCREASE() where it reported a total
   over the selected range, and MAX(LAST_OVER_TIME()) for the cumulative "Total Reloads" metric.
   Output column names, bucketing and result shapes are unchanged so the Lens configurations keep
   working. Both the _dev/shared/kibana/*.yaml authoring sources and the generated
   kibana/dashboard/*.json are updated so regeneration cannot reintroduce the old queries.
2. Narrow the README's cumulativetodelta filter to ['traefik_.*_request_duration_seconds']. The
   processor is genuinely required, but only for the request-duration histograms, which the
   Prometheus receiver emits as cumulative and the exporter drops in otel mapping mode. Applying
   it to the counters was collateral damage.
3. Add a troubleshooting section, since a field's time-series type is fixed when the backing index is
   created. Recovery needs a rollover, and because documents are routed by @timestamp the previous
   backing index keeps accepting writes until its index.time_series.end_time has passed.

Gauge-backed panels and rules are untouched. The SLO templates aggregate through the DSL rather than
ES|QL and were never affected.

TS, RATE, INCREASE and LAST_OVER_TIME are GA since 9.4, matching the package's existing
kibana.version: ^9.4.0 constraint, so no constraint change is required.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Confirm that standardising on the counter contract is the direction we want, rather than keeping the delta contract and fixing only the documentation.
• Decide whether this warrants a breaking-change note and a minor bump instead of the current patch bump to 0.3.2. Existing users following the old README have gauge-mapped counters and will need both the narrowed collector filter and a data stream rollover.
• Confirm ['traefik_.*_request_duration_seconds'] covers every histogram metric that the latency SLO template depends on (traefik_service_request_duration_seconds).
• Confirm the rewritten panels render as expected in Kibana, in particular that RATE() vs the previous SUM()/DATE_DIFF() produces the intended values on the "Over Time" panels.

How to test this PR locally

Executed on this branch (Elasticsearch 9.5.0-SNAPSHOT via elastic-package stack up, OTel-native metrics-traefik.otel-* fixture with counter-mapped fields):

Every dashboard panel and alerting rule query was run against the stack, on the base commit and on this branch:

base main : TOTAL: ok=19 fail=27
this branch: TOTAL: ok=46 fail=0

Rewritten panels return values consistent with the analytically known rates in the fixture:

Overview / Bandwidth (bytes/sec)                394792.960  394992.529  0.05%  PASS
Services / Total Service Requests (req/sec)        135.460     135.299  0.12%  PASS
TLS / TLS Request Rate (req/sec)                    72.000      71.709  0.40%  PASS
Process / CPU Rate (cores)                           0.350       0.347  0.72%  PASS
Overview / Requests by Status Code, 200 (count) 594405.000  593568.371  0.14%  PASS
failures: 0

The YAML authoring sources reconstruct the JSON queries byte for byte, so regeneration is safe:

checked=37 mismatched=0

Package validation:

elastic-package lint   -> Done
elastic-package check  -> Package built: build/packages/traefik_otel-0.3.2.zip

The mapping claim behind the README change was verified with a real
otel/opentelemetry-collector-contrib:0.160.0 scraping a Prometheus endpoint into the same cluster,
varying only the cumulativetodelta filter:

Filter	Resulting time_series_metric	SUM(field)	SUM(RATE(field))
Broad, matches counters (old README)	gauge	works	fails
Scoped to .*_request_duration_seconds (new README)	counter	fails	works

For the reviewer (not yet run in this session):

1. Install the built package and configure a collector using the README snippet in this PR, pointed at a Traefik instance with metrics.prometheus enabled.
2. Confirm GET /metrics-traefik.otel-default/_mapping/field/*requests_total* reports time_series_metric: counter.
3. Open the Overview, Services, TLS & Config and Process dashboards and confirm no panel shows verification_exception and that rate panels show plausible values.
4. Repeat on a 9.4.x stack, the minimum the package supports and the version reported in the SDH.

Related issues

• Relates elastic/obs-integration-team#1205
• Relates elastic/sdh-beats#7557

Screenshots

Pending — the four affected dashboards should be captured against a counter-mapped data stream before this leaves draft.

[github-actions]

Elastic Docs Style Checker (Vale)

Summary: 1 warning, 1 suggestion found

⚠️ Warnings (1): Fix when the suggestion improves clarity or correctness.

File	Line	Rule	Message
packages/traefik_otel/docs/README.md	79	Elastic.QuotesPunctuation	Place punctuation inside closing quotation marks.

💡 Suggestions (1): Optional style improvements. Apply when helpful.

File	Line	Rule	Message
packages/traefik_otel/docs/README.md	79	Elastic.Ellipses	Use ellipses sparingly. Remove the ellipsis unless it appears in UI text.

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 6d52dc8

cc @giorgi-imerlishvili-elastic