Mark otel performance suite WIP, drop lower bounds, and set up stats experimenting

[fareedah999]

Description of the issue

The otel performance test was added last week and due to the tight thresholds calibrated, it is not passing. Raising this PR to put in WIP while the issues are sorted.

Description of changes

• Marked the suite WIP (wip: true) and pinned instanceType: t3.medium in the generator entry. WIP so failing runs are overruled in CI while the stat/thresholds are still being tuned; the instance type is pinned because the regression baseline and %-of-node thresholds are keyed to node size and shouldn't drift with the default.
• Gave cwagent_image_tag an empty-string default (removed the validation block). Matches the other otel suites having a default, but empty so a real run still fails without a tag via the existing require.NotEmpty(cwaCommitSha) check. Needed so the shared terraform destroy (which passes no -var) doesn't prompt and hang.
• Removed the lower bounds so threshold test no longer fails below the band (upper-bound only); regression test no longer fails on large drops (only growth > 30% fails now).
• Changed the sampling step from 30s to 1s, to get more values.
• Added a summaryStat selector + percentile/expectedShortfall helpers for experimentation. This will allow us find the best stat to compare. Originally avg + max; now also p90, p95, p99, es90, es95 (es = expected shortfall/CVaR). Added per-pod logging of avg/max/p95/p99/es95 side by side so one run shows all stats. We need to find a stat that is stable, not too noisy, and a good representation of CPU/mem usage.
• Both tests currently gate on p95.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

go vet -tags=integration ./test/otel/performance/... — clean.
go test -c -tags=integration ./test/otel/performance/... — the test binary compiles.
gofmt -l test/otel/performance/ generator/ — no formatting issues.

[fareedah999]

Added zero handling on two layers in the second commit.

• Skip all-zero series: If a pod's entire series over the window is zeros, that means no data was collected for it, so it's skipped rather than scored, otherwise those zeros would drag the stats down. A series with a mix of zeros and real values is kept and scored normally (

• A zero still fails — we don't silently pass on "0".
  Threshold test: if a scored pod's computed statistic comes out as 0, it fails with "no data collected for this pod."
  Regression test: if skipping the all-zero series leaves a whole pod class with no usable data, the existing presence checks fail the test ("no DaemonSet/scraper series observed").