Mark otel performance suite WIP, drop lower bounds, and set up stats experimenting

[fareedah999]

Description of the issue

The otel performance test was added last week and due to the tight thresholds calibrated, it is not passing. Raising this PR to put in WIP while the issues are sorted.

Description of changes

• Marked the suite WIP (wip: true) and pinned instanceType: t3.medium in the generator entry. WIP so failing runs are overruled in CI while the stat/thresholds are still being tuned; the instance type is pinned because the regression baseline and %-of-node thresholds are keyed to node size and shouldn't drift with the default.
• Gave cwagent_image_tag an empty-string default (removed the validation block). Matches the other otel suites having a default, but empty so a real run still fails without a tag via the existing require.NotEmpty(cwaCommitSha) check. Needed so the shared terraform destroy (which passes no -var) doesn't prompt and hang.
• Removed the lower bounds so threshold test no longer fails below the band (upper-bound only); regression test no longer fails on large drops (only growth > 30% fails now).
• Changed the sampling step from 30s to 1s, to get more values.
• Added a summaryStat selector + percentile/expectedShortfall helpers for experimentation. This will allow us find the best stat to compare. Originally avg + max; now also p90, p95, p99, es90, es95 (es = expected shortfall/CVaR). Added per-pod logging of avg/max/p95/p99/es95 side by side so one run shows all stats. We need to find a stat that is stable, not too noisy, and a good representation of CPU/mem usage.
• Both tests currently gate on p95.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

go vet -tags=integration ./test/otel/performance/... — clean.
go test -c -tags=integration ./test/otel/performance/... — the test binary compiles.
gofmt -l test/otel/performance/ generator/ — no formatting issues.

PR Checklist

• Commits are squashed into a logical, reviewable set (one commit for a single change)
• Commits and PR description comply with Amazon internal guidelines
• make passes locally (build, unit tests, lint)
• All GitHub Actions checks on the PR are passing
• Integration test evidence: link to a passing run, or state N/A with the reason
  N/A — EKS OTel performance suite is intentionally marked wip: true while the
  gating statistic and thresholds are tuned, so it cannot produce a passing run yet.
  New pure-logic helpers are covered by offline unit tests (stats_test.go).
• New or updated integration test coverage: link to the corresponding
  amazon-cloudwatch-agent-test PR, or state N/A
  N/A — this PR is the amazon-cloudwatch-agent-test change (test-harness only).
• Related PRs in any repo are linked in this description and link back to this PR,
  with any required merge or release order stated; or state N/A
  N/A — no related PRs in other repos; self-contained test-suite change.
• New functionality has unit tests; bug fixes have a reproducing test
  Pure stat helpers (percentile, expectedShortfall, summaryStat, calcStats,
  isAllZero) extracted to untagged stats.go with table-driven tests in stats_test.go.
• Config translation changes include updated golden files
  N/A — no translator/ changes.
• Breaking or customer-visible changes are called out in the PR description
  N/A — test-only; no customer-visible or agent-runtime change.

[fareedah999]

Added zero handling on two layers in the second commit.

• Skip all-zero series: If a pod's entire series over the window is zeros, that means no data was collected for it, so it's skipped rather than scored, otherwise those zeros would drag the stats down. A series with a mix of zeros and real values is kept and scored normally (

• A zero still fails — we don't silently pass on "0".
  Threshold test: if a scored pod's computed statistic comes out as 0, it fails with "no data collected for this pod."
  Regression test: if skipping the all-zero series leaves a whole pod class with no usable data, the existing presence checks fail the test ("no DaemonSet/scraper series observed").