Skip to content

Add OTel Container Insights filters EKS integration test - #782

Open
louisall wants to merge 1 commit into
mainfrom
otel-ci-filters-e2e
Open

louisall wants to merge 1 commit into
mainfrom
otel-ci-filters-e2e

Conversation

@louisall

@louisall louisall commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Description of the issue

The Helm chart adds otelContainerInsights.filters for metrics and application logs: namespaces (aws-observability/helm-charts#393), node and pod labels (aws-observability/helm-charts#394) and metric names (aws-observability/helm-charts#395). Those PRs test how the values translate into the generated OTel config. This PR checks end to end, on a real EKS cluster, that the filtered telemetry reaching CloudWatch is what the values ask for.

Merge order: aws-observability/helm-charts#393, aws-observability/helm-charts#394 and aws-observability/helm-charts#395 (stacked in that order) merge first. The Terraform clones the chart from helm_chart_branch (default main), so this suite fails in CI until they land.

Description of changes

  • terraform/eks/daemon/otel-filters, patterned on otel-attr-limit: its own cluster with one 2-node group carrying test node labels, the chart installed with an include and an exclude (at least one prefix each) for every filter setting, and a logging workload with test pod labels in each of 4 namespaces (2 kept, 2 filtered).
  • test/otel/filters:
    • Namespaces: kube_pod_info (kube-state-metrics, cluster scraper), container_cpu_usage_seconds_total (cAdvisor) and k8s.pod.cpu.usage (kubelet stats) are present for included namespaces and absent for excluded ones. Application logs are present and absent per namespace in the same way.
    • Labels, for metrics and logs: included labels are present and excluded or not-included labels are absent. pod-template-hash is included but is also a recommended exclusion, so it is absent from metrics (default on) and present on logs (default off).
    • Metric names: included names, including the dotted prefix k8s.pod.*, are present. Excluded and not-included names are absent, from both the node agent and the cluster scraper (kube_pod_status_phase).
    • Each absence check runs only after the matching kept data has been seen, so a pipeline that is not reporting fails the test instead of passing it.
    • Metrics are read with client.Query per name rather than queryCache: the absence checks need fresh results, and the source registry expects every metric of a source, which a metric name filter removes by design.
  • Generator entry under eks_daemon, next to otel/attr_limit.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

PR Checklist

  • Commits are squashed into a logical, reviewable set (one commit for a single change)
  • Commits and PR description comply with Amazon internal guidelines
  • make passes locally (build, unit tests, lint): go vet, terraform fmt and terraform validate
  • All GitHub Actions checks on the PR are passing
  • Integration test evidence: link to a passing run: https://github.com/aws/amazon-cloudwatch-agent/actions/runs/37945145915
  • New or updated integration test coverage: this PR
  • New functionality has unit tests; bug fixes have a reproducing test: N/A, test-only
  • Config translation changes include updated golden files: N/A
  • Breaking or customer-visible changes are called out in the PR description: none

Install the chart with an include and an exclude, each with at least one
prefix, for every otelContainerInsights.filters setting, and check in
CloudWatch that kept namespaces, labels and metric names are present and
filtered ones are absent, for metrics and application logs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant