Skip to content

test(otel): cover Karpenter node-label and pod-label counterpart removal - #775

Open
petruanica wants to merge 5 commits into
aws:mainfrom
petruanica:otel-ci-drop-karpenter-labels
Open

petruanica wants to merge 5 commits into
aws:mainfrom
petruanica:otel-ci-drop-karpenter-labels

Conversation

@petruanica

@petruanica petruanica commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Description of the issue

The Container Insights attribute denylist in awsattributelimit/cw_k8s_ci_v0 is gaining 25 Karpenter node labels plus five pod labels, and had no integration coverage. The karpenter.k8s.aws/ and karpenter.sh/ domains are deliberately not denied by prefix, because 8 keys in them are retained — the 22 instance labels are denied by the narrower karpenter.k8s.aws/instance- prefix, which no retained key matches, and the 3 karpenter.sh/ keys are enumerated. So the interesting failure mode is not "a key survived" but "a retained key was removed too", which nothing tested.

Description of changes

Adds four tests to the standard suite, next to the existing nfd_labels_dropped subtest of TestNodeLabelEnrichment (node-feature-discovery, the feature.node.kubernetes.io/ family), which exercises the same denylist path. Two cover the Karpenter keys and two cover the pod-label counterparts.

TestKarpenterInstanceLabelsRemoved asserts that none of the 25 denied keys reach resource scope on any node-label-enriched metric. 22 are karpenter.k8s.aws/instance-* keys that restate host.type; 3 are karpenter.sh/ node-management flags — initialized, registered and do-not-sync-taints.

TestKarpenterGroupingLabelsRetained asserts that the three keys Karpenter sets on every node it manages — karpenter.sh/nodepool, karpenter.sh/capacity-type and karpenter.k8s.aws/ec2nodeclass — still arrive on at least one node. It requires one node rather than every node, because a cluster can mix Karpenter-managed nodes with a static node group.

The pair is the point: a regression that widened the narrow instance- prefix to the whole karpenter.k8s.aws/ and karpenter.sh/ domains would satisfy the first test and fail the second. The retention test gates on the result set it is already checking — it collects the nodes carrying any k8s.node.label.karpenter.* attribute and requires that set to be non-empty. So the suite needs at least one Karpenter-managed node and fails rather than skips without one. That is deliberate: a domain-prefix regression strips every Karpenter attribute, and a skip-on-absence gate would hide exactly the case the test exists to catch.

capacity-reservation-id, -type, -interruptible and placement-group-id, -partition are retained by the denylist but not asserted, because Karpenter only sets them on nodes launched into a capacity reservation or a placement group, so they are absent from an ordinary cluster.

TestPodLabelCounterpartsRemoved asserts none of the five new k8s.pod.label.* keys reach resource scope on any label-enriched metric: the four pod-side counterparts of node labels the denylist already removed — topology.kubernetes.io/region, topology.kubernetes.io/zone, helm.sh/chart, release — plus pod-template-generation.

TestPodLabelsStillEnriched is the complement, and the same shape of guard as the Karpenter pair. It requires at least one k8s.pod.label.* attribute other than the five denied keys to survive, so widening the denylist to the k8s.pod.label. prefix fails here instead of quietly satisfying the removal assertion while stripping every pod label. It iterates the pod-scoped metric names only, since node-scoped metrics have no pod and so never carry k8s.pod.label.*.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

gofmt clean and go vet -tags integration ./test/otel/standard/ passes; the test binary builds with all four test functions registered.

These are integration tests, so they need a live cluster running the paired configuration change. They will fail against a cluster that has not yet taken the denylist update — that is the intended behaviour, and the reason this PR is paired one-to-one with the config change rather than bundled with the other Container Insights test additions.

Related pull requests

Chart change aws-observability/helm-charts#383
Sibling Container Insights changes #776, #777

PR Checklist

  • Commits are squashed into a logical, reviewable set (one commit for a single change)
  • Commits and PR description comply with Amazon internal guidelines
  • gofmt clean, go vet -tags integration passes, test binary builds
  • All GitHub Actions checks on the PR are passing
  • Integration test evidence: N/A — this PR is the integration test coverage. The tests require a cluster running the paired configuration change and will fail until it is deployed, which is intentional.
  • New or updated integration test coverage: N/A — this is the corresponding amazon-cloudwatch-agent-test PR
  • New functionality has unit tests; bug fixes have a reproducing test — N/A, test-only change
  • Config translation changes include updated golden files — N/A, no translator in this repo
  • Breaking or customer-visible changes are called out in the PR description — N/A, test-only change

Adds two integration tests for the Container Insights Karpenter node-label denylist.

TestKarpenterInstanceLabelsRemoved asserts that none of the 25 denied karpenter.k8s.aws/instance-* and karpenter.sh/ keys reach CloudWatch on any node-label-enriched metric.

TestKarpenterGroupingLabelsRetained asserts the complement: the three keys the denylist deliberately keeps -- karpenter.sh/nodepool, karpenter.sh/capacity-type and karpenter.k8s.aws/ec2nodeclass -- are still present on at least one node. It gates on karpenterNodeNames, which collects the nodes carrying any k8s.node.label.karpenter.* attribute from the result set already being checked, and requires that set to be non-empty. Finding no karpenter.* attribute at all fails rather than skips, because that is the signature of a denylist switched from exact keys to the karpenter.k8s.aws/ and karpenter.sh/ prefixes, which would strip the retained keys along with the denied ones.

Requiring at least one node per retained key rather than every node lets the tests run on a cluster that mixes Karpenter-managed nodes with a static node group.
Adds two integration tests for the six node and pod label keys added to the Container Insights denylist alongside the Karpenter keys.

TestPodLabelCounterpartsRemoved asserts none of the six reach resource scope on any node-label-enriched metric: the four pod-side counterparts of node labels the denylist already removed, plus topology.k8s.aws/zone-id and pod-template-generation.

TestPodLabelsStillEnriched is the complement. It requires at least one k8s.pod.label.* attribute other than the six denied keys to survive somewhere in the result set, so widening the denylist to the k8s.pod.label. prefix fails here rather than silently satisfying the removal assertions while stripping every pod label. That is the same failure mode the Karpenter pair guards against for the karpenter.k8s.aws/ and karpenter.sh/ prefixes.
@petruanica petruanica changed the title test(otel): cover Karpenter node-label removal and retention test(otel): cover Karpenter node-label and pod-label counterpart removal Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant