[petruanica]

Important

This is a breaking change and must ship in a major version bump. It removes 25 node-label dimensions from every Container Insights metric emitted from a Karpenter-managed node. Any dashboard, alarm, or query that selects or groups on one of them stops matching the moment the new config rolls out. There is no deprecation window and no per-customer opt-out in this change: the attributes are dropped at the processor, so they disappear in a single release. The chart counterpart therefore needs a major chart version bump (6.x to 7.0.0), and neither change should be picked into a minor or patch release.

Note

Release coordination (working assumption, not yet decided). This PR is written assuming it ships together with the two sibling Container Insights changes in a single major chart release, v7.0.0. That bundling has not been agreed yet — it is recorded here so reviewers know what the description assumes, and it needs confirming before any of the three merge. If they end up shipping separately, each breaking change still needs its own major bump.

Description of the issue

Karpenter applies its well-known labels to every node it manages, and the Kubernetes attribute enrichment in the OTel Container Insights pipelines copies them onto the resource block — so their key text is attached to every datapoint of every series emitted from a Karpenter-managed node, across all receivers. Most of them are derived from instance type details, so they restate host.type, which these pipelines already publish.

The awsattributelimit/cw_k8s_ci_v0 denylist already removes comparable platform-injected label families (feature.node.kubernetes.io/, beta.kubernetes.io/, alpha.eksctl.io/, sourceLaunchTemplateId). The Karpenter equivalents were never added.

Description of changes

Adds 25 Karpenter node-label keys to awsattributelimit/cw_k8s_ci_v0.unconditional_removal_keys — 22 karpenter.k8s.aws/instance-* and 3 karpenter.sh/ lifecycle labels — across the 12 Container Insights pipeline configs that carry the processor and the 7 translator/tocwconfig/sampleConfig/opentelemetry/ golden files that contain it. The two filelog_* pipelines have no attribute-limit processor and are unchanged. Additive only, 25 lines per file, no Go changes.

These labels appear only as resource attributes, so unconditional_removal_keys — which is resource-scoped — is sufficient; no datapoint-level delete_key transform is needed.

What is removed and what is kept

The list is now complete against upstream's WellKnownLabels. Karpenter defines 28 labels in the karpenter.k8s.aws/ domain and 5 in karpenter.sh/; this denies 22 and 3 of them respectively, and retains the remaining 8 deliberately.

Retained, because they are not instance properties and are legitimate grouping dimensions:

• karpenter.sh/nodepool, karpenter.k8s.aws/ec2nodeclass, karpenter.sh/capacity-type — provisioning identity, with capacity-type the only signal distinguishing spot from on-demand capacity.
• karpenter.k8s.aws/capacity-reservation-id, -type, -interruptible — which capacity reservation a node consumed, whether it is an ordinary On-Demand Capacity Reservation or a Capacity Block, and whether EC2 may reclaim it. Not derivable from the instance type.
• karpenter.k8s.aws/placement-group-id, -partition — which EC2 placement group the instance launched into, and for partition placement groups which partition. A partition is an independent failure domain, so this is a physical fault boundary rather than an instance property.

That is also why the change lists exact keys rather than adding karpenter.k8s.aws/ and karpenter.sh/ to unconditional_removal_prefixes, which would strip all 8 retained keys as well.

GPU and accelerator labels

The four instance-gpu-* and three instance-accelerator-* keys were added after review feedback. Karpenter populates them conditionally — computeRequirements sets the GPU labels only when the instance type reports GPU info, and the accelerator labels only for Inferentia or Neuron devices — so they are absent from clusters with no accelerated nodes and were missed when the list was first assembled. They are the same class as instance-cpu and instance-memory: properties of the instance type that restate host.type.

GPU and Neuron dashboards are unaffected, because device identity travels on the DCGM_FI_DEV_* and neuroncore_* metrics' own datapoint attributes, which this change does not touch. Note that the Neuron branch also populates the accelerator labels, so this covers Trainium and Inferentia2 as well as Inferentia1.

Breaking change and release sequencing

Customers see the removed dimensions on two surfaces: the k8s_node_label_karpenter_* metric label, and the @resource. prefix in PromQL. Both stop resolving for the 25 denied keys. Concretely, a query grouping by karpenter.k8s.aws/instance-family to compare utilisation across instance families returns a single unlabelled series afterwards, and an alarm scoped to a specific instance-size stops matching any series at all.

Because the keys are listed in a static config with no conditional and no values-level toggle, a customer who relies on one of them has no way to retain it in the release that removes it. That is the reason this needs a major version bump rather than a minor, and it should be called out in the release notes alongside the chart counterpart.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

go test ./translator/tocwconfig/ — the golden tests covering the edited sample configs pass (TestContainerInsightsConfig, TestContainerInsightsAKSGKEConfig, TestDefaultOtelConfigAKSTranslation, TestDefaultOtelConfigGKETranslation), rendering the pipeline templates and diffing them against the goldens.

New integration coverage in aws/amazon-cloudwatch-agent-test#775. That PR also carries two tests for the pod-label counterpart keys, which this change does not implement; they assert behaviour introduced by the chart counterpart and will fail until the equivalent keys land here too. The two tests covering this change: TestKarpenterInstanceLabelsRemoved asserts none of the 25 denied keys reach resource scope on any node-label-enriched metric, and TestKarpenterGroupingLabelsRetained asserts the three retained keys still arrive. The pair is what makes the exact-keys decision testable — a regression replacing the 25 entries with the karpenter.k8s.aws/ and karpenter.sh/ prefixes would pass the first and fail the second. The retention test requires at least one node in the result set to carry a k8s.node.label.karpenter.* attribute, so a change that stripped every Karpenter attribute fails rather than skips.

Both were validated on a live EKS cluster, since the claim is about what arrives in CloudWatch rather than what the config renders to. The changed configuration was applied to the running agent and cluster scraper, the pods were restarted, and metrics were given 30 minutes to settle; test/otel/standard -computeType=EKS then asserts on datapoints read back from CloudWatch. The suite was also run before the change, because an "attribute is absent" assertion passes trivially against empty data.

Assertion	Before	After
TestKarpenterInstanceLabelsRemoved	18 fail	18 pass
TestKarpenterGroupingLabelsRetained (guard)	18 pass	18 pass

No failures, no skips. The guard passing both before and after shows only the intended keys were removed, and establishes that the result set genuinely contained Karpenter labels.

Requirements

Before commiting your code, please do the following steps.

1. Run make fmt and make fmt-sh
2. Run make lint

Related pull requests

Chart change	aws-observability/helm-charts#383
Integration tests	aws/amazon-cloudwatch-agent-test#775
Sibling Container Insights changes	#2321, #2322

Scope note: this change covers the Karpenter keys only; the chart counterpart additionally denies six pod-label counterpart keys, so the two are not yet identical.

PR Checklist

• Commits are squashed into a logical, reviewable set (one commit for a single change)
• Commits and PR description comply with Amazon internal guidelines
• make passes locally (build, unit tests, lint)
• All GitHub Actions checks on the PR are passing
• Integration test evidence: validated end-to-end on a live EKS cluster; baseline and post-change assertion counts are in the Tests section above. There is no public CI link because the integration suite needs a provisioned cluster and credentials that fork PRs cannot reach.
• New or updated integration test coverage: test(otel): cover Karpenter node-label and pod-label counterpart removal amazon-cloudwatch-agent-test#775
• New functionality has unit tests; bug fixes have a reproducing test
• Config translation changes include updated golden files
• Breaking or customer-visible changes are called out in the PR description

---

Integration Tests

To run integration tests against this PR, add the ready for testing label.

[louisall]

Call out a bit more that this is breaking and has to be part of major version bump. Just highlight it more in description