Repository navigation
feat(container-insights): drop deprecated and duplicated resource attributes - #2321
Closed
petruanica wants to merge 2 commits into
Closed
petruanica wants to merge 2 commits into
petruanica wants to merge 2 commits into
Conversation
…butes Agent-side counterpart to the equivalent change in the amazon-cloudwatch-observability Helm chart. The chart and this translator each carry their own copy of the CI pipeline config and must stay in sync. Deprecated semantic conventions. The Prometheus receiver injects net.host.name, net.host.port and http.scheme on every scrape target alongside their current-spelling twins server.address, server.port and url.scheme. Each pair carries an identical value on the same series. Add transform/cw_k8s_ci_v0_drop_deprecated_semconv, deleting the deprecated form and keeping the current one, to all ten Prometheus-receiver pipelines. This also corrects an inversion. karpenter.yaml and keda.yaml already deleted net.host.name and net.host.port in transform/..._solution_promote, but deleted url.scheme — the current name — while keeping http.scheme, the deprecated one. Those inline deletes are replaced by the shared processor, so all ten pipelines now agree. Doubled raw Prometheus labels. groupbyattrs moves each raw label to resource scope; the promote transforms then rename it to its semantic convention name and copy the raw name back down to the datapoint, but never remove the resource-level copy, leaving three billed copies of the same fact. Delete the resource-level copy in the five promote transforms that leave it behind: cadvisor, dcgm, kube-state-metrics, and the shared solution promote used by karpenter and keda. This brings them in line with the neuron, efa, ebs-csi and lis-csi promotes, which already promote-and-delete correctly. The datapoint copy is kept for dashboard compatibility and the semantic convention names are untouched, so no consumer loses a dimension. The new deletes sit in a trailing `context: resource` statement group that runs after the datapoint copy-back, since transform statement groups execute in declaration order. Nothing downstream reads the raw resource copies: k8sattributes keys its pod_association on k8s.pod.name / k8s.namespace.name / k8s.node.name, transform/cw_k8s_ci_v0_set_workload reads the k8s.*.name forms, and awsattributelimit keys on k8s.node.label.* and k8s.pod.label.* prefixes. Note that every CI pipeline defines these processors under the same name and they are merged into one collector config, so all copies must be identical or the merged result depends on pipeline order. Golden sample configs regenerated via the GENERATE_GOLDEN path, limited to the CI tests so the ~30 unrelated goldens that verifyToYamlTranslation also writes are left alone — rewriting those only reorders lists that the tests compare with cmpopts.SortSlices, so the churn would be pure noise. Verified with AWS_EC2_METADATA_DISABLED=true go test ./translator/... — 85 packages ok. Without that flag several unrelated tests fail on an IMDS lookup and TestOtlpMetricsConfigKubernetes panics, which aborts the test binary and silently skips everything after it; that behaviour is on main.
petruanica
force-pushed
the
otel-ci-l1-drop-redundant-attributes
branch
from
October 5, 2026 13:24
37d78ed to
79e28cd
Compare
This was referenced Oct 7, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Release coordination (working assumption, not yet decided). This PR is written assuming it ships together with the two sibling Container Insights changes in a single major chart release,
v7.0.0. This change is not itself breaking, but it is customer-visible, so bundling it with the major release avoids a customer-visible dimension change landing in a minor version. That bundling has not been agreed yet and needs confirming before merge.Description of the issue
Two groups of resource attributes on the Container Insights pipelines duplicate values that are already being sent.
The Prometheus receiver injects
net.host.name,net.host.portandhttp.schemeon every scrape target alongside their current-spelling twinsserver.address,server.portandurl.scheme; each pair carries an identical value on the same series.karpenter.yamlandkeda.yamlalready deleted some of these, but deletedurl.scheme— the current name — while keepinghttp.scheme, the deprecated one.Separately,
groupbyattrsmoves each raw Prometheus label to resource scope and the promote transforms then rename it to its semantic-convention name and copy the raw name back down to the datapoint, without removing the resource-level copy.Description of changes
Adds
transform/cw_k8s_ci_v0_drop_deprecated_semconv, deleting the three deprecated attributes and keeping their current-spelling twins, wired into all ten Prometheus-receiver pipelines. The inline deletes inkarpenter.yamlandkeda.yamlare replaced by it, which corrects the inversion described above. These processors are merged by name into one collector config, so all copies are kept identical.Deletes the resource-level raw copies in the five promote transforms that leave them behind: cadvisor, dcgm, kube-state-metrics, and the shared
transform/cw_k8s_ci_v0_solution_promoteused by karpenter and keda. This brings them in line with the neuron, efa, ebs-csi and lis-csi promotes, which already promote-and-delete.The datapoint copies are kept for dashboard compatibility and the semantic-convention names are untouched, so no consumer loses a dimension. Nothing downstream reads the raw resource copies:
k8sattributeskeys itspod_associationonk8s.pod.name/k8s.namespace.name/k8s.node.name,transform/cw_k8s_ci_v0_set_workloadreads thek8s.*.nameforms, andawsattributelimitkeys on thek8s.node.label.*andk8s.pod.label.*prefixes.The new delete statements sit in a trailing
context: resourceblock that runs after the datapoint copy-back, since transform statement groups execute in declaration order.Golden sample configs regenerated.
The chart counterpart is linked below.
Customer-visible changes
Three resource attributes disappear from every metric emitted by a Prometheus-receiver pipeline:
net.host.name,net.host.portandhttp.scheme. A query or alarm that selects on one of them stops matching. The replacement is a rename, not a removal —server.address,server.portandurl.schemecarry identical values on the same series and are untouched — so the migration is a one-line edit on the consumer side.The raw-label deletions are narrower: the raw Prometheus names (
pod,namespace,containerand the kube-state-metrics set) remain available at datapoint scope, which is where dashboards read them. Only the duplicate resource-scope copy goes, so@resource.pod-style selectors stop resolving while the plain datapoint label continues to work.Both are customer-visible and worth release-note treatment, but neither removes information from the pipeline.
License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Tests
go test ./translator/...passes, including the regeneratedtocwconfiggoldens.New integration coverage in
aws/amazon-cloudwatch-agent-test(aws/amazon-cloudwatch-agent-test#776):TestDeprecatedSemconvAttributesRemovedasserts the three deprecated attributes are gone whileurl.schemeis still present on at least one series — the assertion that would have caught the original karpenter/keda inversion — andTestRawLabelsNotBilledTwiceasserts the raw Prometheus labels exist only at datapoint scope for the cadvisor and kube-state-metrics pipelines.Both were validated on a live EKS cluster, since the claim is about what arrives in CloudWatch rather than what the config renders to. The changed configuration was applied to the running agent and cluster scraper, the pods were restarted, and metrics were given 30 minutes to settle;
test/otel/standard -computeType=EKSthen asserts on datapoints read back from CloudWatch. The suite was also run before the change, because an "attribute is absent" assertion passes trivially against empty data.TestDeprecatedSemconvAttributesRemovedTestRawLabelsNotBilledTwiceNo failures, no skips.
TestDeprecatedSemconvAttributesRemovedfailing before the change is the result that matters most here: it is the assertion that would have caught the original inversion, and it only passes once the current-spelling twins are the ones left in place.Requirements
Before commiting your code, please do the following steps.
make fmtandmake fmt-sh— pass, no files modifiedmake lint— 0 issuesRelated pull requests
Scope note: this change covers the deprecated and doubled attributes only; the chart counterpart additionally removes the cAdvisor cgroup path, so the two are not yet identical.
PR Checklist
makepasses locally (build, unit tests, lint)Integration Tests
To run integration tests against this PR, add the
ready for testinglabel.