Start supporting the use case of no cluster agent by agagniere · Pull Request #3549 · DataDog/datadog-operator

agagniere · 2026-10-02T22:43:38Z

What does this PR do?

When the cluster agent is disabled, others components like the core agent should know about it, to stop trying to contact it.

DD_CLUSTER_AGENT_ENABLED hardcoded "true" — the foundational switch the node agent uses to decide whether to even try reaching a cluster agent. Wrong on every single node-agent container and the init-config init container, regardless of config. Root cause behind several of the symptoms below.
admissionController — runs a webhook server inside the cluster agent. With it disabled, the Service/RBAC still got created and webhooks still registered pointing at nothing — the Kubernetes API server would block pod admission trying to reach a webhook with no backend. Most severe single-feature bug; this is an outage risk, not just an orphan.
instrumentationCrd — defaults to enabled (agent ≥ 7.82.0). Node agent kept registering the instrumentation-checks autodiscovery provider and spamming connection errors against a cluster agent Service that doesn't exist.
serviceDiscovery — explicit false silently dropped; agent's own ≥7.78.0 auto-on default won instead.
kubeStateMetricsCore — default-enabled, orphaned RBAC+ConfigMap bound to a nonexistent ServiceAccount.
eventCollection — default-enabled, same RBAC-orphan shape.
orchestratorExplorer — default-enabled, RBAC orphan (node-side disable env var was already correct, only the cluster-agent-side RBAC leaked).
controlPlaneMonitoring — default-enabled, orphaned ConfigMaps.
CSPM — opt-in, but ties node-side check requirement to the cluster agent requirement; same RBAC-orphan shape.
autoscaling — opt-in, RBAC orphan (already tagged for component-scoped cleanup, so lower practical risk, but still shouldn't be created).
helmCheck — opt-in, RBAC orphan.
kubernetesActions — opt-in, RBAC orphan.
externalMetricsServer — opt-in, orphaned Service/RBAC/APIService.
appsec injector — opt-in (annotation/CRD-gated), RBAC orphan; lowest blast radius since it requires the most explicit user configuration to trigger at all.
serviceDiscovery — the disabled-state env var fix (item 4) only covered ManageNodeAgent; ManageSingleContainerNodeAgent was still a no-op, so under SingleContainerStrategy a disabled config never reached the unprivileged-single-agent container and the agent's own ≥7.78.0 auto-on default won anyway.
DCA token Secret and Service — created unconditionally in manageDDADependenciesWithDDAI regardless of override.clusterAgent.disabled, left behind with no consumer.
DD_CLUSTER_AGENT_AUTH_TOKEN — written on the node agent based on merged feature requirements only, not the override; after fixing item 16 this left a secretKeyRef pointing at a Secret that no longer exists, which would have broken pod startup.
enabledefault — always set ClusterAgent.IsRequired: true, so merging it with override.clusterAgent.disabled made checkComponentEnabledWithOverride treat every no-Cluster-Agent DatadogAgent as a feature/override conflict, persisting an OverrideReconcileConflict status condition on an otherwise supported configuration. Found by Copilot/Codex PR review.
clusterChecks — default-enabled, no clusterAgent.disabled gate at all; missed in the original audit. Left DD_EXTRA_CONFIG_PROVIDERS=clusterchecks endpointschecks on the node agent pointing at a Cluster Agent that would never dispatch those checks. Found by Codex PR review.

Motivation

For reInvent we want to provide DD features on top of the DDOT collector, and so require the DD agent, while features that can be done via OTel like k8s explorer are handled by DDOT. In particular this means that the cluster agent should not be present.

OTAGENT-1387

Testing

Added testdata/no-cluster-agent-dda.yaml + a golden-file fixture (internal/controller/testutils/renderer) that renders a DatadogAgent with the cluster agent disabled and several cluster-agent-dependent features (kubeStateMetricsCore, orchestratorExplorer, serviceDiscovery) also enabled, to lock in that no Cluster Agent Deployment/Service/RBAC/token Secret is produced and that the node agent resolves correctly on its own.

Docs

Added a "Running without the Cluster Agent" section to docs/cluster_agent_setup.md plus an example manifest (examples/datadogagent/datadog-agent-without-clusteragent.yaml) — previously override.clusterAgent.disabled was undocumented.

Checklist

PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
PR has a milestone or the qa/skip-qa label
All commits are signed (see: signing commits)

…covery is disabled Configure() previously left the Agent component unrequired when service discovery resolved to disabled, so ManageNodeAgent never ran and the env var was simply omitted instead of written as "false". The agent defaults service discovery to on for versions >= 7.78.0, so an explicit false was silently overridden by that default. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The instrumentation-checks autodiscovery provider requires a live Cluster Agent to serve configs to the node agent. Configure() only checked agent/cluster-agent version floors, never whether the Cluster Agent was actually going to be deployed (override.clusterAgent.disabled). Without one, the node agent kept registering the provider and spamming connection errors trying to reach a Cluster Agent Service that doesn't exist. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…bled kubeStateMetricsCore only ever runs inside the Cluster Agent or Cluster Check Runners it dispatches to — there's no code path for it to run without one. Configure() gated purely on the feature's own enabled flag, never on override.clusterAgent.disabled, so disabling the cluster agent while the feature stayed enabled created a ClusterRole/ClusterRoleBinding bound to a nonexistent ServiceAccount plus a stray ConfigMap. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

… disabled orchestratorExplorer runs inside the Cluster Agent (or Cluster Check Runners it dispatches to). Configure() never checked override.clusterAgent.disabled, so ManageDependencies still created a ClusterRole/ClusterRoleBinding bound to a nonexistent cluster-agent ServiceAccount. The node agent's explicit DD_ORCHESTRATOR_EXPLORER_ENABLED disable write is preserved. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Event collection is configured using the Cluster Agent only, and defaults to enabled. Configure() never checked override.clusterAgent.disabled, so ManageDependencies still created RBAC bound to a nonexistent cluster-agent ServiceAccount. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The admission controller webhook server runs inside the Cluster Agent; without one there's nothing to back the webhook Service, and the API server would block pod creation trying to reach it. Configure() never checked override.clusterAgent.disabled, so the Service and RBAC were still created and the admission webhooks still registered. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

AppSec injection is configured and served by the Cluster Agent. Configure() never checked override.clusterAgent.disabled, so ManageDependencies still created RBAC bound to a nonexistent cluster-agent ServiceAccount. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

CSPM dispatches its checks via the Cluster Agent and already ties its node-side (Agent) requirement to the Cluster Agent requirement. Configure() never checked override.clusterAgent.disabled, so ManageDependencies still created RBAC bound to a nonexistent cluster-agent ServiceAccount. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…abled The external metrics server is served by the Cluster Agent. Configure() never checked override.clusterAgent.disabled, so ManageDependencies still created a Service, RBAC, and an APIService with nothing to back them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The helm check only ever runs inside the Cluster Agent (or Cluster Check Runners it dispatches to). Configure() never checked override.clusterAgent.disabled, so ManageDependencies still created RBAC bound to a nonexistent cluster-agent ServiceAccount. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Kubernetes Actions runs inside the Cluster Agent. Configure() never checked override.clusterAgent.disabled, so ManageDependencies still created RBAC bound to a nonexistent cluster-agent ServiceAccount. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Autoscaling decisions are computed and served by the Cluster Agent. Configure() never checked override.clusterAgent.disabled, so ManageDependencies still created RBAC bound to a nonexistent cluster-agent ServiceAccount (its RBAC was already tagged for component-scoped cleanup, but creating it at all was still wrong). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…sabled Control plane monitoring is driven by the Cluster Agent. Configure() never checked override.clusterAgent.disabled, so ManageDependencies still created ConfigMaps for a feature that has nothing to run it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The default pod template scaffolding (component/agent/default.go) hardcodes DD_CLUSTER_AGENT_ENABLED to "true" on every agent container and the init-config init container, regardless of whether a Cluster Agent is actually being deployed. This is the switch the node agent uses to decide whether to try reaching a cluster agent at all, and was the likely root cause behind several of the per-feature symptoms already fixed on this branch. The enabledefault feature now overrides it to "false" when the Cluster Agent is disabled via override. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

datadog-datadog-prod-us1-2 · 2026-10-02T22:56:50Z

Pipelines

🎯 Code Coverage (details)
• Patch Coverage: 92.86%
• Overall Coverage: 53.95% (+1.43%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 4d3b88e | Docs | View more details | Give us feedback!}

Copilot

Copilot review overview

🟡 Changes recommended

Disabled service discovery is not propagated when using the single-container strategy.

Review effort: Balanced
Findings: 1

Open (1)

Apply discovery-disabled override for single-container agents · New

What changed in this PR

Adds support for running node agents without a Cluster Agent.

Changes:

Propagates the disabled Cluster Agent state to node-agent containers.
Suppresses Cluster Agent-dependent features and resources.
Adds regression coverage for disabled configurations.

File	Description
`feature/servicediscovery/feature.go`	Explicitly configures service discovery state.
`feature/servicediscovery/feature_test.go`	Tests explicit disabling.
`feature/orchestratorexplorer/feature.go`	Skips Cluster Agent resources when disabled.
`feature/orchestratorexplorer/feature_test.go`	Tests disabled Cluster Agent behavior.
`feature/kubernetesstatecore/feature.go`	Suppresses KSM Core resources.
`feature/kubernetesstatecore/feature_test.go`	Tests suppression.
`feature/kubernetesactions/feature.go`	Disables Kubernetes Actions.
`feature/kubernetesactions/feature_test.go`	Tests disabling.
`feature/instrumentationcrd/feature.go`	Disables Instrumentation CRD integration.
`feature/instrumentationcrd/feature_test.go`	Tests disabling.
`feature/helmcheck/feature.go`	Suppresses Helm check resources.
`feature/helmcheck/feature_test.go`	Tests suppression.
`feature/externalmetrics/feature.go`	Suppresses external metrics resources.
`feature/externalmetrics/feature_test.go`	Tests suppression.
`feature/eventcollection/feature.go`	Suppresses event collection resources.
`feature/eventcollection/feature_test.go`	Tests suppression.
`feature/enabledefault/feature.go`	Overrides Cluster Agent availability environment variables.
`feature/enabledefault/feature_test.go`	Tests environment overrides.
`feature/cspm/feature.go`	Disables CSPM without a Cluster Agent.
`feature/cspm/feature_test.go`	Tests disabling.
`feature/controlplanemonitoring/feature.go`	Suppresses control-plane monitoring resources.
`feature/controlplanemonitoring/feature_test.go`	Tests suppression.
`feature/autoscaling/feature.go`	Disables Cluster Agent-backed autoscaling.
`feature/autoscaling/feature_test.go`	Tests disabling.
`feature/appsec/feature.go`	Disables AppSec injection.
`feature/appsec/feature_test.go`	Tests disabling.
`feature/admissioncontroller/feature.go`	Prevents unbacked admission webhooks.
`feature/admissioncontroller/feature_test.go`	Tests disabling.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Copilot review overview

🟡 Changes recommended

Cluster-side private action runner configuration remains enabled and can create dependencies or block reconciliation when the Cluster Agent is disabled.

Review effort: Balanced
Findings: 1

Open (1)

Cluster Agent override leaves private action runner dependencies enabled · New

Resolved since last review (1)

Apply discovery-disabled override for single-container agents

… agent is disabled privateactionrunner's Cluster Agent config is annotation-gated rather than spec.features-gated, so it was missed in the earlier feature audit. It had no clusterAgent.disabled check, so it kept requiring a Cluster Agent and creating its ConfigMap (and optional RBAC) with nothing to deploy them onto. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…t bugs asm, prometheusscrape and remoteconfig still set ClusterAgent.IsRequired unconditionally without checking override.clusterAgent.disabled. Unlike the 20 previously fixed instances, these are currently harmless: none creates orphaned dependencies, and enabledefault's explicit false already wins in the merge. Document the invariant they rely on so a future change to the merge order or to enabledefault doesn't silently reintroduce the override-conflict bug here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

asm.Configure read admissionController.enabled directly rather than checking override.clusterAgent.disabled, so it kept requiring the Cluster Agent even when disabled. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…abled remoteconfig.Configure set ClusterAgent.IsRequired/Containers unconditionally whenever the feature was enabled, regardless of override.clusterAgent.disabled. The Agent requirement stays, since remote config still runs node-side. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

… disabled prometheusscrape.Configure set ClusterAgent.IsRequired unconditionally whenever the feature was enabled, regardless of override.clusterAgent.disabled. The Agent requirement stays, since Prometheus scrape still runs node-side. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…ig are gated The merge-priority safety net was documented as something those three features relied on instead of checking the override themselves. Now that they do, reframe the comment as a safety net for future features, not a substitute for the per-feature check. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Copilot

Copilot review overview

🟡 Changes recommended

APM single-step instrumentation remains partially configured node-side even though its Cluster Agent requirement is masked.

Review effort: Balanced
Findings: 1

Open (1)

Gate APM SSI when Cluster Agent is explicitly disabled · New

Resolved since last review (1)

Cluster Agent override leaves private action runner dependencies enabled

A Copilot review comment questioned whether APM's single-step instrumentation (language detection in particular) gets left half configured when override.clusterAgent.disabled is set. Verified it doesn't: shouldConfigureSSI already calls validateSSISharedComponentPrerequisites, which rejects SSI when the Cluster Agent is disabled, so singleStepInstrumentation stays nil and shouldEnableLanguageDetection() is false. No production code change; adding a regression test since nothing previously covered this case. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

gabedos · 2026-10-08T19:10:38Z

+	managers.EnvVar().AddEnvVar(envVar)
+	// AddEnvVar only reaches regular containers; the init-config init container builds its
+	// env vars from the same hardcoded-true default and needs the override too.
+	managers.EnvVar().AddEnvVarToInitContainer(apicommon.InitConfigContainerName, envVar)


minor nit: unsure if the init containers care about this value

gabedos · 2026-10-08T19:13:38Z

 			IsRequired: new(true),
 			Containers: []apicommon.AgentContainerName{apicommon.CoreAgentContainerName, apicommon.SystemProbeContainerName},
 		}
+	} else {


I don't think the service discovery feature is related to the Cluster Agent. This feature isn't about k8s services. For whatever reason a service == a process which can be detected on the Agent via system-probe / system-probe-lite. This should remain on without the cluster agent.

gabedos · 2026-10-08T20:07:52Z

+// overrideClusterAgentEnabledEnvVar corrects DD_CLUSTER_AGENT_ENABLED, which the default pod
+// template scaffolding (component/agent/default.go) hardcodes to "true" on every agent
+// container regardless of whether a Cluster Agent is actually being deployed.
+func (f *defaultFeature) overrideClusterAgentEnabledEnvVar(managers feature.PodTemplateManagers) error {


edge case: the datadog agent profiles set cluster agent to be disabled on the profiles because we don't want to re-create the DCA. However, the user may still have the DCA enabled in the original DDA. This would result in all Agents in profiles not being able to contact the cluster agent?

#3586 would fix that edge case and needs to be merged alongside (ideally this is before/same day so that nightly images don't bug)... still thinking about other potential solutions.

levan-m · 2026-10-13T22:10:59Z

@agagniere, I changed milestone to 1.33 for to have a bit more time for reviewing and possibly changing the approach.

override.Component.disabled conflates 'don't create component' with 'component doesn't exist' which leads to issues @gabedos mentioned above.
DCA disabled check is in the wrong place in my opinion. Bunch of features duplicate identical check to compute required components based on the override and exit early. requiredComponent is more nuanced, based on what product/feature/component is enabled since features may still work without some of the components. E.g. kubernetesstatecore can run on node agents without DCA and collect limited telemetry.

I think handling clusterAgent.disabled shouldn't be solved on feature level, rather on DDA/DDAI where each DDAI can get a clear signal whether it owns the component, uses existing component or component doesn't exist at all.

agagniere · 2026-10-16T15:07:14Z

@levan-m So if I understand correctly, you would prefer a toggle to disable the cluster agent outside of override ? Works for me, which team shall own this change ?

agagniere and others added 14 commits October 2, 2026 23:19

agagniere added this to the v1.32.0 milestone Oct 2, 2026

agagniere requested a review from cyrille-leclerc October 2, 2026 22:43

agagniere self-assigned this Oct 2, 2026

agagniere added bug Something isn't working enhancement New feature or request labels Oct 2, 2026

github-actions Bot added team/agent-discovery team/agent-security team/asm-go team/container-platform team/container-autoscaling team/container-integrations team/kubernetes-experiences labels Oct 2, 2026

agagniere requested a balanced review from Copilot October 5, 2026 11:32

Copilot started reviewing on behalf of agagniere October 5, 2026 11:33

View session

Copilot AI reviewed Oct 5, 2026

View reviewed changes

Comment thread internal/controller/datadogagent/feature/servicediscovery/feature.go Outdated

Copilot started reviewing on behalf of agagniere October 6, 2026 13:00

View session

Copilot AI reviewed Oct 6, 2026

View reviewed changes

Comment thread internal/controller/datadogagent/feature/enabledefault/feature.go

agagniere and others added 6 commits October 6, 2026 15:10

agagniere requested review from a team as code owners October 6, 2026 16:00

agagniere requested review from dd-gplassard and removed request for a team October 6, 2026 16:00

github-actions Bot added team/action-platform team/remote-config labels Oct 6, 2026

agagniere requested a balanced review from Copilot October 6, 2026 16:01

Copilot started reviewing on behalf of agagniere October 6, 2026 16:01

View session

dd-gplassard approved these changes Oct 6, 2026

View reviewed changes

Copilot AI reviewed Oct 6, 2026

View reviewed changes

Comment thread internal/controller/datadogagent/feature/enabledefault/feature.go

agagniere requested a review from a team as a code owner October 7, 2026 22:06

github-actions Bot added the team/agent-apm label Oct 7, 2026

gabedos mentioned this pull request Oct 9, 2026

Stop force-disabling cluster-wide components on profile DDAIs #3586

Draft

3 tasks done

gabedos reviewed Oct 9, 2026

View reviewed changes

kangyili approved these changes Oct 12, 2026

View reviewed changes

levan-m modified the milestones: v1.32.0, v1.33.0 Oct 13, 2026

levan-m added the do-not-merge label Oct 14, 2026

agagniere closed this Oct 16, 2026

Navigation Menu

Repository navigation

Start supporting the use case of no cluster agent - #3549

Start supporting the use case of no cluster agent#3549
agagniere wants to merge 28 commits into
mainfrom
agagniere/no-cluster-agent

agagniere commented Oct 2, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Oct 2, 2026 •

edited by datadog-official Bot

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

gabedos Oct 8, 2026

Uh oh!

gabedos Oct 8, 2026

Uh oh!

gabedos Oct 8, 2026 •

edited

Loading

Uh oh!

levan-m commented Oct 13, 2026

Uh oh!

agagniere commented Oct 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

agagniere commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Testing

Docs

Checklist

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Oct 2, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

gabedos Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

gabedos Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

gabedos Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

levan-m commented Oct 13, 2026

Uh oh!

agagniere commented Oct 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

agagniere commented Oct 2, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Oct 2, 2026 •

edited by datadog-official Bot

Loading

gabedos Oct 8, 2026 •

edited

Loading