[CONTP-2125] Add opt-in APM tracing for DDA and DDAI reconciles

[nlchung]

What does this PR do?

Adds opt-in APM tracing (dd-trace-go v2) to the DDA and DDAI reconcile loops.

• Enable with --tracing-enabled or DD_TRACING_ENABLED=true. Off by default.
• One root span per reconcile: datadogagent.reconcile / datadogagentinternal.reconcile, tagged with kind, name, namespace and reconcileID.
• Child spans:
  • DDA: manageExperiment, manageRevision, reconcileProfiles, manageDDADependenciesWithDDAI, createOrUpdateDDAI, updateStatusIfNeeded
  • DDAI: applyAndCleanupDependencies, reconcileComponent / Cleanup (tagged agent.component), reconcileV2Agent, cleanupExtraneousResources, updateStatusIfNeeded
• Kubernetes API requests made with the reconcile context are traced (cached reads don't reach the API server). By default, only 5xx responses count as errors (override with DD_TRACE_HTTP_CLIENT_ERROR_STATUSES).
• Logs written through the context logger (ctrl.LoggerFrom(ctx)) include dd.trace_id and the current span's dd.span_id, in dd-trace-go's trace ID format so they link to traces. Logs still written through the base logger (r.log) don't; moving them is tracked in a backlog card.
• The profiler moves to dd-trace-go v2 and the v1 dependency is removed. Its service name is unchanged (manager unless DD_SERVICE is set).

Motivation

CONTP-2125. Existing metrics show total reconcile duration and error counts, but not which step inside a reconcile is slow or failing. Tracing breaks each reconcile into spans, including the Kubernetes API calls, so a slow or failed step can be found for a specific reconcile. This builds on the prototype in #2744.

Additional Notes

• Users need to point the tracer at an Agent: DD_AGENT_HOST (e.g. from status.hostIP), DD_TRACE_AGENT_URL, or the APM socket at /var/run/datadog/apm.socket. The Helm chart sets none of these.
• The binary grows by about 7.5 MB (+7%).
• When tracing is disabled, the tracer isn't started and API calls aren't wrapped. The main cost is binary size.
• If DD_TRACE_ENABLED=false is set alongside --tracing-enabled, no spans are created and logs aren't tagged with trace IDs.
• The k8s client-go integration sets each traced request's Audit-Id header to the trace ID, so API calls in one reconcile share an audit ID.
• Not traced: calls that don't use the reconcile context, e.g. the OpenShift etcd secret copy in control plane monitoring.
• A reconcile that panics is recorded as a successful span (controller-runtime recovers the panic after the spans finish).
• Log trace IDs are 128-bit hex only when DD_TRACE_128_BIT_TRACEID_LOGGING_ENABLED is on (the default) and the ID has upper bits set. Otherwise they are 64-bit decimal, so pipelines that only parse decimal IDs still correlate.

Minimum Agent Versions

• Agent: N/A
• Cluster Agent: N/A

Describe your test plan

• Unit tests in pkg/trace cover spans, tags, errors, log fields, the transport, and a tracer that isn't running.
• TestReconcileTracing runs a DDA → DDAI reconcile and checks the span tree.
• Manual check: deploy with tracing enabled next to an Agent with APM. Change a DatadogAgent, then confirm the traces show up in APM under datadog-operator.

Checklist

• PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• PR has a milestone or the qa/skip-qa label
• All commits are signed (see: signing commits)

[datadog-datadog-prod-us1-2]

🛑 Gate Violations

🎯 1 Code Coverage issue detected

A Patch coverage percentage gate may be blocking this PR.

• Patch coverage: 59.43% (threshold: 80.00%)

ℹ️ Info

🎯 Code Coverage (details)
• Patch Coverage: 59.43%
• Overall Coverage: 52.45% (+0.00%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c78f27d | Docs | Give us feedback!}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-01T17:43:58.712787Z	1014596	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 101459672e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".