[lucastemb]

Summary

Adds a manually-triggered OTLP metrics ingestion platform comparison: Core Agent vs DDOT vs ADP, as three SMP experiment legs over the same converged image:

• otlp_ingest_metrics_agent_5mb — the Core Agent's own OTLP receiver handles ingest (no sidecars). This leg doubles as the sidecar-less deployment baseline.
• otlp_ingest_metrics_ddot_5mb — the otel-agent sidecar (DDOT) handles ingest, supervised by s6 alongside the Core Agent, configured via otel-config.yaml.
• otlp_ingest_metrics_adp_5mb — ADP handles ingest as the data-plane sidecar in converged mode.

All three legs run the same Core Agent, the same lading load (identical seed, 5 MiB/s, single connection — byte-identical payload streams; verified by diffing the generated lading.yamls), and differ only in which process owns OTLP and which port lading targets (4317/4317/6317). Their raw CPU/PSS numbers are directly comparable with no idle-baseline subtraction or time-shifting, replacing the manual notebook methodology used previously.

Supporting changes:

• New on_demand: true experiment attribute; the generator now produces a third on-demand/ suite excluded from both the nightly full/ run and the quality-gates/ PR gate (predicates updated, docs in the generator and test/smp/README.md).
• New manual-only CI job run-benchmarks-adp-on-demand in .gitlab/benchmark.yml (never scheduled, never gates; posts its report to the PR).
• Retires the standalone-mode otlp_ingest_metrics_5mb experiment: standalone is not a deployment shape we ship, and the converged ADP leg replaces it. (The remaining standalone experiments — DSD, OTLP logs/traces — are unchanged and out of scope here.)

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

How did you test this PR?

• make generate-smp-experiments and make check-smp-experiments: all 72 experiment configurations across 3 suites up-to-date.
• Verified the three legs' generated lading.yamls are byte-identical except target_uri, and that the merged datadog.yaml/env toggles match the intended per-leg process sets (Core Agent receiver on 4317; DDOT via DD_OTELCOLLECTOR_ENABLED + otelcol config; ADP via the data-plane env set with cont-init redirecting the Core Agent receiver out of the way).
• First live validation is the manual run-benchmarks-adp-on-demand job on this PR's pipeline — that is its intended usage.

References

• Replaces the one-shot methodology of chore: add OTLP metrics experiment #921 + (ignore) Trigger ci datadog-agent#41416 (benchmarking notebook: https://app.datadoghq.com/notebook/13229113).
• DDOT supervision rides the otel s6 service shipped in the official Agent image; the Core Agent provides the IPC certificate in converged mode (hence dropping the standalone dummy cert, matching the existing tagfilter_base precedent).

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: 56af3d0 · Comparison: 761701f · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 38.58 MiB (baseline) vs 38.58 MiB (comparison)
Size Change: +8 B (+0.00%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
anon.2030b05ca6c6373049acad230a4fc299.1119.llvm.10857003849299697151	+4.39 KiB	1
anon.2030b05ca6c6373049acad230a4fc299.1119.llvm.15908324372486280926	-4.39 KiB	1
anon.2030b05ca6c6373049acad230a4fc299.5.llvm.10857003849299697151	+888 B	1
anon.2030b05ca6c6373049acad230a4fc299.5.llvm.15908324372486280926	-888 B	1
_RNvXsZ_NtCsgW8esjfipvk_5alloc6stringNtB5_6StringNtNtCs27Vx93FoQ6z_4core3fmt5Write10write_char.llvm.10857003849299697151	+758 B	1
_RNvXsZ_NtCsgW8esjfipvk_5alloc6stringNtB5_6StringNtNtCs27Vx93FoQ6z_4core3fmt5Write10write_char.llvm.15908324372486280926	-758 B	1
_RNvXsZ_NtNtCs27Vx93FoQ6z_4core3fmt3numjNtB7_5Debug3fmt.llvm.10857003849299697151	+579 B	1
_RNvXsZ_NtNtCs27Vx93FoQ6z_4core3fmt3numjNtB7_5Debug3fmt.llvm.15908324372486280926	-579 B	1
anon.2030b05ca6c6373049acad230a4fc299.42.llvm.10857003849299697151	+555 B	1
anon.2030b05ca6c6373049acad230a4fc299.42.llvm.15908324372486280926	-555 B	1
anon.2030b05ca6c6373049acad230a4fc299.534.llvm.10857003849299697151	+552 B	1
anon.2030b05ca6c6373049acad230a4fc299.534.llvm.15908324372486280926	-552 B	1
anon.2030b05ca6c6373049acad230a4fc299.1052.llvm.10857003849299697151	+549 B	1
anon.2030b05ca6c6373049acad230a4fc299.1052.llvm.15908324372486280926	-549 B	1
anon.2030b05ca6c6373049acad230a4fc299.134.llvm.10857003849299697151	+476 B	1
anon.2030b05ca6c6373049acad230a4fc299.134.llvm.15908324372486280926	-476 B	1
anon.2030b05ca6c6373049acad230a4fc299.132.llvm.10857003849299697151	+474 B	1
anon.2030b05ca6c6373049acad230a4fc299.132.llvm.15908324372486280926	-474 B	1
anon.2030b05ca6c6373049acad230a4fc299.255.llvm.10857003849299697151	+465 B	1
anon.2030b05ca6c6373049acad230a4fc299.255.llvm.15908324372486280926	-465 B	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +4.39Ki  [NEW]     +74    anon.2030b05ca6c6373049acad230a4fc299.1119.llvm.10857003849299697151
  [NEW]    +888  [NEW]     +76    anon.2030b05ca6c6373049acad230a4fc299.5.llvm.10857003849299697151
  [NEW]    +758  [NEW]    +613    _RNvXsZ_NtCsgW8esjfipvk_5alloc6stringNtB5_6StringNtNtCs27Vx93FoQ6z_4core3fmt5Write10write_char.llvm.10857003849299697151
  [NEW]    +579  [NEW]    +473    _RNvXsZ_NtNtCs27Vx93FoQ6z_4core3fmt3numjNtB7_5Debug3fmt.llvm.10857003849299697151
  [NEW]    +555  [NEW]    +104    anon.2030b05ca6c6373049acad230a4fc299.42.llvm.10857003849299697151
  [NEW]    +552  [NEW]      +4    anon.2030b05ca6c6373049acad230a4fc299.534.llvm.10857003849299697151
  [NEW]    +549  [NEW]      +4    anon.2030b05ca6c6373049acad230a4fc299.1052.llvm.10857003849299697151
  [NEW]    +476  [NEW]     +22    anon.2030b05ca6c6373049acad230a4fc299.134.llvm.10857003849299697151
  [NEW]    +474  [NEW]     +20    anon.2030b05ca6c6373049acad230a4fc299.132.llvm.10857003849299697151
  [NEW]    +465  [NEW]    +101    anon.2030b05ca6c6373049acad230a4fc299.255.llvm.10857003849299697151
  +0.0%      +8  [ = ]       0    [401 Others]
  [DEL]    -465  [DEL]    -101    anon.2030b05ca6c6373049acad230a4fc299.255.llvm.15908324372486280926
  [DEL]    -474  [DEL]     -20    anon.2030b05ca6c6373049acad230a4fc299.132.llvm.15908324372486280926
  [DEL]    -476  [DEL]     -22    anon.2030b05ca6c6373049acad230a4fc299.134.llvm.15908324372486280926
  [DEL]    -549  [DEL]      -4    anon.2030b05ca6c6373049acad230a4fc299.1052.llvm.15908324372486280926
  [DEL]    -552  [DEL]      -4    anon.2030b05ca6c6373049acad230a4fc299.534.llvm.15908324372486280926
  [DEL]    -555  [DEL]    -104    anon.2030b05ca6c6373049acad230a4fc299.42.llvm.15908324372486280926
  [DEL]    -579  [DEL]    -473    _RNvXsZ_NtNtCs27Vx93FoQ6z_4core3fmt3numjNtB7_5Debug3fmt.llvm.15908324372486280926
  [DEL]    -758  [DEL]    -613    _RNvXsZ_NtCsgW8esjfipvk_5alloc6stringNtB5_6StringNtNtCs27Vx93FoQ6z_4core3fmt5Write10write_char.llvm.15908324372486280926
  [DEL]    -888  [DEL]     -76    anon.2030b05ca6c6373049acad230a4fc299.5.llvm.15908324372486280926
  [DEL] -4.39Ki  [DEL]     -74    anon.2030b05ca6c6373049acad230a4fc299.1119.llvm.15908324372486280926
  +0.0%      +8  [ = ]       0    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: 837d594f-31ba-41f9-8efe-ac65d57e7913
Baseline: 56af3d03 · Comparison: 761701ff · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.74	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ -0.12	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ -0.59	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ -0.76	metrics profiles logs
quality_gates_rss_idle	memory	⚪ -1.22	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 230 MiB ≤ 250 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 52 MiB ≤ 60 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 91.9 MiB ≤ 100 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 387 MiB ≤ 420 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 33.5 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression. Experiments tagged (no analysis) show ⚠️ n/a: SMP ran them but produced no analysis, usually because a replicate failed and exhausted its retries. Check the SMP report for that experiment's replicate failures.

[pr-commenter]

Regression Detector (Agent Data Plane, on-demand suite)

Run ID: 23f759e0-ec8c-4b69-962f-ffe400b23c60
Baseline: 56af3d03 · Comparison: 761701ff · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (3)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
otlp_ingest_metrics_adp_5mb	memory	⚪ +0.13	metrics profiles logs
otlp_ingest_metrics_ddot_5mb (erratic)	memory	⚪ -0.46	metrics profiles logs
otlp_ingest_metrics_agent_5mb (erratic)	memory	⚪ -0.51	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression. Experiments tagged (no analysis) show ⚠️ n/a: SMP ran them but produced no analysis, usually because a replicate failed and exhausted its retries. Check the SMP report for that experiment's replicate failures.

[pr-commenter]

Regression Detector (Agent Data Plane, full suite)

This comment was omitted because it was over 65,536 characters.Please check the Gitlab Job logs to see its output.