feat: collect the OTLP data exported by the benchmarks by lvaroqui · Pull Request #589 · CodSpeedHQ/codspeed

lvaroqui · 2026-10-16T12:56:23Z

Add an experimental --experimental-otel flag (CODSPEED_EXPERIMENTAL_OTEL) that collects the OpenTelemetry data the benchmarks export, and uploads it with the results.

When the flag is set, each run part starts an OTLP/HTTP receiver on a free localhost port and passes its URL to the benchmark in CODSPEED_OTEL_ENDPOINT. The receiver accepts protobuf and JSON exports of traces, logs and metrics, plain or gzipped, on /v1/{traces,logs,metrics}.

While the benchmark runs, the receiver only writes each request body to disk as received, so it adds no compression work next to the measured process. A request is stored before it is acknowledged, so everything the process sent before exiting is kept. Once the process exits, the requests are decompressed and packed into a single otel.tar.zst in the profile folder. Compressing them as one stream with long-distance matching takes advantage of the content that repeats across export batches. Nothing is written when the benchmark exports nothing.

To put the spans and logs on the same clock as the benchmark markers, instrument-hooks is bumped for instrument_hooks_monotonic_offset: integrations timestamp with the monotonic clock shifted to read as wall clock, and record the offset in the codspeed.clock.monotonic_offset_ns resource attribute. The bindings expose it as InstrumentHooks::monotonic_offset and MONOTONIC_OFFSET_ATTRIBUTE.

tests/otel.rs runs the divan otel_example end to end through codspeed run --experimental-otel in walltime mode, and checks that every span and log record, shifted back by its resource's offset, falls within the benchmark markers. It needs a codspeed-rust checkout next to this repository, so it is ignored by default: cargo test --test otel -- --ignored. Set CODSPEED_OTEL_TEST_PROFILE_FOLDER to keep the profile folder for inspection.

Stacked on #587.

Closes COD-3863

codspeed · 2026-10-16T13:06:23Z

Merging this PR will not alter performance

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 8 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 41 untouched benchmarks
⏩ 6 skipped benchmarks¹

_{Comparing cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet (5c8bbe0) with main (d9612e3)}

6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Add the files holding the OpenTelemetry spans and logs the benchmarks emit, shared by the runner that writes them and the parsers that read them: `otel/spans.msgpack.zst` and `otel/logs.msgpack.zst` in the profile folder. Each file is a zstd stream of positional MessagePack records, one per span or log, like the memtrack artifacts. Rows are flattened for the tables they are loaded into: ids as hex, attribute values as text, and the times of a span or log on the monotonic clock of the benchmark markers, when the benchmark recorded them, in their own fields. zstd runs with a 16 MB window and long-distance matching, since rows repeat their resource attributes far apart. Versions follow `UnwindData`: each record is wrapped in a versioned enum (`SpanRecord`, `LogRecord`), `SpanRow` and `LogRow` alias the current version, and older versions are upgraded through `From`. Version 1 fixtures must keep decoding. `TelemetryReader` decodes a file from any `Read`, such as an archive entry, as its bytes arrive. Unlike a truncated memtrack stream, a file cut in the middle of a row ends with an error. Refs COD-3863 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Add `codspeed run --mode telemetry`, an experimental mode collecting the OpenTelemetry spans and logs the benchmarks export. It is a separate pass, so nothing has to keep the export out of the measured runs, and the benchmarks run once each instead of once per walltime iteration. The mode runs on a new analysis executor: it runs the benchmarks once each without a profiler, answering the integration as an analysis mode, with analysis instruments hooked onto it. An instrument sets up before the process is spawned, gets the integration's start and stop profiler commands, and stops once the process exits, with the benchmark markers. The executor saves those markers as `ExecutionTimestamps.msgpack` in the profile folder, and fails when no benchmark ran, unless `--allow-empty` is set. Uploads carry `executor: "analysis"` and the enabled instruments, so later analysis instruments only add an instrument. The telemetry mode enables the telemetry instrument: - It starts an OTLP/HTTP receiver on a free localhost port, passed to the benchmark in CODSPEED_OTEL_ENDPOINT. The receiver accepts protobuf and JSON exports, plain or gzipped, and writes each trace and log request to disk as received. Metrics are acknowledged and dropped. - Once the process exits, the requests are decoded and written as the runner-shared telemetry files, one row per span or log, formatted like the OpenTelemetry Collector's ClickHouse exporter. A request that fails to decode is skipped with a warning. - The `codspeed.monotonic_start_ns` and `codspeed.monotonic_end_ns` attributes of a span and the `codspeed.monotonic_time_ns` attribute of a log, their times on the monotonic clock of the benchmark markers, move to their own fields of the rows: along with the markers, they tell the benchmark that emitted each span or log. A warning counts the spans and logs without a start or time. `CODSPEED_RUNNER_MODE` is `telemetry` in this mode, which `cargo codspeed run` reads to pick its telemetry build. exec-harness runs it like memory mode. Refs COD-3863 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Build the divan `telemetry` bench of a codspeed-rust checkout next to this repository with `cargo codspeed build -m telemetry`, run it through `codspeed run --mode telemetry`, and read back its telemetry files. Its benchmarks emit `tracing` spans and events, a `tracing` event outside of any span, records of the `log` crate, and nothing at all. Each span and log is timestamped within the run on the wall clock, and falls within the rounds of the benchmark that emitted it by its time on the monotonic clock. Records keep their severity and call site, the ones below `info` are filtered out, and the benchmark without telemetry has no rows. The test is ignored by default since CI has no codspeed-rust checkout. Run it with `cargo test --test otel -- --ignored`. `scripts/run-otel-example.sh` runs the same example and uploads it. Refs COD-3863 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

greptile-apps · 2026-10-23T16:42:01Z

Confidence Score: 4/5

[Medium impact] The change adds an opt-in telemetry collection and upload path for benchmark runs.

Fix stale telemetry in reused profile folders before merging.

Findings

Old telemetry enters new runs ▶
Gzip can exhaust runner memory ▶
Test command names missing target ▶

Fix with agent prompt

### Issue 1
src/instruments/otel/files.rs:106-112
Reusing `--profile-folder` can upload telemetry from the previous run. `LazyWriter` only opens a file when new rows arrive, so a run with no logs leaves the old `otel/logs.msgpack.zst` untouched; the same happens for spans. The uploader then archives the whole folder alongside the new benchmark timestamps. Remove the previous telemetry output before collecting a new run, and add a test that reuses a folder with fewer signals.

### Issue 2
src/instruments/otel/receiver.rs:236-240
`MAX_REQUEST_BODY_BYTES` caps the compressed body, but `read_to_end` expands it into an unbounded `Vec` before any OTLP decoding. A local client can send a small, highly compressed body that makes the runner run out of memory. Cap the decompressed size too and reject requests that exceed it.

**How this was verified:** The localhost receiver accepts gzip bodies and expands them into an unbounded buffer before checking their OTLP contents.

### Issue 3
tests/telemetry.rs:5
The documented command still names `otel`, but this file is now `tests/telemetry.rs` and there is no `otel` test target. Change it to `cargo test --test telemetry -- --ignored` so contributors can run the ignored end-to-end test.

```suggestion
//! `cargo test --test telemetry -- --ignored`.
```

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds experimental telemetry mode to collect benchmark traces and logs without measuring performance.

CodSpeed collects benchmark traces and logs during telemetry runs.
Telemetry files travel with the uploaded run profile.

Review used source inspection; no builds or tests were run.

Diagram

sequenceDiagram
  participant R as Runner
  participant B as Benchmark
  participant O as Local OTLP receiver
  participant F as Profile folder
  participant U as Upload
  R->>O: Start receiver
  R->>B: Run with CODSPEED_OTEL_ENDPOINT
  B->>O: Export traces and logs
  O->>F: Store request bodies
  O-->>B: Acknowledge export
  B-->>R: Exit
  R->>O: Stop receiver
  R->>F: Decode and write compressed telemetry
  R->>U: Upload analysis archive

_{Reviews (1) · Last reviewed commit: "test: run the divan telemetry example en..." · Reviewed by Greptile}

greptile-apps · 2026-10-23T16:42:04Z

+        let body = if is_gzipped {
+            let mut decoded = Vec::new();
+            if let Err(error) = GzipDecoder::new(body.as_ref())
+                .read_to_end(&mut decoded)
+                .await


Gzip can exhaust runner memory

MAX_REQUEST_BODY_BYTES caps the compressed body, but read_to_end expands it into an unbounded Vec before any OTLP decoding. A local client can send a small, highly compressed body that makes the runner run out of memory. Cap the decompressed size too and reject requests that exceed it.

How this was verified: The localhost receiver accepts gzip bodies and expands them into an unbounded buffer before checking their OTLP contents.

Prompt To Fix With AI

This is a comment left during a code review. Path: src/instruments/otel/receiver.rs Line: 236-240 Comment: **Gzip can exhaust runner memory** `MAX_REQUEST_BODY_BYTES` caps the compressed body, but `read_to_end` expands it into an unbounded `Vec` before any OTLP decoding. A local client can send a small, highly compressed body that makes the runner run out of memory. Cap the decompressed size too and reject requests that exceed it. **How this was verified:** The localhost receiver accepts gzip bodies and expands them into an unbounded buffer before checking their OTLP contents. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-10-23T16:42:04Z

+    fn write(&mut self, rows: Vec<R>) -> Result<()> {
+        if rows.is_empty() {
+            return Ok(());
+        }
+        let writer = match &mut self.writer {
+            Some(writer) => writer,
+            None => self.writer.insert(TelemetryWriter::create(&self.path)?),


Old telemetry enters new runs

Reusing --profile-folder can upload telemetry from the previous run. LazyWriter only opens a file when new rows arrive, so a run with no logs leaves the old otel/logs.msgpack.zst untouched; the same happens for spans. The uploader then archives the whole folder alongside the new benchmark timestamps. Remove the previous telemetry output before collecting a new run, and add a test that reuses a folder with fewer signals.

Prompt To Fix With AI

This is a comment left during a code review. Path: src/instruments/otel/files.rs Line: 106-112 Comment: **Old telemetry enters new runs** Reusing `--profile-folder` can upload telemetry from the previous run. `LazyWriter` only opens a file when new rows arrive, so a run with no logs leaves the old `otel/logs.msgpack.zst` untouched; the same happens for spans. The uploader then archives the whole folder alongside the new benchmark timestamps. Remove the previous telemetry output before collecting a new run, and add a test that reuses a folder with fewer signals. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-10-23T16:42:04Z

+//! checkout, written as telemetry files that line up with the benchmark markers.
+//!
+//! Needs codspeed-rust checked out next to this repository. Run with
+//! `cargo test --test otel -- --ignored`.


Test command names missing target

The documented command still names otel, but this file is now tests/telemetry.rs and there is no otel test target. Change it to cargo test --test telemetry -- --ignored so contributors can run the ignored end-to-end test.

Suggested change

//! `cargo test --test otel -- --ignored`.

//! `cargo test --test telemetry -- --ignored`.

Prompt To Fix With AI

This is a comment left during a code review. Path: tests/telemetry.rs Line: 5 Comment: **Test command names missing target** The documented command still names `otel`, but this file is now `tests/telemetry.rs` and there is no `otel` test target. Change it to `cargo test --test telemetry -- --ignored` so contributors can run the ignored end-to-end test. ```suggestion //! `cargo test --test telemetry -- --ignored`. ``` --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

lvaroqui added this pull request to stack #590 October 16, 2026 12:56

Base automatically changed from cod-3863-chore-avoid-blocking-in-runner to main October 16, 2026 13:07

lvaroqui force-pushed the cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet branch 3 times, most recently from 33db967 to cf0d278 Compare October 20, 2026 14:33

lvaroqui force-pushed the cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet branch from cf0d278 to 7058ea3 Compare October 23, 2026 13:16

lvaroqui mentioned this pull request Oct 23, 2026

feat: export the telemetry of divan benchmarks over OTLP CodSpeedHQ/codspeed-rust#197

Open

lvaroqui and others added 3 commits October 23, 2026 18:35

lvaroqui force-pushed the cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet branch from 7058ea3 to 5c8bbe0 Compare October 23, 2026 16:35

lvaroqui marked this pull request as ready for review October 23, 2026 16:36

greptile-apps Bot reviewed Oct 23, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat: collect the OTLP data exported by the benchmarks - #589

feat: collect the OTLP data exported by the benchmarks#589
lvaroqui wants to merge 3 commits into
mainfrom
cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet

lvaroqui commented Oct 16, 2026

Uh oh!

codspeed Bot commented Oct 16, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Oct 23, 2026 •

edited

Loading

Diagram

Uh oh!

greptile-apps Bot Oct 23, 2026

Uh oh!

greptile-apps Bot Oct 23, 2026

Uh oh!

greptile-apps Bot Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	//! `cargo test --test otel -- --ignored`.
	//! `cargo test --test telemetry -- --ignored`.

Conversation

lvaroqui commented Oct 16, 2026

Uh oh!

codspeed Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

greptile-apps Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Diagram

Uh oh!

greptile-apps Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codspeed Bot commented Oct 16, 2026 •

edited

Loading

greptile-apps Bot commented Oct 23, 2026 •

edited

Loading