Skip to content

Guest telemetry: tracing to X-Ray, metrics to CloudWatch - #4

Merged
aruokhai merged 3 commits into
mainfrom
telemetry
Oct 11, 2026
Merged

aruokhai merged 3 commits into
mainfrom
telemetry

Conversation

@aruokhai

Copy link
Copy Markdown
Contributor

Adds enclave:telemetry (wit/telemetry/telemetry.wit), a host interface a guest imports to report spans, events and metrics. Spans and events go to AWS X-Ray. Metrics go to CloudWatch through PutMetricData.

What a guest gets

  • tracing
    • event(level, message, fields)
    • A span resource. Spans nest by the order the guest opens them, and close on end() or on drop.
  • metrics
    • counter-add and gauge-set
    • histogram-record, sent as CloudWatch's values/counts distribution
  • No function takes a tenant id. The runtime attributes everything to the tenant it already authenticated, as with tasks, notify and streams.
  • When telemetry is switched off, calls still succeed and do less. Events and closed spans are always mirrored to the console under the guest target.

How it works

  • One trace per invocation. A request, task run or stream message becomes one X-Ray trace. Its root is annotated with tenant and invocation, which X-Ray indexes. Spans become subsegments, and events and fields go in metadata.
  • The trace ships when the store drops. That covers every exit, timeouts and traps included, so no call site has to remember to flush.
  • Metrics are aggregated, not queued. Each 30 s window is summed, keeps the last value, or keeps the distribution. Nothing is sent for an empty window. The runtime adds no tenant dimension.
  • Host calls are synchronous and never touch the network.
    • Two forwarders are shaped like the guest-log forwarder: a bounded queue, batches, two backed-off retries, and drops counted and reported once a minute.
    • Anything X-Ray or CloudWatch would refuse is dropped or shed when it is recorded, instead of failing a whole request. Examples are an out-of-range value, a repeated dimension, or a trace document over 64 KiB.
  • Hand-wired, like the other interfaces (no bindgen), including the span resource.
  • Deployment
    • deployment.nix gains xray and metricsNamespace. They are measured into PCR0, and their or defaults keep an older deployment.nix building to the same derivation.
    • deploy/tofu gains a telemetry variable that grants xray:PutTraceSegments and cloudwatch:PutMetricData, the latter limited to the one namespace. Nothing is granted when telemetry is off.
  • Dependencies: aws-sdk-xray 1.109 and aws-sdk-cloudwatch 1.127 are the releases that match the locked smithy runtime, so no existing crate moves.

See docs/TELEMETRY.md for limits, delivery, and exactly what leaves the enclave.

Testing

  • scripts/ci-check.sh is green: rustfmt, clippy -D warnings, 414 lib tests (16 of them new), bin, and boot_origin.
  • tests/telemetry.rs serves the example guest's new /telemetry route through the real linker, with X-Ray and CloudWatch replaced by in-memory destinations. It checks attribution, nesting, fields, events, timings, metric aggregation and units. This is what proves the hand-wired ABI matches what wit-bindgen generates.
  • Every suite that serves the example guests passes with --include-ignored. That is guest_logs, harness, serve_auth, serve_guest, serve_h2, serve_tls, serve_grpc and serve_grpc_wire.
  • scripts/wit-drift.sh is clean. tofu validate and tofu fmt are clean.

Not covered

  • No run against real X-Ray or CloudWatch, and none against an emulator stand-in such as LocalStack.
  • No sampling: every invocation is traced. At $5 per million traces, with 100k free each month, it isn't needed yet.

🤖 Generated with Claude Code

aruokhai and others added 3 commits October 10, 2026 23:23
A new host interface, enclave:telemetry (wit/telemetry), that a guest
imports to report what it is doing beyond its stdout:

- tracing: events, and a `span` resource that nests by the order a guest
  opens spans and closes on `end` or on drop. Each invocation (request,
  task run, stream message) is one X-Ray trace, its root annotated with
  the tenant and the invocation kind, spans as subsegments.
- metrics: counter-add, gauge-set, histogram-record, aggregated over a
  30 s window (sum, last value, values/counts distribution) and published
  with PutMetricData. No tenant dimension; CloudWatch bills per series.

No function takes a tenant id: the serve layer puts a TelemetryContext on
the store, and the trace ships when the store drops, so every exit
(timeout and trap included) flushes and no call site has to remember to.
Host calls are synchronous and never touch the network. Shipping is two
forwarders shaped like the guest-log one: bounded queue, batches, two
backed-off retries, drops counted and reported once a minute. What
CloudWatch or X-Ray would refuse (an out-of-range value, a repeated
dimension, a 64 KiB+ document) is dropped or shed when recorded rather
than failing a whole request. With telemetry off the calls still succeed,
and events and spans are mirrored to the console either way.

Hand-wired like tasks/notify/streams (no bindgen), including the span
resource. tests/telemetry.rs serves the example guest's /telemetry route
through the real linker with in-memory X-Ray and CloudWatch, so the ABI
names and types are checked against what wit-bindgen generates.

aws-sdk-xray 1.109 and aws-sdk-cloudwatch 1.127 are the releases that
match the locked smithy runtime; no existing crate moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- flake.nix: ENCLAVE_XRAY and ENCLAVE_METRICS_NAMESPACE from deployment
  `xray` and `metricsNamespace`, defaulted with `or`, so a deployment.nix
  written before telemetry still builds, to the same derivation, with it
  off. Measured into PCR0 like every other image setting.
- deploy/tofu: a `telemetry` variable granting the parent's role exactly
  what the runtime calls: xray:PutTraceSegments (X-Ray cannot be scoped to
  a resource) and cloudwatch:PutMetricData held to the one namespace by
  condition. Nothing is granted when telemetry is off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The benchmarks job failed whenever the cache restored a partial
target/criterion. Criterion found a `base/` without its `sample.json`,
printed "Criterion.rs ERROR: Failed to access file ..." onto each
result's line, and github-action-benchmark could not parse bench.txt.
The numbers themselves were fine. It failed this way on main on Oct 4
and Oct 6, and on this branch.

History is the tracker's (gh-pages), not Criterion's, so the script now
removes target/criterion before benchmarking. Checked by planting a
half-restored state and running scripts/ci-bench.sh: three bencher lines,
no errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aruokhai
aruokhai merged commit c4a0b96 into main Oct 11, 2026
3 checks passed
Dunsin-cyber pushed a commit to Dunsin-cyber/MerlinWallet that referenced this pull request Nov 9, 2026
- enclave-bundle.lock: dev-enclave-dev-c4a0b96, built from enclave-runtime
  main after vtxos/runtime#4 (guest telemetry). This moves the e2e
  from be65bd7 across 26 runtime commits, the per-tenant ZFS pools of vtxos#3
  among them. The bundle's dev-enclave.sh interface is unchanged.
- Vendor the runtime's wit/telemetry/telemetry.wit and include
  enclave:telemetry/observed in the cosigner's world, so the cosigner can
  report spans to X-Ray and metrics to CloudWatch. Not called yet, and free
  until it is: the release build's imports are identical with and without
  the include, so it still instantiates on a runtime without telemetry.
- wit/deps/stream/stream.wit: re-synced to the new bundle's copy (doc
  comment only).
- Makefile BUNDLE_URL and the lock's comment name vtxos/enclave-runtime
  after the org move.

`make enclave-bundle` fetches and verifies the new tarball, and
`make wit-drift` is clean against it. 231 cosigner tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant