Exact metric values, stricter fail-fast, and an opt-in client derivation - #208
Merged
Merged
Conversation
…re slipping through Four defects from a post-merge review of #199, all in the application-metrics path, all producing the failure mode that code's own comments name as the worst case: a scrape Prometheus rejects in full, with no in-process consumer to notice. FormatNumber built a string from snprintf's would-have-written length, an out-of-bounds read for any value whose "%.6f" form exceeds the 64-byte buffer — gauge.Set(1e60), or a bucket bound of 1e60, which the ladder validation happily admits. And "%.6f" is lossy below its granularity: {1e-8, 1e-7} passes the strictly-ascending check while both bounds render "0", two identical le labels, a duplicate series, the whole scrape refused. The rule now: the fixed six-decimal spelling is kept only when it parses back to exactly the value it claims — so every historical series string is byte-stable, which matters because those strings are series identity — and everything else gets a shortest round-trip form. One rule, both bugs. Three fail-fast gaps, closed to match the posture the file already has (ADR-0009): duplicate label names aborted nowhere and rendered {a="1",a="2"}; a user label named le on a histogram doubled the label the bucket lines append themselves (reserved only there — on a counter it is legal Prometheus, and a test pins the scoping); and re-registering a histogram compared kind and help but not buckets, beneath a comment promising "a mismatch aborts rather than picking one". Counter::Increment now aborts on a negative amount. The decrease fails nowhere visible — rate() reads it as a counter reset and extrapolates from zero, inflating exactly the panel someone is staring at. client_golang panics here for the same reason. Checked before the disabled-registry early-return, so enabling metrics in production is never the first time it runs. Also replaces a vacuous transport assertion: the scrape-does-not-count- itself check searched for operation=""/status= spellings that cannot appear in this exposition at all, so it passed under any composition. It now asserts route="/metrics" is absent, which the endpoint's self-label makes falsifiable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014C7WdBD99mUFWGxMvGSjSU
Observe derived the ADR-0012 client on every request — a header walk, an
address parse, and string building — even in chains whose sinks never
read it, which is every metrics-only composition: RecordMetrics
deliberately ignores observation.client, and the loopback paths this
repo times are the ones that notice.
The trust boundary is now optional, defaulting to none-supplied, which
skips the derivation entirely and leaves client value-initialized at
Source::kUnknown. Any supplied boundary derives, including
TrustedProxies::None(), which keeps its meaning as the explicit
direct-connect statement rather than doubling as "I didn't think about
it". A sink that wants the client states the boundary, the way
PerClientRateLimit already must — and both of the access-log formatter's
call sites already do.
Also from the same review: the guide's application-metrics example
called NewHistogram without the buckets argument the API deliberately
requires, so the documented snippet did not compile (and named its
metric _seconds in a microseconds dialect); the CHANGELOG introduced the
endpoint as "three families" and enumerated five in the same sentence;
max_series was documented against the {method,route,status} keying an
earlier commit removed; and BuiltInLabels promised an empty-name drop
its rewrite no longer performs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014C7WdBD99mUFWGxMvGSjSU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Eleven findings from a critical post-merge review of #199, in two commits. Four are real correctness defects in the application-metrics path, and all four produce the failure mode that code's own comments name as the worst case: a scrape Prometheus rejects in full, with no in-process consumer to notice.
Every rendered value is exact now
FormatNumberhad two related holes:snprintf's would-have-written length, unclamped against its 64-byte buffer — an out-of-bounds read on every scrape for any value whose%.6fform is longer than that.gauge.Set(1e60)reaches it; so does a bucket bound of1e60, which the ladder validation happily admits.%.6fis lossy below its granularity.NewHistogram("s","S",{1e-8, 1e-7})passes the strictly-ascending check while both bounds render"0"— two identicallelabels, a duplicate series, the whole scrape refused. Same trap one decimal further in:1.1e-6and1.2e-6both rendered"0.000001".One rule fixes both: the fixed six-decimal spelling is kept only when it parses back to exactly the value it claims, and everything else gets a shortest round-trip form (
%.15g/%.16g/%.17g, verified by parse-back). The stability half matters as much as the exactness half — rendered strings are series identity, sole="2500000"must never becomele="2.5e+06". A test pins the historical spellings (0.005,2500000,-0.25) untouched; the ASan job is what makes the out-of-bounds test a proof rather than a hope.Misuse aborts instead of corrupting the scrape
Three gaps where the file's own fail-fast posture (ADR-0009) didn't reach:
{a="1",a="2"}— a parse error that fails the entire scrape — while the invalid-name check beside them aborted.leon a histogram doubled the label the bucket lines append themselves. Reserved only where it collides: on a counter or gauge,leis legal Prometheus and a test pins that the reservation stays scoped.And one delegated design call:
Counter::Incrementnow aborts on a negative amount. The decrease fails nowhere visible —rate()reads it as a counter reset and extrapolates from zero, inflating exactly the panel someone is staring at. prometheus/client_golang panics here for the same reason. The check runs before the disabled-registry early-return, so enabling metrics in production is never the first time it fires.The client derivation is opt-in (the other delegated call)
Observederived the ADR-0012 client on every request — header walk, address parse, string building — even in chains whose sinks never read it, which is every metrics-only composition. The trust boundary is nowstd::optional, defaulting to skip:clientstays value-initialized atSource::kUnknownand the request pays nothing. Any supplied boundary derives, includingTrustedProxies::None(), which keeps its meaning as the explicit direct-connect statement rather than doubling as "I didn't think about it". Both of the access-log formatter's call sites (#206) already pass a boundary explicitly, so nothing out of tree changes behavior.Cleanup from the same review
operation=""/status=spellings that cannot appear in this exposition at all, so it passed under any composition. It now assertsroute="/metrics"absence, which the endpoint's self-label makes falsifiable.NewHistogramwithout the buckets argument the API deliberately requires — the documented snippet didn't compile, and named its metric_secondsin a microseconds dialect.max_serieswas documented against the removed{method,route,status}keying;BuiltInLabelspromised an empty-name drop its rewrite no longer performs.Testing
Fifteen new tests: four value-formatting (huge exact, tiny buckets distinct, past-six-decimals distinct, historical spellings byte-stable), six fail-fast death tests plus the two scoping/idempotence positives, the derivation-skip pin (peer and header present,
clientstayskUnknown), and the two updated tests now stating their boundary explicitly.Locally:
make verify(129 tests under--config=werror, lockfiles, codegen, goldens, clang-format),make noexcept, gcc ASan and TSan on the touched targets, clang-tidy clean on the touched files, buildifier via npx, and the 15-target consumer suite. The onlyverifyfailure is the sandbox's missing buildifier binary, verified vianpx @bazel/buildifier@8.2.1instead.Checklist
bazel test //...and(cd codegen && gradle build spotlessCheck)pass locally🤖 Generated with Claude Code
https://claude.ai/code/session_014C7WdBD99mUFWGxMvGSjSU
Generated by Claude Code