fix(observability): sample rollup misses per reason per interval, not 1-in-512

[tonyalaribe]

The problem

rollup_miss_sampled renders the refused plan and is the tool that previously turned a "97% missing_project" mystery into a one-line fix. It currently logs nothing.

const ROLLUP_MISS_SAMPLE: u64 = 512;   // 1 in 512 misses

That constant was right for the regime it was written in — the call site records ~2.7 misses/second (~9,700/hour), where 1:512 yields ~19 samples/hour. Prod on 2026-08-21 ran ~170 misses/hour, so it fires about once every three hours, and a 30-minute diagnostic window contained zero events.

A fixed sampling ratio goes silent exactly when the diagnostic becomes most useful. While misses are a flood, 1:512 is right. Once they are a trickle, the survivors are the interesting ones — and the sampler hides them.

Why per-reason and not just a smaller constant

The actionable reason is usually not the loudest one. The window that prompted this:

filter_not_eligible  38   (37%)  ← a correct refusal
unknown_filter       28   (27%)  ← a GAP IN THE MATCHER, the addressable category
not_built            24   (23%)
tiny_interior        12   (12%)  ← a correct refusal

A global sampler spends its budget on whichever reason dominates. Lowering the constant fixes today's rate and breaks again at the next traffic level.

The change

• sample_rollup_miss(reason) — at most one sample per MissReason per 60s. Rate-based, so it cannot go quiet as traffic falls; per-reason, so a loud refusal cannot crowd out a rare actionable one.
• MissReason::ALL is declared [Self; Self::COUNT], so adding a variant fails the build rather than silently indexing out of range — a compile-time guarantee rather than a test that can rot.
• Three call sites updated. The two diagnostic sites in database/mod.rs that borrow this limiter now pass the semantically closest reason (IncompleteCoverage, TinyInterior).

Verification

• new test sweeps MissReason::ALL: every reason samples once, then rate-limits
• verified the test FAILS against the old ratio sampler — it is load-bearing, not decorative
• 82 rollup lib tests pass; cargo lint clean (only pre-existing vendored walrus-rust warnings); cargo fmt applied

Note on scope

Deliberately small and observability-only — it changes no query or maintenance behaviour. Context for why this matters now is in #206: routing hit rate reads ~7% at some hours versus 91.5% at others, and this sampler is the blocker on diagnosing which, because it is the only thing that renders the plan that was refused.

[claude]

Reviewed the diff. This is a well-scoped, well-motivated change — the per-reason/rate-based redesign is the right fix for the problem described (a fixed ratio going silent as traffic drops), the atomic CAS logic in sample_rollup_miss is race-free, and the call sites are threaded through correctly. Two things worth a look before merge:

1. The MissReason::COUNT/ALL "compile-time guarantee" doesn't actually hold (src/rollup.rs:46-66, src/observability.rs:680)

The doc comments (and the PR description) claim adding a new MissReason variant "fails the build rather than silently indexing out of range." That's not what happens. COUNT is a hand-written literal (= 15) and ALL is a hand-written array — neither is derived from the enum. If someone adds a 16th variant and forgets to update COUNT/ALL:

• The ALL: [Self; Self::COUNT] array literal still has 15 elements matching COUNT = 15 — it compiles fine.
• LAST in sample_rollup_miss is sized [AtomicI64; MissReason::COUNT] = 15 slots.
• The moment sample_rollup_miss(new_variant) is called, reason as usize can be 15 (or shift others if inserted mid-enum), and LAST[reason as usize] panics with an out-of-bounds index at runtime — in production, on the exact path this PR is trying to make more reliable.

Contrast with record_rollup_miss's match a few lines below, which is genuinely exhaustive and would fail to compile on a missing arm — that one keeps its promise, the array-based one doesn't.

Suggest either deriving COUNT/ALL (e.g. strum::EnumCount/EnumIter) so they can't drift from the enum, or replacing the fixed-size array with something keyed by an exhaustive match (so the compiler enforces coverage the way record_rollup_miss does). At minimum, the doc comment's claim should be corrected if the array stays hand-maintained.

2. Test only covers "samples once," not "resamples after the interval" (src/observability.rs:1247-1254)

every_reason_samples_once_then_rate_limits_independently verifies the first call succeeds and the immediate second call is suppressed, but never advances time to confirm a reason samples again after SAMPLE_INTERVAL_SECS. The codebase already has a frozen-clock test seam for exactly this (crate::support::set_micros/advance_micros, used the same way in hot_tier.rs around line 1770 to jump past a cooldown and assert a gate reopens) — worth using here too so the interval-elapses half of the behavior is actually verified, not just the interval-doesn't-elapse half.

Minor/non-blocking: the LAST array is a function-local static, so it's process-global shared state; if this test ever runs more than once in the same process (e.g. via a flaky-test retry harness) the second run's first assertion would fail spuriously since state persists from the first run. Low risk given there's only one test touching it today, just flagging for awareness.

Nothing else stood out — no security concerns (all inputs are internal enum values, not attacker-controlled), and the per-reason design genuinely solves the "loud reason crowds out rare actionable one" problem described in the PR.