feat: alert when an audit sidecar stops delivering

[scotwells]

Summary

Production lost about 75% of its audit events for three hours today and nothing paged. Two of the three audit collectors stalled, the third kept working, and every alert averaged across all three — so the pipeline looked healthy the whole time. It was spotted by eye on a dashboard.

After this, a single stalled collector pages within about five minutes, and whoever answers gets a runbook that walks them from the alert to the fix.

While checking why nothing fired, four existing rules turned out to be incapable of firing at all — three referencing metrics that no longer exist, one comparing two things that can never match. A fifth was set to page only after roughly 83 minutes of a completely stopped consumer, a number chosen against a metric that never existed and so never checked against real traffic.

• Two new alerts covering the collectors, evaluated per pod so one bad pod is visible behind two healthy ones
• Four repaired rules, plus three more carrying the same stale metric name
• The critical backlog alert now pages in minutes rather than an hour and a half
• A runbook per new alert: what broke, what to check, how to recover, how to confirm the backlog cleared

Note

The backlog alert fires the moment this merges — a collector is still working through events left over from today's incident. That is correct, and it clears itself as the backlog drains.

Root cause of the outage is fixed separately. This is detection and response only; no application code changes.

Related to #226

Validation

Worth being precise about, since the bug being fixed is rules that quietly never fire — proving a rule stays quiet proves very little on its own.

Confirmed to fire on real failure data:

• VectorSidecarNATSPublishStalled — went true at 18:50 and 18:52 UTC during today's outage, 5–6 minutes after each collector stalled
• VectorSidecarBufferBacklog — firing now on genuine leftover backlog

Structurally verified only. The repaired rules cover conditions that have not occurred in 30 days, so there is no real failure to test them against. For each, the metric, labels, and thresholds were checked to be live and reachable — but none has been observed firing:

• ActivityDataPipelineStalled, ActivityGenerationStalled — the quantities they watch have never touched zero (11.75/s and 11.24/s minimums over 30 days), so the rules are sound but unexercised
• VectorClickHouseWritesStopped — label sets intersect on the three aggregator pods; collectors drop out on their own, having no ClickHouse stage
• ActivityPipelineBacklogCritical, NATSConsumerLagHigh — consumer names match a real consumer, and the new 50k threshold sits 25,000x above the highest backlog seen in 30 days

Every new and repaired rule produces zero firings across 30 days of history.

Test plan

• Both alert files parse and kustomize build config/components/observability is clean
• Every metric, label value, and regex referenced by a new or edited rule confirmed present in production
• Both runbook links resolve to files added here, and every command in them was run against production
• No new or repaired rule fires against 30 days of healthy history
• Confirm in staging that the rules load and appear in the alert list
• Exercise the repaired rules against an induced failure in staging — none can be true-positive tested against production history

Warning

Two of these dead expressions also exist in the Jsonnet mixin. #155 needs to fix them there before it moves alert authoring into Jsonnet, or they will come back.