From 2e081e548668ff5d8e9f19d7450559a8dacd83eb Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Mon, 24 Aug 2026 05:06:18 +0000 Subject: [PATCH] feat(alerts): a dead man's switch on the notification path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AlertmanagerNotificationsFailing catches delivery ERRORS — a refused connection, a 5xx. It cannot catch a webhook URL that is well-formed, reachable, and pointed at nothing: a 200 into a deleted ntfy topic is a successful notification by every measure Alertmanager has. That is not hypothetical here. The webhook was the ntfy.example.invalid placeholder for the entire life of the stack and nothing noticed, because the only symptom is that alerts stop arriving — which is also what a healthy week looks like. This repository already applies the reasoning elsewhere. security.rules.yaml has FirewallLogsStopped because a firewall passing traffic for the whole house is never silent for thirty minutes, and it spends ten lines explaining why there is deliberately no SuricataStopped rule: absence of alerts is indistinguishable from absence of the service, and detecting that needs a heartbeat rather than a threshold. The notification path was the one place the argument had never been turned on itself. So: one rule, `vector(1)`, firing unconditionally and forever. Its firing carries no information. Its ABSENCE is the entire signal, and one `continue: true` — the only one in the routing tree — makes that absence observable two ways: heartbeat external cron-monitor ping 5m Prometheus stopped evaluating, Alertmanager died, no egress default the real alert channel 24h the alert channel itself is a 200 into nothing Neither half substitutes for the other. The heartbeat proves delivery to a DIFFERENT url than real alerts use, so it cannot see a deleted topic. The daily notification travels the identical url your warnings travel, but nothing machine-checks its absence — you do. The watcher lives off this host by necessity: one here fails at the same moment as the thing it watches. `severity: none` on the rule is load-bearing, not a placeholder. It keeps the alert out of the bare severity routes, where `info` would send it to the "null" receiver; and out of the second inhibit rule, which matches severity =~ "warning|critical" with equal: ["instance"] — Alertmanager treats absent == absent as equal, so a Watchdog carrying a warning severity and no instance label would be silenced every time InstanceDown fired. A dead man's switch that goes quiet exactly when a host goes down is worse than none. That is what the unit test asserts. It has no paired quiet case, deliberately — `vector(1)` is unconditional, so a silent case would be asserting a bug — and it checks the label set instead, which is the part that can break without promtool or the Prometheus status page noticing. Confirmed by mutation: changing severity to `warning` fails the test. The routing assertions gained a row for the same reason, and it is the only one expecting two receivers. Rule counts updated where they are live claims: 34 → 35 metric, 47 → 48 total. docs/roadmap.md:182 is left alone; it records what was built at the time, and #72 owns that drift. Closes #67 Co-Authored-By: Claude Opus 5 --- .github/workflows/ci.yml | 1 + README.md | 4 +- docs/observability.md | 49 ++++++- docs/runbooks/deploy-stack.md | 2 +- docs/runbooks/verify-the-alert-path.md | 121 ++++++++++++++++++ docs/security.md | 2 +- scripts/render-config.sh | 2 + scripts/validate.sh | 8 +- scripts/verify-key-backup.sh | 1 + secrets/README.md | 5 +- secrets/observability.example.yaml | 16 +++ .../alertmanager/alertmanager.yaml | 45 ++++++- .../loki/rules/security.rules.yaml | 5 + .../prometheus/rules/watchdog.rules.yaml | 65 ++++++++++ .../prometheus/tests/watchdog.test.yaml | 71 ++++++++++ 15 files changed, 384 insertions(+), 13 deletions(-) create mode 100644 docs/runbooks/verify-the-alert-path.md create mode 100644 stacks/observability/prometheus/rules/watchdog.rules.yaml create mode 100644 stacks/observability/prometheus/tests/watchdog.test.yaml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 658ec21..e674871 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -166,6 +166,7 @@ jobs: --verify.receivers="$expected" $labels \ || { echo "::error::expected $expected for $labels"; fail=1; } done <<'ROUTES' + heartbeat,default alertname=Watchdog severity=none category=monitoring urgent severity=critical category=power security severity=critical category=security security severity=warning category=security diff --git a/README.md b/README.md index 5fbe32f..b2f9a4d 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ incident. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 5 provisioned dashboards, 84 panels, and - 47 alert rules — 34 metric-based in Prometheus, 13 log-based in Loki — sharing + 48 alert rules — 35 metric-based in Prometheus, 13 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -143,7 +143,7 @@ rack; a dashed border means egress only. Full topology and data flow in . ├── stacks/observability/ # the deployed stack — one compose file, six services │ ├── compose.yaml -│ ├── prometheus/ # config, file_sd targets, 34 alert rules +│ ├── prometheus/ # config, file_sd targets, 35 alert rules │ ├── alertmanager/ # routing and inhibition │ ├── loki/ # single-binary config + 13 LogQL rules │ ├── alloy/ # one agent config, used on every host diff --git a/docs/observability.md b/docs/observability.md index 9eff662..3245a7f 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -70,7 +70,7 @@ expression in every panel is syntactically valid. ## Alerting -47 rules in total: 34 metric-based in `prometheus/rules/`, and 13 log-based in +48 rules in total: 35 metric-based in `prometheus/rules/`, and 13 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -99,7 +99,7 @@ boot check. ### Metric-based (Prometheus) -34 rules across four files in `prometheus/rules/`: +35 rules across five files in `prometheus/rules/`: | File | Covers | | --- | --- | @@ -107,6 +107,7 @@ boot check. | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware | | `ups.rules.yaml` | On battery, low battery, runtime, load, temperature | | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling, and the stack watching itself | +| `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | `promtool check rules` validates that these parse. It does not — and cannot — tell you whether a rule can ever be true: `ContainerHighMemory` passed it for @@ -115,9 +116,9 @@ and healthy and could not fire for any input ([#63](https://github.com/Gerrrt/Ho `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is one rule of 34 so far: -`ContainerHighMemory`. The other 33 are still validated for syntax only, which -is exactly the standing #63 had. +against the broken rule too. Coverage is two rules of 35 so far: +`ContainerHighMemory` and `Watchdog`. The other 33 are still validated for +syntax only, which is exactly the standing #63 had. Disk alerting is predictive rather than a fixed threshold — `predict_linear` over a 6-hour window, firing when the extrapolation reaches zero within a day *and* @@ -160,6 +161,44 @@ Inhibit rules stop cascades: a down host suppresses its own disk warnings, and a dead `snmp-exporter` suppresses the "every device is unreachable" storm that would otherwise follow. +### The dead man's switch + +`AlertmanagerNotificationsFailing` catches delivery *errors*. It cannot catch a +webhook URL that is well-formed, reachable, and pointed at nothing — a 200 into a +deleted ntfy topic is a successful notification by every measure Alertmanager +has. That is not hypothetical: the webhook was the `ntfy.example.invalid` +placeholder for the entire life of the stack and nothing noticed, because the +only symptom is that alerts stop arriving, which is also what a healthy week +looks like ([#67](https://github.com/Gerrrt/HomeLab/issues/67)). + +`prometheus/rules/watchdog.rules.yaml` holds one rule, `Watchdog`, whose +expression is `vector(1)`. It fires unconditionally and forever. **Its firing +carries no information; its absence is the entire signal.** One `continue: true` +— the only one in the tree — sends it to two places: + +| Route | Destination | Cadence | Catches | +| --- | --- | --- | --- | +| `heartbeat` | external cron-monitor ping | 5m | Prometheus stopped evaluating, Alertmanager died, no outbound network | +| `default` | the real alert channel | 24h | the alert channel itself is a 200 into nothing | + +Neither half substitutes for the other. The heartbeat proves delivery to a +*different* URL than real alerts use, so it cannot see a deleted topic; the daily +notification travels the identical URL your warnings travel, but nothing +machine-checks its absence. + +The watcher lives off this host by necessity — a watcher here fails at the same +moment as the thing it is watching. Setting it up, the coupling between +`repeat_interval` and the external check's period and grace, and how to read +which half went quiet are in +[`runbooks/verify-the-alert-path.md`](runbooks/verify-the-alert-path.md). + +This is the same reasoning `loki/rules/security.rules.yaml` already applies to +the firewall with `FirewallLogsStopped`, and the reason it gives for deliberately +*not* writing a `SuricataStopped` rule: absence of alerts is indistinguishable +from absence of the service, and detecting that needs a heartbeat rather than a +threshold. The notification path was the one place that argument had not been +turned on itself. + ## Adding a monitored device See [`runbooks/add-monitored-device.md`](runbooks/add-monitored-device.md). In diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index a3455d3..24a7e00 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -43,7 +43,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. -2. **Prometheus → Status → Rules.** 34 rules loaded, none in error. +2. **Prometheus → Status → Rules.** 35 rules loaded, none in error. 3. **Grafana → Dashboards → HomeLab.** Five dashboards, populated. 4. **Grafana → Explore → Loki**, run `{host=~".+"}`. Logs should be arriving. 5. Confirm level normalisation is working — this has been silently broken diff --git a/docs/runbooks/verify-the-alert-path.md b/docs/runbooks/verify-the-alert-path.md new file mode 100644 index 0000000..57cad93 --- /dev/null +++ b/docs/runbooks/verify-the-alert-path.md @@ -0,0 +1,121 @@ +# Verify the alert path + +**Who watches the thing that tells you something is wrong.** + +`AlertmanagerNotificationsFailing` catches delivery *errors* — a refused +connection, a 5xx. It cannot catch a webhook URL that is well-formed, reachable, +and pointed at nothing. A 200 into a deleted ntfy topic is a successful +notification by every measure Alertmanager has, and the only symptom is that +alerts stop arriving, which is also what a healthy week looks like. + +This lab has already lived through that failure: the webhook was the +`ntfy.example.invalid` placeholder for the entire life of the stack and nothing +noticed. See [#67](https://github.com/Gerrrt/HomeLab/issues/67). + +## The two halves + +`prometheus/rules/watchdog.rules.yaml` holds one rule, `Watchdog`, whose +expression is `vector(1)`. It fires unconditionally and forever. Its firing +carries no information; **its absence is the entire signal.** + +Alertmanager sends it to two places, from one rule, via the only `continue: true` +in the routing tree: + +| Route | Destination | Cadence | Catches | +| --- | --- | --- | --- | +| `heartbeat` | external cron-monitor ping | every 5m | Prometheus stopped evaluating, Alertmanager died, the host lost outbound network | +| `default` | the real alert channel | every 24h | the alert channel itself is a 200 into nothing | + +Both halves are needed, and neither substitutes for the other. The heartbeat +route proves delivery to a *different* URL than real alerts use, so it cannot +see a deleted ntfy topic. The daily route travels the identical URL your warnings +travel, but nothing machine-checks its absence — you do. + +## Setting up the external watcher + +The watcher has to live somewhere other than the monitoring host. A watcher on +this host fails at the same moment as the thing it is watching, which is not +watching at all. + +A cron-monitor / heartbeat service is the least effort: +[healthchecks.io](https://healthchecks.io) (free tier is enough for one check), +Cronitor, or an Uptime Kuma "push" monitor on any other machine. + +1. Create one check. Name it so a 3am notification is self-explanatory — + `homelab alerting path`, not `check 1`. +2. Set **period 5m** and **grace 15m**. See "The timing is coupled" below before + changing either. +3. Point the check's own notification at something that is **not** the webhook + this stack uses. If both go to the same ntfy topic, a deleted topic takes out + the alert and the warning about the alert together. Email is fine here; it + fails independently. +4. Put the ping URL into the encrypted secrets file and render: + + ```bash + make secrets-edit # set ALERTMANAGER_HEARTBEAT_URL + make up + ``` + +5. Confirm the check goes green within one `repeat_interval`. + +## The timing is coupled + +Alertmanager sends the first notification after `group_wait` and then re-sends +every `repeat_interval`. The heartbeat route uses `group_wait: 0s` and +`repeat_interval: 5m`. + +- **External period ≥ `repeat_interval`.** A period shorter than 5m expects pings + that are never sent, and the check alarms on a perfectly healthy stack. +- **External grace ≥ 2 × `repeat_interval`.** One missed ping is a hiccup — a + reload, a restart, a slow scrape. Two consecutive misses is a fault. A grace + under 10m turns every `make reload` into a page. + +Change `repeat_interval` in `alertmanager/alertmanager.yaml` and the external +check's period and grace move with it. Nothing enforces that from here, which is +why it is written down. + +## Confirming it actually works + +Do not trust a green check you have never seen go red. + +```bash +docker stop alertmanager +# wait out the grace window — 15m by default +# the external check must report DOWN and notify you +docker start alertmanager +# it must return to green within one repeat_interval +``` + +Doing this once is worth more than the rule is. A dead man's switch nobody has +ever seen trip is indistinguishable from a dead man's switch that does not work. + +To confirm the daily half without waiting a day, temporarily lower +`repeat_interval` on the second Watchdog route, `make reload`, and check the +notification arrives on your normal alert channel: + +```bash +amtool alert query --alertmanager.url=http://localhost:9093 alertname=Watchdog +``` + +Put it back to `24h` afterwards. + +## Reading the failure + +| What you see | What it means | +| --- | --- | +| External check DOWN, daily heartbeat still arriving | The heartbeat URL is wrong or that specific destination is unreachable. The alert path itself is fine. | +| External check UP, daily heartbeat stopped | The **real alert channel** is broken — a deleted topic, a rotated URL. This is #67's original failure, and every real alert is being lost right now. | +| Both stopped | Prometheus, Alertmanager, or this host. Start with `docker compose ps` and `curl -s localhost:9093/-/healthy`. | +| Both fine, but you expected an alert about something else | Not this runbook. The path works; check the rule, then the routing tree with `amtool config routes test`. | + +The second row is the one this whole arrangement exists for, and it is the one +that looks like nothing is wrong. + +## Related + +- `prometheus/rules/watchdog.rules.yaml` — the rule, and why `severity: none` is + load-bearing rather than a placeholder +- `loki/rules/security.rules.yaml` — `FirewallLogsStopped`, and the comment + explaining why Suricata deliberately has no equivalent rule. Same reasoning, + applied to a different silent component +- [`docs/observability.md`](../observability.md#routing) — the full routing table diff --git a/docs/security.md b/docs/security.md index a61844d..7ea8885 100644 --- a/docs/security.md +++ b/docs/security.md @@ -14,7 +14,7 @@ What this network is actually built to survive: | A smart TV's firmware phoning somewhere unexpected | VLAN 40 is terminal, egress only | | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | -| Losing visibility of a failure | 34 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 35 alert rules, 30 days of metrics and logs | | Mains power loss | **Not currently defended.** `mjolnir` has no battery installed — see below | What it explicitly does **not** defend against: a determined attacker with diff --git a/scripts/render-config.sh b/scripts/render-config.sh index 00720b1..1f6eb10 100755 --- a/scripts/render-config.sh +++ b/scripts/render-config.sh @@ -45,6 +45,7 @@ REQUIRED=( ALERTMANAGER_WEBHOOK_URL ALERTMANAGER_URGENT_WEBHOOK_URL ALERTMANAGER_SECURITY_WEBHOOK_URL + ALERTMANAGER_HEARTBEAT_URL SNMP_COMMUNITY_PFSENSE SNMP_COMMUNITY_APC SNMP_COMMUNITY_MOKERLINK @@ -124,6 +125,7 @@ AM_CHANNELS=( "ALERTMANAGER_WEBHOOK_URL:webhook_url" "ALERTMANAGER_URGENT_WEBHOOK_URL:urgent_url" "ALERTMANAGER_SECURITY_WEBHOOK_URL:security_url" + "ALERTMANAGER_HEARTBEAT_URL:heartbeat_url" ) AM_OUT_DIR="${STACK_DIR}/alertmanager/.rendered" if [[ -f "${STACK_DIR}/alertmanager/alertmanager.yaml" ]]; then diff --git a/scripts/validate.sh b/scripts/validate.sh index c6dbde1..5d8f106 100755 --- a/scripts/validate.sh +++ b/scripts/validate.sh @@ -153,6 +153,11 @@ if ((${#AMTOOL[@]})); then # # Each line below is one row of the table in alertmanager.yaml. # --verify.receivers exits non-zero when the resolved receiver differs. + # + # The Watchdog row expects TWO receivers because its first route sets + # `continue: true` — the one place in the tree that does. That row is also + # what catches a `severity` slip on the watchdog rule: with `info` it would + # resolve to "null" and the dead man's switch would be silently disarmed. routes_ok=1 while read -r expected labels; do [[ -n "${expected}" ]] || continue @@ -166,6 +171,7 @@ if ((${#AMTOOL[@]})); then routes_ok=0 fi done <<'ROUTES' +heartbeat,default alertname=Watchdog severity=none category=monitoring urgent severity=critical category=power security severity=critical category=security security severity=warning category=security @@ -175,7 +181,7 @@ null severity=info category=correctness ROUTES if ((routes_ok)); then - pass "amtool config routes test (6 assertions)" + pass "amtool config routes test (7 assertions)" else fail "amtool config routes test" fi diff --git a/scripts/verify-key-backup.sh b/scripts/verify-key-backup.sh index 3c888b7..2fb86bb 100755 --- a/scripts/verify-key-backup.sh +++ b/scripts/verify-key-backup.sh @@ -218,6 +218,7 @@ REQUIRED=( ALERTMANAGER_WEBHOOK_URL ALERTMANAGER_URGENT_WEBHOOK_URL ALERTMANAGER_SECURITY_WEBHOOK_URL + ALERTMANAGER_HEARTBEAT_URL SNMP_COMMUNITY_PFSENSE SNMP_COMMUNITY_APC SNMP_COMMUNITY_MOKERLINK diff --git a/secrets/README.md b/secrets/README.md index df8a7d7..55f2010 100644 --- a/secrets/README.md +++ b/secrets/README.md @@ -85,8 +85,9 @@ file, exports the values as environment variables, and: - renders `snmp-exporter/snmp.yaml`'s `${SNMP_COMMUNITY_*}` placeholders into `snmp-exporter/.rendered/snmp.yaml`, which is what the container mounts; - writes one file per notification channel into `alertmanager/.rendered/` — - `webhook_url`, `urgent_url` and `security_url` — because Alertmanager does not - expand environment variables and reads receiver URLs via `url_file`. + `webhook_url`, `urgent_url`, `security_url` and `heartbeat_url` — because + Alertmanager does not expand environment variables and reads receiver URLs + via `url_file`. All of them are gitignored, and each secret is written to exactly one of them. Nothing writes a secret into a tracked file. diff --git a/secrets/observability.example.yaml b/secrets/observability.example.yaml index b241f31..250a96b 100644 --- a/secrets/observability.example.yaml +++ b/secrets/observability.example.yaml @@ -52,3 +52,19 @@ SNMP_COMMUNITY_ILO: change-me-ilo ALERTMANAGER_WEBHOOK_URL: https://ntfy.example.invalid/homelab-alerts ALERTMANAGER_URGENT_WEBHOOK_URL: https://ntfy.example.invalid/homelab-urgent ALERTMANAGER_SECURITY_WEBHOOK_URL: https://ntfy.example.invalid/homelab-security + +# The dead man's switch (#67). NOT an ntfy topic — this is a cron-monitor ping +# URL from something that lives OFF this host: healthchecks.io, Cronitor, an +# Uptime Kuma push monitor, or your own endpoint on another machine. +# +# The Watchdog rule fires forever and Alertmanager posts here every 5 minutes. +# Its arrival means nothing. Its absence means the alerting path is broken, and +# only something outside this stack can notice that — a watcher on the +# monitoring host fails at the same moment as the thing it is watching. +# +# Set the external check's period to 5m and its grace to 15m, and read +# docs/runbooks/verify-the-alert-path.md before changing either: the period is +# tied to `repeat_interval` on the heartbeat route in alertmanager.yaml, and +# raising one without the other produces a check that alarms on a healthy stack +# or one that never alarms at all. +ALERTMANAGER_HEARTBEAT_URL: https://hc-ping.example.invalid/00000000-0000-0000-0000-000000000000 diff --git a/stacks/observability/alertmanager/alertmanager.yaml b/stacks/observability/alertmanager/alertmanager.yaml index 315da4b..1c373c9 100644 --- a/stacks/observability/alertmanager/alertmanager.yaml +++ b/stacks/observability/alertmanager/alertmanager.yaml @@ -11,7 +11,7 @@ # at /etc/alertmanager/secrets/. Both the rendered directory and its contents # are gitignored. # -# There are three channels and three destinations. They used to be three names +# There are four channels and four destinations. They used to be three names # for one URL, which meant `urgent` and `default` differed only in how often # they repeated — a UPS on battery and a slow scrape landed in the same place # (#66). Point them at three separate topics on whatever receiver you use, so @@ -35,6 +35,40 @@ route: # config still loads and still validates. scripts/validate.sh asserts the # whole table with `amtool config routes test` for exactly that reason. routes: + # --- The dead man's switch, and it has to be first --------------------- + # + # Watchdog fires unconditionally and forever (prometheus/rules/watchdog.rules.yaml). + # Its firing means nothing; its ABSENCE is the signal, so both of these + # routes exist to make the absence observable in two different ways. + # + # `continue: true` on the first is what lets ONE rule produce TWO + # notifications on two cadences, rather than duplicating the expression in + # the rule file. It is the only `continue` in this tree — everything below + # is first-match-wins. + # + # 5m to the heartbeat URL, which is a cron-monitor ping watched from off + # this host. Keep this at or under the external check's period; the runbook + # states the relationship, so retuning one without the other is visible. + - receiver: heartbeat + matchers: + - alertname = "Watchdog" + group_wait: 0s + group_interval: 5m + repeat_interval: 5m + continue: true + + # 24h to the REAL alert channel. This is the half that catches #67's actual + # failure: a webhook that is well-formed, reachable, returns 200 and is + # pointed at nothing. It travels the identical URL real warnings travel, so + # a deleted topic shows up as a missing daily heartbeat instead of as + # nothing at all. The heartbeat route above cannot see that, because it + # proves delivery to a different URL. + - receiver: default + matchers: + - alertname = "Watchdog" + group_wait: 0s + repeat_interval: 24h + # Power events page immediately and repeat aggressively — a UPS on battery # with no spare battery installed is a short fuse. - receiver: urgent @@ -113,3 +147,12 @@ receivers: webhook_configs: - url_file: /etc/alertmanager/secrets/security_url send_resolved: true + + # send_resolved: false, deliberately. If Prometheus stops sending Watchdog, + # Alertmanager expires it after resolve_timeout and the pings simply stop — + # which is precisely the signal the external check is waiting for. A resolved + # notification here would be one last successful ping on the way out the door. + - name: heartbeat + webhook_configs: + - url_file: /etc/alertmanager/secrets/heartbeat_url + send_resolved: false diff --git a/stacks/observability/loki/rules/security.rules.yaml b/stacks/observability/loki/rules/security.rules.yaml index 6fe1e13..92effe6 100644 --- a/stacks/observability/loki/rules/security.rules.yaml +++ b/stacks/observability/loki/rules/security.rules.yaml @@ -248,6 +248,11 @@ groups: # died needs a heartbeat or a process metric, not a log rule, and inventing # a threshold here would produce exactly the sort of alert that cannot fire # while looking like one that can. Tracked in docs/roadmap.md. + # + # prometheus/rules/watchdog.rules.yaml is the worked example of the heartbeat + # this comment is describing, built for the notification path (#67). Suricata + # would need the same shape: something that reports in when healthy, and an + # observer that acts on the silence. # --------------------------------------------------------------------------- - name: ids interval: 1m diff --git a/stacks/observability/prometheus/rules/watchdog.rules.yaml b/stacks/observability/prometheus/rules/watchdog.rules.yaml new file mode 100644 index 0000000..06aaee2 --- /dev/null +++ b/stacks/observability/prometheus/rules/watchdog.rules.yaml @@ -0,0 +1,65 @@ +--- +# A dead man's switch on the notification path. +# +# AlertmanagerNotificationsFailing (containers.rules.yaml) catches delivery +# ERRORS — a refused connection, a 5xx. It cannot catch a webhook URL that is +# well-formed, reachable, and pointed at nothing: a 200 into a deleted ntfy +# topic is a successful notification by every measure Alertmanager has. +# +# That is not hypothetical here. The webhook was the ntfy.example.invalid +# placeholder for the entire life of the stack and nothing noticed, because the +# only symptom is that alerts stop arriving — which is also what a healthy week +# looks like. +# +# This repository already applies exactly this reasoning elsewhere. +# security.rules.yaml has FirewallLogsStopped because a firewall passing traffic +# for the whole house is never silent for thirty minutes, and it explains at +# length why there is deliberately NO SuricataStopped rule: "absence of alerts is +# indistinguishable from absence of the service", and detecting that needs a +# heartbeat, not a log rule. The notification path was the one place the +# reasoning was never turned on itself. Tracked in #67. +# +# THE INVERSION THAT MAKES THIS RULE MAKE SENSE: its firing carries no +# information whatsoever. Its ABSENCE is the entire signal. Nothing inside this +# stack can act on that, because a watcher on this host fails at the same moment +# as the thing it is watching — so the watching is done by an external service, +# and docs/runbooks/verify-the-alert-path.md is where that half is written down. +groups: + - name: watchdog + interval: 1m + rules: + - alert: Watchdog + # Unconditional, by design. `vector(1)` has no inputs, so there is no + # data problem, no scrape, no label join and no threshold that can stop + # this from firing. Anything that stops the heartbeat arriving is + # therefore a fault in the path itself: rule evaluation, the send to + # Alertmanager, routing, or delivery. + expr: vector(1) + for: 0s + labels: + component: watchdog + # severity: none is load-bearing, not a placeholder. + # + # 1. It keeps the alert out of the three bare `severity` routes at the + # bottom of alertmanager.yaml. With `info` it would be swallowed by + # the "null" receiver; with `warning` or `critical` it would page. + # Its own routes match on alertname and sit above all of them. + # + # 2. It keeps the alert out of both inhibit_rules. The second matches + # target_matchers: [severity =~ "warning|critical"] with + # equal: ["instance"] — and Alertmanager treats absent == absent as + # equal, so a Watchdog carrying a warning severity and no instance + # label would be silenced every single time InstanceDown fired. + # A dead man's switch that goes quiet exactly when a host goes down + # is worse than none: it would report the path healthy right up to + # the moment it stopped, and then look like the host's fault. + severity: none + category: monitoring + annotations: + summary: "Alerting path is alive" + description: >- + Always firing. This exists so that its absence is detectable — + nothing here needs looking at, and its arrival means Prometheus + evaluated a rule, reached Alertmanager, and Alertmanager delivered. + If you are reading this because it STOPPED, see + docs/runbooks/verify-the-alert-path.md. diff --git a/stacks/observability/prometheus/tests/watchdog.test.yaml b/stacks/observability/prometheus/tests/watchdog.test.yaml new file mode 100644 index 0000000..5b8a755 --- /dev/null +++ b/stacks/observability/prometheus/tests/watchdog.test.yaml @@ -0,0 +1,71 @@ +--- +# Unit test for watchdog.rules.yaml. +# +# There is deliberately no paired "stays quiet" case here, and that is a +# departure from containers.test.yaml, which pairs every firing case with a +# silent one. The reasoning there was that a test which only ever expects +# silence would have passed against the unfireable rule in #63. Watchdog is the +# mirror image: `vector(1)` is unconditional by design, so there is no input +# that should silence it and a quiet case would be asserting a bug. +# +# What IS under test is the label set, which is the part that can break +# silently. `severity: none` is what keeps this alert out of the bare `severity` +# routes and, more importantly, out of the second inhibit rule — which matches +# severity =~ "warning|critical" with equal: ["instance"], and Alertmanager +# treats absent == absent as equal. A drive-by edit setting `severity: warning` +# here would produce a dead man's switch that goes silent every time +# InstanceDown fires, while promtool check rules still passed and the Prometheus +# status page still showed it loaded, healthy and firing. +# +# What this does NOT prove: that the notification reaches anything. Nothing +# offline can prove that — it is the whole reason #67 needs a watcher outside +# this host. See docs/runbooks/verify-the-alert-path.md. +rule_files: + - ../rules/watchdog.rules.yaml + +evaluation_interval: 1m + +tests: + # --- Watchdog: fires with no input series at all -------------------------- + # No input_series on purpose. A rule that needs nothing to fire is exactly + # what a heartbeat has to be: no scrape to fail, no series to go stale, no + # label join to collapse. + - interval: 1m + input_series: [] + alert_rule_test: + # for: 0s, so it is firing at the first evaluation rather than pending. + - eval_time: 1m + alertname: Watchdog + exp_alerts: + - exp_labels: + alertname: Watchdog + component: watchdog + severity: none + category: monitoring + exp_annotations: + summary: "Alerting path is alive" + description: >- + Always firing. This exists so that its absence is detectable — + nothing here needs looking at, and its arrival means Prometheus + evaluated a rule, reached Alertmanager, and Alertmanager + delivered. If you are reading this because it STOPPED, see + docs/runbooks/verify-the-alert-path.md. + + # Still firing an hour later. A heartbeat that fires once and resolves is + # a heartbeat that trips its own watcher. + - eval_time: 60m + alertname: Watchdog + exp_alerts: + - exp_labels: + alertname: Watchdog + component: watchdog + severity: none + category: monitoring + exp_annotations: + summary: "Alerting path is alive" + description: >- + Always firing. This exists so that its absence is detectable — + nothing here needs looking at, and its arrival means Prometheus + evaluated a rule, reached Alertmanager, and Alertmanager + delivered. If you are reading this because it STOPPED, see + docs/runbooks/verify-the-alert-path.md.