feat(alerts): a dead man's switch on the notification path

.github/workflows/ci.yml

@@ -166,6 +166,7 @@ jobs:
               --verify.receivers="$expected" $labels \
               || { echo "::error::expected $expected for $labels"; fail=1; }
           done <<'ROUTES'
+          heartbeat,default alertname=Watchdog severity=none category=monitoring
           urgent    severity=critical category=power
           security  severity=critical category=security
           security  severity=warning category=security

README.md

@@ -45,7 +45,7 @@ incident.
   can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed
   identically everywhere. [How](docs/architecture.md#observability-data-flow)
 - **Dashboards and alerting as code.** 5 provisioned dashboards, 84 panels, and
-  47 alert rules — 34 metric-based in Prometheus, 13 log-based in Loki — sharing
+  48 alert rules — 35 metric-based in Prometheus, 13 log-based in Loki — sharing
   one Alertmanager routing tree. No dashboard exists only in a database.
 - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials,
   decrypted at deploy time into gitignored paths, with `git log` showing which
@@ -143,7 +143,7 @@ rack; a dashed border means egress only. Full topology and data flow in
 .
 ├── stacks/observability/     # the deployed stack — one compose file, six services
 │   ├── compose.yaml
-│   ├── prometheus/           # config, file_sd targets, 34 alert rules
+│   ├── prometheus/           # config, file_sd targets, 35 alert rules
 │   ├── alertmanager/         # routing and inhibition
 │   ├── loki/                 # single-binary config + 13 LogQL rules
 │   ├── alloy/                # one agent config, used on every host

docs/observability.md

@@ -70,7 +70,7 @@ expression in every panel is syntactically valid.
 
 ## Alerting
 
-47 rules in total: 34 metric-based in `prometheus/rules/`, and 13 log-based in
+48 rules in total: 35 metric-based in `prometheus/rules/`, and 13 log-based in
 `loki/rules/`.
 
 ### Log-based (Loki ruler)
@@ -99,14 +99,15 @@ boot check.
 
 ### Metric-based (Prometheus)
 
-34 rules across four files in `prometheus/rules/`:
+35 rules across five files in `prometheus/rules/`:
 
 | File | Covers |
 | --- | --- |
 | `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots |
 | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware |
 | `ups.rules.yaml` | On battery, low battery, runtime, load, temperature |
 | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling, and the stack watching itself |
+| `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable |
 
 `promtool check rules` validates that these parse. It does not — and cannot —
 tell you whether a rule can ever be true: `ContainerHighMemory` passed it for
@@ -115,9 +116,9 @@ and healthy and could not fire for any input ([#63](https://github.com/Gerrrt/Ho
 `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which
 feed a rule synthetic series and assert it fires — paired with a case asserting
 it stays quiet, because a test that only ever expects silence would have passed
-against the broken rule too. Coverage is one rule of 34 so far:
-`ContainerHighMemory`. The other 33 are still validated for syntax only, which
-is exactly the standing #63 had.
+against the broken rule too. Coverage is two rules of 35 so far:
+`ContainerHighMemory` and `Watchdog`. The other 33 are still validated for
+syntax only, which is exactly the standing #63 had.
 
 Disk alerting is predictive rather than a fixed threshold — `predict_linear` over
 a 6-hour window, firing when the extrapolation reaches zero within a day *and*
@@ -160,6 +161,44 @@ Inhibit rules stop cascades: a down host suppresses its own disk warnings, and a
 dead `snmp-exporter` suppresses the "every device is unreachable" storm that
 would otherwise follow.
 
+### The dead man's switch
+
+`AlertmanagerNotificationsFailing` catches delivery *errors*. It cannot catch a
+webhook URL that is well-formed, reachable, and pointed at nothing — a 200 into a
+deleted ntfy topic is a successful notification by every measure Alertmanager
+has. That is not hypothetical: the webhook was the `ntfy.example.invalid`
+placeholder for the entire life of the stack and nothing noticed, because the
+only symptom is that alerts stop arriving, which is also what a healthy week
+looks like ([#67](https://github.com/Gerrrt/HomeLab/issues/67)).
+
+`prometheus/rules/watchdog.rules.yaml` holds one rule, `Watchdog`, whose
+expression is `vector(1)`. It fires unconditionally and forever. **Its firing
+carries no information; its absence is the entire signal.** One `continue: true`
+— the only one in the tree — sends it to two places:
+
+| Route | Destination | Cadence | Catches |
+| --- | --- | --- | --- |
+| `heartbeat` | external cron-monitor ping | 5m | Prometheus stopped evaluating, Alertmanager died, no outbound network |
+| `default` | the real alert channel | 24h | the alert channel itself is a 200 into nothing |
+
+Neither half substitutes for the other. The heartbeat proves delivery to a
+*different* URL than real alerts use, so it cannot see a deleted topic; the daily
+notification travels the identical URL your warnings travel, but nothing
+machine-checks its absence.
+
+The watcher lives off this host by necessity — a watcher here fails at the same
+moment as the thing it is watching. Setting it up, the coupling between
+`repeat_interval` and the external check's period and grace, and how to read
+which half went quiet are in
+[`runbooks/verify-the-alert-path.md`](runbooks/verify-the-alert-path.md).
+
+This is the same reasoning `loki/rules/security.rules.yaml` already applies to
+the firewall with `FirewallLogsStopped`, and the reason it gives for deliberately
+*not* writing a `SuricataStopped` rule: absence of alerts is indistinguishable
+from absence of the service, and detecting that needs a heartbeat rather than a
+threshold. The notification path was the one place that argument had not been
+turned on itself.
+
 ## Adding a monitored device
 
 See [`runbooks/add-monitored-device.md`](runbooks/add-monitored-device.md). In

docs/runbooks/deploy-stack.md

@@ -43,7 +43,7 @@ Then in the UI:
 
 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets
    take up to 45 seconds on their first scrape.
-2. **Prometheus → Status → Rules.** 34 rules loaded, none in error.
+2. **Prometheus → Status → Rules.** 35 rules loaded, none in error.
 3. **Grafana → Dashboards → HomeLab.** Five dashboards, populated.
 4. **Grafana → Explore → Loki**, run `{host=~".+"}`. Logs should be arriving.
 5. Confirm level normalisation is working — this has been silently broken

docs/runbooks/verify-the-alert-path.md

@@ -0,0 +1,121 @@
+# Verify the alert path
+
+**Who watches the thing that tells you something is wrong.**
+
+`AlertmanagerNotificationsFailing` catches delivery *errors* — a refused
+connection, a 5xx. It cannot catch a webhook URL that is well-formed, reachable,
+and pointed at nothing. A 200 into a deleted ntfy topic is a successful
+notification by every measure Alertmanager has, and the only symptom is that
+alerts stop arriving, which is also what a healthy week looks like.
+
+This lab has already lived through that failure: the webhook was the
+`ntfy.example.invalid` placeholder for the entire life of the stack and nothing
+noticed. See [#67](https://github.com/Gerrrt/HomeLab/issues/67).
+
+## The two halves
+
+`prometheus/rules/watchdog.rules.yaml` holds one rule, `Watchdog`, whose
+expression is `vector(1)`. It fires unconditionally and forever. Its firing
+carries no information; **its absence is the entire signal.**
+
+Alertmanager sends it to two places, from one rule, via the only `continue: true`
+in the routing tree:
+
+| Route | Destination | Cadence | Catches |
+| --- | --- | --- | --- |
+| `heartbeat` | external cron-monitor ping | every 5m | Prometheus stopped evaluating, Alertmanager died, the host lost outbound network |
+| `default` | the real alert channel | every 24h | the alert channel itself is a 200 into nothing |
+
+Both halves are needed, and neither substitutes for the other. The heartbeat
+route proves delivery to a *different* URL than real alerts use, so it cannot
+see a deleted ntfy topic. The daily route travels the identical URL your warnings
+travel, but nothing machine-checks its absence — you do.
+
+## Setting up the external watcher
+
+The watcher has to live somewhere other than the monitoring host. A watcher on
+this host fails at the same moment as the thing it is watching, which is not
+watching at all.
+
+A cron-monitor / heartbeat service is the least effort:
+[healthchecks.io](https://healthchecks.io) (free tier is enough for one check),
+Cronitor, or an Uptime Kuma "push" monitor on any other machine.
+
+1. Create one check. Name it so a 3am notification is self-explanatory —
+   `homelab alerting path`, not `check 1`.
+2. Set **period 5m** and **grace 15m**. See "The timing is coupled" below before
+   changing either.
+3. Point the check's own notification at something that is **not** the webhook
+   this stack uses. If both go to the same ntfy topic, a deleted topic takes out
+   the alert and the warning about the alert together. Email is fine here; it
+   fails independently.
+4. Put the ping URL into the encrypted secrets file and render:
+
+   ```bash
+   make secrets-edit     # set ALERTMANAGER_HEARTBEAT_URL
+   make up
+   ```
+
+5. Confirm the check goes green within one `repeat_interval`.
+
+## The timing is coupled
+
+Alertmanager sends the first notification after `group_wait` and then re-sends
+every `repeat_interval`. The heartbeat route uses `group_wait: 0s` and
+`repeat_interval: 5m`.
+
+- **External period ≥ `repeat_interval`.** A period shorter than 5m expects pings
+  that are never sent, and the check alarms on a perfectly healthy stack.
+- **External grace ≥ 2 × `repeat_interval`.** One missed ping is a hiccup — a
+  reload, a restart, a slow scrape. Two consecutive misses is a fault. A grace
+  under 10m turns every `make reload` into a page.
+
+Change `repeat_interval` in `alertmanager/alertmanager.yaml` and the external
+check's period and grace move with it. Nothing enforces that from here, which is
+why it is written down.
+
+## Confirming it actually works
+
+Do not trust a green check you have never seen go red.
+
+```bash
+docker stop alertmanager
+# wait out the grace window — 15m by default
+# the external check must report DOWN and notify you
+docker start alertmanager
+# it must return to green within one repeat_interval
+```
+
+Doing this once is worth more than the rule is. A dead man's switch nobody has
+ever seen trip is indistinguishable from a dead man's switch that does not work.
+
+To confirm the daily half without waiting a day, temporarily lower
+`repeat_interval` on the second Watchdog route, `make reload`, and check the
+notification arrives on your normal alert channel:
+
+```bash
+amtool alert query --alertmanager.url=http://localhost:9093 alertname=Watchdog
+```
+
+Put it back to `24h` afterwards.
+
+## Reading the failure
+
+| What you see | What it means |
+| --- | --- |
+| External check DOWN, daily heartbeat still arriving | The heartbeat URL is wrong or that specific destination is unreachable. The alert path itself is fine. |
+| External check UP, daily heartbeat stopped | The **real alert channel** is broken — a deleted topic, a rotated URL. This is #67's original failure, and every real alert is being lost right now. |
+| Both stopped | Prometheus, Alertmanager, or this host. Start with `docker compose ps` and `curl -s localhost:9093/-/healthy`. |
+| Both fine, but you expected an alert about something else | Not this runbook. The path works; check the rule, then the routing tree with `amtool config routes test`. |
+
+The second row is the one this whole arrangement exists for, and it is the one
+that looks like nothing is wrong.
+
+## Related
+
+- `prometheus/rules/watchdog.rules.yaml` — the rule, and why `severity: none` is
+  load-bearing rather than a placeholder
+- `loki/rules/security.rules.yaml` — `FirewallLogsStopped`, and the comment
+  explaining why Suricata deliberately has no equivalent rule. Same reasoning,
+  applied to a different silent component
+- [`docs/observability.md`](../observability.md#routing) — the full routing table

docs/security.md

@@ -14,7 +14,7 @@ What this network is actually built to survive:
 | A smart TV's firmware phoning somewhere unexpected | VLAN 40 is terminal, egress only |
 | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access |
 | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it |
-| Losing visibility of a failure | 34 alert rules, 30 days of metrics and logs |
+| Losing visibility of a failure | 35 alert rules, 30 days of metrics and logs |
 | Mains power loss | **Not currently defended.** `mjolnir` has no battery installed — see below |
 
 What it explicitly does **not** defend against: a determined attacker with

scripts/render-config.sh

@@ -45,6 +45,7 @@ REQUIRED=(
   ALERTMANAGER_WEBHOOK_URL
   ALERTMANAGER_URGENT_WEBHOOK_URL
   ALERTMANAGER_SECURITY_WEBHOOK_URL
+  ALERTMANAGER_HEARTBEAT_URL
   SNMP_COMMUNITY_PFSENSE
   SNMP_COMMUNITY_APC
   SNMP_COMMUNITY_MOKERLINK
@@ -124,6 +125,7 @@ AM_CHANNELS=(
   "ALERTMANAGER_WEBHOOK_URL:webhook_url"
   "ALERTMANAGER_URGENT_WEBHOOK_URL:urgent_url"
   "ALERTMANAGER_SECURITY_WEBHOOK_URL:security_url"
+  "ALERTMANAGER_HEARTBEAT_URL:heartbeat_url"
 )
 AM_OUT_DIR="${STACK_DIR}/alertmanager/.rendered"
 if [[ -f "${STACK_DIR}/alertmanager/alertmanager.yaml" ]]; then

scripts/validate.sh

@@ -153,6 +153,11 @@ if ((${#AMTOOL[@]})); then
   #
   # Each line below is one row of the table in alertmanager.yaml.
   # --verify.receivers exits non-zero when the resolved receiver differs.
+  #
+  # The Watchdog row expects TWO receivers because its first route sets
+  # `continue: true` — the one place in the tree that does. That row is also
+  # what catches a `severity` slip on the watchdog rule: with `info` it would
+  # resolve to "null" and the dead man's switch would be silently disarmed.
   routes_ok=1
   while read -r expected labels; do
     [[ -n "${expected}" ]] || continue
@@ -166,6 +171,7 @@ if ((${#AMTOOL[@]})); then
       routes_ok=0
     fi
   done <<'ROUTES'
+heartbeat,default alertname=Watchdog severity=none category=monitoring
 urgent    severity=critical category=power
 security  severity=critical category=security
 security  severity=warning category=security
@@ -175,7 +181,7 @@ null      severity=info category=correctness
 ROUTES
 
   if ((routes_ok)); then
-    pass "amtool config routes test (6 assertions)"
+    pass "amtool config routes test (7 assertions)"
   else
     fail "amtool config routes test"
   fi

scripts/verify-key-backup.sh

@@ -218,6 +218,7 @@ REQUIRED=(
   ALERTMANAGER_WEBHOOK_URL
   ALERTMANAGER_URGENT_WEBHOOK_URL
   ALERTMANAGER_SECURITY_WEBHOOK_URL
+  ALERTMANAGER_HEARTBEAT_URL
   SNMP_COMMUNITY_PFSENSE
   SNMP_COMMUNITY_APC
   SNMP_COMMUNITY_MOKERLINK

secrets/README.md

@@ -85,8 +85,9 @@ file, exports the values as environment variables, and:
 - renders `snmp-exporter/snmp.yaml`'s `${SNMP_COMMUNITY_*}` placeholders into
   `snmp-exporter/.rendered/snmp.yaml`, which is what the container mounts;
 - writes one file per notification channel into `alertmanager/.rendered/` —
-  `webhook_url`, `urgent_url` and `security_url` — because Alertmanager does not
-  expand environment variables and reads receiver URLs via `url_file`.
+  `webhook_url`, `urgent_url`, `security_url` and `heartbeat_url` — because
+  Alertmanager does not expand environment variables and reads receiver URLs
+  via `url_file`.
 
 All of them are gitignored, and each secret is written to exactly one of them.
 Nothing writes a secret into a tracked file.

secrets/observability.example.yaml

@@ -52,3 +52,19 @@ SNMP_COMMUNITY_ILO: change-me-ilo
 ALERTMANAGER_WEBHOOK_URL: https://ntfy.example.invalid/homelab-alerts
 ALERTMANAGER_URGENT_WEBHOOK_URL: https://ntfy.example.invalid/homelab-urgent
 ALERTMANAGER_SECURITY_WEBHOOK_URL: https://ntfy.example.invalid/homelab-security
+
+# The dead man's switch (#67). NOT an ntfy topic — this is a cron-monitor ping
+# URL from something that lives OFF this host: healthchecks.io, Cronitor, an
+# Uptime Kuma push monitor, or your own endpoint on another machine.
+#
+# The Watchdog rule fires forever and Alertmanager posts here every 5 minutes.
+# Its arrival means nothing. Its absence means the alerting path is broken, and
+# only something outside this stack can notice that — a watcher on the
+# monitoring host fails at the same moment as the thing it is watching.
+#
+# Set the external check's period to 5m and its grace to 15m, and read
+# docs/runbooks/verify-the-alert-path.md before changing either: the period is
+# tied to `repeat_interval` on the heartbeat route in alertmanager.yaml, and
+# raising one without the other produces a check that alarms on a healthy stack
+# or one that never alarms at all.
+ALERTMANAGER_HEARTBEAT_URL: https://hc-ping.example.invalid/00000000-0000-0000-0000-000000000000

stacks/observability/alertmanager/alertmanager.yaml

@@ -11,7 +11,7 @@
 # at /etc/alertmanager/secrets/. Both the rendered directory and its contents
 # are gitignored.
 #
-# There are three channels and three destinations. They used to be three names
+# There are four channels and four destinations. They used to be three names
 # for one URL, which meant `urgent` and `default` differed only in how often
 # they repeated — a UPS on battery and a slow scrape landed in the same place
 # (#66). Point them at three separate topics on whatever receiver you use, so
@@ -35,6 +35,40 @@ route:
   # config still loads and still validates. scripts/validate.sh asserts the
   # whole table with `amtool config routes test` for exactly that reason.
   routes:
+    # --- The dead man's switch, and it has to be first ---------------------
+    #
+    # Watchdog fires unconditionally and forever (prometheus/rules/watchdog.rules.yaml).
+    # Its firing means nothing; its ABSENCE is the signal, so both of these
+    # routes exist to make the absence observable in two different ways.
+    #
+    # `continue: true` on the first is what lets ONE rule produce TWO
+    # notifications on two cadences, rather than duplicating the expression in
+    # the rule file. It is the only `continue` in this tree — everything below
+    # is first-match-wins.
+    #
+    # 5m to the heartbeat URL, which is a cron-monitor ping watched from off
+    # this host. Keep this at or under the external check's period; the runbook
+    # states the relationship, so retuning one without the other is visible.
+    - receiver: heartbeat
+      matchers:
+        - alertname = "Watchdog"
+      group_wait: 0s
+      group_interval: 5m
+      repeat_interval: 5m
+      continue: true
+
+    # 24h to the REAL alert channel. This is the half that catches #67's actual
+    # failure: a webhook that is well-formed, reachable, returns 200 and is
+    # pointed at nothing. It travels the identical URL real warnings travel, so
+    # a deleted topic shows up as a missing daily heartbeat instead of as
+    # nothing at all. The heartbeat route above cannot see that, because it
+    # proves delivery to a different URL.
+    - receiver: default
+      matchers:
+        - alertname = "Watchdog"
+      group_wait: 0s
+      repeat_interval: 24h
+
     # Power events page immediately and repeat aggressively — a UPS on battery
     # with no spare battery installed is a short fuse.
     - receiver: urgent
@@ -113,3 +147,12 @@ receivers:
     webhook_configs:
       - url_file: /etc/alertmanager/secrets/security_url
         send_resolved: true
+
+  # send_resolved: false, deliberately. If Prometheus stops sending Watchdog,
+  # Alertmanager expires it after resolve_timeout and the pings simply stop —
+  # which is precisely the signal the external check is waiting for. A resolved
+  # notification here would be one last successful ping on the way out the door.
+  - name: heartbeat
+    webhook_configs:
+      - url_file: /etc/alertmanager/secrets/heartbeat_url
+        send_resolved: false