diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 658ec21..e674871 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -166,6 +166,7 @@ jobs: --verify.receivers="$expected" $labels \ || { echo "::error::expected $expected for $labels"; fail=1; } done <<'ROUTES' + heartbeat,default alertname=Watchdog severity=none category=monitoring urgent severity=critical category=power security severity=critical category=security security severity=warning category=security diff --git a/README.md b/README.md index 5fbe32f..b2f9a4d 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ incident. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 5 provisioned dashboards, 84 panels, and - 47 alert rules — 34 metric-based in Prometheus, 13 log-based in Loki — sharing + 48 alert rules — 35 metric-based in Prometheus, 13 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -143,7 +143,7 @@ rack; a dashed border means egress only. Full topology and data flow in . ├── stacks/observability/ # the deployed stack — one compose file, six services │ ├── compose.yaml -│ ├── prometheus/ # config, file_sd targets, 34 alert rules +│ ├── prometheus/ # config, file_sd targets, 35 alert rules │ ├── alertmanager/ # routing and inhibition │ ├── loki/ # single-binary config + 13 LogQL rules │ ├── alloy/ # one agent config, used on every host diff --git a/docs/observability.md b/docs/observability.md index 9eff662..3245a7f 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -70,7 +70,7 @@ expression in every panel is syntactically valid. ## Alerting -47 rules in total: 34 metric-based in `prometheus/rules/`, and 13 log-based in +48 rules in total: 35 metric-based in `prometheus/rules/`, and 13 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -99,7 +99,7 @@ boot check. ### Metric-based (Prometheus) -34 rules across four files in `prometheus/rules/`: +35 rules across five files in `prometheus/rules/`: | File | Covers | | --- | --- | @@ -107,6 +107,7 @@ boot check. | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware | | `ups.rules.yaml` | On battery, low battery, runtime, load, temperature | | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling, and the stack watching itself | +| `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | `promtool check rules` validates that these parse. It does not — and cannot — tell you whether a rule can ever be true: `ContainerHighMemory` passed it for @@ -115,9 +116,9 @@ and healthy and could not fire for any input ([#63](https://github.com/Gerrrt/Ho `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is one rule of 34 so far: -`ContainerHighMemory`. The other 33 are still validated for syntax only, which -is exactly the standing #63 had. +against the broken rule too. Coverage is two rules of 35 so far: +`ContainerHighMemory` and `Watchdog`. The other 33 are still validated for +syntax only, which is exactly the standing #63 had. Disk alerting is predictive rather than a fixed threshold — `predict_linear` over a 6-hour window, firing when the extrapolation reaches zero within a day *and* @@ -160,6 +161,44 @@ Inhibit rules stop cascades: a down host suppresses its own disk warnings, and a dead `snmp-exporter` suppresses the "every device is unreachable" storm that would otherwise follow. +### The dead man's switch + +`AlertmanagerNotificationsFailing` catches delivery *errors*. It cannot catch a +webhook URL that is well-formed, reachable, and pointed at nothing — a 200 into a +deleted ntfy topic is a successful notification by every measure Alertmanager +has. That is not hypothetical: the webhook was the `ntfy.example.invalid` +placeholder for the entire life of the stack and nothing noticed, because the +only symptom is that alerts stop arriving, which is also what a healthy week +looks like ([#67](https://github.com/Gerrrt/HomeLab/issues/67)). + +`prometheus/rules/watchdog.rules.yaml` holds one rule, `Watchdog`, whose +expression is `vector(1)`. It fires unconditionally and forever. **Its firing +carries no information; its absence is the entire signal.** One `continue: true` +— the only one in the tree — sends it to two places: + +| Route | Destination | Cadence | Catches | +| --- | --- | --- | --- | +| `heartbeat` | external cron-monitor ping | 5m | Prometheus stopped evaluating, Alertmanager died, no outbound network | +| `default` | the real alert channel | 24h | the alert channel itself is a 200 into nothing | + +Neither half substitutes for the other. The heartbeat proves delivery to a +*different* URL than real alerts use, so it cannot see a deleted topic; the daily +notification travels the identical URL your warnings travel, but nothing +machine-checks its absence. + +The watcher lives off this host by necessity — a watcher here fails at the same +moment as the thing it is watching. Setting it up, the coupling between +`repeat_interval` and the external check's period and grace, and how to read +which half went quiet are in +[`runbooks/verify-the-alert-path.md`](runbooks/verify-the-alert-path.md). + +This is the same reasoning `loki/rules/security.rules.yaml` already applies to +the firewall with `FirewallLogsStopped`, and the reason it gives for deliberately +*not* writing a `SuricataStopped` rule: absence of alerts is indistinguishable +from absence of the service, and detecting that needs a heartbeat rather than a +threshold. The notification path was the one place that argument had not been +turned on itself. + ## Adding a monitored device See [`runbooks/add-monitored-device.md`](runbooks/add-monitored-device.md). In diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index a3455d3..24a7e00 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -43,7 +43,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. -2. **Prometheus → Status → Rules.** 34 rules loaded, none in error. +2. **Prometheus → Status → Rules.** 35 rules loaded, none in error. 3. **Grafana → Dashboards → HomeLab.** Five dashboards, populated. 4. **Grafana → Explore → Loki**, run `{host=~".+"}`. Logs should be arriving. 5. Confirm level normalisation is working — this has been silently broken diff --git a/docs/runbooks/verify-the-alert-path.md b/docs/runbooks/verify-the-alert-path.md new file mode 100644 index 0000000..57cad93 --- /dev/null +++ b/docs/runbooks/verify-the-alert-path.md @@ -0,0 +1,121 @@ +# Verify the alert path + +**Who watches the thing that tells you something is wrong.** + +`AlertmanagerNotificationsFailing` catches delivery *errors* — a refused +connection, a 5xx. It cannot catch a webhook URL that is well-formed, reachable, +and pointed at nothing. A 200 into a deleted ntfy topic is a successful +notification by every measure Alertmanager has, and the only symptom is that +alerts stop arriving, which is also what a healthy week looks like. + +This lab has already lived through that failure: the webhook was the +`ntfy.example.invalid` placeholder for the entire life of the stack and nothing +noticed. See [#67](https://github.com/Gerrrt/HomeLab/issues/67). + +## The two halves + +`prometheus/rules/watchdog.rules.yaml` holds one rule, `Watchdog`, whose +expression is `vector(1)`. It fires unconditionally and forever. Its firing +carries no information; **its absence is the entire signal.** + +Alertmanager sends it to two places, from one rule, via the only `continue: true` +in the routing tree: + +| Route | Destination | Cadence | Catches | +| --- | --- | --- | --- | +| `heartbeat` | external cron-monitor ping | every 5m | Prometheus stopped evaluating, Alertmanager died, the host lost outbound network | +| `default` | the real alert channel | every 24h | the alert channel itself is a 200 into nothing | + +Both halves are needed, and neither substitutes for the other. The heartbeat +route proves delivery to a *different* URL than real alerts use, so it cannot +see a deleted ntfy topic. The daily route travels the identical URL your warnings +travel, but nothing machine-checks its absence — you do. + +## Setting up the external watcher + +The watcher has to live somewhere other than the monitoring host. A watcher on +this host fails at the same moment as the thing it is watching, which is not +watching at all. + +A cron-monitor / heartbeat service is the least effort: +[healthchecks.io](https://healthchecks.io) (free tier is enough for one check), +Cronitor, or an Uptime Kuma "push" monitor on any other machine. + +1. Create one check. Name it so a 3am notification is self-explanatory — + `homelab alerting path`, not `check 1`. +2. Set **period 5m** and **grace 15m**. See "The timing is coupled" below before + changing either. +3. Point the check's own notification at something that is **not** the webhook + this stack uses. If both go to the same ntfy topic, a deleted topic takes out + the alert and the warning about the alert together. Email is fine here; it + fails independently. +4. Put the ping URL into the encrypted secrets file and render: + + ```bash + make secrets-edit # set ALERTMANAGER_HEARTBEAT_URL + make up + ``` + +5. Confirm the check goes green within one `repeat_interval`. + +## The timing is coupled + +Alertmanager sends the first notification after `group_wait` and then re-sends +every `repeat_interval`. The heartbeat route uses `group_wait: 0s` and +`repeat_interval: 5m`. + +- **External period ≥ `repeat_interval`.** A period shorter than 5m expects pings + that are never sent, and the check alarms on a perfectly healthy stack. +- **External grace ≥ 2 × `repeat_interval`.** One missed ping is a hiccup — a + reload, a restart, a slow scrape. Two consecutive misses is a fault. A grace + under 10m turns every `make reload` into a page. + +Change `repeat_interval` in `alertmanager/alertmanager.yaml` and the external +check's period and grace move with it. Nothing enforces that from here, which is +why it is written down. + +## Confirming it actually works + +Do not trust a green check you have never seen go red. + +```bash +docker stop alertmanager +# wait out the grace window — 15m by default +# the external check must report DOWN and notify you +docker start alertmanager +# it must return to green within one repeat_interval +``` + +Doing this once is worth more than the rule is. A dead man's switch nobody has +ever seen trip is indistinguishable from a dead man's switch that does not work. + +To confirm the daily half without waiting a day, temporarily lower +`repeat_interval` on the second Watchdog route, `make reload`, and check the +notification arrives on your normal alert channel: + +```bash +amtool alert query --alertmanager.url=http://localhost:9093 alertname=Watchdog +``` + +Put it back to `24h` afterwards. + +## Reading the failure + +| What you see | What it means | +| --- | --- | +| External check DOWN, daily heartbeat still arriving | The heartbeat URL is wrong or that specific destination is unreachable. The alert path itself is fine. | +| External check UP, daily heartbeat stopped | The **real alert channel** is broken — a deleted topic, a rotated URL. This is #67's original failure, and every real alert is being lost right now. | +| Both stopped | Prometheus, Alertmanager, or this host. Start with `docker compose ps` and `curl -s localhost:9093/-/healthy`. | +| Both fine, but you expected an alert about something else | Not this runbook. The path works; check the rule, then the routing tree with `amtool config routes test`. | + +The second row is the one this whole arrangement exists for, and it is the one +that looks like nothing is wrong. + +## Related + +- `prometheus/rules/watchdog.rules.yaml` — the rule, and why `severity: none` is + load-bearing rather than a placeholder +- `loki/rules/security.rules.yaml` — `FirewallLogsStopped`, and the comment + explaining why Suricata deliberately has no equivalent rule. Same reasoning, + applied to a different silent component +- [`docs/observability.md`](../observability.md#routing) — the full routing table diff --git a/docs/security.md b/docs/security.md index a61844d..7ea8885 100644 --- a/docs/security.md +++ b/docs/security.md @@ -14,7 +14,7 @@ What this network is actually built to survive: | A smart TV's firmware phoning somewhere unexpected | VLAN 40 is terminal, egress only | | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | -| Losing visibility of a failure | 34 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 35 alert rules, 30 days of metrics and logs | | Mains power loss | **Not currently defended.** `mjolnir` has no battery installed — see below | What it explicitly does **not** defend against: a determined attacker with diff --git a/scripts/render-config.sh b/scripts/render-config.sh index 00720b1..1f6eb10 100755 --- a/scripts/render-config.sh +++ b/scripts/render-config.sh @@ -45,6 +45,7 @@ REQUIRED=( ALERTMANAGER_WEBHOOK_URL ALERTMANAGER_URGENT_WEBHOOK_URL ALERTMANAGER_SECURITY_WEBHOOK_URL + ALERTMANAGER_HEARTBEAT_URL SNMP_COMMUNITY_PFSENSE SNMP_COMMUNITY_APC SNMP_COMMUNITY_MOKERLINK @@ -124,6 +125,7 @@ AM_CHANNELS=( "ALERTMANAGER_WEBHOOK_URL:webhook_url" "ALERTMANAGER_URGENT_WEBHOOK_URL:urgent_url" "ALERTMANAGER_SECURITY_WEBHOOK_URL:security_url" + "ALERTMANAGER_HEARTBEAT_URL:heartbeat_url" ) AM_OUT_DIR="${STACK_DIR}/alertmanager/.rendered" if [[ -f "${STACK_DIR}/alertmanager/alertmanager.yaml" ]]; then diff --git a/scripts/validate.sh b/scripts/validate.sh index c6dbde1..5d8f106 100755 --- a/scripts/validate.sh +++ b/scripts/validate.sh @@ -153,6 +153,11 @@ if ((${#AMTOOL[@]})); then # # Each line below is one row of the table in alertmanager.yaml. # --verify.receivers exits non-zero when the resolved receiver differs. + # + # The Watchdog row expects TWO receivers because its first route sets + # `continue: true` — the one place in the tree that does. That row is also + # what catches a `severity` slip on the watchdog rule: with `info` it would + # resolve to "null" and the dead man's switch would be silently disarmed. routes_ok=1 while read -r expected labels; do [[ -n "${expected}" ]] || continue @@ -166,6 +171,7 @@ if ((${#AMTOOL[@]})); then routes_ok=0 fi done <<'ROUTES' +heartbeat,default alertname=Watchdog severity=none category=monitoring urgent severity=critical category=power security severity=critical category=security security severity=warning category=security @@ -175,7 +181,7 @@ null severity=info category=correctness ROUTES if ((routes_ok)); then - pass "amtool config routes test (6 assertions)" + pass "amtool config routes test (7 assertions)" else fail "amtool config routes test" fi diff --git a/scripts/verify-key-backup.sh b/scripts/verify-key-backup.sh index 3c888b7..2fb86bb 100755 --- a/scripts/verify-key-backup.sh +++ b/scripts/verify-key-backup.sh @@ -218,6 +218,7 @@ REQUIRED=( ALERTMANAGER_WEBHOOK_URL ALERTMANAGER_URGENT_WEBHOOK_URL ALERTMANAGER_SECURITY_WEBHOOK_URL + ALERTMANAGER_HEARTBEAT_URL SNMP_COMMUNITY_PFSENSE SNMP_COMMUNITY_APC SNMP_COMMUNITY_MOKERLINK diff --git a/secrets/README.md b/secrets/README.md index df8a7d7..55f2010 100644 --- a/secrets/README.md +++ b/secrets/README.md @@ -85,8 +85,9 @@ file, exports the values as environment variables, and: - renders `snmp-exporter/snmp.yaml`'s `${SNMP_COMMUNITY_*}` placeholders into `snmp-exporter/.rendered/snmp.yaml`, which is what the container mounts; - writes one file per notification channel into `alertmanager/.rendered/` — - `webhook_url`, `urgent_url` and `security_url` — because Alertmanager does not - expand environment variables and reads receiver URLs via `url_file`. + `webhook_url`, `urgent_url`, `security_url` and `heartbeat_url` — because + Alertmanager does not expand environment variables and reads receiver URLs + via `url_file`. All of them are gitignored, and each secret is written to exactly one of them. Nothing writes a secret into a tracked file. diff --git a/secrets/observability.example.yaml b/secrets/observability.example.yaml index b241f31..250a96b 100644 --- a/secrets/observability.example.yaml +++ b/secrets/observability.example.yaml @@ -52,3 +52,19 @@ SNMP_COMMUNITY_ILO: change-me-ilo ALERTMANAGER_WEBHOOK_URL: https://ntfy.example.invalid/homelab-alerts ALERTMANAGER_URGENT_WEBHOOK_URL: https://ntfy.example.invalid/homelab-urgent ALERTMANAGER_SECURITY_WEBHOOK_URL: https://ntfy.example.invalid/homelab-security + +# The dead man's switch (#67). NOT an ntfy topic — this is a cron-monitor ping +# URL from something that lives OFF this host: healthchecks.io, Cronitor, an +# Uptime Kuma push monitor, or your own endpoint on another machine. +# +# The Watchdog rule fires forever and Alertmanager posts here every 5 minutes. +# Its arrival means nothing. Its absence means the alerting path is broken, and +# only something outside this stack can notice that — a watcher on the +# monitoring host fails at the same moment as the thing it is watching. +# +# Set the external check's period to 5m and its grace to 15m, and read +# docs/runbooks/verify-the-alert-path.md before changing either: the period is +# tied to `repeat_interval` on the heartbeat route in alertmanager.yaml, and +# raising one without the other produces a check that alarms on a healthy stack +# or one that never alarms at all. +ALERTMANAGER_HEARTBEAT_URL: https://hc-ping.example.invalid/00000000-0000-0000-0000-000000000000 diff --git a/stacks/observability/alertmanager/alertmanager.yaml b/stacks/observability/alertmanager/alertmanager.yaml index 315da4b..1c373c9 100644 --- a/stacks/observability/alertmanager/alertmanager.yaml +++ b/stacks/observability/alertmanager/alertmanager.yaml @@ -11,7 +11,7 @@ # at /etc/alertmanager/secrets/. Both the rendered directory and its contents # are gitignored. # -# There are three channels and three destinations. They used to be three names +# There are four channels and four destinations. They used to be three names # for one URL, which meant `urgent` and `default` differed only in how often # they repeated — a UPS on battery and a slow scrape landed in the same place # (#66). Point them at three separate topics on whatever receiver you use, so @@ -35,6 +35,40 @@ route: # config still loads and still validates. scripts/validate.sh asserts the # whole table with `amtool config routes test` for exactly that reason. routes: + # --- The dead man's switch, and it has to be first --------------------- + # + # Watchdog fires unconditionally and forever (prometheus/rules/watchdog.rules.yaml). + # Its firing means nothing; its ABSENCE is the signal, so both of these + # routes exist to make the absence observable in two different ways. + # + # `continue: true` on the first is what lets ONE rule produce TWO + # notifications on two cadences, rather than duplicating the expression in + # the rule file. It is the only `continue` in this tree — everything below + # is first-match-wins. + # + # 5m to the heartbeat URL, which is a cron-monitor ping watched from off + # this host. Keep this at or under the external check's period; the runbook + # states the relationship, so retuning one without the other is visible. + - receiver: heartbeat + matchers: + - alertname = "Watchdog" + group_wait: 0s + group_interval: 5m + repeat_interval: 5m + continue: true + + # 24h to the REAL alert channel. This is the half that catches #67's actual + # failure: a webhook that is well-formed, reachable, returns 200 and is + # pointed at nothing. It travels the identical URL real warnings travel, so + # a deleted topic shows up as a missing daily heartbeat instead of as + # nothing at all. The heartbeat route above cannot see that, because it + # proves delivery to a different URL. + - receiver: default + matchers: + - alertname = "Watchdog" + group_wait: 0s + repeat_interval: 24h + # Power events page immediately and repeat aggressively — a UPS on battery # with no spare battery installed is a short fuse. - receiver: urgent @@ -113,3 +147,12 @@ receivers: webhook_configs: - url_file: /etc/alertmanager/secrets/security_url send_resolved: true + + # send_resolved: false, deliberately. If Prometheus stops sending Watchdog, + # Alertmanager expires it after resolve_timeout and the pings simply stop — + # which is precisely the signal the external check is waiting for. A resolved + # notification here would be one last successful ping on the way out the door. + - name: heartbeat + webhook_configs: + - url_file: /etc/alertmanager/secrets/heartbeat_url + send_resolved: false diff --git a/stacks/observability/loki/rules/security.rules.yaml b/stacks/observability/loki/rules/security.rules.yaml index 6fe1e13..92effe6 100644 --- a/stacks/observability/loki/rules/security.rules.yaml +++ b/stacks/observability/loki/rules/security.rules.yaml @@ -248,6 +248,11 @@ groups: # died needs a heartbeat or a process metric, not a log rule, and inventing # a threshold here would produce exactly the sort of alert that cannot fire # while looking like one that can. Tracked in docs/roadmap.md. + # + # prometheus/rules/watchdog.rules.yaml is the worked example of the heartbeat + # this comment is describing, built for the notification path (#67). Suricata + # would need the same shape: something that reports in when healthy, and an + # observer that acts on the silence. # --------------------------------------------------------------------------- - name: ids interval: 1m diff --git a/stacks/observability/prometheus/rules/watchdog.rules.yaml b/stacks/observability/prometheus/rules/watchdog.rules.yaml new file mode 100644 index 0000000..06aaee2 --- /dev/null +++ b/stacks/observability/prometheus/rules/watchdog.rules.yaml @@ -0,0 +1,65 @@ +--- +# A dead man's switch on the notification path. +# +# AlertmanagerNotificationsFailing (containers.rules.yaml) catches delivery +# ERRORS — a refused connection, a 5xx. It cannot catch a webhook URL that is +# well-formed, reachable, and pointed at nothing: a 200 into a deleted ntfy +# topic is a successful notification by every measure Alertmanager has. +# +# That is not hypothetical here. The webhook was the ntfy.example.invalid +# placeholder for the entire life of the stack and nothing noticed, because the +# only symptom is that alerts stop arriving — which is also what a healthy week +# looks like. +# +# This repository already applies exactly this reasoning elsewhere. +# security.rules.yaml has FirewallLogsStopped because a firewall passing traffic +# for the whole house is never silent for thirty minutes, and it explains at +# length why there is deliberately NO SuricataStopped rule: "absence of alerts is +# indistinguishable from absence of the service", and detecting that needs a +# heartbeat, not a log rule. The notification path was the one place the +# reasoning was never turned on itself. Tracked in #67. +# +# THE INVERSION THAT MAKES THIS RULE MAKE SENSE: its firing carries no +# information whatsoever. Its ABSENCE is the entire signal. Nothing inside this +# stack can act on that, because a watcher on this host fails at the same moment +# as the thing it is watching — so the watching is done by an external service, +# and docs/runbooks/verify-the-alert-path.md is where that half is written down. +groups: + - name: watchdog + interval: 1m + rules: + - alert: Watchdog + # Unconditional, by design. `vector(1)` has no inputs, so there is no + # data problem, no scrape, no label join and no threshold that can stop + # this from firing. Anything that stops the heartbeat arriving is + # therefore a fault in the path itself: rule evaluation, the send to + # Alertmanager, routing, or delivery. + expr: vector(1) + for: 0s + labels: + component: watchdog + # severity: none is load-bearing, not a placeholder. + # + # 1. It keeps the alert out of the three bare `severity` routes at the + # bottom of alertmanager.yaml. With `info` it would be swallowed by + # the "null" receiver; with `warning` or `critical` it would page. + # Its own routes match on alertname and sit above all of them. + # + # 2. It keeps the alert out of both inhibit_rules. The second matches + # target_matchers: [severity =~ "warning|critical"] with + # equal: ["instance"] — and Alertmanager treats absent == absent as + # equal, so a Watchdog carrying a warning severity and no instance + # label would be silenced every single time InstanceDown fired. + # A dead man's switch that goes quiet exactly when a host goes down + # is worse than none: it would report the path healthy right up to + # the moment it stopped, and then look like the host's fault. + severity: none + category: monitoring + annotations: + summary: "Alerting path is alive" + description: >- + Always firing. This exists so that its absence is detectable — + nothing here needs looking at, and its arrival means Prometheus + evaluated a rule, reached Alertmanager, and Alertmanager delivered. + If you are reading this because it STOPPED, see + docs/runbooks/verify-the-alert-path.md. diff --git a/stacks/observability/prometheus/tests/watchdog.test.yaml b/stacks/observability/prometheus/tests/watchdog.test.yaml new file mode 100644 index 0000000..5b8a755 --- /dev/null +++ b/stacks/observability/prometheus/tests/watchdog.test.yaml @@ -0,0 +1,71 @@ +--- +# Unit test for watchdog.rules.yaml. +# +# There is deliberately no paired "stays quiet" case here, and that is a +# departure from containers.test.yaml, which pairs every firing case with a +# silent one. The reasoning there was that a test which only ever expects +# silence would have passed against the unfireable rule in #63. Watchdog is the +# mirror image: `vector(1)` is unconditional by design, so there is no input +# that should silence it and a quiet case would be asserting a bug. +# +# What IS under test is the label set, which is the part that can break +# silently. `severity: none` is what keeps this alert out of the bare `severity` +# routes and, more importantly, out of the second inhibit rule — which matches +# severity =~ "warning|critical" with equal: ["instance"], and Alertmanager +# treats absent == absent as equal. A drive-by edit setting `severity: warning` +# here would produce a dead man's switch that goes silent every time +# InstanceDown fires, while promtool check rules still passed and the Prometheus +# status page still showed it loaded, healthy and firing. +# +# What this does NOT prove: that the notification reaches anything. Nothing +# offline can prove that — it is the whole reason #67 needs a watcher outside +# this host. See docs/runbooks/verify-the-alert-path.md. +rule_files: + - ../rules/watchdog.rules.yaml + +evaluation_interval: 1m + +tests: + # --- Watchdog: fires with no input series at all -------------------------- + # No input_series on purpose. A rule that needs nothing to fire is exactly + # what a heartbeat has to be: no scrape to fail, no series to go stale, no + # label join to collapse. + - interval: 1m + input_series: [] + alert_rule_test: + # for: 0s, so it is firing at the first evaluation rather than pending. + - eval_time: 1m + alertname: Watchdog + exp_alerts: + - exp_labels: + alertname: Watchdog + component: watchdog + severity: none + category: monitoring + exp_annotations: + summary: "Alerting path is alive" + description: >- + Always firing. This exists so that its absence is detectable — + nothing here needs looking at, and its arrival means Prometheus + evaluated a rule, reached Alertmanager, and Alertmanager + delivered. If you are reading this because it STOPPED, see + docs/runbooks/verify-the-alert-path.md. + + # Still firing an hour later. A heartbeat that fires once and resolves is + # a heartbeat that trips its own watcher. + - eval_time: 60m + alertname: Watchdog + exp_alerts: + - exp_labels: + alertname: Watchdog + component: watchdog + severity: none + category: monitoring + exp_annotations: + summary: "Alerting path is alive" + description: >- + Always firing. This exists so that its absence is detectable — + nothing here needs looking at, and its arrival means Prometheus + evaluated a rule, reached Alertmanager, and Alertmanager + delivered. If you are reading this because it STOPPED, see + docs/runbooks/verify-the-alert-path.md.