feat(ci): nothing was watching whether the ecosystem still holds the contract

.github/scripts/check-health-conformance.py

@@ -0,0 +1,257 @@
+#!/usr/bin/env python3
+"""Probe every registered /api/health and check it against HEALTH_CONTRACT.md.
+
+Why this exists, and why here rather than in each service's own repo:
+
+Conformance is not a property of any service's source. For most of the
+ecosystem the `Access-Control-Allow-Origin` header comes from nginx or Caddy on
+a hand-managed box, not from the app — measured 2026-09-10, `passport`'s origin
+at 127.0.0.1:3000 sends no ACAO at all, and `agora`'s arrives only because an
+API Gateway CorsConfiguration was deleted from the AWS console. None of that is
+in any repository, no service's own test suite can reach it, and every deploy
+gate probes an origin *behind* the component that adds the header. So each
+repo's CI is structurally blind to the failure this checks for.
+
+The failure is also silent by construction. A consumer that cannot read a
+health endpoint does not see red: monitor.moss.land's map falls back to city's
+aggregate, and the service keeps a green dot sourced from a probe of an
+unrelated data URL. Nothing anywhere turns a colour. That is exactly how
+`signalmap` sat green for weeks while its `/api/health` was unreadable.
+
+This runs from the registry rather than a list of ids, so a service added to
+`ecosystem-registry.json` with a `statusUrl` is covered the day it lands.
+
+What makes the build red:
+
+  * a service that is expected to conform stops conforming — the regression
+    this exists to catch;
+  * a recorded exception starts conforming, so EXCEPTIONS cannot quietly rot
+    into a list of things nobody rechecks;
+  * every probe fails, which means the check itself is broken rather than the
+    ecosystem being entirely down.
+
+What does not: a single service being unreachable. This checks the contract,
+not uptime — a timeout is reported and skipped, because failing the gate for
+an outage teaches everyone to ignore it.
+
+Usage:  python3 .github/scripts/check-health-conformance.py [--registry PATH]
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+CONTRACT_STATUS = {"ok", "degraded", "down"}
+TIMEOUT_S = 20
+
+# A browser sends one; the header must not depend on it, so we probe with a
+# foreign origin. A service that reflects only allow-listed origins passes a
+# same-origin curl and still fails every real consumer.
+PROBE_ORIGIN = "https://monitor.moss.land"
+
+# Services known not to satisfy some rule, with the reason and who can fix it.
+# A service listed here is still probed: if it starts conforming, the build goes
+# red so the entry gets removed rather than outliving its reason.
+#
+# Keys are registry ids; values are the rule numbers from HEALTH_CONTRACT.md
+# that entry is exempt from, and why.
+# RULE 6 IS OPEN ACROSS MOST OF THE ECOSYSTEM, measured 2026-09-10. Seven of the
+# sixteen send no `Cache-Control` at all: the 2026-09-09 nginx session that added
+# `Access-Control-Allow-Origin` and `Vary` to eight vhosts did not add this one,
+# and no app sets it either. Without it a browser applies heuristic freshness, so
+# a consumer polling every 60s can be served an answer it already had — which is
+# the one thing rule 6 exists to prevent. It is recorded rather than enforced
+# because a gate that is red on the day it ships is a gate nobody reads; fixing it
+# is one `add_header Cache-Control "no-cache" always;` per vhost, on the box
+# reachable as `ssh mossland`. Remove each entry as it lands — and note the check
+# goes red if you fix one without removing it, which is the point.
+EXCEPTIONS: dict[str, dict] = {
+    "alpha":  {"rules": {6}, "why": "no Cache-Control at the edge — see the note above"},
+    "signal": {"rules": {6}, "why": "no Cache-Control at the edge — see the note above"},
+    "npc":    {"rules": {1, 3, 6}, "why": "no `service`, no `timestamp`, no Cache-Control. "
+                                          "Manual-deploy service, no local checkout."},
+    "ao":     {"rules": {6}, "why": "no Cache-Control at the edge — see the note above"},
+    "bridge": {"rules": {6}, "why": "no Cache-Control at the edge — see the note above"},
+    "city": {
+        "rules": {1, 2, 3, 6},
+        "why": "its /api/health is the cross-service aggregate (checkedAt/summary/services), "
+               "not a report about itself, and it is served `public, s-maxage=60, "
+               "stale-while-revalidate=120` from its own Next app — the only entry here "
+               "that is cached on purpose. Manual-deploy service, no local checkout.",
+    },
+    "recipe": {
+        "rules": {1},
+        "why": "no `service`. Manual-deploy service, no local checkout.",
+    },
+    "algora": {
+        "rules": {1, 2, 6},
+        "why": 'answers status "running", outside the enum, and sends no `service`. '
+               "lifecycle: archive — owner decision 2026-08-23, the code is frozen. "
+               "Deliberately left visible rather than mapped to ok.",
+    },
+    "signalmap": {
+        "rules": {1, 3, 5},
+        "why": "no ACAO, no `service`, no `timestamp`. Fix merged in "
+               "MosslandCore/signalmap#134; signalmap is manual-deploy on its own box, "
+               "so remove this entry once that deploy lands.",
+    },
+}
+
+# Rule 4's documented exception: these answer 503 when unhealthy because a probe
+# watches the status code and alarms on it. A non-200 from them is not a
+# contract violation, and the body still has to be readable.
+STATUS_CODE_SIGNALLERS = {"signal", "signalmap"}
+
+
+class Probe:
+    """One service's answer, and what the contract makes of it."""
+
+    def __init__(self, sid: str, url: str):
+        self.id = sid
+        self.url = url
+        self.reachable = False
+        self.error: str | None = None
+        self.code = 0
+        self.headers: dict[str, str] = {}
+        self.body: dict | None = None
+
+    def fetch(self) -> None:
+        req = urllib.request.Request(self.url, headers={"Origin": PROBE_ORIGIN})
+        try:
+            with urllib.request.urlopen(req, timeout=TIMEOUT_S) as res:
+                self._absorb(res.status, res.headers, res.read())
+        except urllib.error.HTTPError as e:
+            # A 4xx/5xx is an answer, not a failure to reach — rule 4 puts the
+            # verdict in the body, so read it.
+            self._absorb(e.code, e.headers, e.read())
+        except Exception as e:                                  # noqa: BLE001
+            self.error = f"{type(e).__name__}: {e}"
+
+    def _absorb(self, code, headers, raw) -> None:
+        self.reachable = True
+        self.code = code
+        self.headers = {k.lower(): v for k, v in headers.items()}
+        try:
+            self.body = json.loads(raw)
+        except Exception:                                       # noqa: BLE001
+            self.body = None
+
+    def violations(self) -> list[tuple[int, str]]:
+        """(rule number, what is wrong). Empty means conformant."""
+        out: list[tuple[int, str]] = []
+        b = self.body
+
+        if b is None:
+            out.append((4, f"HTTP {self.code} with no JSON body to read a verdict from"))
+            return out
+
+        if b.get("service") != self.id:
+            out.append((1, f"`service` is {b.get('service')!r}, expected the registry id {self.id!r}"))
+
+        status = b.get("status")
+        if status not in CONTRACT_STATUS:
+            out.append((2, f"`status` is {status!r}, not one of ok/degraded/down"))
+
+        # Presence only. Rule 3 sanctions a build stamp for a static artifact —
+        # "the document *is* the content, so its generation time is the only
+        # 'now' it has" — which is why `links` and `monitor` return a timestamp
+        # that does not move between calls and are conformant anyway. Asserting
+        # that it advances would fail exactly the two the rule carves out.
+        if not isinstance(b.get("timestamp"), str):
+            out.append((3, "no `timestamp`"))
+
+        if self.code != 200 and self.id not in STATUS_CODE_SIGNALLERS:
+            out.append((4, f"HTTP {self.code}; the contract asks for an unconditional 200"))
+
+        if self.headers.get("access-control-allow-origin") != "*":
+            got = self.headers.get("access-control-allow-origin")
+            out.append((5, f"Access-Control-Allow-Origin is {got!r}, expected '*'"
+                           " — a browser consumer cannot read this at all"))
+
+        cache = (self.headers.get("cache-control") or "").lower()
+        if "no-store" not in cache and "no-cache" not in cache:
+            out.append((6, f"Cache-Control is {self.headers.get('cache-control')!r}; "
+                           "a cached health response reports the past"))
+
+        return out
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--registry", default=str(ROOT / "ecosystem-registry.json"))
+    args = ap.parse_args()
+
+    services = json.loads(Path(args.registry).read_text())["services"]
+    targets = [(s["id"], s["statusUrl"]) for s in services if s.get("statusUrl")]
+    if not targets:
+        print("::error::registry lists no statusUrl — the check found nothing to probe")
+        return 1
+
+    failures: list[str] = []
+    unreachable: list[str] = []
+    conformant: list[str] = []
+    excepted: list[str] = []
+
+    print(f"Probing {len(targets)} health endpoints with Origin: {PROBE_ORIGIN}\n")
+
+    for sid, url in targets:
+        p = Probe(sid, url)
+        p.fetch()
+
+        if not p.reachable:
+            unreachable.append(sid)
+            print(f"  ?  {sid:<11} unreachable — {p.error}")
+            continue
+
+        broken = p.violations()
+        exempt = EXCEPTIONS.get(sid, {}).get("rules", set())
+        unexpected = [(n, msg) for n, msg in broken if n not in exempt]
+        fixed = exempt - {n for n, _ in broken}
+
+        if unexpected:
+            failures.append(sid)
+            print(f"  ✗  {sid:<11} HTTP {p.code}")
+            for n, msg in unexpected:
+                print(f"       rule {n}: {msg}")
+        elif sid in EXCEPTIONS:
+            excepted.append(sid)
+            print(f"  –  {sid:<11} known exception (rules {sorted(exempt)}) — {EXCEPTIONS[sid]['why']}")
+        else:
+            conformant.append(sid)
+            print(f"  ✓  {sid:<11} HTTP {p.code}")
+
+        if fixed:
+            failures.append(sid)
+            print(f"::error::{sid} now satisfies rule(s) {sorted(fixed)} — remove them from "
+                  f"EXCEPTIONS in this script so the entry cannot outlive its reason")
+
+    # Everything failing to answer means this check is broken, not that the
+    # whole ecosystem is down. Silence would read as "nothing to report".
+    if len(unreachable) == len(targets):
+        print("\n::error::every probe failed — the check itself is broken, not the ecosystem")
+        return 1
+
+    print(f"\n{len(conformant)} conformant · {len(excepted)} known exceptions · "
+          f"{len(unreachable)} unreachable (not graded) · {len(failures)} failing")
+
+    if unreachable:
+        print("::notice::not graded because they did not answer: " + ", ".join(unreachable))
+
+    if failures:
+        print("\n::error::health contract regression: " + ", ".join(sorted(set(failures))))
+        print("See HEALTH_CONTRACT.md. Note the header is usually set at the edge "
+              "(nginx/Caddy/API Gateway), not in the app — check there first.")
+        return 1
+
+    print("\nAll registered health endpoints hold the contract.")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

.github/workflows/health-conformance.yml

@@ -0,0 +1,47 @@
+# Does the live ecosystem still hold HEALTH_CONTRACT.md?
+#
+# Deliberately not `on: push`. This probes production, so a push-triggered run
+# would report the state of the world at the moment someone edited a README —
+# and would go red for an outage that has nothing to do with the change. Daily
+# is the right cadence for drift that arrives from outside any repository:
+# an nginx edit, a Caddyfile restored from a backup, an AWS console change.
+#
+# It lives here because conformance is not a property of any service's source.
+# The CORS header usually comes from the edge, not the app, and every service's
+# own deploy gate probes an origin *behind* that edge — so each repo's CI is
+# structurally blind to this. The registry is here, so the check is here.
+name: health conformance
+
+on:
+  schedule:
+    # 03:00 UTC — noon KST, when someone is awake to act on a red.
+    - cron: "0 3 * * *"
+  workflow_dispatch:
+  pull_request:
+    # Only when the check or the thing it reads changes. A registry edit that
+    # adds a statusUrl should be probed before it merges, not the day after.
+    paths:
+      - "ecosystem-registry.json"
+      - "HEALTH_CONTRACT.md"
+      - ".github/scripts/check-health-conformance.py"
+      - ".github/workflows/health-conformance.yml"
+
+permissions:
+  contents: read
+
+jobs:
+  conformance:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      # No dependencies on purpose: urllib only. A daily probe that installs
+      # from PyPI can fail for reasons that have nothing to do with what it
+      # measures, and a check nobody trusts is a check nobody reads.
+      - name: Every registered /api/health holds the contract
+        run: python3 .github/scripts/check-health-conformance.py

HEALTH_CONTRACT.md

@@ -115,3 +115,23 @@ A service that serves this endpoint gets a `statusUrl` in `ecosystem-registry.js
 The registry does not poll it and does not derive `status` from it — `statusUrl` is
 a pointer, not a measurement. That non-goal is deliberate and is documented in
 `ecosystem-registry.schema.json`.
+
+## Checking it
+
+`.github/scripts/check-health-conformance.py` probes every `statusUrl` in the
+registry against the rules above and runs daily from `.github/workflows/
+health-conformance.yml`. It sends a foreign `Origin`, because a service that
+reflects only allow-listed origins passes a same-origin `curl` and still fails
+every real consumer.
+
+It lives here rather than in each service's repo because conformance is mostly
+not a property of any service's source: the CORS header usually comes from
+nginx, Caddy or an API gateway, and every service's own deploy gate probes an
+origin *behind* that edge. Measured 2026-09-10, `passport`'s app sends no
+`Access-Control-Allow-Origin` at all — it arrives from a Caddyfile on the box.
+
+Services that do not yet satisfy a rule are listed in the script's `EXCEPTIONS`
+with the reason and who can fix it. The build goes red if one of them starts
+conforming, so an entry cannot outlive its reason. A service that simply does
+not answer is reported and skipped: this checks the contract, not uptime, and a
+gate that goes red for an outage is a gate people learn to ignore.