From eaa125526d4e818f9f137409127cfa9065622d90 Mon Sep 17 00:00:00 2001 From: mikim <1441941+mikim@users.noreply.github.com> Date: Thu, 10 Sep 2026 13:36:13 +0900 Subject: [PATCH] feat(ci): nothing was watching whether the ecosystem still holds the contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Conformance to HEALTH_CONTRACT.md is mostly not a property of any service's source, so no service's CI can check it. Measured 2026-09-10: passport's own origin at 127.0.0.1:3000 sends no Access-Control-Allow-Origin — the header comes from a hand-edited /etc/caddy/Caddyfile that exists in no repository — and agora's arrives only because an API Gateway CorsConfiguration was deleted from the AWS console. Every service's deploy gate probes an origin *behind* the component that adds the header, so each repo is structurally blind to it. The failure is silent by construction, which is the actual problem. A consumer that cannot read a health endpoint does not go red: monitor.moss.land falls back to city's aggregate and the service keeps a green dot sourced from a probe of an unrelated data URL. signalmap sat green that way for weeks with an unreadable /api/health. So the check belongs where the registry is. It reads statusUrl from ecosystem-registry.json rather than a list of ids, so a service registered tomorrow is covered tomorrow, and probes with a foreign Origin, because a service reflecting only allow-listed origins passes a same-origin curl and still fails every real consumer. Red on: a conformant service regressing; a recorded exception starting to conform, so EXCEPTIONS cannot rot into a list nobody rechecks; or every probe failing, which means the check is broken rather than the ecosystem being down. Not red on: one service being unreachable. This checks the contract, not uptime, and a gate that goes red for an outage is one people learn to ignore. Daily rather than on push — it probes production, and drift here arrives from outside every repository. Also on PRs that touch the registry, the contract or the check itself. FOUND WHILE WRITING IT, and recorded rather than enforced: rule 6 is open across most of the ecosystem. Seven of the sixteen (alpha, signal, npc, ao, algora, bridge, and city differently) send no Cache-Control at all — the 2026-09-09 nginx session that added ACAO and Vary to eight vhosts did not add this one, and no app sets it. city is the odd one, deliberately cached as `public, s-maxage=60, stale-while-revalidate=120` by its own Next app. These ship as EXCEPTIONS because a gate that is red on the day it ships is a gate nobody reads; the fix is one add_header per vhost. Standing state: 7 conformant, 9 recorded exceptions, 0 failing. Co-Authored-By: Claude Opus 5 --- .github/scripts/check-health-conformance.py | 257 ++++++++++++++++++++ .github/workflows/health-conformance.yml | 47 ++++ HEALTH_CONTRACT.md | 20 ++ 3 files changed, 324 insertions(+) create mode 100644 .github/scripts/check-health-conformance.py create mode 100644 .github/workflows/health-conformance.yml diff --git a/.github/scripts/check-health-conformance.py b/.github/scripts/check-health-conformance.py new file mode 100644 index 0000000..a145fa9 --- /dev/null +++ b/.github/scripts/check-health-conformance.py @@ -0,0 +1,257 @@ +#!/usr/bin/env python3 +"""Probe every registered /api/health and check it against HEALTH_CONTRACT.md. + +Why this exists, and why here rather than in each service's own repo: + +Conformance is not a property of any service's source. For most of the +ecosystem the `Access-Control-Allow-Origin` header comes from nginx or Caddy on +a hand-managed box, not from the app — measured 2026-09-10, `passport`'s origin +at 127.0.0.1:3000 sends no ACAO at all, and `agora`'s arrives only because an +API Gateway CorsConfiguration was deleted from the AWS console. None of that is +in any repository, no service's own test suite can reach it, and every deploy +gate probes an origin *behind* the component that adds the header. So each +repo's CI is structurally blind to the failure this checks for. + +The failure is also silent by construction. A consumer that cannot read a +health endpoint does not see red: monitor.moss.land's map falls back to city's +aggregate, and the service keeps a green dot sourced from a probe of an +unrelated data URL. Nothing anywhere turns a colour. That is exactly how +`signalmap` sat green for weeks while its `/api/health` was unreadable. + +This runs from the registry rather than a list of ids, so a service added to +`ecosystem-registry.json` with a `statusUrl` is covered the day it lands. + +What makes the build red: + + * a service that is expected to conform stops conforming — the regression + this exists to catch; + * a recorded exception starts conforming, so EXCEPTIONS cannot quietly rot + into a list of things nobody rechecks; + * every probe fails, which means the check itself is broken rather than the + ecosystem being entirely down. + +What does not: a single service being unreachable. This checks the contract, +not uptime — a timeout is reported and skipped, because failing the gate for +an outage teaches everyone to ignore it. + +Usage: python3 .github/scripts/check-health-conformance.py [--registry PATH] +""" + +from __future__ import annotations + +import argparse +import json +import sys +import urllib.error +import urllib.request +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +CONTRACT_STATUS = {"ok", "degraded", "down"} +TIMEOUT_S = 20 + +# A browser sends one; the header must not depend on it, so we probe with a +# foreign origin. A service that reflects only allow-listed origins passes a +# same-origin curl and still fails every real consumer. +PROBE_ORIGIN = "https://monitor.moss.land" + +# Services known not to satisfy some rule, with the reason and who can fix it. +# A service listed here is still probed: if it starts conforming, the build goes +# red so the entry gets removed rather than outliving its reason. +# +# Keys are registry ids; values are the rule numbers from HEALTH_CONTRACT.md +# that entry is exempt from, and why. +# RULE 6 IS OPEN ACROSS MOST OF THE ECOSYSTEM, measured 2026-09-10. Seven of the +# sixteen send no `Cache-Control` at all: the 2026-09-09 nginx session that added +# `Access-Control-Allow-Origin` and `Vary` to eight vhosts did not add this one, +# and no app sets it either. Without it a browser applies heuristic freshness, so +# a consumer polling every 60s can be served an answer it already had — which is +# the one thing rule 6 exists to prevent. It is recorded rather than enforced +# because a gate that is red on the day it ships is a gate nobody reads; fixing it +# is one `add_header Cache-Control "no-cache" always;` per vhost, on the box +# reachable as `ssh mossland`. Remove each entry as it lands — and note the check +# goes red if you fix one without removing it, which is the point. +EXCEPTIONS: dict[str, dict] = { + "alpha": {"rules": {6}, "why": "no Cache-Control at the edge — see the note above"}, + "signal": {"rules": {6}, "why": "no Cache-Control at the edge — see the note above"}, + "npc": {"rules": {1, 3, 6}, "why": "no `service`, no `timestamp`, no Cache-Control. " + "Manual-deploy service, no local checkout."}, + "ao": {"rules": {6}, "why": "no Cache-Control at the edge — see the note above"}, + "bridge": {"rules": {6}, "why": "no Cache-Control at the edge — see the note above"}, + "city": { + "rules": {1, 2, 3, 6}, + "why": "its /api/health is the cross-service aggregate (checkedAt/summary/services), " + "not a report about itself, and it is served `public, s-maxage=60, " + "stale-while-revalidate=120` from its own Next app — the only entry here " + "that is cached on purpose. Manual-deploy service, no local checkout.", + }, + "recipe": { + "rules": {1}, + "why": "no `service`. Manual-deploy service, no local checkout.", + }, + "algora": { + "rules": {1, 2, 6}, + "why": 'answers status "running", outside the enum, and sends no `service`. ' + "lifecycle: archive — owner decision 2026-08-23, the code is frozen. " + "Deliberately left visible rather than mapped to ok.", + }, + "signalmap": { + "rules": {1, 3, 5}, + "why": "no ACAO, no `service`, no `timestamp`. Fix merged in " + "MosslandCore/signalmap#134; signalmap is manual-deploy on its own box, " + "so remove this entry once that deploy lands.", + }, +} + +# Rule 4's documented exception: these answer 503 when unhealthy because a probe +# watches the status code and alarms on it. A non-200 from them is not a +# contract violation, and the body still has to be readable. +STATUS_CODE_SIGNALLERS = {"signal", "signalmap"} + + +class Probe: + """One service's answer, and what the contract makes of it.""" + + def __init__(self, sid: str, url: str): + self.id = sid + self.url = url + self.reachable = False + self.error: str | None = None + self.code = 0 + self.headers: dict[str, str] = {} + self.body: dict | None = None + + def fetch(self) -> None: + req = urllib.request.Request(self.url, headers={"Origin": PROBE_ORIGIN}) + try: + with urllib.request.urlopen(req, timeout=TIMEOUT_S) as res: + self._absorb(res.status, res.headers, res.read()) + except urllib.error.HTTPError as e: + # A 4xx/5xx is an answer, not a failure to reach — rule 4 puts the + # verdict in the body, so read it. + self._absorb(e.code, e.headers, e.read()) + except Exception as e: # noqa: BLE001 + self.error = f"{type(e).__name__}: {e}" + + def _absorb(self, code, headers, raw) -> None: + self.reachable = True + self.code = code + self.headers = {k.lower(): v for k, v in headers.items()} + try: + self.body = json.loads(raw) + except Exception: # noqa: BLE001 + self.body = None + + def violations(self) -> list[tuple[int, str]]: + """(rule number, what is wrong). Empty means conformant.""" + out: list[tuple[int, str]] = [] + b = self.body + + if b is None: + out.append((4, f"HTTP {self.code} with no JSON body to read a verdict from")) + return out + + if b.get("service") != self.id: + out.append((1, f"`service` is {b.get('service')!r}, expected the registry id {self.id!r}")) + + status = b.get("status") + if status not in CONTRACT_STATUS: + out.append((2, f"`status` is {status!r}, not one of ok/degraded/down")) + + # Presence only. Rule 3 sanctions a build stamp for a static artifact — + # "the document *is* the content, so its generation time is the only + # 'now' it has" — which is why `links` and `monitor` return a timestamp + # that does not move between calls and are conformant anyway. Asserting + # that it advances would fail exactly the two the rule carves out. + if not isinstance(b.get("timestamp"), str): + out.append((3, "no `timestamp`")) + + if self.code != 200 and self.id not in STATUS_CODE_SIGNALLERS: + out.append((4, f"HTTP {self.code}; the contract asks for an unconditional 200")) + + if self.headers.get("access-control-allow-origin") != "*": + got = self.headers.get("access-control-allow-origin") + out.append((5, f"Access-Control-Allow-Origin is {got!r}, expected '*'" + " — a browser consumer cannot read this at all")) + + cache = (self.headers.get("cache-control") or "").lower() + if "no-store" not in cache and "no-cache" not in cache: + out.append((6, f"Cache-Control is {self.headers.get('cache-control')!r}; " + "a cached health response reports the past")) + + return out + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--registry", default=str(ROOT / "ecosystem-registry.json")) + args = ap.parse_args() + + services = json.loads(Path(args.registry).read_text())["services"] + targets = [(s["id"], s["statusUrl"]) for s in services if s.get("statusUrl")] + if not targets: + print("::error::registry lists no statusUrl — the check found nothing to probe") + return 1 + + failures: list[str] = [] + unreachable: list[str] = [] + conformant: list[str] = [] + excepted: list[str] = [] + + print(f"Probing {len(targets)} health endpoints with Origin: {PROBE_ORIGIN}\n") + + for sid, url in targets: + p = Probe(sid, url) + p.fetch() + + if not p.reachable: + unreachable.append(sid) + print(f" ? {sid:<11} unreachable — {p.error}") + continue + + broken = p.violations() + exempt = EXCEPTIONS.get(sid, {}).get("rules", set()) + unexpected = [(n, msg) for n, msg in broken if n not in exempt] + fixed = exempt - {n for n, _ in broken} + + if unexpected: + failures.append(sid) + print(f" ✗ {sid:<11} HTTP {p.code}") + for n, msg in unexpected: + print(f" rule {n}: {msg}") + elif sid in EXCEPTIONS: + excepted.append(sid) + print(f" – {sid:<11} known exception (rules {sorted(exempt)}) — {EXCEPTIONS[sid]['why']}") + else: + conformant.append(sid) + print(f" ✓ {sid:<11} HTTP {p.code}") + + if fixed: + failures.append(sid) + print(f"::error::{sid} now satisfies rule(s) {sorted(fixed)} — remove them from " + f"EXCEPTIONS in this script so the entry cannot outlive its reason") + + # Everything failing to answer means this check is broken, not that the + # whole ecosystem is down. Silence would read as "nothing to report". + if len(unreachable) == len(targets): + print("\n::error::every probe failed — the check itself is broken, not the ecosystem") + return 1 + + print(f"\n{len(conformant)} conformant · {len(excepted)} known exceptions · " + f"{len(unreachable)} unreachable (not graded) · {len(failures)} failing") + + if unreachable: + print("::notice::not graded because they did not answer: " + ", ".join(unreachable)) + + if failures: + print("\n::error::health contract regression: " + ", ".join(sorted(set(failures)))) + print("See HEALTH_CONTRACT.md. Note the header is usually set at the edge " + "(nginx/Caddy/API Gateway), not in the app — check there first.") + return 1 + + print("\nAll registered health endpoints hold the contract.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/workflows/health-conformance.yml b/.github/workflows/health-conformance.yml new file mode 100644 index 0000000..968ff29 --- /dev/null +++ b/.github/workflows/health-conformance.yml @@ -0,0 +1,47 @@ +# Does the live ecosystem still hold HEALTH_CONTRACT.md? +# +# Deliberately not `on: push`. This probes production, so a push-triggered run +# would report the state of the world at the moment someone edited a README — +# and would go red for an outage that has nothing to do with the change. Daily +# is the right cadence for drift that arrives from outside any repository: +# an nginx edit, a Caddyfile restored from a backup, an AWS console change. +# +# It lives here because conformance is not a property of any service's source. +# The CORS header usually comes from the edge, not the app, and every service's +# own deploy gate probes an origin *behind* that edge — so each repo's CI is +# structurally blind to this. The registry is here, so the check is here. +name: health conformance + +on: + schedule: + # 03:00 UTC — noon KST, when someone is awake to act on a red. + - cron: "0 3 * * *" + workflow_dispatch: + pull_request: + # Only when the check or the thing it reads changes. A registry edit that + # adds a statusUrl should be probed before it merges, not the day after. + paths: + - "ecosystem-registry.json" + - "HEALTH_CONTRACT.md" + - ".github/scripts/check-health-conformance.py" + - ".github/workflows/health-conformance.yml" + +permissions: + contents: read + +jobs: + conformance: + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + + # No dependencies on purpose: urllib only. A daily probe that installs + # from PyPI can fail for reasons that have nothing to do with what it + # measures, and a check nobody trusts is a check nobody reads. + - name: Every registered /api/health holds the contract + run: python3 .github/scripts/check-health-conformance.py diff --git a/HEALTH_CONTRACT.md b/HEALTH_CONTRACT.md index b14bb8f..b7d6d94 100644 --- a/HEALTH_CONTRACT.md +++ b/HEALTH_CONTRACT.md @@ -115,3 +115,23 @@ A service that serves this endpoint gets a `statusUrl` in `ecosystem-registry.js The registry does not poll it and does not derive `status` from it — `statusUrl` is a pointer, not a measurement. That non-goal is deliberate and is documented in `ecosystem-registry.schema.json`. + +## Checking it + +`.github/scripts/check-health-conformance.py` probes every `statusUrl` in the +registry against the rules above and runs daily from `.github/workflows/ +health-conformance.yml`. It sends a foreign `Origin`, because a service that +reflects only allow-listed origins passes a same-origin `curl` and still fails +every real consumer. + +It lives here rather than in each service's repo because conformance is mostly +not a property of any service's source: the CORS header usually comes from +nginx, Caddy or an API gateway, and every service's own deploy gate probes an +origin *behind* that edge. Measured 2026-09-10, `passport`'s app sends no +`Access-Control-Allow-Origin` at all — it arrives from a Caddyfile on the box. + +Services that do not yet satisfy a rule are listed in the script's `EXCEPTIONS` +with the reason and who can fix it. The build goes red if one of them starts +conforming, so an entry cannot outlive its reason. A service that simply does +not answer is reported and skipped: this checks the contract, not uptime, and a +gate that goes red for an outage is a gate people learn to ignore.