Skip to content

Alerting stack: Grafana alerts, synthetic-check script, ONCALL.md runbook #166

Description

@truthixify

Tier: M (2-4 days) | Type: ops

Context. stellar/scripts/metrics-indexer/ ships a Prometheus scraper and a pre-built grafana-dashboard.json, and stellar/scripts/keeper/ runs the TTL keeper, but there is no alert layer on top of them. stellar/MAINNET_READINESS.md has Ops → Incident Response Runbook and Ops → On-call Rotation both unchecked. stellar/DEPLOYMENT.md contains zero references to monitor, on-call, alert, runbook, or incident. Wave 8 turns the metrics into a real alerting stack before mainnet.

Scope.

  • Add Grafana alert rules alongside the dashboard: rate spike on send_count, drop on resolve_hit_count, missed keeper heartbeat, name-registration failure spike.
  • Add stellar/scripts/synthetic-check.sh: end-to-end announcement / resolution / name-lookup ping against a target network, with pass / fail exit codes suitable for a cron.
  • Publish stellar/ONCALL.md: escalation ladder, paging targets, incident-response checklist (declare, contain, communicate, resolve, postmortem), links to the pause and multisig runbooks.
  • Update stellar/DEPLOYMENT.md with a new "Post-deploy monitoring" section pointing at all three.

Acceptance.

  • Grafana alert rules JSON committed under stellar/scripts/metrics-indexer/ and loaded by the docker-compose in that dir
  • synthetic-check.sh returns nonzero on injected failure and zero on happy path (verified against futurenet in CI)
  • stellar/ONCALL.md reviewed and linked from stellar/README.md operations section
  • stellar/MAINNET_READINESS.md Incident Response Runbook row checked with a link to ONCALL.md

Files. stellar/scripts/metrics-indexer/alerts.json (new), stellar/scripts/metrics-indexer/docker-compose.yml (or existing config file), stellar/scripts/synthetic-check.sh (new), stellar/ONCALL.md (new), stellar/DEPLOYMENT.md, stellar/MAINNET_READINESS.md, stellar/README.md.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Stellar WaveIssues in the Stellar wave programdocsDocumentationdripsFunded via Drips Networkhelp wantedExtra attention is neededobservabilityAuto-created for Wave 8opsstellarTouches Stellar / Soroban codestellar-waveAuto-created for Wave 8wave-8Auto-created for Wave 8

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions