fix(monitoring): scrape each DAO API replica and alert on open circuits - #2151
Merged
Merged
Conversation
DAO APIs run with several Railway replicas behind one private hostname, and a static target was answered by a different replica at every scrape, so their counters looked like they reset constantly and rate() reported thousands of times the real traffic (32,000 req/min for the ENS API against ~8 req/min in its logs). The API job now resolves the hostname's AAAA records and scrapes every replica as its own target. Adds GatefulCircuitOpen (a DAO route is being rejected by the gateway) and DaoApiRouteFailing (one route failing most of the time for an hour), the two signals that would have pointed straight at the causes of the late-August outages. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
brunod-e
requested review from
LeonardoVieira1630 and
PedroBinotto
as code owners
September 3, 2026 12:23
|
🚅 Deployed to the anticapture-pr-2151 environment in anticapture-infra
|
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2151
September 3, 2026 12:24
Destroyed
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4394684da9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…tune DaoApiRouteFailing Review feedback: a DNS-discovered job with no replicas has no up series, so add a Prometheus self-scrape and an ApiNoReplicas alert on prometheus_sd_discovered_targets. DaoApiRouteFailing already smooths over a 1h window, so its for clause drops to 10m (a short burst of failures now alerts instead of aging out of the window first), and the ratio is expressed in percent so the annotation prints the right number. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2151
September 3, 2026 12:30
Destroyed
Contributor
|
🔍 Vercel preview: https://anticapture-lmbcuhnq4-ful.vercel.app |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 53f65db6b4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
With one up series per discovered replica, a single replica restarting during a rolling deploy would have fired the critical ApiDown. ApiDown now requires every replica of the job to be down; ApiReplicaDown (warning) keeps the per-instance signal. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2151
September 3, 2026 12:54
Destroyed
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 45c79fdb7a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
pikonha
approved these changes
Sep 9, 2026
… by job DAO API jobs can have several replicas via DNS discovery, each with its own process_cpu_time_seconds_total series. Summing by job before comparing to 80% let two healthy replicas at 45% CPU each falsely alert. Aggregate by (job, instance) instead, matching the ApiReplicaDown per-instance pattern. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2151
September 14, 2026 17:05
Destroyed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problema
http_server_requests_total(e o histograma de duração) das DAO APIs estava inflada em ~4.000x. As APIs rodam com 2+ réplicas na Railway atrás de um hostname privado, e o job usavastatic_configscom um único targethost:porta: cada scrape cai numa réplica diferente, o contador "cai", erate()/increase()tratam como reset e somam o valor inteiro (9.875 resets/dia só na ENS API). Valores brutos do contador alternando entre duas sequências (11785, 11787, 11790/11746, 11747, 11749) confirmaram. Ground truth: ~8 req/min nos logs da ENS API contra 32.000/min na métrica.Isso levou a um diagnóstico errado durante a investigação da DEV-1163 (o notification-system foi apontado como causa das quedas por "2.600 req/s" que nunca existiram).
Mudanças
entrypoint.prometheus.sh: o jobanticapture-<dao>-apiusadns_sd_configs(AAAA) no hostname de<DAO>_API_ENDPOINT, um target por réplica. Indexers seguem estáticos (Ponder é instância única).<DAO>_API_ENDPOINTcontinua emhost:porta.alerts.yml:circuit_breaker_state == 2por 2 min, por nome do breaker. É o sinal direto de "usuários vendo 503 para este DAO/rota".HighErrorRateglobal não destaca.dns_sde o pitfall da métrica.Observação para o time
HighErrorRateeHighLatencydispararam em ~30 dos últimos 45 dias (incluindo 26-28/08). Eles funcionam, mas viraram ruído; vale rever thresholds/roteamento em separado. Não alterei nada nesses dois.Verificação
alerts.ymlparseia como YAML válido;sh -nno entrypoint OK; trecho gerado por DAO validado.promtool check rules infra/monitoring/alerts.ymlno CI/local antes do merge.🤖 Generated with Claude Code