fix: contain upstream failures in gateful and keep RPC out of the proposals request path - #2148
Merged
Merged
Conversation
Serve the latest block stale-while-revalidate so requests never wait on eth_getBlockByNumber once the cache is warm, back off after a failed refresh, and degrade to indexed proposal statuses when the RPC is down instead of failing the request. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Replace the consecutive-failure counter with a sliding-window failure rate that needs a minimum sample before opening, so a partially failing reload burst no longer takes a DAO offline while a real outage still trips within seconds. Proxy breakers are now keyed per DAO and route group so one slow route cannot block the others, and the default cooldown drops from 5min to 30s (max 5min) so recovery follows the upstream instead of lagging it by up to 40min. Health reports the worst per-route state for each DAO. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Keep Redis entries for a 5min grace period past their max-age and, when revalidation hits a 5xx, a timeout or an open circuit, answer with the stale body and Cache-Status: Redis; stale instead of an error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
brunod-e
requested review from
LeonardoVieira1630,
PedroBinotto,
alextnetto and
pikonha
as code owners
September 3, 2026 12:17
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
|
🚅 Deployed to the anticapture-pr-2148 environment in anticapture-infra
|
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2148
September 3, 2026 12:18
Destroyed
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 82179d9e76
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Contributor
|
🔍 Vercel preview: https://anticapture-4s7efbbia-ful.vercel.app |
…lth, no-cache on stale Review feedback on the per-route breakers: route keys derive from client controlled paths, so the proxy now falls back to the DAO breaker once the registry holds 128 entries; an OPEN circuit whose cooldown has elapsed no longer makes /health report the DAO as down (it only transitions on its next real request, which idle routes may never see); and stale cache responses are served with Cache-Control: no-cache so browsers do not pin an expired body for another full max-age. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2148
September 3, 2026 12:35
Destroyed
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c36a332432
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…ups, guard per-proposal status Review feedback: the failure ratio is now checked after successes too, so settlement order under concurrency cannot leave an over-threshold window closed; per-route breaker keys are limited to the API's known route groups (unknown segments share the DAO breaker) instead of a global cap a client could exhaust; and one proposal whose status reads fail (TORN reads quorum and timelock without a fallback) keeps its indexed status instead of failing the whole listing or activity request. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2148
September 3, 2026 12:54
Destroyed
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 10990894a2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…bound block staleness Addresses the review on the resilience PR. gateful - The breaker had one trip rule (failure rate over a window with at least 10 requests), so keys that never see 10 requests in 30s (relayer, address-enrichment, fan-out) could no longer open at all: a dead relayer made every vote wait the full 30s proxy timeout. Quiet keys now open after CIRCUIT_BREAKER_CONSECUTIVE_FAILURES (default 5) straight failures; busy keys keep the rate rule, so a partially failing reload still does not trip. - Route breakers are derived from the request path instead of a hardcoded list of API routes: any route-shaped first segment gets its own key the first time it is hit, ids/hashes/addresses stay on the DAO key, and the number of route breakers per DAO is capped at 64 so client-controlled paths cannot grow the registry or the metric series without bound. - Fan-out and the per-DAO health probe use the same per-route breaker as the proxy (`ens:dao`, `ens:health`), so an open route short-circuits them too instead of them reading a bare DAO key nobody writes to any more. - Window outcomes are counted in 10 fixed time buckets (Map keyed by bucket index) rather than an unbounded array of timestamps. api - The stale-while-revalidate latest block is bounded to 60s of age. Past that the call waits for the RPC again and a failure surfaces, so proposal services fall back to indexed statuses (a proposal whose endBlock passed no longer reads ACTIVE for hours) and /health reports chain.head as null. Inside the retry backoff of a failed refresh the call fails fast instead of re-probing the RPC. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2148
September 10, 2026 18:00
Destroyed
Evicting idle route breakers made the per-DAO cap stop bounding metric cardinality: every made-up path that took a slot recorded a new circuit_breaker_state series, and the OTel SDK keeps every attribute set for the life of the process. Route breakers now record their state lazily, from the first transition out of CLOSED onwards, so a series exists only for routes that actually tripped. DAO, relayer and service keys come from configuration and keep recording eagerly, so dashboards still list them while healthy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2148
September 14, 2026 17:22
Destroyed
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d0a395fdee
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Two ways a breaker could lose the failures it had counted. A batch of slow calls kept recording after one of them opened the circuit, pushing the cooldown forward or reopening a circuit that a probe had already closed; every state transition now starts a new generation and a call whose generation has moved on is not counted. Eviction also treated any closed breaker as idle, so made-up paths interleaved with a failing route could drop that route before it tripped; only breakers with no live failures and no failure streak are evicted now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2148
September 14, 2026 17:42
Destroyed
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d758f0a80f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
A breaker whose slow request had not settled yet still looked idle, so another route could take its slot; the timeout then landed on a detached instance while the next request for that path got a clean breaker, and slow failures never added up to a trip. Execution now counts in-flight calls in a finally block on every state path, and a breaker with anything in flight is not idle. Every call site passes an abort timeout, so a slot cannot be pinned indefinitely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2148
September 14, 2026 17:59
Destroyed
…ery path shape A route breaker kept its slot for good once it had failed: a streak only ended on a success and an open circuit only moved on traffic, so one failing call to each of 64 made-up segments pinned a DAO's slots through an outage and left the real routes sharing the DAO breaker afterwards. A streak now ends when the window empties, and a breaker with nothing in flight, no traffic for a window and no cooldown left is reclaimable. Path keys are also decoded and lowercased before matching, with every non-route shape on one key per DAO, and the proxy drops the DAO segment by position so a percent-encoded DAO cannot slip through. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…breakers The per-DAO health probe used a route key, so probe failures reached gateway readiness through the DAO summary, the back door the probe was written to avoid; it now has its own key outside the DAO namespace. Fan-out aborts six times sooner than the proxy and no longer shares the proxy's key, so its deadline cannot open a circuit for calls that had far more budget. The circuit breaker windows, counts and cooldowns are now rejected at boot when not above zero. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… a failing RPC The timelock delay and grace period are read only for queued proposals now, so a listing with nothing queued makes no extra eth_call and a failing read cannot downgrade every proposal to its indexed status. A cold quorum cache shares one in-flight read per key instead of firing one identical call per finished proposal, and the latest-block retry deadline is kept outside the cache entry so an RPC outage at boot is backed off rather than re-probed on every request. Chain head reads without a timestamp are guarded instead of asserted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…out a timestamp Aging the failure streak with the window disabled the consecutive-failure rule for any upstream called less than once per window, which is exactly the set it exists for: a relayer, a health probe, a fan-out, a quiet route. The streak is kept and stale eviction alone reclaims slots. A chain head whose timestamp is missing no longer discards the block number either, so block-based statuses are still computed and only the queued comparisons are skipped. The failure rate is rejected at boot when zero, and the fan-out test now trips a fan-out key. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2148
September 14, 2026 21:40
Destroyed
Collaborator
Author
|
Summary of the hardening pushed today (677e5bd through 191dddc), following the review rounds:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 191dddc3e0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
… of an evicted route Pairing a block number with a timestamp fetched separately put an RPC read back on the warm path: a background refresh landing between the two calls replaced the single cache entry, so the timestamp lookup for the older block missed the cache. Callers now take both from one read through getChainHead. On the gateway side, a route breaker that had reported OPEN and was then evicted left its series stuck at open forever, since the gauge is synchronous and the lazy replacement says nothing while healthy; eviction now publishes a final CLOSED for any breaker that had published at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2148
September 14, 2026 21:51
Destroyed
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4cafa27593
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
… correctly A revalidation that ended on the stale body was counted twice, first as a miss and then as a stale serve, so the denominator of the cache hit rate doubled exactly while an upstream was failing. The result label is now decided once the call has settled and incremented once. The gateful changeset also named env vars that do not exist; it now points at the CIRCUIT_BREAKER prefix and the config file that reads them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2148
September 14, 2026 22:05
Destroyed
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f2bf3c8760
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contexto
Investigação da DEV-1163 sobre as quedas de julho (25-27/07), 19/08 e 26-28/08. Em todos os casos o encadeamento foi o mesmo: um punhado de falhas reais (RPC com blip, Postgres lento, rota com dependência externa quebrada) -> circuit breaker por DAO abre com 5 falhas consecutivas -> todas as rotas do DAO respondem 503 por 5 a 40 minutos, com o probe de recuperação caindo justamente na rota quebrada. Em 26/08 17-19h, 63 falhas reais viraram 722 x 503 no ENS; o FLUID ficou 13h fora com a API saudável.
Este PR reduz o raio da explosão. Não corrige as causas (essas estão no PR da DEV-1163).
Mudanças
gateful
ens:proposals,ens:revenue...): uma rota quebrada não derruba as demais do DAO./healthreporta o pior estado entre as rotas do DAO.max-agee serve stale quando o upstream falha (5xx, timeout, circuito aberto) em vez de erro.CIRCUIT_BREAKER_WINDOW_MS,CIRCUIT_BREAKER_MIN_REQUESTS,CIRCUIT_BREAKER_FAILURE_RATEsubstituemCIRCUIT_BREAKER_FAILURE_THRESHOLD(atualizar na Railway se estiver definida).api
/proposals,/proposals/{id}e/proposals-activityrespondem com o status indexado em vez de falhar (incidente de 19/08: blip de 20min do eRPC derrubou todos os DAOs por >1h).Verificação
pnpm api typecheck/pnpm gateful typecheck: OKpnpm api lint: OK (gateful via lint-staged)🤖 Generated with Claude Code