fix(api): serve stale or empty data instead of 5xx when Dune or CoinGecko fail (DEV-1163) - #2149
Merged
Merged
Conversation
Dune (revenue) and CoinGecko (token price history) failures now serve the last successful result, or an empty series when there is none, instead of 503/500. Those two routes produced 95% of all DAO API 5xx in the last two weeks and each 5xx counts against the gateway circuit breaker for the whole DAO, which is how ENS and FLUID went offline in late August. Also maps FLUID to its actual CoinGecko id (instadapp): the previous id 404ed on every call. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
brunod-e
requested review from
LeonardoVieira1630,
PedroBinotto,
alextnetto and
pikonha
as code owners
September 3, 2026 12:17
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
|
🚅 Deployed to the anticapture-pr-2149 environment in anticapture-infra
|
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2149
September 3, 2026 12:21
Destroyed
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f144c5c5e7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Contributor
|
🔍 Vercel preview: https://anticapture-dqz8k8txw-ful.vercel.app |
…ut of caches Review feedback: fetchJson is now generic over the row type so the empty fallback needs no unknown cast, setCacheControl no longer overrides a Cache-Control header set by the handler, and the empty series returned when CoinGecko is unavailable is sent with no-store so the gateway does not cache it for the route's one hour max-age after the provider recovers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2149
September 3, 2026 12:37
Destroyed
pikonha
approved these changes
Sep 9, 2026
pikonha
reviewed
Sep 9, 2026
pikonha
requested changes
Sep 9, 2026
The Dune and CoinGecko fallbacks answer with 200s, so the stale and empty responses they serve are invisible to the HTTP error metrics and to the HighErrorRate alert. Every fallback site now increments the degraded_upstream_responses_total counter, labelled by upstream, resource and mode, alongside the structured warn log carrying the upstream error. The DegradedUpstreamData rule in infra/monitoring/alerts.yml fires on that counter and routes through the existing Alertmanager Slack receiver. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2149
September 14, 2026 17:36
Destroyed
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a4ae59291c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The token history route degraded on any non-client error. For NOUNS and LIL_NOUNS it is wired with NFTPriceService, which reads auction prices from PostgreSQL before calling CoinGecko, so a database error or a mapping bug was returned as an empty 200 labelled as a CoinGecko failure and hidden from error monitoring. UpstreamUnavailableError now carries the failing upstream and is thrown only by the provider calls themselves: the CoinGecko market chart request, the ETH price requests in the NFT path, and the Dune result requests. The controller and both degraded clients fall back only on that error and rethrow everything else, so our own failures stay a 5xx and are counted by the HTTP metrics. It extends HTTPException with 503, so callers that do not degrade keep the status and body they returned before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2149
September 14, 2026 18:07
Destroyed
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e0a61b1482
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Malformed 2xx bodies escaped the upstream classification. The Dune revenue client accepted and cached any JSON, so a body without result.rows was kept for 24h and failed in the mapper on every later call, outside the classified path. It now validates the envelope before anything is cached and degrades on a mismatch. The NFT price wrapper returned CoinGecko's body unchecked, so a missing, mistyped or too short prices array blew up in the mapping and became a 500 instead of degrading; it now validates the shape and the minimum length the caller's mapping needs. The CoinGecko token price call is now classified like the others, and every live provider call has a 15s deadline, an AbortSignal for fetch and the axios timeout option for the axios clients. Without one a hanging provider is neither degraded nor errored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…routes Dune rows are now validated per query key inside the fetch, before anything is cached, so a renamed or nulled column degrades once instead of being stored for 24h and then failing in the mappers on every later request. The row cast is gone and nullable aggregate columns are modelled explicitly, with the mappers defaulting them to zero. Only transport errors, timeouts, 408, 429 and 5xx count as an outage. A provider rejecting our request, an expired key or a deleted query id or a wrong token id, now surfaces as a 502 with an error log naming the upstream, status and path, so HighErrorRate catches it rather than an empty 200 being served behind a warning. Treasury and token properties degrade the same way, which is what stops a CoinGecko outage from opening the DAO circuit breaker in the gateway. Stale responses carry no-store like the empty ones, so day-old prices do not sit in a downstream cache for the route's hour. The per-days price cache became one bounded last-good series with a 24h max age, sliced to the window being asked for, and the NFT path gained the same. A Dune outage now leases the stale value for 60s and shares one in-flight call per key, instead of every request opening its own. Also removes the dead DuneService, drops `for` from the DegradedUpstreamData alert so it pages on the first degraded response, and restores spies between tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…outage The treasury path degraded straight to empty while the price routes served their last good value, so an outage blanked the chart rather than showing slightly old numbers. TreasuryProviderCache now keeps fetchedAt and exposes a bounded stale read, capped at the fresh TTL plus one day, the same rule and reasoning as the revenue cache. All three providers expose it through getStaleTreasury, and the one degrade path in the service serves it marked degraded, which the controller answers with no-store. Empty is the fallback only when nothing fresh enough is held. Nothing is written to the cache on a failure, so a degraded response can never become the next response's source of truth. The token treasury path needs no change: its staleness lives in the price provider, and its other input is the database. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2149
September 14, 2026 22:06
Destroyed
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a2c7e27a87
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The fake now implements ITreasuryRepository, the interface TreasuryService depends on, so the five `as unknown as TreasuryRepository` casts go away and the test double is type-checked against what the service reads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2149
September 14, 2026 22:31
Destroyed
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6370a90d10
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…nd label stale treasury honestly Follow-ups from a regression pass over yesterday's commits: - Splitting the empty check in two made the token treasury return nothing whenever no transfer fell inside the window, skipping the balance carried in from before it. It now returns empty only when there is no balance to carry either, as dev did. - A 404 from CoinGecko, DefiLlama or Compound became a 502, which the gateway counts against the DAO's breaker. It degrades under `not_found` now, the rule the Dune client already followed; 401 and 403 stay a 502. - A cached empty treasury series was reported as `mode=stale` while serving nothing; it is reported and served as `empty`. - A vacuous `every()` over an empty array in the unit tests asserts the empty result instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2149
September 15, 2026 12:08
Destroyed
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 98e5254409
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…he controller for real The NFT price service used a hand-rolled copy of the axios classification that never got the 404 branch, so a delisted ETH price returned a 502 for NOUNS and LIL_NOUNS while every other DAO degraded; it now uses the shared classifyAxiosFailure. The historical-data controller passes the upstream reason to the degraded counter, so a persistent not_found is not counted as a transient outage. The controller test runs the real CoingeckoService with MSW at the HTTP boundary instead of a fake client, the CoinGecko test asserts the whole result, and redactUrl keeps the path of the relative axios paths every provider logs instead of a placeholder. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2149
September 15, 2026 17:17
Destroyed
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2bdee77394
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…e and harden fallbacks Upstream failures are logged as a description (status, code, redacted url, cause chain capped at three) instead of the raw AxiosError, whose config carried the CoinGecko and Dune API keys. DegradedUpstreamData gains an arm for counter series born inside the window, so the first fallback of a new label set pages. Dune month values must be real calendar dates before they are cached. The empty Dune fallback is leased like the stale one, for transport and status failures only. The NFT spot price is quoted in the requested currency so its stale tag is truthful. The total treasury refuses a half that degraded to nothing instead of serving a token-only sum, joins the two halves by date rather than by position, and a DAO holding none of its token no longer loses its liquid half in a CoinGecko outage. The stale treasury test runs through the real provider factory with MSW. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2149
September 15, 2026 18:38
Destroyed
…osals-endpoint # Conflicts: # infra/monitoring/alerts.yml
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e8b51a66af
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2149
September 15, 2026 18:51
Destroyed
…t NFT results whole
A DAO with neither in-window transfers nor a carried balance has nothing to
price, yet the token treasury still waited on CoinGecko and recorded a
degraded treasury on outage before discarding the result. Fetch the balances
first and return before asking the provider.
The NFT price happy-path tests destructured `degraded` away, so a regression
flagging every healthy response as degraded would have passed. Assert the
full `{ data, degraded: false }` object.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
railway-app
Bot
temporarily deployed
to
anticapture-infra / anticapture-pr-2149
September 15, 2026 18:55
Destroyed
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1b229951cf
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Causa raiz das quedas de 26-28/08
Reconstruído pelos access logs do gateful e logs/traces das DAO APIs (a métrica
http_server_requests_totalestá inflada por scrape de réplicas e não foi usada). O volume da plataforma é ~3 req/s, flat. As falhas reais vieram de duas rotas com dependência externa quebrada, que juntas são 95% de todos os 5xx das DAO APIs em 14 dias:/ens/revenue/renewal-tenureHTTP 404para a query derenewalTenure/ENS/*em 26/08 17-19h/fluid/token/historical-data404em/coins/fluid/market_chart(id errado)Cada 5xx conta no circuit breaker por DAO do gateful (5 consecutivas -> 5-40min de 503 para todas as rotas). O dashboard faz 3 retries por padrão, então um único usuário abrindo a página de Revenue do ENS gera as falhas necessárias.
Mudanças
FLUIDmapeado parainstadapp(id real;fluidnão existe). O serviço guarda a última resposta boa por janela e a serve quando o CoinGecko falha; a rota/token/historical-dataresponde200 []em vez de 5xx quando não há dado (404 de token não listado é preservado).RevenueDuneClientserve a última resposta boa (cache de 24h passa a reter entradas expiradas para isso) ou um result set vazio quando o Dune falha, em vez de 503. As 7 rotas/revenue/*passam a degradar para200comitems: [], igual ao comportamento quando o client não está configurado. O erro continua logado com a URL.Pendente fora do código (Railway)
REVENUE_DUNE_RENEWAL_TENURE_QUERY_IDaponta para uma query que o Dune responde 404. Com este PR a rota deixa de derrubar o ENS, mas o gráfico de renewal tenure só volta a ter dados quando o id for corrigido.Verificação
pnpm api typecheck,pnpm api lint: OKRelacionado: PR de resiliência do gateful/API (breaker por rota, RPC fora do hot path).
🤖 Generated with Claude Code