Make the production deploy check the site, and restart once if it is down - #905
Open
edwh wants to merge 2 commits into
Open
Make the production deploy check the site, and restart once if it is down#905edwh wants to merge 2 commits into
edwh wants to merge 2 commits into
Conversation
…down Today's outage: production was pushed at 12:02, deploy-fly-prod built the image and ran a rolling update, the machine came up at 12:08 without passing its health check, and flyctl gave up: Error: failed to update machine 7813112c002178: Unrecoverable error: timeout reached waiting for health checks to pass The job exited 1 and stopped there, leaving the machine started but out of the proxy pool. With one machine that is a total outage, and it stayed down for 43 minutes. Restarting that same image brought it back immediately, so nothing was wrong with the build - the deploy simply had everything it needed to fix this and instead walked away. So don't stop at flyctl's exit code. Ask the question that actually matters - is the site serving? - by fetching robots.txt from outside Fly, the same static path the health check uses, so the answer doesn't depend on the database. If it isn't serving, restart each machine once, re-checking after each so we stop as soon as it recovers. Never loop: a restart that doesn't help means something a restart cannot fix, and that needs a person. A deploy that only passes because of the restart says so loudly. This path existing is not a reason to stop asking why a machine failed to come up healthy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The post-deploy probe only fetched robots.txt. That is static, and it is also the path the Fly health check uses - so it reports success in exactly the case it most needs to catch. On 2026-09-03 restarters-dev served robots.txt with a passing health check while every database-backed page returned 500, because it pointed at a database app that no longer existed. This check would have called that deploy fine. Two probes now, answering different questions. robots.txt still asks whether anything is serving, because that is what decides a restart is worth trying. /api/homepage_data - public, cheap, aggregates over the database - asks whether the app actually works, and that is what decides pass or fail, including after a restart. Serving but not working is called out and deliberately does not restart: a restart cannot fix a bad config or an unreachable database, and cycling a machine there would only hide the fault. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened today
productionwas pushed at 12:02:30.deploy-fly-prodbuilt the image and started a rolling update. The machine came up at 12:08:07 without passing its health check, and flyctl gave up:The job exited 1 and stopped, leaving the machine
startedbut out of the proxy pool. Withmin_machines_running = 1that is a total outage — every request gotno known healthy instances found for route tcp/443. It stayed down 43 minutes, until the machine was restarted by hand.Restarting that same image brought it straight back. Nothing was wrong with the build. The deploy had everything it needed to recover and instead walked away.
Change
Stop treating flyctl's exit code as the verdict, and ask the question that actually matters: is the site serving?
https://restarters.net/robots.txtfrom outside Fly — the same static path the Fly health check uses, so the answer doesn't depend on the database being reachable.flyctl statusand the last 50 log lines.Notes
bash -n.jqisn't used anywhere else in this config, so its presence is no longer assumed — it's installed if missing.productionbranch to take effect, so it follows the usual develop → master → production route.