feat(remote): report why the relay connection died - #229
Merged
Conversation
The persistent analyzer's connection to the relay breaks and reconnects every 65 seconds or so, and neither side says why. capnweb reports every death as the same "WebSocket connection failed", which cannot distinguish the server reaping us as a half-open client from an ordinary close. The socket is now constructed here rather than opened by capnweb from a URL, so its close frame reaches the log. A close with code 1006 and no reason is a terminate; anything else is not. watchEventLoopLag warns when the loop stops turning for more than five seconds. The relay's pong is answered by the transport on that same loop, and the server terminates any client that misses one 30-second heartbeat, so a stall here looks exactly like a dead process from the outside. Nothing in the logs currently distinguishes the two. It runs only on the persistent path; a CI session is over in seconds. Neither line changes behaviour. Both exist to answer a question the current logs cannot. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.

Query Doctor — 6 successful checks
More details via MCP → get_ci_run({ runId: "019ff786-1330-72f2-b94f-1ce4f44f1054" }) · view run · docs
3 queries read against main on assumed statistics of 10,000,000 rows per table. Sync production stats for costs measured against your real data.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
The persistent analyzer's connection to the relay breaks and reconnects roughly every 65 seconds, and has done for at least 19 days. Neither side records why. This PR adds the two lines needed to find out. It changes no behaviour and fixes nothing on its own.
The follow-up is the delivery fix in the PR stacked on top of this one, which stops a push lost to that connection from being recorded as delivered.
What
Before: a broken connection logged
Connection broken: Error: WebSocket connection failed.and nothing else. That message is the same whether the server terminated the client, the network dropped, or the process closed the socket itself.After: the WebSocket close code and reason are logged alongside it. A stalled event loop is reported once it passes five seconds.
How
ApiClient.connectnow constructs theWebSocketand hands it to capnweb, instead of passing a URL string for capnweb to open. That is the only way to see the close frame, since capnweb collapses every failure into one message. Verified againstapi.querydoctor.comthat capnweb drives a caller-constructed socket: the server's auth check replies in 1.2s.watchEventLoopLagis a one-second interval that measures its own drift. It exists because the relay's pong is answered by the transport on the same event loop, and the server terminates any client that misses one 30-second heartbeat. A process pinned by CPU work is therefore indistinguishable, from the server's side, from one that has died, and nothing currently records the difference. It runs only on the persistent path, holds the process open viaunref, and costs one subtraction per second.Tests
No new tests. Both changes are log lines, and the one behavioural risk — whether capnweb accepts a socket it did not open — was checked against production rather than mocked.
Existing suite passes: 440 tests across 43 files.
npm run typecheckandnpm run buildare clean.What these lines are expected to show, once deployed: a close code of 1006 with an empty reason at each break, preceded by an event-loop stall over 30 seconds. If the stall is absent, the diagnosis is wrong and the cause is on the network rather than in the process.