Skip to content

feat(remote): report why the relay connection died - #229

Merged
veksen merged 1 commit into
mainfrom
feat-relay-observability
Aug 12, 2026
Merged

feat(remote): report why the relay connection died#229
veksen merged 1 commit into
mainfrom
feat-relay-observability

Conversation

@veksen

@veksen veksen commented Aug 12, 2026

Copy link
Copy Markdown
Member

Goal

The persistent analyzer's connection to the relay breaks and reconnects roughly every 65 seconds, and has done for at least 19 days. Neither side records why. This PR adds the two lines needed to find out. It changes no behaviour and fixes nothing on its own.

The follow-up is the delivery fix in the PR stacked on top of this one, which stops a push lost to that connection from being recorded as delivered.

What

Before: a broken connection logged Connection broken: Error: WebSocket connection failed. and nothing else. That message is the same whether the server terminated the client, the network dropped, or the process closed the socket itself.

After: the WebSocket close code and reason are logged alongside it. A stalled event loop is reported once it passes five seconds.

How

ApiClient.connect now constructs the WebSocket and hands it to capnweb, instead of passing a URL string for capnweb to open. That is the only way to see the close frame, since capnweb collapses every failure into one message. Verified against api.querydoctor.com that capnweb drives a caller-constructed socket: the server's auth check replies in 1.2s.

watchEventLoopLag is a one-second interval that measures its own drift. It exists because the relay's pong is answered by the transport on the same event loop, and the server terminates any client that misses one 30-second heartbeat. A process pinned by CPU work is therefore indistinguishable, from the server's side, from one that has died, and nothing currently records the difference. It runs only on the persistent path, holds the process open via unref, and costs one subtraction per second.

Tests

No new tests. Both changes are log lines, and the one behavioural risk — whether capnweb accepts a socket it did not open — was checked against production rather than mocked.

Existing suite passes: 440 tests across 43 files. npm run typecheck and npm run build are clean.

What these lines are expected to show, once deployed: a close code of 1006 with an empty reason at each break, preceded by an event-loop stall over 30 seconds. If the stall is absent, the diagnosis is wrong and the cause is on the network rather than in the process.

The persistent analyzer's connection to the relay breaks and reconnects every
65 seconds or so, and neither side says why. capnweb reports every death as the
same "WebSocket connection failed", which cannot distinguish the server reaping
us as a half-open client from an ordinary close.

The socket is now constructed here rather than opened by capnweb from a URL, so
its close frame reaches the log. A close with code 1006 and no reason is a
terminate; anything else is not.

watchEventLoopLag warns when the loop stops turning for more than five seconds.
The relay's pong is answered by the transport on that same loop, and the server
terminates any client that misses one 30-second heartbeat, so a stall here looks
exactly like a dead process from the outside. Nothing in the logs currently
distinguishes the two. It runs only on the persistent path; a CI session is over
in seconds.

Neither line changes behaviour. Both exist to answer a question the current logs
cannot.

Co-Authored-By: Claude <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Query Doctor — 6 successful checks

 Cost regression — No query went up more than 5%
 Untested data access — No changed data-access file without a test
 New query — No new queries
 New query with index recommendation — No new query ships an index recommendation
 Schema drift — No schema changes
 High-value nudge — No index or rewrite past the threshold


More details via MCP → get_ci_run({ runId: "019ff786-1330-72f2-b94f-1ce4f44f1054" }) · view run · docs
3 queries read against main on assumed statistics of 10,000,000 rows per table. Sync production stats for costs measured against your real data.

@veksen
veksen merged commit 32d082b into main Aug 12, 2026
6 checks passed
@veksen
veksen deleted the feat-relay-observability branch August 12, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant