Skip to content

fix: ignore callbacks from a signal socket that was already replaced - #1211

Open
hiroshihorie wants to merge 9 commits into
hiroshi/complete-disconnect-on-socket-closefrom
hiroshi/signal-socket-generation
Open

hiroshihorie wants to merge 9 commits into
hiroshi/complete-disconnect-on-socket-closefrom
hiroshi/signal-socket-generation

Conversation

@hiroshihorie

@hiroshihorie hiroshihorie commented Sep 16, 2026 •

Copy link
Copy Markdown
Member

Stacked on #1208.

Problem

A real socket reports its close some time after the SDK disposes it (websocket/io.dart onDone, websocket/web.dart onClose). SignalClient routed that report through the same handler as a live server close. When it landed after the next connection was up, it set the signal state back to disconnected and the engine started a reconnect against a healthy socket.

How late the report comes decides how often this bites:

  • In a browser the close event fires when the server drops TCP. When the server keeps TCP open after the close handshake that is about 2 s later.
  • dart:io reports it when the server answers the close frame. That is quick on a healthy path and late on a dead one, for example after a network change.

Any reconnect replaces the socket, so when the old close lands after the reconnect is done, the room reconnects again, and again, every 2 s or so for as long as the session lives. A disconnect() followed by connect() hits the same path.

Fix

  • Each socket's onData, onDispose and onError callbacks carry the id of the socket they were created for. connect() gives each new socket the next id and cleanUp() advances it too. A callback from another id is dropped with a log line. Same as the JS SDK ignoring a close from a transport that has since been replaced.
  • The signalReconnect and fullReconnect debug scenarios, the disconnectSignalOnResume actions and the connection checker's reconnect check used to call signalClient.cleanUp() and rely on the late close to make the engine recover. That close is now ignored by design, so they call a new internal simulateSignalDrop() which disposes the socket and raises the disconnect explicitly. Same observable behaviour.
  • The mock socket reports its close after dispose() with a configurable latency, like the real implementations.

Tests

test/core/stale_socket_close_test.dart:

  • a server Leave asks for a resume, or a full reconnect, while the socket is open, the reconnect completes, then the old socket reports its close: one reconnect, no new connection, room stays connected. Both fail without the guard.
  • disconnect, connect again before the old socket reports its close: no reconnect or disconnect event. Fails without the guard.
  • the debug scenarios and the connection checker still start a reconnect.

e2e-flutter

Scenarios in livekit/e2e-flutter#39. Only the target's signal socket goes through a proxy that makes the old socket's close late: it delays the server dropping TCP, which is what a browser waits for, and the server's answer to the close frame, which is what dart:io waits for.

Scenario 2.14.0 this PR
Late close of a replaced signal socket after a resume fails, resumes every 2 to 2.5 s passes
Late close of a replaced signal socket after a full reconnect fails, same loop passes
A signal connect answered after its timeout (#1219) fails, duplicateIdentity passes
A signal connect that reaches the server after its timeout (#1219) fails still fails, needs the timed-out connect cancelled, follow-up

Run on macOS against a local livekit-server 1.13.11 and LiveKit Cloud, in Chrome against the local server (the two late close scenarios), and on an iPhone 17 Pro Max and an iPhone 16 Pro on iOS 27.2.1 against LiveKit Cloud. The other reconnect, signaling, lifecycle and RPC scenarios give the same results as on 2.14.0 in each setup.

A real socket reports its close a few milliseconds after dispose()
returns. That report went through the same handler as a live close, so
when it landed during the next connect it reset the signal state to
disconnected and started a reconnect against a socket that was fine.
Each socket's callbacks now carry a generation, bumped on cleanUp() and
on every connect, and a callback from an older generation is dropped.

The signalReconnect and fullReconnect debug scenarios relied on that
late close to trigger the engine's recovery. They now call
simulateSignalDrop(), which disposes the socket and raises the
disconnect explicitly.

The mock socket reports its close after dispose() like the real ones, so
the race is reproducible in tests.
…ocket

cleanUp() now ignores the close the disposed socket reports afterwards, so
callers that relied on that late close to make the engine recover stopped
working. The connection checker's reconnect check always reported that no
reconnect was attempted, and the disconnectSignalOnResume debug actions never
started the resume the server is waiting for.

Both now call simulateSignalDrop(), like the signalReconnect and fullReconnect
scenarios already do.
@hiroshihorie
hiroshihorie marked this pull request as ready for review October 16, 2026 21:56

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

@hiroshihorie
hiroshihorie added this pull request to stack #1242 October 16, 2026 21:59
_socketGeneration becomes _socketId. The value is compared to decide whether a
callback came from the current socket, which an id describes more directly than
a generation. No behavior change.
…hi/signal-socket-generation

# Conflicts:
#	test/mock/websocket_mock.dart
The close of a replaced socket can reach the client seconds after the reconnect that replaced it, a browser reports it only once the server drops TCP. On main it was taken for a loss of the new socket and every reconnect started the next one. Both shapes fail with the socket id check removed.
@hiroshihorie

Copy link
Copy Markdown
Member Author

Merged main (through #1208) in, resolved the mock conflict, format passes now. Rewrote the description: the late close is seconds late in a browser, not milliseconds, which turns this into a reconnect every 2 s rather than a rare race. Added tests for the resume and full reconnect shapes and the e2e-flutter results.

…aches the old socket first

A connect whose socket opened only after disconnect() or a newer connect() still installed itself as the current socket, so requests went to the wrong socket and the room could report connected after disconnect() returned. It now closes that socket and fails with SignalConnectSupersededException, which leaves the shared state and events alone and is not a reason to fail over to another region.

cleanUp() detached the socket only after awaiting its close, so a connect() that ran while a dart:io close waited on the server lost its own socket, queue and ping timers when that cleanUp() finished. It now detaches first and awaits the close after.

simulateSignalDrop() does nothing without a socket and reports the loss before the socket has finished closing. The connection checker comment matches what it does now.

Tests: the mock holds old closes until the test releases them, so the late close tests no longer depend on timing, the checker test answers the resume and expects success, and there are new tests for late data from a replaced socket, the superseded connect, the cleanUp() race and a drop with no socket. Each fails without its change. Found in review (Codex).

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 new potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment on lines +169 to +172
// Attempt to connect. Callbacks carry the id of the socket they belong to
// so a late report from a replaced socket is ignored.
final id = socketId = ++_socketId;
final ws = await _wsConnector(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Stale resume rejects a newer join

If a resume finishes closing its old socket during a new join’s handshake, connect gives the resume the newer ID. The new join then rejects its own successful handshake as superseded.

Learn more

A server migration requesting RESUME reaches resumeConnection while the old signal socket is still open. cleanUp detaches that socket before awaiting its disposal, and the native WebSocket disposer awaits the close. The app can complete disconnect and call Room.connect while that detached socket is closing. If the close finishes while the new join is opening its socket, the old resume claims a later ID and the post-handshake check rejects the new join, regardless of whether the resume itself succeeds.

Example: A migration starts a resume and waits for socket A to close. The app leaves and starts joining another room on socket B. A finishes closing before B opens, so the stale resume claims the current ID. B opens successfully but the SDK disposes it and fails the join.

Recommended fix: Reserve ownership of the connection attempt before asynchronous preparation and teardown. Let connect detach the old socket without invalidating its own reservation, and recheck that reservation before invoking the connector as well as after the handshake. Add a gated test where the old close finishes while the new join is opening.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread lib/src/core/engine.dart
Comment on lines +357 to +359
// A superseded attempt reports nothing, the connection that replaced it
// is the one the room has.
if (error is SignalConnectSupersededException) rethrow;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Cancelled reconnect restarts the new room

When a full reconnect opens after the app leaves and rejoins, SignalConnectSupersededException still reaches reconnect recovery. attemptReconnect treats it as retryable and restarts the healthy session.

Learn more

A server-requested full reconnect opens a fresh signaling socket through restartConnection. The app can disconnect during that handshake and reuse the room; Engine.connect clears _isClosed and _connect replaces the URL and token. When the old handshake returns, SignalClient.connect discards it with a superseded exception. Without another region, restartConnection propagates that exception, and attemptReconnect sets fullReconnectOnNext and queues a retry. The engine is open again, so the retry tears down the new room's transports in restartConnection. With a regional fallback, restartConnection can start another handshake directly instead of propagating cancellation.

Example: Room A starts a full reconnect. The app leaves A and successfully joins B using the same Room. A's handshake returns late; the old retry starts a full reconnect of B even though B is healthy.

Recommended fix: Propagate supersession past regional failover and treat it as cancellation, not failure, in attemptReconnect. Scope the attempt's retry bookkeeping and finally cleanup to the session that started it. Test a full reconnect held at the handshake while the app disconnects and joins again, then verify that releasing the stale handshake opens no further sockets.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +184 to +188
if (id != _socketId) {
// disconnect() or a newer connect() took over while this socket was
// opening. Installing it would take the place of the current one.
await ws.dispose();
throw const SignalConnectSupersededException();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Stale join disables Android call routing

When a cancelled join opens after a successful rejoin, SignalConnectSupersededException reaches Room.connect. Its failure cleanup stops the shared Android audio session, releasing the active call’s audio focus and routing.

Learn more

Room.connect starts native audio before it opens the signal socket. Room._cleanUp stops audio when the app disconnects, and the next join starts it again. When the first socket finally opens, the superseded exception passes through Engine._connect, but the original Room.connect still runs its failure cleanup. In SDK-managed Android mode, NativeAudioManagement.stop stops the shared session, and LKAudioSwitchManager.stop tears down focus and routing rather than releasing an attempt-specific resource.

Example: An Android app cancels a slow join and then joins another room. The new call starts with Bluetooth routing. The first handshake completes late and its error cleanup stops the audio session still in use by the new call.

Recommended fix: Track which room connection attempt owns the native audio setup and make failure cleanup conditional on that ownership. A superseded attempt must not stop audio after a newer attempt has taken ownership. Extend the delayed-handshake regression test to assert that the stale completion causes no extra native audio stop.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant