Repository navigation
fix: ignore callbacks from a signal socket that was already replaced - #1211
hiroshihorie wants to merge 9 commits into
Conversation
A real socket reports its close a few milliseconds after dispose() returns. That report went through the same handler as a live close, so when it landed during the next connect it reset the signal state to disconnected and started a reconnect against a socket that was fine. Each socket's callbacks now carry a generation, bumped on cleanUp() and on every connect, and a callback from an older generation is dropped. The signalReconnect and fullReconnect debug scenarios relied on that late close to trigger the engine's recovery. They now call simulateSignalDrop(), which disposes the socket and raises the disconnect explicitly. The mock socket reports its close after dispose() like the real ones, so the race is reproducible in tests.
…ocket cleanUp() now ignores the close the disposed socket reports afterwards, so callers that relied on that late close to make the engine recover stopped working. The connection checker's reconnect check always reported that no reconnect was attempted, and the disconnectSignalOnResume debug actions never started the resume the server is waiting for. Both now call simulateSignalDrop(), like the signalReconnect and fullReconnect scenarios already do.
…hi/signal-socket-generation
There was a problem hiding this comment.
Note
Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.
🔍 Devin Review: 1 flag
Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
_socketGeneration becomes _socketId. The value is compared to decide whether a callback came from the current socket, which an id describes more directly than a generation. No behavior change.
…hi/signal-socket-generation # Conflicts: # test/mock/websocket_mock.dart
The close of a replaced socket can reach the client seconds after the reconnect that replaced it, a browser reports it only once the server drops TCP. On main it was taken for a loss of the new socket and every reconnect started the next one. Both shapes fail with the socket id check removed.
|
Merged main (through #1208) in, resolved the mock conflict, format passes now. Rewrote the description: the late close is seconds late in a browser, not milliseconds, which turns this into a reconnect every 2 s rather than a rare race. Added tests for the resume and full reconnect shapes and the e2e-flutter results. |
…hi/signal-socket-generation
…aches the old socket first A connect whose socket opened only after disconnect() or a newer connect() still installed itself as the current socket, so requests went to the wrong socket and the room could report connected after disconnect() returned. It now closes that socket and fails with SignalConnectSupersededException, which leaves the shared state and events alone and is not a reason to fail over to another region. cleanUp() detached the socket only after awaiting its close, so a connect() that ran while a dart:io close waited on the server lost its own socket, queue and ping timers when that cleanUp() finished. It now detaches first and awaits the close after. simulateSignalDrop() does nothing without a socket and reports the loss before the socket has finished closing. The connection checker comment matches what it does now. Tests: the mock holds old closes until the test releases them, so the late close tests no longer depend on timing, the checker test answers the resume and expects success, and there are new tests for late data from a replaced socket, the superseded connect, the cleanUp() race and a drop with no socket. Each fails without its change. Found in review (Codex).
There was a problem hiding this comment.
Devin Review found 3 new potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| // Attempt to connect. Callbacks carry the id of the socket they belong to | ||
| // so a late report from a replaced socket is ignored. | ||
| final id = socketId = ++_socketId; | ||
| final ws = await _wsConnector( |
There was a problem hiding this comment.
🔴 Stale resume rejects a newer join
If a resume finishes closing its old socket during a new join’s handshake, connect gives the resume the newer ID. The new join then rejects its own successful handshake as superseded.
Learn more
A server migration requesting RESUME reaches resumeConnection while the old signal socket is still open. cleanUp detaches that socket before awaiting its disposal, and the native WebSocket disposer awaits the close. The app can complete disconnect and call Room.connect while that detached socket is closing. If the close finishes while the new join is opening its socket, the old resume claims a later ID and the post-handshake check rejects the new join, regardless of whether the resume itself succeeds.
Example: A migration starts a resume and waits for socket A to close. The app leaves and starts joining another room on socket B. A finishes closing before B opens, so the stale resume claims the current ID. B opens successfully but the SDK disposes it and fails the join.
Recommended fix: Reserve ownership of the connection attempt before asynchronous preparation and teardown. Let connect detach the old socket without invalidating its own reservation, and recheck that reservation before invoking the connector as well as after the handshake. Add a gated test where the old close finishes while the new join is opening.
Was this helpful? React with 👍 or 👎 to provide feedback.
| // A superseded attempt reports nothing, the connection that replaced it | ||
| // is the one the room has. | ||
| if (error is SignalConnectSupersededException) rethrow; |
There was a problem hiding this comment.
🔴 Cancelled reconnect restarts the new room
When a full reconnect opens after the app leaves and rejoins, SignalConnectSupersededException still reaches reconnect recovery. attemptReconnect treats it as retryable and restarts the healthy session.
Learn more
A server-requested full reconnect opens a fresh signaling socket through restartConnection. The app can disconnect during that handshake and reuse the room; Engine.connect clears _isClosed and _connect replaces the URL and token. When the old handshake returns, SignalClient.connect discards it with a superseded exception. Without another region, restartConnection propagates that exception, and attemptReconnect sets fullReconnectOnNext and queues a retry. The engine is open again, so the retry tears down the new room's transports in restartConnection. With a regional fallback, restartConnection can start another handshake directly instead of propagating cancellation.
Example: Room A starts a full reconnect. The app leaves A and successfully joins B using the same Room. A's handshake returns late; the old retry starts a full reconnect of B even though B is healthy.
Recommended fix: Propagate supersession past regional failover and treat it as cancellation, not failure, in attemptReconnect. Scope the attempt's retry bookkeeping and finally cleanup to the session that started it. Test a full reconnect held at the handshake while the app disconnects and joins again, then verify that releasing the stale handshake opens no further sockets.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if (id != _socketId) { | ||
| // disconnect() or a newer connect() took over while this socket was | ||
| // opening. Installing it would take the place of the current one. | ||
| await ws.dispose(); | ||
| throw const SignalConnectSupersededException(); |
There was a problem hiding this comment.
🔴 Stale join disables Android call routing
When a cancelled join opens after a successful rejoin, SignalConnectSupersededException reaches Room.connect. Its failure cleanup stops the shared Android audio session, releasing the active call’s audio focus and routing.
Learn more
Room.connect starts native audio before it opens the signal socket. Room._cleanUp stops audio when the app disconnects, and the next join starts it again. When the first socket finally opens, the superseded exception passes through Engine._connect, but the original Room.connect still runs its failure cleanup. In SDK-managed Android mode, NativeAudioManagement.stop stops the shared session, and LKAudioSwitchManager.stop tears down focus and routing rather than releasing an attempt-specific resource.
Example: An Android app cancels a slow join and then joins another room. The new call starts with Bluetooth routing. The first handshake completes late and its error cleanup stops the audio session still in use by the new call.
Recommended fix: Track which room connection attempt owns the native audio setup and make failure cleanup conditional on that ownership. A superseded attempt must not stop audio after a newer attempt has taken ownership. Extend the delayed-handshake regression test to assert that the stale completion causes no extra native audio stop.
Was this helpful? React with 👍 or 👎 to provide feedback.
Stacked on #1208.
Problem
A real socket reports its close some time after the SDK disposes it (
websocket/io.dartonDone,websocket/web.dartonClose).SignalClientrouted that report through the same handler as a live server close. When it landed after the next connection was up, it set the signal state back todisconnectedand the engine started a reconnect against a healthy socket.How late the report comes decides how often this bites:
Any reconnect replaces the socket, so when the old close lands after the reconnect is done, the room reconnects again, and again, every 2 s or so for as long as the session lives. A
disconnect()followed byconnect()hits the same path.Fix
onData,onDisposeandonErrorcallbacks carry the id of the socket they were created for.connect()gives each new socket the next id andcleanUp()advances it too. A callback from another id is dropped with a log line. Same as the JS SDK ignoring a close from a transport that has since been replaced.signalReconnectandfullReconnectdebug scenarios, thedisconnectSignalOnResumeactions and the connection checker's reconnect check used to callsignalClient.cleanUp()and rely on the late close to make the engine recover. That close is now ignored by design, so they call a new internalsimulateSignalDrop()which disposes the socket and raises the disconnect explicitly. Same observable behaviour.dispose()with a configurable latency, like the real implementations.Tests
test/core/stale_socket_close_test.dart:e2e-flutter
Scenarios in livekit/e2e-flutter#39. Only the target's signal socket goes through a proxy that makes the old socket's close late: it delays the server dropping TCP, which is what a browser waits for, and the server's answer to the close frame, which is what dart:io waits for.
duplicateIdentityRun on macOS against a local livekit-server 1.13.11 and LiveKit Cloud, in Chrome against the local server (the two late close scenarios), and on an iPhone 17 Pro Max and an iPhone 16 Pro on iOS 27.2.1 against LiveKit Cloud. The other reconnect, signaling, lifecycle and RPC scenarios give the same results as on 2.14.0 in each setup.