Repository navigation
fix: disconnect() tears down locally after sending the Leave instead of waiting for the server echo - #1208
fix: disconnect() tears down locally after sending the Leave instead of waiting for the server echo#1208hiroshihorie wants to merge 6 commits into
Conversation
…echoing the leave engine.disconnect() sends the Leave and relies on the server's echo to run cleanup and emit EngineDisconnectedEvent. Media nodes drop queued leave messages when they close the signal sink, so the echo can be lost. The socket close was then ignored because _isClosed was set, and Room.disconnect() waited out its 10 s timeout for an event that never came, leaving the room half torn down. The engine now remembers a disconnect that is waiting for its echo and, on a socket close in that window, runs cleanup and emits the event itself. A close that follows the echo is still ignored, so there is no second event.
…of waiting for the echo The engine used to send the Leave and rely on the server's echo to run cleanup and emit EngineDisconnectedEvent, and Room.disconnect() waited up to 10 s for that event. Media nodes drop queued leave messages when they close the signal sink, so the echo can be lost, and every lost echo was a 10 s stall, a TimeoutException, and a room left half torn down. No other SDK waits on the server here. disconnect() now sends the Leave, runs cleanUp() and emits the event in one go, like JS, Swift, Android and Rust. A Leave that arrives after disconnect() started is the echo and is ignored. Room.disconnect() owns the teardown for the disconnect it started, so the engine event handler skips its own cleanup in that case and a connect() right after disconnect() returns cannot race it. Replaces the earlier socket close fallback on this branch.
|
e2e-flutter suite (livekit/e2e-flutter#7) against this branch, local livekit-server 1.13.6, macOS, three full runs on 2026-09-16:
ParticipantAttributesTest, the scenario whose agent hit the lost Leave echo on every run before, completed its teardown in about 0.5 s on all three runs with no warning. The 10 skips are camera scenarios, not runnable on this machine. |
|
@hiroshihorie any ETA on when this can be reviewed and merged? Some of our users are having this fatal error and this PR fixes it: |
| await _signalListener.cancelAll(); | ||
| _clearPendingReconnect(); | ||
| } | ||
| await cleanUp(); |
There was a problem hiding this comment.
🔴 Old socket close disrupts immediate reconnect
When connect() follows disconnect(), the old socket's delayed close can mark the new connection disconnected. SignalClient._onSocketDispose does not distinguish the old socket from the new one, triggering an unnecessary reconnect.
Learn more
The signaling client stores connection state for the current socket, but each socket retains the same _onSocketDispose callback. On web, LiveKitWebSocketWeb calls WebSocket.close() without waiting for the later onClose event. SignalClient.cleanUp therefore finishes before the old callback runs. A subsequent connect can install a new socket before SignalClient._onSocketDispose receives the old close and marks that new connection disconnected.
Example: Disconnect session A on web, then immediately connect session B on the same Room. If A's onClose fires after B connects, B is marked disconnected and the engine begins reconnecting despite B's live socket.
Recommended fix: Bind close callbacks to the socket or connection generation that created them. Ignore closes from sockets that are no longer current; add a web-style delayed-close regression test after reconnect.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Valid, this is the case #1211 covers. It is stacked on this PR and they merge together. SignalClient now tags each socket's callbacks with a generation that connect() and cleanUp() advance, so a close reported by a socket that was already replaced is ignored. Covered by a close reported by a replaced socket does not disturb the next connection in test/core/stale_socket_close_test.dart.
…onnect-on-socket-close
disconnect() let the engine handler emit RoomDisconnectedEvent as soon as the engine had closed, while the room's own teardown was still running. An app that connects again from that event started its new session in the middle of the old teardown, which then disposed the new engine state and left the room disconnected. The handler now leaves the disconnect it did not start alone, and disconnect() emits the event after _cleanUp() has finished.
…onnect-on-socket-close
|
Merged main in (clean). Full suite 612 passing, analyze and format clean. The e2e-flutter run promised in the description, on macOS against a local livekit-server 1.13.11, with #1211 on top: 13 of the 14 existing scenarios tagged reconnect or signaling pass, including "Disconnect during a full reconnect stays disconnected" and "Dispose during a full reconnect stays gone" (#1264, #1268). The 14th, "stale publisher answer after ice restart", is an expected failure unrelated to this. Of the new scenarios from livekit/e2e-flutter#39, 3 pass unexpectedly (fixed by #1211), 1 is the expected failure left for the #1219 follow-up, and the publisher-down one passes. @bdlukaa this is moving again. It fixes the |
…eport it when a step fails Two disconnect() calls that overlapped both passed the closed check, so each sent a Leave, ran the teardown and emitted RoomDisconnectedEvent. They now share the teardown in flight, released before the event so a rejoin from that event gets a teardown of its own on its next disconnect(). A teardown step that threw skipped the room's cleanup or the only RoomDisconnectedEvent, because the engine event handler stays quiet while disconnect() runs. The room's cleanup now runs after a failed engine teardown, the event is emitted either way, and the error still reaches the caller. The tests now check that the Leave is sent, count engine events in the echo case, and check that the room is torn down when the disconnected event arrives. Found in review (Codex).
Problem
engine.disconnect()sent the Leave and relied on the server's echo to run cleanup and emitEngineDisconnectedEvent, andRoom.disconnect()waited up to 10 s for that event. Media nodes drop queued leave messages when they close the signal sink (pkg/routing/signal.go,signalMessageSink.Close, comment in place), so the echo can be lost. Every lost echo was a 10 s stall, aTimeoutExceptionout ofdisconnect(), and a room left half torn down becauseRoom.disconnect()never reached its own cleanup.Found by the e2e-flutter suite (livekit/e2e-flutter#7): one agent per run against a local livekit-server 1.13.6,
SignalClient did disconnectwith noLeave receivedbefore it while the server loggedsendLeave: true.This design dates from the Dec 2023 engine rewrite (v1.5.5). #559 (v2.2.2, issue #553) made
Room.disconnect()wait for the event, which turned a silently missing event into a visible stall. No other SDK waits on the server here: JS, Swift, Android and Rust all send the Leave and tear down locally.Fix
Flutter now does the same.
engine.disconnect()sends the Leave, runscleanUp()and emitsEngineDisconnectedEventin one go. No dependency on the echo.disconnect()started is the echo, or a server Leave racing ours, and is ignored.Room.disconnect()awaits the engine teardown and then its own cleanup, no wait on the server and no timeout. The engine event handler skips its cleanup whileRoom.disconnect()is running, so teardown has one owner and aconnect()right afterdisconnect()returns cannot race a straggling handler. That is the case behind [bug] Actions stop working after second connect attempt to same room flutter livekit client #553, now covered by a test.The first commit on this branch was a narrower fallback that completed the disconnect on socket close while keeping the echo dependency. The second commit replaces it with the alignment above.
Tests
test/core/disconnect_event_test.dart, four added:disconnect()with oneclientInitiatedeventdisconnect()returns with engine and room torn down when the server sends nothing at alldisconnect()returnsThe last two fail on main.
e2e-flutter
"Disconnect while the server stops answering", a new scenario for livekit/e2e-flutter#39. Only the target's signal socket goes through a proxy, which stops forwarding anything from the server once
disconnect()starts, so neither the Leave echo nor the answer to the WebSocket close frame arrives. That is a lost echo, and also a user who leaves right after the network died.disconnect()must return within 3 s with oneclientInitiatedevent and the room torn down, the other participant must see the leave, and the sameRoommust join again.TimeoutExceptionafter 10.0 sTimeoutExceptionafter 10.0 sThe existing scenarios that exercise
disconnect()pass in the same setups, with #1211 on top: "Connect and disconnect in a loop on one room" (ten connects and disconnects on oneRoom), "Disconnect reasons", "Dispose during stream writes", "RPC recipient leaves mid-call", "Disconnect during a full reconnect stays disconnected" and "Dispose during a full reconnect stays gone" (#1264, #1268).The Cloud runs used a regional URL. Against a region pinned project, every connect on the project URL is refused and goes through region failover, and the connect loop then fails on its third connect on every SDK version, the issue #1204 addresses.