Skip to content

fix(dsh-plugin): reconnect the observation stream after a fatal failure - #315

Merged
iuyo5678 merged 3 commits into
Tencent:mainfrom
djtzemx:fix/dsh-plugin-observation-stream-reconnect
Sep 26, 2026
Merged

iuyo5678 merged 3 commits into
Tencent:mainfrom
djtzemx:fix/dsh-plugin-observation-stream-reconnect

Conversation

@djtzemx

@djtzemx djtzemx commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

The observation store attached only an onmessage handler to the plugin's SSE stream and start() returns early while the store is already marked started, so a fatal EventSource failure left the view empty until the page was reloaded.

An EventSource never retries a non-200 response, which is exactly what happens when the page loads before the observation route is registered, or right after a plugin reload: the console reports 404 on /bsk-observation/events while /bsk-observation/state and a direct SSE request both answer normally, and the sidebar sits at "no session" with no way to recover except a page reload.

  • attach an onerror handler and rebuild the stream with a bounded backoff when readyState is CLOSED; transient drops keep readyState 0/1 and are still left to the EventSource's own retry
  • reset the backoff on any healthy frame, and cancel a pending reconnect in stop()
  • republish the snapshot whenever the stream is (re)created so subscribed reflects reality
  • surface subscribed in the overlay and sidebar empty state: a dead feed now reads "connecting..." / "connection lost - retrying" instead of "no session"
  • keep the floating card mounted while reconnecting (it used to return null as soon as there were no sessions, showing nothing at all)

Follow-up (d102adc)

The reconnect logic above is kept as is. Review found that subscribed stays true for the whole outage, because the failed EventSource is still held until the rebuilt one replaces it. The "connecting…" states therefore never showed during a real failure, while the floating card, gated on subscribed, was committed for one frame on every page load with no sessions. This commit:

  • adds reconnecting to the snapshot: set on any stream error (including transient drops the browser retries itself) and cleared by the next well-formed frame on the current stream; subscribed keeps its original meaning
  • shows "reconnecting…" in the overlay and sidebar header while it is set, instead of the last session status
  • keeps the floating card hidden without sessions, as on main; the sidebar tab still reports the outage
  • cancels a pending reconnect whenever the stream is rebuilt, so a thumbnail switch during the backoff is not torn down by the stale timer
  • removes the extra publish per stream rebuild that only served subscribed

Tests cover the outage state across fatal and transient drops, the thumbnail switch during a pending reconnect, the header text while reconnecting, and that the card is never committed without sessions. Checked on the branch and merged onto current main: typecheck, pnpm test (376 passed after the merge), biome, and stylelint are all clean.

Follow-up (8f485ab)

The expanded header now says "reconnecting…", but the collapsed capsule and both status dots still treated the last session as live. Collapse is the form users leave on the page.

  • reconnecting is its own chrome state, so the header and capsule dots leave the active color
  • the capsule keeps the session count and replaces the action timer with "reconnecting…"

A hard ceiling on retries (permanent 403 / gone route) is left as a later change. Delay already caps at 30s; each store then costs about two local requests a minute. A "give up and retry now" surface would add new store and UI state without changing the recovery this PR is for.

The observation store attached only an onmessage handler to the plugin's SSE
stream and start() returns early while the store is already marked started, so
a fatal EventSource failure left the view empty until the page was reloaded.

An EventSource never retries a non-200 response, which is exactly what happens
when the page loads before the observation route is registered, or right after
a plugin reload: the console reports 404 on /bsk-observation/events while
/bsk-observation/state and a direct SSE request both answer normally, and the
sidebar sits at "no session" with no way to recover except a page reload.

- attach an onerror handler and rebuild the stream with a bounded backoff when
  readyState is CLOSED; transient drops keep readyState 0/1 and are still left
  to the EventSource's own retry
- reset the backoff on any healthy frame, and cancel a pending reconnect in stop()
- republish the snapshot whenever the stream is (re)created so `subscribed`
  reflects reality
- surface `subscribed` in the overlay and sidebar empty state: a dead feed now
  reads "connecting..." / "connection lost - retrying" instead of "no session"
- keep the floating card mounted while reconnecting (it used to return null as
  soon as there were no sessions, showing nothing at all)
@iuyo5678

Copy link
Copy Markdown
Collaborator

Thanks — the failure mode is real, and reconnecting only when EventSource.readyState === CLOSED is the right fix. A non-200 response (the 404 while /bsk-observation/events is not registered yet, or right after a plugin reload) is terminal for EventSource, and start() returns early once the store is marked started, so the view stays empty until a page reload. The retry itself can stay as written: rebuild only on readyState === 2, leave 0/1 to the browser, reset the backoff on a real frame, and cancel the timer in stop().

Two things need to change before this is ready.

subscribed means “an EventSource object is currently held” (this.events !== undefined). After a fatal close that object is still held, including through the backoff, so subscribed stays true. The new “connecting…” / “connection lost — retrying” copy, and the change that keeps the floating card mounted while reconnecting, never run for this outage. They only appear on the first paint, before acquire() runs in the effect, which flashes the failure copy on a healthy load. Please track connection health separately, and drive the empty state and the floating card from that flag.

connectEvents() does not cancel a pending reconnect timer. A thumbnail viewer change calls connectEvents() directly; the leftover timer then closes that healthy stream and opens another. Please clear the timer at the start of
connectEvents(), not only in stop().

Please also cover both cases in tests: a recreated stream that does not immediately emit a snapshot, and
watchThumbnails() while a reconnect timer is still pending.

`subscribed` is derived from whether an EventSource object exists, so it
stays true for the whole outage: the failed stream is still held until the
rebuilt one replaces it. The "connecting…" states therefore never showed
during a real failure, while the floating card, now gated on `subscribed`,
was committed for one frame with "connection lost — retrying" on every
page load that had no sessions.

- add `reconnecting` to the snapshot: set on any stream error, including
  transient drops the browser retries itself, and cleared by the next
  well-formed frame on the current stream
- show "reconnecting…" in the overlay and sidebar header while it is set,
  in place of the last session status that could no longer be trusted
- keep the floating card hidden without sessions, as before this change
- cancel a pending reconnect whenever the stream is rebuilt, so a
  thumbnail switch during the backoff is not torn down by the stale timer
- drop the extra publish per stream rebuild that only served `subscribed`
The expanded header already replaced a stale "s1 · clicking" line with
"reconnecting…", but the collapsed capsule and both status dots still
read the last session as live. Collapse is the form users leave on the
page, so an outage there looked like an in-progress action.

- treat reconnecting as its own chrome state so the header and capsule
  dots leave the active color
- keep the capsule's session count and swap the action timer for
  "reconnecting…" until the feed delivers a frame again
@iuyo5678
iuyo5678 merged commit 8cbcc49 into Tencent:main Sep 26, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants