Summary
On a GitHub-hosted 2-vCPU Linux runner, a web client that opens a new
Firestore channel and registers its listen targets intermittently receives
no listen results for ~5–6 s, then receives them all at once. During the
wait the emulator JVM and the whole machine are idle (≈ 98–99 % CPU idle,
no GC pause over 28 ms, no JVM safepoint for 4.6 s), and the browser has a
backchannel request open to the emulator. It never reproduces on a fast
developer workstation (Windows, same versions). Roughly 2 in 10 full e2e
runs hit it on firebase-tools 15.31.0; roughly 4 in 10 on 15.30.2 (same
emulator jar).
Versions
| Component |
Version |
| firebase-tools |
15.31.0 (also seen on 15.30.2) |
| Firestore emulator |
1.22.0 (the jar both firebase-tools versions ship) |
| Firebase web SDK |
11.10.0 (firebase npm) |
| Transport |
experimentalForceLongPolling: true, useFetchStreams: false (also seen with long polling NOT forced — streaming) |
| Browser |
Chromium (Playwright 1.61.1, chromium-1228) |
| JDK |
Temurin 21 |
| Runner |
GitHub ubuntu-latest (private repository → 2 vCPU) |
What the application does
A React SPA. Each page subscribes 5–15 onSnapshot listeners (simple
where tenantId == X queries, a few with orderBy + limit 500, one
orderBy on a field to collect corrections). Every page load opens a new
channel session; client-side navigation adds and removes targets on the same
session. Collections are small (largest: 3 929 telemetry docs; every query
shape a page issues answers in ≤ 0.4 s against the same emulator when timed
directly over REST).
What happens
- The page loads; the SDK opens a channel (
POST .../Listen/channel?…RID=…)
and posts its targets.
- Intermittently, the backchannel
GET .../Listen/channel?…TYPE=xmlhttp
is held open by the emulator for 5 488 ms (one run) / 5 902 ms
(another) with no data, although targets are registered.
- Then every pending listener resolves at once.
Because the SDK raises snapshot events only once the server reports the batch
consistent, every listener registered in that batch waits — a 1-document
tanks query stalls alongside the rest.
Evidence (all from failing CI runs; artifacts retained 14 days)
- Chrome net log of the failing browser (run 35981785621, attempt 4):
975 backchannel requests in the whole e2e run; exactly one held longer
than 3 s (5 488 ms) — the stall. In that instance the emulator had closed
the previous backchannel mid-response (net_error -100 ERR_CONNECTION_CLOSED)
1.5 s earlier; six other such closes in the run were harmless.
- The same shape without long polling (streaming; run 35985336810,
attempt 6): the stream held open 5 902 ms with no data after the page posted
its targets, no connection close anywhere near it.
- No browser-side queueing: zero
SOCKET_POOL_STALLED_* events in the net
log — it is not Chrome's per-host connection limit.
- Emulator connection log (
firestore-debug.log): the browser opens no new
connection during the stall window (e.g. 09:12:17 → 09:12:25, 09:29:03 →
09:29:08), after ~13–24 in the two seconds before — i.e. the client has its
requests outstanding and is waiting.
- Not GC, not CPU (run 35990275614, attempt 4):
-Xlog:gc*,safepoint for
the emulator JVM — longest pause in the whole run 28 ms; vmstat -t 1 — the
runner was 98–99 % idle from 11:27:09 to 11:27:12, mid-stall; the JVM
logged no safepoint for 4.6 s.
- Named pending reads (the application's stall gate): e.g.
"SILENT STALL on /claim/2026-06 … Reads still waiting: claimPeriods,
deliveries corrections, deliveries [period], dispensingEvents corrections,
… workSessions [period]" (13 reads, the first page load of the suite,
run 36019271182); and after a client-side navigation, only the two reads
added in that navigation's batch (serviceRecords corrections,
serviceRecords, run 36019327566).
Artifacts (per failing attempt): Playwright trace.zip, error context,
chrome-netlog-<pid>.json, firestore-debug.log, emulator-gc-<pid>.log,
vmstat.log. The two most recent failures, on firebase-tools 15.31.0:
The repository is private, so these links will not open for you. They are
kept until 2026-10-08, and I can attach any of the files here on request.
The data in them is synthetic seed data.
Ruled out
The dev server (reproduces on a production bundle served by vite preview);
a slow query (all ≤ 0.4 s); the long-polling transport specifically (also
streaming); Chrome's connection limit; JVM GC; CPU starvation.
Expected
Listen results for newly registered targets are delivered as soon as the
query has run (≤ 0.4 s here), not after an idle ~5 s.
Reproduction (as far as we have it)
Not yet reduced to a minimal case. It needs the slower runner: on a fast
workstation 25 consecutive fresh page loads resolved all listeners together
in a median of 482 ms (max 1 568 ms), never near 5 s. The full suite that
reproduces it is private; a reduced repro (one page, ~13 listeners,
reload in a loop on a 2-vCPU Linux VM) is the next thing I will build, and
I will add it here.
Mitigations in the application meanwhile
None that change the emulator. The e2e gate has no retries and no
inflated timeouts by policy; instead a stall gate fails any page still
"Loading…" after 4 s (below the 5 s assertion timeout) with zero page errors,
naming every listener still waiting — so each occurrence is identified, not
mistaken for an application defect.
Summary
On a GitHub-hosted 2-vCPU Linux runner, a web client that opens a new
Firestore channel and registers its listen targets intermittently receives
no listen results for ~5–6 s, then receives them all at once. During the
wait the emulator JVM and the whole machine are idle (≈ 98–99 % CPU idle,
no GC pause over 28 ms, no JVM safepoint for 4.6 s), and the browser has a
backchannel request open to the emulator. It never reproduces on a fast
developer workstation (Windows, same versions). Roughly 2 in 10 full e2e
runs hit it on firebase-tools 15.31.0; roughly 4 in 10 on 15.30.2 (same
emulator jar).
Versions
firebasenpm)experimentalForceLongPolling: true,useFetchStreams: false(also seen with long polling NOT forced — streaming)ubuntu-latest(private repository → 2 vCPU)What the application does
A React SPA. Each page subscribes 5–15
onSnapshotlisteners (simplewhere tenantId == Xqueries, a few withorderBy+limit 500, oneorderByon a field to collect corrections). Every page load opens a newchannel session; client-side navigation adds and removes targets on the same
session. Collections are small (largest: 3 929 telemetry docs; every query
shape a page issues answers in ≤ 0.4 s against the same emulator when timed
directly over REST).
What happens
POST .../Listen/channel?…RID=…)and posts its targets.
GET .../Listen/channel?…TYPE=xmlhttpis held open by the emulator for 5 488 ms (one run) / 5 902 ms
(another) with no data, although targets are registered.
Because the SDK raises snapshot events only once the server reports the batch
consistent, every listener registered in that batch waits — a 1-document
tanksquery stalls alongside the rest.Evidence (all from failing CI runs; artifacts retained 14 days)
975 backchannel requests in the whole e2e run; exactly one held longer
than 3 s (5 488 ms) — the stall. In that instance the emulator had closed
the previous backchannel mid-response (
net_error -100 ERR_CONNECTION_CLOSED)1.5 s earlier; six other such closes in the run were harmless.
attempt 6): the stream held open 5 902 ms with no data after the page posted
its targets, no connection close anywhere near it.
SOCKET_POOL_STALLED_*events in the netlog — it is not Chrome's per-host connection limit.
firestore-debug.log): the browser opens no newconnection during the stall window (e.g. 09:12:17 → 09:12:25, 09:29:03 →
09:29:08), after ~13–24 in the two seconds before — i.e. the client has its
requests outstanding and is waiting.
-Xlog:gc*,safepointforthe emulator JVM — longest pause in the whole run 28 ms;
vmstat -t 1— therunner was 98–99 % idle from 11:27:09 to 11:27:12, mid-stall; the JVM
logged no safepoint for 4.6 s.
"SILENT STALL on /claim/2026-06 … Reads still waiting: claimPeriods,
deliveries corrections, deliveries [period], dispensingEvents corrections,
… workSessions [period]" (13 reads, the first page load of the suite,
run 36019271182); and after a client-side navigation, only the two reads
added in that navigation's batch (serviceRecords corrections,
serviceRecords, run 36019327566).
Artifacts (per failing attempt): Playwright
trace.zip, error context,chrome-netlog-<pid>.json,firestore-debug.log,emulator-gc-<pid>.log,vmstat.log. The two most recent failures, on firebase-tools 15.31.0:The repository is private, so these links will not open for you. They are
kept until 2026-10-08, and I can attach any of the files here on request.
The data in them is synthetic seed data.
Ruled out
The dev server (reproduces on a production bundle served by
vite preview);a slow query (all ≤ 0.4 s); the long-polling transport specifically (also
streaming); Chrome's connection limit; JVM GC; CPU starvation.
Expected
Listen results for newly registered targets are delivered as soon as the
query has run (≤ 0.4 s here), not after an idle ~5 s.
Reproduction (as far as we have it)
Not yet reduced to a minimal case. It needs the slower runner: on a fast
workstation 25 consecutive fresh page loads resolved all listeners together
in a median of 482 ms (max 1 568 ms), never near 5 s. The full suite that
reproduces it is private; a reduced repro (one page, ~13 listeners,
reload in a loop on a 2-vCPU Linux VM) is the next thing I will build, and
I will add it here.
Mitigations in the application meanwhile
None that change the emulator. The e2e gate has no retries and no
inflated timeouts by policy; instead a stall gate fails any page still
"Loading…" after 4 s (below the 5 s assertion timeout) with zero page errors,
naming every listener still waiting — so each occurrence is identified, not
mistaken for an application defect.