[Vortiago]

An automated test-quality audit flagged 43 tests changed in the last five commits: 23 as "fix" and 20 as "look". I checked each finding against the test and the code it covers. 10 are fixed here. The other 33 are left: 30 are false positives, and 3 are deliberate. None of the 3 involves timing.

The rule for this PR is that a flaky test is fixed when it is found. A bounded poll is not a reason to leave one. Every wait in the tests changed here is on a real event: a process's exit notice, a pipe or an Event, or a mark the page sets. None polls a condition against a clock.

Two small production changes (the second is described in section (e): UtteranceIndex takes a clock). The first: tapscribe/web/js/next/main.js now sets <html data-booted> when boot's awaits finish. By then every view module, template and warmed component has arrived, and the dashboard fetches no module after that point. The dead-cookie e2e test needed a real signal to wait on, and the page had none.

(a) Findings fixed

Test	Finding	What changed
tests/test_live_guard.py::test_the_server_dies_with_a_recorder_that_is_killed	flaky	The test polled ps against a 15 s deadline. It now opens the kernel's exit notice for the server and the guard while both are alive: a pidfd (os.pidfd_open plus select) on Linux, or a kqueue NOTE_EXIT on macOS. It then blocks on that notice. A zombie nobody reaps counts as exited, so the ps check is gone. The pid line is read once the pipe is readable. The stand-ins block in signal.pause() instead of sleeping. Cleanup signals only a process whose exit it has not seen.
tests/test_live_guard.py::test_a_normal_stop_ends_the_guard_with_the_server	flaky	Popen.wait(timeout=) polls internally, so the test now waits on the exit notice for each process. It also asserts that both ended by -SIGTERM, where before is not None passed for any exit.
tests/test_live_guard.py::test_the_guard_leaves_when_the_server_exits_on_its_own	flaky	The server used to exit after time.sleep(1). It now exits when the test closes its stdin. Both exits are waited on through the exit notice.
tests/test_login_links.py::test_an_unknown_token_is_refused	no positive assertion	The test only asserted refusals, which a store that refused everything would pass. It now also spends the link it minted and validates the session.
tests/test_auth_cookie.py::test_a_same_origin_write_with_a_null_origin_passes	conditional logic	The two refused cases ran in a for loop inside the passing test. They moved to a new parametrized test, test_an_opaque_null_origin_write_is_refused, with one id per case.
tests/e2e/test_pipeline_hallucinations.py::test_large_session_merge_preserves_all_speakers_in_order	conditional logic	Two if filters could drop data without failing. A segment from an unknown speaker was skipped. A plain_text line with no clock stamp left the ordering check, and with no stamped lines at all the check compared two empty lists. The test now asserts that every segment's speaker is known and that there is one stamped line for each of the 8 segments.
tests/e2e/test_dashboard_ui.py::test_a_dead_session_cookie_tells_the_operator_instead_of_going_quiet	conditional logic; flaky	The for/else polled the count of /web/js/ responses every 0.5 s until three readings matched. It now waits for html[data-booted], the new boot mark described above, so the loop and its else are gone. Its store swap also goes through monkeypatch, as in the next row.
tests/e2e/test_harness.py::test_recorder_server_retries_past_a_port_taken_on_ipv6	conditional logic (also timing: the harness under test polled)	RecorderServer.start() polled server.started every 20 ms against a deadline. It now wraps uvicorn's startup() and sets a threading.Event once startup has an outcome. A bind failure raises SystemExit out of that same call, and a thread that dies earlier sets the Event too. start() then makes one wait. The pytest.skip branches for hosts without IPv6 stay, since they are the honest form. The test ran and passed here; it was not skipped. Every e2e fixture that starts a server gets this change.
tapscribe/web/js/formatters.test.js "fmtClockZ keeps the ASCII spelling under a non-English viewer locale"	flaky	The test spawned a child node under LANG=da_DK. It now runs in process: Intl.DateTimeFormat's fallback locale is set to da-DK, formatters.js is imported afresh, and the original is restored with t.after. A guard assertion fails if the runtime has no da-DK data. A mutant without the pinned "en-US" fails with +01.00.
tests/e2e/test_dashboard_ui.py::test_a_signed_out_tab_is_never_challenged_and_recovers_whole	leaks state (also flagged: conditional logic, reaches into internals)	The test replaced the global app's login_links store with a plain assignment and never restored it. A new helper, _sign_every_session_out, now makes the swap through monkeypatch, and test_a_dead_session_cookie_tells_the_operator_instead_of_going_quiet uses it too. The other two flags are false positives: the for _ in range(2) loop always runs both of its assertions, and app.state.login_links is the store's documented seam (ADR-0023).

(b) Findings left

Test	Finding	Why left
tests/e2e/test_dashboard_ui.py::test_login_link_signs_the_browser_in_without_an_auth_dialog	flaky	False positive. Every wait is a Playwright wait with a timeout, and there are no sleeps.
tests/e2e/test_dashboard_ui.py::test_a_link_signed_in_dashboard_can_still_write	conditional logic	False positive. The try/except only re-raises a timeout as an AssertionError that names the refused requests, so no assertion is skipped.
tests/test_bringup_defaults.py::test_start_ps1_reads_each_variable_from_its_key	reaches into internals	False positive. _live_code is a helper in the test module, not a production internal.
tests/test_live_supervision.py::test_start_gives_the_server_a_process_group_not_a_session	does not run / marker; reaches into internals	False positive. The skipif is the POSIX gate (process_group exists only on POSIX). Patching _find_exe is the seam this whole file documents and uses.
tests/test_login_routes.py::test_secure_follows_tls	reaches into internals	False positive. It sets the public config.TLS_ENABLED flag through monkeypatch.
tests/test_login_routes.py::test_the_cookie_name_carries_the_port	reaches into internals	False positive. It sets the public config.PORT through monkeypatch.
tests/test_login_routes.py::test_a_spent_link_renders_a_page_and_never_a_bare_401	leaks state	False positive. The client fixture builds a new app for each test, so the replaced store is thrown away with that app.
tests/test_login_routes.py::test_the_access_log_keeps_the_login_line_but_not_the_token	reaches into internals	False positive. _RedactQuerySecrets is the unit under test. Its only public path is the lifespan attaching it to uvicorn.access, which would need a live Recorder. The repo tests private units like this directly.
tests/test_login_routes.py::test_other_paths_keep_their_query_strings	reaches into internals	False positive, for the same reason.
tests/test_login_routes.py::test_a_record_that_is_not_an_access_line_passes_through_untouched	reaches into internals	False positive, for the same reason.
tests/test_preflight.py::test_summarize_probe_agrees_with_resolve_local_backend	leaks state	False positive. The reset_available_backends fixture puts the auto-probe back on teardown.
tests/test_session_merge.py::test_clock_stamps_an_aware_utc_instant_with_its_offset	reaches into internals	False positive. _one_line is a test helper over the public render_transcript_text.
tests/test_session_merge.py::test_clock_treats_a_naive_stamp_as_utc	reaches into internals	False positive, for the same reason.
tapscribe/web/js/formatters.test.js "fmtClockZ returns ? for missing or unparseable input"	look: positive unstable	False positive. It asserts exact values for four inputs.
tests/test_auth_cookie.py::test_a_browser_background_request_is_never_challenged_even_without_a_cookie	look: positive borderline	False positive. It asserts the 401 and that WWW-Authenticate is absent, which is the rule under test.
tests/test_bundle_recorder_contract.py::test_the_quit_probe_reads_a_count_the_recorder_reports	look: asserts unstable	False positive. The route and key are read from ControlClient.cs. The test asserts that the route is served and that the key is an int. A shape check is the contract here, because the count varies and the C# side reads any int.
tests/test_install_matrix.py::test_install_matrix_wheel_index_matches_preflight	look: can_fail unstable	False positive. It asserts equality against preflight.LLAMA_CPP_WHEEL_INDEX, so a drift fails it.
tests/test_live_guard.py::test_no_guard_for_a_child_that_is_not_a_real_process	look: positive / asserts unstable	False positive. is None is the exact contract for a stand-in that is not a real process.
tests/test_login_links.py::test_the_cookie_is_not_the_token	look: positive borderline	False positive. Both assertions pin the rule. The positive half is in test_a_minted_link_spends_once_for_a_session.
tests/test_login_links.py::test_a_respend_inside_the_grace_window_reissues_the_same_session	look	False positive. The clock is injected, and the test asserts equality.
tests/test_login_links.py::test_a_respend_after_the_grace_window_is_refused	look	False positive. It asserts the refusal and that the issued session still validates.
tests/test_login_links.py::test_an_unspent_link_expires	look	False positive. The clock is injected, and the test asserts the refusal.
tests/test_login_links.py::test_an_unspent_link_expires_even_when_the_sweep_never_runs	look; reaches into internals	Deliberate. Stubbing _sweep is the point: the docstring explains that it holds spend's own TTL check to the rule.
tests/test_login_links.py::test_validate_refuses_a_cookie_this_store_never_issued	look	False positive. Its cookie is real, and so is the store that issued it.
tests/test_login_links.py::test_non_ascii_credentials_compare_without_crashing	look	False positive. It asserts that a real session exists first, so the compare path is reached.
tests/test_login_links.py::test_expired_and_used_up_links_are_not_retained	look; reaches into internals	Deliberate. Retention can only be seen through _links, because the public API cannot tell a forgotten link from an expired one.
tests/test_login_links.py::test_a_session_nobody_uses_expires	look	False positive. The clock is injected.
tests/test_login_links.py::test_a_session_in_use_outlives_the_idle_limit	look	False positive. The clock is injected, and the loop is fixed.
tests/test_login_links.py::test_a_session_in_use_still_ends_at_the_hard_limit	look	False positive. The clock is injected, and it asserts both the live and the dead side.
tests/test_login_links.py::test_an_expired_session_is_not_refreshed_back_to_life	look	False positive. The second refusal is the point of the test.
tests/test_login_links.py::test_expired_sessions_are_not_retained	look; reaches into internals	Deliberate. Retention can only be seen through _sessions, and the test also asserts that the live session validates.
tests/test_login_routes.py::test_minting_a_link_needs_the_password	look: positive borderline	False positive. The 401 is the access control under test. The positive mint is covered by _mint in the neighbouring tests.
tests/test_login_routes.py::test_the_redaction_does_not_depend_on_the_parameter_being_called_k	look; reaches into internals	False positive. _RedactQuerySecrets is the unit under test, as above.

(c) Polling waits found along the way

The audit did not flag these. They are the same flaky pattern, so this PR fixes them too.

Where	Was	Now
tests/e2e/harness.py::wait_until, with about 80 e2e callers	Re-read the predicate every 50 ms until a deadline.	Re-reads it only when watched state changes. tests/change_signal.py (new) provides CHANGES, a version counter that any thread may bump and any loop or thread may wait on. A waiter reads the version before it checks, so no change can slip between the check and the wait. build_tap_recorder wraps the public mutators of each Recorder it builds so they bump: streams, utterance index, tap settings, jobs, pipeline results, live transcripts, and the recording and session toggles. FakeWlkThread bumps when a relay connects, sends a frame, or goes away. timeout only bounds a hang.
Three test_dashboard_ui.py callers that read state the signal does not cover	Polled HTTP, the file system, or the DOM.	The session-meta save waits on the PUT's response. The WAV count waits on utterance_released, after which the WAV is closed and on disk. The .wavrow check uses page.wait_for_function.
tests/e2e/test_bridge_extension_e2e.py	6 wait_until waits on an unsignalled fake server, 7 asyncio.sleep(0.3/0.4) pauses before checking that frames flow, and an asyncio.sleep(0.15) before the drain test's mute.	FakeTapServer bumps CHANGES, and frames_flowing() waits for the first PCM frame. The drain test holds the redial's handshake and waits for it to arrive, which means onclose has fired and PCM is buffering. It then mutes inside expect_console_message("draining before close"), which proves the drain branch ran.
tests/test_tap_endpoint.py::_wait_until and its 4 helpers	Polled every 20 ms.	Uses CHANGES.wait_until, the blocking twin. The resume-window test no longer sleeps 0.25 s: it moves the closed record's last_close back past the real window. The rotation test's 10 ms glob loop now waits on the signal.
tests/test_live_supervision.py::_wait_until, 9 callers	Polled every 5 ms.	The fake stdout counts how often the pump comes back for another line, so wait_processed() blocks until every pushed line is handled. The two start() tests join the pump thread. The TailLog stress test made as many reads as 0.3 s allowed; it now makes a fixed 2,000 reads once the log is full.
tests/test_tap_live_gate_real_audio.py::_wait_until_relay_settles	Took three equal byte counts, 0.2 s apart, as "done".	Counts frames the /tap loop has handled (TapFanOut.write_frame) and bytes relayed (WlKRelay.send). It waits, with the tap still open, until every frame is handled and the fake WlK holds every relayed byte. The wait has to be inside the block, because leaving it makes TestClient cancel the app and drop queued frames. Waiting outside forwarded 0 to 17 KB; waiting inside forwards 85,120 bytes every run.

(d) The remaining fixed sleeps

Where	Was	Now
test_dashboard_ui.py: 12 waits of the form "let a poll pass"	wait_for_timeout(250…10000) / asyncio.sleep(1.0…1.2)	_cross_polls(page, n) waits for n whole /api/state ticks that started after the call. The poll loop runs one tick at a time, so the next poll starting proves the previous tick, render included, has finished. It counts the page's own fetches (_COUNT_STATE_304S_JS) and fails loudly if the counter is missing. It covers the focus sweep, the renderList focus hold, the selection hold, the virtualized-row poll, the player stage walk, and idle churn (six counted ticks instead of ~10 s). The rest wait on their own events. Job ticks wait for the job bar to show each tick. Paced-retry nudges wait for the label to render, then on the route's retried event. The log dialog uses _COUNT_LIVE_LOG_HANDLED_JS, which counts bodies once renderLogInto has run. The people rename waits for the PUT response. The stage walk waits for the player's currentTime to advance while away.
test_bridge_meeting_e2e.py: 6	A 0.3 s /health poll, 2–2.5 s after tapping, 1 s after each mute, and a 1 s pipeline poll	Startup waits for uvicorn's "Uvicorn running on" line, read through a pipe that still writes recorder.log. Streaming waits until the bridge's published status shows frames sent on the tapped channels. Muting waits until every channel is muted, drained and closed. The pipeline waits for the popup card to show the summary or a failure, then reads the endpoint once.
test_pipeline_resilience.py (5) and test_pipeline_tls.py (3)	5 ms per frame, plus a 20 ms started poll	None of these were real-time audio pacing: real pacing is 20 ms per frame. The abort helper and the WlK-crash test now wait until the Recorder has handled every frame they sent. The new tap_frames fixture counts TapFanOut.write_frame and signals CHANGES. The clean-close and refused streams needed no wait, because one WebSocket delivers in order. The TLS server start uses harness.signal_startup.
Found along the way: async predicates to wait_for_function (dashboard 2, bridge extension 1, bridge meeting 1)	Playwright takes the returned Promise as truthy, so these returned at once and checked nothing. I confirmed it in isolation: wait_for_function("async () => false") returns immediately.	They are now one read after the event, a wait on the PUT response, or a sync predicate over _MIRROR_STORAGE_JS. That script mirrors chrome.storage into the popup and keeps it fresh from onChanged.

Kept on purpose:

• The frame_interval_s pacing in harness.stream_wav_via_tap simulates real-time audio.
• test_next_perf_soak.py is opt-in and measures over time by design.
• The 0.5 s retry backoff in test_pipeline_e2e.py's Parakeet weight prefetch is the only sleep left that waits for something. It retries after an OSError from a concurrent writer of the Hugging Face cache, inside a lock that writer does not take. No event or signal reports when that writer finishes. The sleep runs only on that error path, in the opt-in real-audio Parakeet test.

(e) Cleanup round: the audit of this PR's own tests

A fresh audit of the 31 tests this PR touches, taken before 8fc85f9, found 23 "fix" and 5 "look" findings. I checked each against the branch head. 14 are fixed in this round, 1 was already fixed by 8fc85f9, and 13 are false positives.

One production change: UtteranceIndex(clock=...) in tapscribe/recorder.py. It defaults to datetime.now(UTC) and is exposed as a public clock attribute. It stamps last_close and decides the resume window, so a test can step past the window. Behaviour is unchanged.

Why the pump is tested through start(): the log pump (_pump_logs) is private, so I drove it through the public start() rather than making it a public seam. The behaviour under test is "what start() does with the child's output", and start() is already public. The fake child reaches start() through public seams only: a whisperlivekit-server on PATH, which is how start() finds the real one, and subprocess.Popen patched for the call. The assertions read only the channel's public info and log.

Fixed

Test	Finding	What changed
test_live_supervision.py::test_pump_logs_promotes_starting_to_running_on_uvicorn_banner	reaches into internals	Through start() via the start_with fixture. It no longer calls _pump_logs, sets chan._proc, or patches _find_exe.
…::test_pump_logs_promotes_on_application_startup_complete_line_too	reaches into internals	Same.
…::test_pump_logs_overwrites_seeded_device_with_child_accelerator_report	leaks state; reaches into internals	Same. The seeded device is the one start() sets, not a hand-written info["device"].
…::test_pump_logs_sets_error_state_with_last_error_tail_on_nonzero_exit	reaches into internals	Same.
…::test_pump_logs_skips_info_update_when_proc_was_replaced	reaches into internals	Driven as it happens: the old child has exited with its tail unread, and a second start() spawns the new one before the old pump drains. Assertions are on info and log only.
…::test_start_wires_a_background_pump_that_promotes_state_to_running	reaches into internals	start_with records the threads start() creates. The test no longer filters threads by chan._pump_logs or patches _find_exe.
…::test_start_gives_the_server_a_process_group_not_a_session	reaches into internals (also flagged: marker)	Same. The skipif stays: process_group is POSIX-only.
test_tap_endpoint.py::test_tap_resume_past_window_starts_fresh_wav	reaches into internals	Steps UtteranceIndex.clock past the window instead of mutating the record that snapshot() handed back. Live is turned off through PUT /api/tap-settings, not live._proc = None.
test_tap_endpoint.py::…::test_open_tap_keeps_its_session_across_rotation	reaches into internals	The _utc_session_id patch is gone: the rotation happens only once the WAV is on disk, and rotate_session never re-mints an id whose folder exists. Live is turned off through the public route. The same file's client / auth_client fixtures now restore the global app's override and app.state.recorder through monkeypatch.
test_dashboard_ui.py::test_transcript_transcribe_saves_languages_first_wysiwyg	leaks state	The click also starts a transcribe job. The test now waits for it to release, so it can't run on into the next test's teardown.
test_dashboard_ui.py::test_dashboard_live_channel_start_stop	leaks state; reaches into internals	Fakes the channel only through start, stop and running() (monkeypatched) and info (monkeypatch.setitem). rec.live._proc is no longer touched.
test_dashboard_ui.py::test_dashboard_renders_real_end_of_meeting_pipeline_summary	reaches into internals (also flagged: conditional, leaks state)	Sets the default summarizer through PUT /api/summarize/config instead of writing summarizer.json into the config directory. The other two flags are false positives: the assert inside the wait predicate fails the test with the pipeline's reason, and the config directory is per test.
test_bridge_extension_e2e.py::test_closed_spatialchat_tab_flips_popup_to_no_active_tab	flaky (also flagged: conditional)	The popup opens with Playwright's clock installed. After the tab closes, the test fast-forwards the clock past STALE_AFTER_MS, read from taps-view.js, instead of waiting it out in real time against a 12 s ceiling. The conditional flag is the DOM predicate's own checks.
test_tap_live_gate_real_audio.py::test_tapscribe_gate_forwards_real_speech_and_drops_silence_end_to_end (look)	leaks state; conditional	gate_client restores the global app's override and app.state.recorder through monkeypatch; before, a failing test left them set. The faster-whisper check is a skipif marker, not an if … pytest.skip() in the body. The internals flag is a false positive: the frame ledger wraps public methods, and monkeypatch restores them.

Already fixed by 8fc85f9

Test	Finding	Note
test_bridge_extension_e2e.py::test_room_disconnect_cleans_up_audio_and_presence_only_taps	flaky (also flagged: conditional, internals)	Its popup wait used an async predicate, which never waited. It now reads the storage mirror, a fix in 8fc85f9, after this audit's snapshot. window.__tsTest is the mock page's own test driver, not production.

False positives

Test	Finding	Why
test_bridge_extension_e2e.py::test_pill_transitions_idle_to_ok_to_warn_on_tap_drop	conditional; flaky; leaks state	Every wait is an event wait (pill DOM, frames_flowing). refuse_connections is set on a per-test fixture server. The conditions are inside the DOM predicate.
…::test_pcm_frames_carry_resolved_display_name_after_sidebar_rerender	flaky; reaches into internals	Frames are waited for on CHANGES. window.__tsTest is the mock page's test driver.
…::test_popup_token_rotation_triggers_reconnect_with_new_subprotocol	flaky	The redial is waited for on CHANGES. No sleeps are left.
…::test_mute_drain_reconnect_continues_same_utterance	flaky	It uses the handshake hold plus expect_console_message. No sleeps are left.
…::test_window_room_cleared_without_disconnect_closes_taps	flaky	The close is waited for on CHANGES.
test_dashboard_ui.py::test_next_failed_files_fetch_still_reconciles_after_a_stage_switch	conditional; flaky; leaks state	The nudge loop's break is followed by an unconditional wait on retried. Every wait is on an event. The route and its state live on a per-test page.
test_dashboard_ui.py::test_a_dead_session_cookie_tells_the_operator_instead_of_going_quiet	conditional; flaky; reaches into internals	The try/except re-raises as an AssertionError that names recent traffic. Boot is waited for on data-booted. app.state.login_links is the store's documented seam (the login_links docstring, ADR-0023), swapped through monkeypatch.
test_dashboard_ui.py::test_a_signed_out_tab_is_never_challenged_and_recovers_whole	conditional; reaches into internals	The comprehension filter is the assertion's subject, and the two-poll loop always runs both checks. The store seam is the same as above.
test_live_guard.py::test_the_server_dies_with_a_recorder_that_is_killed	conditional; reaches into internals	The only branches are cleanup in finally; no assertion sits behind them. _ExitWatch and _read_pids are helpers inside the test module.
test_pipeline_hallucinations.py::test_large_session_merge_preserves_all_speakers_in_order (look)	not judged	The audit timed out on it. Its silent filters were removed in round 1.
test_auth_cookie.py::test_an_opaque_null_origin_write_is_refused (look)	positive borderline	The 403 is the rule under test.
test_live_supervision.py::test_log_iteration_is_safe_while_a_thread_appends (look)	can_fail; flaky	A fixed 2,000 reads, started once the writer has filled the log. No clock is involved.
test_login_links.py::test_an_unknown_token_is_refused (look)	positive unstable	It asserts that the real link spends and validates.

Verification

• This round: ruff is clean, and python -m pytest tests -q gives 3341 passed, 28 skipped, 0 failed.
• The changed unit files (test_live_supervision, test_tap_endpoint, test_recorder_subcomponents, test_tap_live_gate_real_audio, test_live_guard): 10 of 10 runs passed.
• The changed e2e files (test_dashboard_ui, plus test_bridge_extension_e2e headed under Xvfb): 10 of 10 runs, 97 passed each. The runs went five at a time in parallel.
• ruff check and ruff format --check over tapscribe tools tests benchmarks bridges/local-test-bridge: clean. The frontend typecheck and the web checks were clean when main.js last changed.
• python -m pytest tests -q: 3341 passed, 28 skipped, 0 failed. This ran with Playwright Chromium and the whisper-cpu extra installed. The headed bridge tests skip there because the run has no display; they ran under Xvfb in the stability runs below.
• CI on the previous push was green on every job, including tests on macOS (the kqueue NOTE_EXIT path in test_live_guard.py) and bridge E2E (extension + meeting).
• Stability runs:
  • This round, the six changed e2e files together (dashboard UI, resilience, TLS, harness, bridge extension, and bridge meeting with real whisper, all headed under Xvfb): 10 of 10 runs, 104 passed each. The runs went five at a time in parallel.
  • The only failure was the same one in every run: test_bridge_meeting_e2e.py::test_multi_person_multi_language_meeting_produces_a_summary, with "English content missing from transcript". It fails the same way, at the same assertion, with the old sleep-based code too, and it passes in CI's bridge E2E job. It comes from the local whisper model, not from timing.
  • Previous round, every e2e file that uses wait_until: 10 of 10 runs, 120 tests each. test_tap_endpoint.py: 10 of 10. test_live_supervision.py: 15 of 15. The real-audio gate test: 10 of 10.
  • Earlier rounds: test_live_guard.py 45×; test_harness.py 15×; the two signed-out dashboard tests 12×, plus 6× with every CPU core pinned by busy loops.

🤖 Generated with Claude Code