Repository navigation
[DO NOT MERGE] Test runner: hang watchdog that dumps state + all-thread stacks on zero progress - #1840
Closed
pa-devbot wants to merge 3 commits into
Closed
[DO NOT MERGE] Test runner: hang watchdog that dumps state + all-thread stacks on zero progress#1840pa-devbot wants to merge 3 commits into
pa-devbot wants to merge 3 commits into
Conversation
If no test COMPLETES for 60 s (PA_HANG_WATCHDOG overrides in seconds; 0 disables), the command-line test runner now dumps its own state (pending/running/completed counts, per-running-test elapsed time, thread pool queue/busy counts, file logger queue depth) and a stack trace of every thread in the process straight to the stderr file descriptor, then exits with code 124. This turns a hung CI runner - which would otherwise sit until the 8 h job timeout - into a failed job with a full diagnostic dump. - HangWatchdog: dedicated watchdog thread (not on the test pools, so it fires even when every test thread is blocked). POSIX dumps capture each thread's backtrace() by sending it SIGUSR2 (Linux signals every /proc tid; macOS signals a registry of program-created threads); Windows suspends each thread and walks it with StackWalk64 + SymFromAddr. - ParallelUnitTestRunner: diagnostics_state() snapshot and note_test_completed() per finished test. - ThreadPool: queued_tasks()/busy_threads() accessors; FileLogger: pending_count()/max_queue_size(). CI note (separate change, needs workflow-scope push): to make the dump readable in CI logs, keep the Windows PDBs, add -g on ubuntu-clang and -rdynamic on Linux/macOS, and upload the Linux/macOS builds for offline addr2line. Without those, frames resolve to module+offset only. [DEBUG: do not merge] -- part of the CI-hang investigation; see qt-socket-close-send-race-only for the suspected root-cause fix.
…gistry THREADLIST32/Thread32First are not available in the CI SDK/MSVC configuration (and SymFunction is not in the public dbghelp header, and STACKFRAME64 has no Machine field on x64), so enumerate the dump's threads from the program's own registry instead (register_thread() now records the thread id on Windows, as it does on macOS): the runner, pool, event-thread and logger threads - i.e., the ones that matter for a hang - are all program-created and registered. Library-created threads are not captured, noted in the dump as on macOS. The StackWalk64 callback is now a local stub that just continues.
The CI SDK's dbghelp.h has the 9-parameter StackWalk64 (no frame-walk callback, stack frame before context) and lacks the PSTACKFRAME64 / PSYMBOL_INFO pointer typedefs, so drop the callback entirely and stop using the pointer types (SYMBOL_INFO / STACKFRAME64 are used by value).
This was referenced Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Diagnostics for the intermittent CI test-runner hang. If no test completes for 60 s (PA_HANG_WATCHDOG env var overrides, in seconds; 0 disables), the command-line test runner dumps, straight to the stderr file descriptor:
then terminates with exit code 124 (the conventional timeout code). A hung CI runner becomes a failed job with a full diagnostic dump ~60 s after progress stops, instead of sitting until the 8 h timeout.
CI note (pushed separately, needs workflow scope)
CI currently strips the debug info the dump needs to symbolize (no Windows PDBs, no -g on ubuntu-clang, no -rdynamic on Linux/macOS). A patch with those CMake flag changes + build uploads exists (watchdog-ci-flags.patch); it needs a token with the workflow scope to push. Without it, frames resolve to module+offset only - the state dump is unaffected.
Expectation on this branch's CI
This branch does NOT contain the suspected root-cause fix (#1839), so on a loaded CI machine the socket send/close race may still fire; if it does, the watchdog should stop the run ~60 s after progress stops and the dump will show the parked threads. That would both prove the watchdog works end-to-end in CI and re-confirm the root cause.
[DO NOT MERGE] - debug/investigation PR; the mergeable content is the watchdog itself.
CI verification (this branch, commit cb5c191)
The watchdog works end-to-end in CI. Three of the six platform runs
(Ubuntu Default, Ubuntu clang, Mac Intel) hit the suspected root-cause
hang - which this branch intentionally does NOT contain - and the
watchdog stopped each one ~60 s after progress stopped, with a dump that
identifies it exactly, e.g. (Ubuntu clang):
and the all-thread dump shows the send task parked in
std::condition_variable::waitinsideClientSocket_Qt::send(symbolized on macOS). The Mac/Windows runs where the race did not fire
completed normally. So the red checks on this PR are the watchdog doing
its job; the fix is in #1839, whose CI is green on every platform so far.