Skip to content

Tests: dump a backtrace to stderr on a fatal signal in test mode - #1819

Merged
Mysticial merged 1 commit into
mainfrom
tests-crash-backtrace
Nov 1, 2026
Merged

Mysticial merged 1 commit into
mainfrom
tests-crash-backtrace

Conversation

@pa-devbot

Copy link
Copy Markdown
Collaborator

Follow-up from the Mac Intel PaddleOCR crash investigation (#1811, #1812, #1817).

When the command-line test runner dies on a fatal signal, the CI log ends right before the crash — the process's last stdio buffers are lost, and on macOS the .ips crash report doesn't land in the log. This made the #1811 crash take multiple CI runs to pin down.

What this adds (command-line test mode only, no GUI impact):

  • Handlers for SIGSEGV/SIGBUS/SIGABRT/SIGILL/SIGFPE (Apple/Linux) that write a native backtrace of the crashing thread straight to fd 2 — unbuffered, and bypassing the app's std::cerr redirection into its logger — then restore the default disposition and re-raise, so the OS still writes its crash report and the exit code (139) still reflects the crash. A re-entrancy guard makes a crash inside the handler just die immediately. No-op on Windows (no portable async-signal-safe backtrace API there; OS crash reports cover it).
  • setvbuf(stdout, _IOLBF, 8192) in test mode so the test lines reach the log as they happen instead of being lost in the full pipe buffer. The explicit size matters: with 0, MSVC's ucrt ends up with a zero-size stdout buffer and the first write crashes with 0xC0000409 (hit this during the investigation).

Proof it works: the #1817 A/B run used this handler to capture the backtrace that identified the Mac Intel crash root cause (a stack-use-after-return in the Qt socket tests, since fixed on main):

[PA-CRASH] Fatal signal, backtrace of the crashing thread:
0   SerialPrograms  ... crash_backtrace_handler + 168
1   libsystem_platform.dylib  _sigtramp + 29
3   SerialPrograms  std::function<void()>::~function() + 21
6   SerialPrograms  PokemonAutomation::(anonymous namespace)::Waiter::wait_for(...)
7   SerialPrograms  PokemonAutomation::(anonymous namespace)::Test_Sockets_RapidReconnect<ClientSocket_Qt, ServerSocket_Qt>::run()
8   SerialPrograms  UnitTestRunner::dispatch_test(...)

When the command-line test runner crashes (e.g. a segfault in CI), the
process's last stdio buffers are lost, so the log ends just before the
crash with no indication of where it happened.

Install handlers for SIGSEGV/SIGBUS/SIGABRT/SIGILL/SIGFPE (command-line
test mode only, Apple/Linux; no-op on Windows) that write a native
backtrace of the crashing thread straight to fd 2 - which also bypasses
the app's std::cerr redirection into its logger - then restore the
default disposition and re-raise, so the OS still writes its crash
report and the exit code still reflects the crash.

Also keep stdout line-buffered in test mode, so the test output reaches
the log as it happens instead of being lost in the full buffer. An
explicit 8192-byte buffer is used: with 0, MSVC's ucrt can end up with
a zero-size buffer, which corrupts the next stdout write.
@Mysticial
Mysticial merged commit 7d0cae3 into main Nov 1, 2026
7 checks passed
@Mysticial
Mysticial deleted the tests-crash-backtrace branch November 1, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants