Repository navigation
[DO NOT MERGE] PaddleOCR crash A/B: same 3x diagnostics on the pre-fix sockets base (77e2f4f10) - #1817
Closed
pa-devbot wants to merge 6 commits into
Closed
[DO NOT MERGE] PaddleOCR crash A/B: same 3x diagnostics on the pre-fix sockets base (77e2f4f10)#1817pa-devbot wants to merge 6 commits into
pa-devbot wants to merge 6 commits into
Conversation
While investigating the intermittent Mac Intel PaddleOCR segfault: - Print the ONNX Runtime version, hardware_concurrency() and the chosen intra/inter-op thread counts to stderr (the file logger is not in the CI log). - Run the global Ort::Env at ORT_LOGGING_LEVEL_INFO so ONNX Runtime's own session-creation progress (Session Options, thread pool params, graph transforms, BFCArena, ...) lands in the CI log. - Add [ORT-DIAG] stderr markers around create_session() and the first PaddleOCRPipeline::recognize_line() Run() calls, to pin down which stage the crash is in. All of this is temporary: revert when the crash is understood.
…t mode The CI log lost the output right up to the segfault because stdout is fully buffered when piped: in the #1812 Mac Intel run the crash happened ~2.5 s after 'Running parallel unit tests...' and not a single line after that was captured. Two temporary diagnostics: - A SIGSEGV/SIGBUS/SIGABRT/SIGILL/SIGFPE handler (command-line test mode only) that writes a native backtrace of the crashing thread straight to fd 2 with backtrace_symbols_fd(), then restores the default disposition and re-raises so the OS crash report and exit code are unchanged. - setvbuf(stdout, _IOLBF) in test mode, so the test 'Starting:'/'Passed:' lines reach the log as they happen and the log shows the last thing the crashing process was doing.
Apple's execinfo.h declares backtrace_symbols_fd() as void (glibc's returns int), so ignore its result on both. Also restore the CRLF line endings the previous commit accidentally rewrote to LF.
setvbuf(stdout, nullptr, _IOLBF, 0) crashed the Windows CI runner with 0xC0000409 (stack-buffer-overrun) ~0.7 s into the test run, on the first stdout write after the call: with size 0, MSVC's ucrt appears to install a zero-size stdio buffer for stdout. 8192 is the conventional default.
The Mac Intel crash is probabilistic (~60% of runs) and lands in the PaddleOCR ONNX session creation, which happens only once per process because the sessions are cached. Run the full suite three times and clear the PaddleOCR cache between runs so the sessions are re-created each time, raising the per-CI-run reproduction odds to ~93%. Stop and fail on the first reported test failure.
pa-devbot
changed the base branch from
sockets-server-posix
to
mac-paddleocr-oldbase-base
November 1, 2026 10:30
Collaborator
Author
|
A/B test complete - see #1812 for the full investigation. |
Collaborator
Author
|
A/B result: confirmed. Mac Intel on this pre-fix base crashed during suite run 2, ~2.4 s into "Sockets (Qt) - Rapid reconnect". The That is the Qt socket test harness itself, not PaddleOCR: the old base has the stack-use-after-return in the Qt server-socket close-completion path (fixed by 6f1eadf "ServerSocket_Qt: give signal completions shared ownership" in the current base), and the corrupt write exploded on this pool thread's stack inside wait_for. Suite run 1 passed all ~26k tests; run 2 died in the socket test. All other platforms passed even on the old base (the bug is probabilistic and showed up only on the Mac Intel runner). No merge target for this branch — please close it. The full write-up is on #1812. |
This was referenced Nov 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A/B test branch for the Mac Intel PaddleOCR crash investigation (#1811, #1812).
Same 5 diagnostic commits as #1812 (ORT stderr markers, fatal-signal backtrace handler, setvbuf line-buffering, 3x suite loop with PaddleOCR session re-creation), but based on the old sockets-server-posix tip 77e2f4f — i.e. before the 'ServerSocket_Qt: give signal completions shared ownership' fix (6f1eadf) that landed in the current base.
Expectation: Mac Intel segfaults during this run (the old base crashed ~3 of 5 runs without the 3x loop). If it passes, the 3x loop is not a sufficient reproduction; if it crashes, #1812's base contains the fix.