[pa-devbot]

What this is

A [DO NOT MERGE] experiment branch for the handoff in #1811 (PaddleOCR/ONNX Runtime segfault in the C++ CI Serial Programs Mac Intel job, exit 139 during the first PaddleOCR ONNX session creation).

Based on sockets-server-posix @ 77e2f4f because that is the tree where the crash has been observed (runs 41266441051 / 41275162708). The diff against that base is diagnostics only - no behavior changes.

What it does

1. Temporary [ORT-DIAG] stderr markers (revert when done):

   • ML/Models/ML_OrtEnv.cpp: prints the ONNX Runtime version, hardware_concurrency() and the chosen intra/inter-op thread counts (the file logger output is not in the CI log), and runs the global Ort::Env at ORT_LOGGING_LEVEL_INFO so ONNX Runtime's own session-creation log (Session Options trace, thread pool params, graph transforms, BFCArena, ...) lands in the CI log.
   • ML/Models/ML_ONNXRuntimeHelpers.cpp: markers around clean_up_old_model_cache / the Ort::Session constructor in create_session().
   • ML/Inference/ML_PaddleOCRPipeline.cpp: markers at the end of the PaddleOCRPipeline ctor and before the first session->Run() calls.

2. (Pending - see below) CI workflow changes that, on a Mac test failure:

   • dump the macOS crash reports ($HOME/Library/Logs/DiagnosticReports/*.ips - symbol names + imageOffset + image UUIDs, matchable against 3rdPartyBinaries/onnxruntime-osx-x86_64-1.23.2.tgz),
   • upload them as an artifact,
   • rerun the suite under lldb with 'thread backtrace all'.

Status / next steps

• The workflow part of this PR is not pushed yet: the bot's token is missing the 'workflow' scope, so pushes that modify .github/workflows are rejected by GitHub. @kuroneko could add the 'workflow' scope to the pa-devbot PAT (or push the workflow commit himself - it is commit 37133db on this branch's history, also attached as a patch in the comments).
• Each push re-runs Mac Intel; if the run passes, I push a trivial commit to get another attempt (the crash is probabilistic, ~3/5 runs on this tree).
• Once the crash reproduces with this instrumentation, the log should show the exact stage (ORT INFO lines + [ORT-DIAG] markers), and the .ips / lldb backtrace the faulting thread.
• No code here changes PaddleOCR behavior; the only functional difference vs the base is the ORT env logging level and extra stderr lines.

[pa-devbot]

Pending: the crash-report / lldb workflow commit

The token used here lacks the `workflow\ scope, so I could not push the CI-instrumentation commit. It is commit \37133db\ (against this branch). Once the pa-devbot PAT has the \workflow\` scope (or this commit is pushed by someone else), it can be applied with:

git fetch origin mac-paddleocr-crash-diagnostics  # or apply the patch below
git am /path/to/0001-Mac-CI-capture-crash-reports.patch
git push

Patch (commit 37133db):

From 37133db7a3f157c97eb34f5ac199a4318986c18c Mon Sep 17 00:00:00 2001
From: PA Dev Bot <344001113+pa-devbot@users.noreply.github.com>
Date: Sun, 1 Nov 2026 06:47:46 +0000
Subject: [PATCH] Mac CI: capture crash reports and lldb backtrace when the
 test run dies

On a segfault the macOS crash reporter writes a .ips report to
~/Library/Logs/DiagnosticReports/, which is not in the Actions log and
the runner is wiped afterwards. Dump the SerialPrograms crash reports
to the log (and upload them as an artifact) on test failure, then rerun
the suite under lldb and print a backtrace of every thread.

Used to investigate the intermittent PaddleOCR/ONNX Runtime segfault in
the Mac Intel CI; see the PR description.
---
 .../workflows/cpp-ci-serial-programs-base.yml | 54 +++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/.github/workflows/cpp-ci-serial-programs-base.yml b/.github/workflows/cpp-ci-serial-programs-base.yml
index 58f1a9d89..b71425319 100644
--- a/.github/workflows/cpp-ci-serial-programs-base.yml
+++ b/.github/workflows/cpp-ci-serial-programs-base.yml
@@ -293,7 +293,61 @@ jobs:
         if: startsWith(inputs.os, 'mac') && inputs.run-tests
         run: |
           cd Arduino-Source/SerialPrograms/bin
+          set +e
           ./SerialPrograms.app/Contents/MacOS/SerialPrograms --command-line-test-mode
+          TEST_EXIT=$?
+          set -e
+          echo "Tests exited with code $TEST_EXIT (139 = segfault, 134 = abort)"
+
+      # The macOS crash reporter (ReportCrash) writes .ips reports for crashed processes to
+      # ~/Library/Logs/DiagnosticReports/. They are not part of the Actions log and the runner
+      # is wiped, so dump them here when the test run dies. Each .ips holds the faulting
+      # thread's stack (symbol names + imageOffset) and the image UUIDs, which can be matched
+      # against 3rdPartyBinaries/onnxruntime-osx-x86_64-*.tgz.
+      - name: Dump macOS crash reports (on test failure)
+        if: startsWith(inputs.os, 'mac') && inputs.run-tests && failure()
+        run: |
+          #  ReportCrash writes the report shortly after the crash; give it a moment.
+          sleep 10
+          mkdir -p crash-reports
+          echo "=== $HOME/Library/Logs/DiagnosticReports ==="
+          ls -la "$HOME/Library/Logs/DiagnosticReports/" || true
+          FOUND=0
+          for f in "$HOME/Library/Logs/DiagnosticReports/"*.ips; do
+            [ -e "$f" ] || continue
+            case "$(basename "$f")" in
+              SerialPrograms*|onnxruntime*) ;;
+              *) continue ;;
+            esac
+            FOUND=1
+            cp "$f" crash-reports/
+            echo "=== $f ==="
+            cat "$f"
+            echo
+          done
+          if [ "$FOUND" -eq 0 ]; then
+            echo "No SerialPrograms crash reports found in DiagnosticReports."
+          fi
+
+      - name: Upload crash reports
+        if: startsWith(inputs.os, 'mac') && inputs.run-tests && failure()
+        uses: actions/upload-artifact@v7
+        with:
+          name: Mac crash reports
+          path: crash-reports/
+          if-no-files-found: ignore
+
+      # Rerun the test suite under lldb and print a backtrace of every thread when it dies.
+      # This step is expected to fail again; the value is the "thread backtrace all" output.
+      - name: Rerun tests under lldb for a backtrace (on test failure)
+        if: startsWith(inputs.os, 'mac') && inputs.run-tests && failure()
+        run: |
+          cd Arduino-Source/SerialPrograms/bin
+          lldb -b \
+            -o run \
+            -k "thread backtrace all" \
+            -k quit \
+            ./SerialPrograms.app/Contents/MacOS/SerialPrograms --command-line-test-mode
 
       # The list of files that passed clang-query, from the latest run on this branch or on main.
       # See .github/scripts/clang-query-cached.py. Unlike ccache, every run saves it: it is only
-- 
2.43.0

[pa-devbot]

Reopening to re-trigger CI

[pa-devbot]

Investigation complete — the "PaddleOCR crash" was a stack-use-after-return in the Qt socket tests

Root cause. The probabilistic exit-139 on macos-15-intel was not in PaddleOCR/ONNX Runtime. It was a stack-use-after-return in the new Qt server-socket close-completion path (the PR #1793 test harness era): the connection-closed completion handler wrote to a stack-local CloseCompletion after the close_connection() frame that owned it had returned. The corrupt write landed on pool-thread stacks and exploded where that memory was next used — in the handoff runs that looked like PaddleOCR session creation; in my A/B run it crashed directly inside the socket test.

Proof (A/B).

• This branch's diagnostics (stderr markers, fatal-signal backtrace handler, 3× suite loop with PaddleOCR cache cleared between runs so the ONNX sessions are re-created every time) were cherry-picked onto the old base 77e2f4f10 (PR [DO NOT MERGE] PaddleOCR crash A/B: same 3x diagnostics on the pre-fix sockets base (77e2f4f10) #1817): Mac Intel crashed in suite run 2, ~2.4 s into "Sockets (Qt) - Rapid reconnect":
  Test_Sockets_RapidReconnect<ClientSocket_Qt, ServerSocket_Qt>::run() → Waiter::wait_for → ~std::function<void()> (SIGSEGV).
• On the current sockets-server-posix tip (which now contains 6f1eadf6d "ServerSocket_Qt: give signal completions shared ownership" plus the regression test "Sockets (Qt) - Completion lifetime under event-thread stall"), this branch's 3× loop passed 3 full suites (~26k tests each, 6 fresh PaddleOCR session creations) with zero failures. The base branch's own CI is green at its current tip too.

Also fixed along the way (in this branch's commits, in case you want to cherry-pick any):

• setvbuf(stdout, nullptr, _IOLBF, 0) crashed the Windows CI runner with 0xC00000409 → 0xC0000409 (stack-buffer-overrun): with size 0, MSVC's ucrt ends up with a zero-size stdout buffer and the first write corrupts the stack. Use an explicit size (8192).
• Apple's execinfo.h declares backtrace_symbols_fd() as void (glibc's returns int) — don't use its return value portably.

Recommendation. Nothing here needs merging: the crash is already fixed on the sockets-server-posix tip. Please close this PR and #1817. If you'd like, the fatal-signal backtrace handler (commit eee9c0d, ~40 lines in CommandLineTests.cpp) is a useful permanent addition for the test runner — say the word and I'll split it out as a clean PR against main.