Skip to content

chore: fix gRPC fork deadlock in Ubuntu unit tests - #9633

Closed
bshaffer wants to merge 1 commit into
move-authfrom
fix-auth-fork-deadlock
Closed

chore: fix gRPC fork deadlock in Ubuntu unit tests#9633
bshaffer wants to merge 1 commit into
move-authfrom
fix-auth-fork-deadlock

Conversation

@bshaffer

@bshaffer bshaffer commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Description

This PR fixes the issue causing the Unit tests in Ubuntu (and Package tests) on PR #9514 to hang and time out after 6 hours.

Root Cause

  1. Auth/tests/Cache/RaceConditionTest.php uses pcntl_fork() to test concurrency across 4 child processes.
  2. In the monorepo CI, the PHP grpc extension is installed for Ubuntu runners.
  3. When the grpc C-core extension is loaded in a PHP process, background threads and mutexes are initialized. When pcntl_fork() is called, POSIX fork() only clones the calling thread, leaving unreleased locks and absent background worker threads in the child process.
  4. When the child process calls exit(0), PHP runs module shutdown, which deadlocks indefinitely on a futex (futex_do_wait) waiting on mutexes/threads that do not exist in the child.
  5. The parent process is stuck in pcntl_waitpid(), and PHPUnit is stuck waiting for the test process to finish, causing the 6-hour timeout in CI.
  6. This also hung run-package-tests.sh because Auth is included in the package test directory loop and run under xargs.
  7. This passed on Windows because pcntl_fork is unsupported on Windows (test was skipped), and passed on PHP 8.1 Unit Test (no gRPC) because grpc was not installed.

Solution

  1. Add GRPC_ENABLE_FORK_SUPPORT=1 to unit-tests.yaml, run-package-tests.sh, phpunit.xml.dist, and Auth/phpunit.xml.dist. This enables gRPC's pthread_atfork handlers to properly reset C-core state post-fork so child processes exit cleanly.
  2. Add a defensive skip check in Auth/tests/Cache/RaceConditionTest.php so that if a developer runs PHPUnit locally with grpc loaded and without GRPC_ENABLE_FORK_SUPPORT=1, the test skips with an informative message instead of hanging.

When the PHP gRPC extension is loaded, calling pcntl_fork() and exiting in the
child causes a futex deadlock in gRPC C-core due to unreleased locks / missing
background threads in the child process.

Enabling GRPC_ENABLE_FORK_SUPPORT=1 registers gRPC's pthread_atfork handlers,
allowing child processes to exit cleanly. Also adds a skip guard to
RaceConditionTest if gRPC is loaded without fork support enabled.
@bshaffer
bshaffer marked this pull request as ready for review September 2, 2026 21:03
@bshaffer
bshaffer requested a review from a team as a code owner September 2, 2026 21:03
@bshaffer bshaffer closed this Sep 2, 2026
@bshaffer
bshaffer deleted the fix-auth-fork-deadlock branch September 2, 2026 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant