Skip to content

Don't report "No leader was elected" while the leader is still streaming - #438

Merged
tekmaven merged 1 commit into
mainfrom
rh/report-while-streaming
Oct 8, 2026
Merged

tekmaven merged 1 commit into
mainfrom
rh/report-while-streaming

Conversation

@tekmaven

@tekmaven tekmaven commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Why

In lazy-load mode (--lazy-load, or --preresolved-tests, which implies it), the leader streams tests into the queue in batches. Workers start running tests as soon as the first batch arrives. The leader only sets the master status to ready after the last batch (worker.rb).

minitest-queue report can stop waiting before then, when --max-test-failed is reached or the report times out (supervisor.rb). It then checks queue_initialized?, which is only true for ready or finished (runner.rb, base.rb). While the leader is still streaming, that check is false, so the report aborts with exit status 40:

No leader was elected. This typically means no worker was able to start. Were there any errors during application boot?

But a leader was elected, and workers were running tests. When --max-test-failed ends the run, the report never prints the failure that ended it. A run that never elects a leader also exits with 40, so anything that keys off that status, such as retries or metrics, puts these runs in the wrong bucket.

We hit this on a large suite where the leader took about five minutes to stream every test. The first failure came about ten seconds before streaming finished.

What

  • report only aborts with "No leader was elected" when the leader is neither streaming nor finished. With a streaming leader, it falls through to BuildStatusReporter, as it already does for an initialized queue that wasn't exhausted. That prints the failures and exits with 44 (--max-test-failed) or 43 (timeout).
  • streaming? is checked before queue_initialized?. The status only moves from streaming to ready, so this order can't miss a leader that finishes between the two reads.
  • Tests the leader hasn't streamed yet aren't in the queue. While it's streaming, the last line reads "At least N tests weren't run (the leader hadn't finished streaming tests to the queue)." Otherwise the line is unchanged.

queue_initialized? itself doesn't change. Workers use it through exhausted? (base.rb). Counting a streaming queue as initialized would let a worker treat a momentarily empty queue as finished and exit early.

Behavior

Scenario: a --lazy-load worker hits --max-test-failed 1 while the leader is still streaming, then report runs.

Before (exit status 40):

Waiting for workers to complete
No leader was elected. This typically means no worker was able to start. Were there any errors during application boot?

After (exit status 44):

Waiting for workers to complete
Ran 1 tests, 1 assertions, 1 failures, 0 errors, 0 skips, 0 requeues in 0.00s (aggregated)

Encountered too many failed tests. Test run was ended early.


================================================================================
FAILED TESTS SUMMARY:
================================================================================
  test/failing_test.rb
================================================================================

--------------------------------------------------------------------------------
Error 1 of 1
--------------------------------------------------------------------------------
FAIL FailingTest#test_failing_0
Expected false to be truthy.
    test/failing_test.rb:7:in 'block (2 levels) in <class:FailingTest>'

================================================================================
At least 9 tests weren't run (the leader hadn't finished streaming tests to the queue).

Other cases:

  • No leader: if no leader ever starts streaming or pushing tests, report still exits 40 with "No leader was elected" (test_all_workers_died).
  • Leader dies mid-stream: the status stays streaming. As before, the report waits for its timeout, because a streaming status keeps resetting the inactive-workers countdown. It now exits 43 ("Timed out waiting for tests to be executed.") instead of 40.
  • Not affected:
    • Runs without lazy loading never set the status to streaming, so their output is unchanged.
    • rspec-queue doesn't support lazy loading.
    • The Python client doesn't stream.

Tests

Two integration tests in ruby/test/integration/minitest_redis_test.rb. A helper runs a real leader with stream_populate and pauses it mid-stream in a Fiber. The queue stays in the streaming state while the test runs minitest-queue subprocesses, with no sleeps.

  • test_max_test_failed_while_leader_is_streaming: a --lazy-load worker fails a test with --max-test-failed 1. report exits 44, prints the failure, and says at least 9 of the 10 tests weren't run.
  • test_report_timeout_while_leader_is_streaming: no workers run. report --timeout 1 exits 43 with "Timed out waiting for tests to be executed."

Both fail on main (exit status 40) and pass with this change.

Results locally on Redis 8.6.3:

  • Ruby 4.0.7, minitest 5.27.0: 329 tests, 0 failures.
  • Ruby 3.3.10, minitest 6.0.6: 329 runs, 0 failures.
  • Repeat runs: the two new tests passed 15 times in a row.

Rollout

There's no version bump in this PR. Only minitest-queue report changes; workers and the Redis layout don't. A reporter and workers on different versions can share a queue.

In lazy-load mode the leader streams tests to the queue in batches and
only marks the queue as initialized after the last batch, while workers
start running tests as soon as the first batch arrives. If the report's
wait ended before that, because --max-test-failed was reached or the
report timed out, `queue_initialized?` was still false, so `report`
aborted with "No leader was elected" and exit status 40 instead of
reporting the failures.

Treat a streaming leader as present, so the report falls through to
BuildStatusReporter: it prints the failures and exits with 44 (too many
failed tests) or 43 (timed out). Tests the leader hasn't streamed yet
aren't in the queue, so the "tests weren't run" count becomes "at
least".
@tekmaven
tekmaven marked this pull request as ready for review October 8, 2026 03:17
@tekmaven
tekmaven merged commit 4c8a831 into main Oct 8, 2026
34 checks passed
@tekmaven
tekmaven deleted the rh/report-while-streaming branch October 8, 2026 16:30

This branch was successfully deployed

1 active deployment
rubygems — a60a9bb8 Deployed Oct 8, 2026 by shopify-shipit[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants