Repository navigation
Don't report "No leader was elected" while the leader is still streaming - #438
Merged
Merged
Conversation
In lazy-load mode the leader streams tests to the queue in batches and only marks the queue as initialized after the last batch, while workers start running tests as soon as the first batch arrives. If the report's wait ended before that, because --max-test-failed was reached or the report timed out, `queue_initialized?` was still false, so `report` aborted with "No leader was elected" and exit status 40 instead of reporting the failures. Treat a streaming leader as present, so the report falls through to BuildStatusReporter: it prints the failures and exits with 44 (too many failed tests) or 43 (timed out). Tests the leader hasn't streamed yet aren't in the queue, so the "tests weren't run" count becomes "at least".
tekmaven
marked this pull request as ready for review
October 8, 2026 03:17
bmaynard
approved these changes
Oct 8, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
In lazy-load mode (
--lazy-load, or--preresolved-tests, which implies it), the leader streams tests into the queue in batches. Workers start running tests as soon as the first batch arrives. The leader only sets the master status toreadyafter the last batch (worker.rb).minitest-queue reportcan stop waiting before then, when--max-test-failedis reached or the report times out (supervisor.rb). It then checksqueue_initialized?, which is only true forreadyorfinished(runner.rb, base.rb). While the leader is still streaming, that check is false, so the report aborts with exit status 40:But a leader was elected, and workers were running tests. When
--max-test-failedends the run, the report never prints the failure that ended it. A run that never elects a leader also exits with 40, so anything that keys off that status, such as retries or metrics, puts these runs in the wrong bucket.We hit this on a large suite where the leader took about five minutes to stream every test. The first failure came about ten seconds before streaming finished.
What
reportonly aborts with "No leader was elected" when the leader is neither streaming nor finished. With a streaming leader, it falls through toBuildStatusReporter, as it already does for an initialized queue that wasn't exhausted. That prints the failures and exits with 44 (--max-test-failed) or 43 (timeout).streaming?is checked beforequeue_initialized?. The status only moves fromstreamingtoready, so this order can't miss a leader that finishes between the two reads.queue_initialized?itself doesn't change. Workers use it throughexhausted?(base.rb). Counting a streaming queue as initialized would let a worker treat a momentarily empty queue as finished and exit early.Behavior
Scenario: a
--lazy-loadworker hits--max-test-failed 1while the leader is still streaming, thenreportruns.Before (exit status 40):
After (exit status 44):
Other cases:
reportstill exits 40 with "No leader was elected" (test_all_workers_died).streaming. As before, the report waits for its timeout, because a streaming status keeps resetting the inactive-workers countdown. It now exits 43 ("Timed out waiting for tests to be executed.") instead of 40.streaming, so their output is unchanged.rspec-queuedoesn't support lazy loading.Tests
Two integration tests in
ruby/test/integration/minitest_redis_test.rb. A helper runs a real leader withstream_populateand pauses it mid-stream in aFiber. The queue stays in thestreamingstate while the test runsminitest-queuesubprocesses, with no sleeps.test_max_test_failed_while_leader_is_streaming: a--lazy-loadworker fails a test with--max-test-failed 1.reportexits 44, prints the failure, and says at least 9 of the 10 tests weren't run.test_report_timeout_while_leader_is_streaming: no workers run.report --timeout 1exits 43 with "Timed out waiting for tests to be executed."Both fail on
main(exit status 40) and pass with this change.Results locally on Redis 8.6.3:
Rollout
There's no version bump in this PR. Only
minitest-queue reportchanges; workers and the Redis layout don't. A reporter and workers on different versions can share a queue.