Repository navigation
Stop idle workers from keeping unreachable actors alive - #3435
Merged
Merged
Conversation
_test_Cleanup waited for 81 of its 100 actors. It now waits for all of them. New tests wait for every actor whose timer has run, every actor whose timer raised an exception that nothing caught, and every actor that becomes unreachable while other actors raise such exceptions. The four tests time out on this commit, in ordinary mode and with --gc-check. Idle workers keep pointers to some of the actors, so these are never finalized. And when a finalizer runs in the worker loop right after a continuation, its cleanup message can wait in the outgoing queue of an actor that never runs again.
A worker with nothing to run waits in libuv's poll. The poll's stack frame holds an array of 1024 events, 32 KiB with kqueue and 12 to 16 KiB with epoll, and the kernel writes only the events it returns. The rest of the array keeps what the worker's earlier calls left at that depth of the stack: pointers to the actors, messages and continuations it ran last, and to an exception that a continuation raised. The collector scans the stack of a waiting thread from its stack pointer up. So these pointers kept their objects alive for as long as the worker waited, and an actor with a __cleanup__ that they reached was not finalized. One such pointer can keep many actors. When the timer queue grows, the old array stays in the heap with every message it held. One stale pointer to it kept 64 actors whose timers had run. A worker whose continuation raised an exception that nothing caught left eight copies of the actor's pointer in the array. A prepare handle on each worker loop now zeroes 64 KiB of stack right before the loop polls, in the frame that the poll then reuses. That takes a few hundred nanoseconds per loop iteration. The scheduling benchmarks pipeline and ring_tokens show no change beyond their noise.
A finalizer runs on the thread that allocates when the collector has objects to finalize, and the __cleanup__ it calls sends a message. The worker loop kept the actor it had run as self until the end of the iteration. An allocation after the continuation could run a finalizer there, for example when the loop formats an exception that nothing caught. The message then joined the outgoing queue of that actor after the loop had sent the queue. It waited there until that actor finished another message, which an actor with nothing more to do never does. The loop now clears self as soon as the continuation returns, and such a message is sent at once.
An exception that no try catches returns to the worker loop's exception frame, jump0 in the worker context. The first continuation on a worker pushed it with a setjmp, and it stayed pushed for the life of the worker. The setjmp saves the registers that the worker loop keeps across calls, and in an optimized build these can hold the continuation that the worker is about to run and its argument. The collector scans the worker context. So the first continuation that each worker ran, and everything it reaches, stayed alive for as long as the program ran. In an optimized build on arm64, the idle workers of a small program kept three continuations and a list alive this way. The prepare handle that clears the stack right before the loop polls now also drops the frame and zeroes it. The first continuation after the poll pushes it again with a new setjmp. That is one setjmp each time a worker wakes up. A raise that nothing catches now returns to a setjmp made in the same call of the worker loop, so the loop no longer relies on libuv calling it at the same stack depth every time.
This was referenced Nov 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Actors that have a cleanup method were sometimes never finalized. Idle workers kept pointers to them in stale stack words under libuv's poll and in the registers saved by the worker loop's exception frame, and a cleanup message sent right after a continuation could wait in the outgoing queue of an actor that never ran again. Workers now clear that stack and drop the frame right before they poll, and clear self as soon as a continuation returns.