Skip to content

Stop idle workers from keeping unreachable actors alive - #3435

Merged
plajjan merged 4 commits into
mainfrom
cleanup-stall
Nov 20, 2026
Merged

plajjan merged 4 commits into
mainfrom
cleanup-stall

Conversation

@plajjan

@plajjan plajjan commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Actors that have a cleanup method were sometimes never finalized. Idle workers kept pointers to them in stale stack words under libuv's poll and in the registers saved by the worker loop's exception frame, and a cleanup message sent right after a continuation could wait in the outgoing queue of an actor that never ran again. Workers now clear that stack and drop the frame right before they poll, and clear self as soon as a continuation returns.

_test_Cleanup waited for 81 of its 100 actors. It now waits for all of
them. New tests wait for every actor whose timer has run, every actor
whose timer raised an exception that nothing caught, and every actor
that becomes unreachable while other actors raise such exceptions.

The four tests time out on this commit, in ordinary mode and with
--gc-check. Idle workers keep pointers to some of the actors, so these
are never finalized. And when a finalizer runs in the worker loop right
after a continuation, its cleanup message can wait in the outgoing
queue of an actor that never runs again.
A worker with nothing to run waits in libuv's poll. The poll's stack
frame holds an array of 1024 events, 32 KiB with kqueue and 12 to 16 KiB
with epoll, and the kernel writes only the events it returns. The rest
of the array keeps what the worker's earlier calls left at that depth of
the stack: pointers to the actors, messages and continuations it ran
last, and to an exception that a continuation raised. The collector
scans the stack of a waiting thread from its stack pointer up. So these
pointers kept their objects alive for as long as the worker waited, and
an actor with a __cleanup__ that they reached was not finalized.

One such pointer can keep many actors. When the timer queue grows, the
old array stays in the heap with every message it held. One stale
pointer to it kept 64 actors whose timers had run. A worker whose
continuation raised an exception that nothing caught left eight copies
of the actor's pointer in the array.

A prepare handle on each worker loop now zeroes 64 KiB of stack right
before the loop polls, in the frame that the poll then reuses. That
takes a few hundred nanoseconds per loop iteration. The scheduling
benchmarks pipeline and ring_tokens show no change beyond their noise.
A finalizer runs on the thread that allocates when the collector has
objects to finalize, and the __cleanup__ it calls sends a message. The
worker loop kept the actor it had run as self until the end of the
iteration. An allocation after the continuation could run a finalizer
there, for example when the loop formats an exception that nothing
caught. The message then joined the outgoing queue of that actor after
the loop had sent the queue. It waited there until that actor finished
another message, which an actor with nothing more to do never does.

The loop now clears self as soon as the continuation returns, and such
a message is sent at once.
An exception that no try catches returns to the worker loop's exception
frame, jump0 in the worker context. The first continuation on a worker
pushed it with a setjmp, and it stayed pushed for the life of the
worker. The setjmp saves the registers that the worker loop keeps across
calls, and in an optimized build these can hold the continuation that
the worker is about to run and its argument. The collector scans the
worker context. So the first continuation that each worker ran, and
everything it reaches, stayed alive for as long as the program ran. In
an optimized build on arm64, the idle workers of a small program kept
three continuations and a list alive this way.

The prepare handle that clears the stack right before the loop polls
now also drops the frame and zeroes it. The first continuation after the
poll pushes it again with a new setjmp. That is one setjmp each time a
worker wakes up. A raise that nothing catches now returns to a setjmp
made in the same call of the worker loop, so the loop no longer relies
on libuv calling it at the same stack depth every time.
@plajjan
plajjan merged commit cbb0194 into main Nov 20, 2026
53 checks passed
@plajjan
plajjan deleted the cleanup-stall branch November 20, 2026 21:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant