Skip to content

Fix debug-mode segfault when the asyncgen finalizer hook runs during frame teardown#756

Open
david-runlayer wants to merge 1 commit into
MagicStack:masterfrom
david-runlayer:fix-asyncgen-finalizer-upstream
Open

Fix debug-mode segfault when the asyncgen finalizer hook runs during frame teardown#756
david-runlayer wants to merge 1 commit into
MagicStack:masterfrom
david-runlayer:fix-asyncgen-finalizer-upstream

Conversation

@david-runlayer

@david-runlayer david-runlayer commented Jul 20, 2026

Copy link
Copy Markdown

Summary

In debug mode, Loop._asyncgen_finalizer_hook can segfault the interpreter. The hook is invoked by the GC from a dealloc — which can happen while CPython is mid-way through clearing the current interpreter frame — and the call_soon_threadsafe it performs creates a Handle whose debug source-traceback capture calls sys._getframe(), dereferencing the half-cleared frame.

We hit this in production on CPython 3.13.14 with uvloop 0.22.1: enabling loop.set_debug(True) on a Starlette/FastAPI service with streaming request bodies (request.stream() is an async generator; under disconnect/cancellation churn they are GC'd constantly) produced worker SIGSEGVs within seconds. Reproduces on 3.13.14 and 3.14.4, x86_64 and aarch64. It appears to be the same underlying weakness as #715 (there, on 3.14, the capture path surfaces a non-frame object to traceback.walk_stack).

Symbolized backtrace (uvloop built -O0 -g3, CPython 3.13.14)

#3  ?? libpython3.13 (sys._getframe impl; SEGV reading frame->frame_obj)
#4  ?? libpython3.13
#5  __pyx_f_6uvloop_4loop_extract_stack ()                    at cbhandles.pyx (extract_stack)
#6  __pyx_f_6uvloop_4loop_6Handle__set_loop ()                at cbhandles.pyx (Handle._set_loop)
#7  __pyx_f_6uvloop_4loop_new_Handle ()
#8  __pyx_pf_6uvloop_4loop_4Loop_14call_soon_threadsafe ()
#11 __pyx_pf_6uvloop_4loop_4Loop_140_asyncgen_finalizer_hook ()
#14 PyObject_CallOneArg
#16 PyObject_CallFinalizerFromDealloc          <- asyncgen GC'd from a dealloc...
#19 _PyEval_FrameClearAndPop                   <- ...while this frame is being cleared
#20 _PyEval_EvalFrameDefault

A production core dump shows the same failure from the release wheel: SIGSEGV (SEGV_MAPERR) inside sys._getframe incref'ing frame->frame_obj containing a torn value (0x3000000010).

Fix

  • _asyncgen_finalizer_hook sets a per-thread flag (the hook can run on any thread) so extract_stack() skips frame introspection for the handle it creates. The source traceback of a GC point is arbitrary victim code anyway, so nothing meaningful is lost.
  • extract_stack() additionally tolerates AttributeError/ValueError/TypeError from StackSummary.extract/walk_stack, returning None instead of propagating — this also addresses the 3.14 failure mode reported in uvloop debug mode crashes on Python 3.14 during async generator finalization (traceback.walk_stack sees non-frame object) #715 (_asyncio.TaskStepMethWrapper lacking f_back).

Reproducer

Crashes in <15 s unpatched (verified end-to-end from a clean directory containing only these files: 271 Fatal Python error lines in a 60 s window, current PyPI starlette/fastapi/uvicorn); clean with this fix (10 min, ~259k streamed requests, verified on 3.13.14 aarch64 and x86_64).

app.py + load.py + docker command
# app.py — needs: pip install uvicorn starlette fastapi httpx uvloop
from fastapi import FastAPI, Request
from starlette.middleware.base import BaseHTTPMiddleware

app = FastAPI()

class Passthrough(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        return await call_next(request)

app.add_middleware(Passthrough)
app.add_middleware(Passthrough)

@app.post("/upload")
async def upload(request: Request) -> dict:
    total = 0
    async for chunk in request.stream():
        total += len(chunk)
    return {"received": total}

# loop_factory.py
import uvloop
def new_event_loop():
    loop = uvloop.new_event_loop()
    loop.set_debug(True)          # <- the trigger
    return loop
# load.py — 32 concurrent chunked uploads in a loop
import asyncio, httpx

async def body():
    for _ in range(200):
        yield b"x" * 2048

async def worker(client):
    while True:
        try:
            await client.post("http://localhost:8000/upload", content=body())
        except Exception:
            await asyncio.sleep(0.05)

async def main():
    async with httpx.AsyncClient(timeout=30) as c:
        await asyncio.gather(*[worker(c) for _ in range(32)])

asyncio.run(main())
docker run --rm -v $PWD:/r python:3.13.14-slim sh -c '
  pip install -q uvloop==0.22.1 uvicorn starlette fastapi httpx httptools
  cd /r
  PYTHONFAULTHANDLER=1 uvicorn app:app --workers 4 --port 8000 \
    --loop loop_factory:new_event_loop --http httptools &
  sleep 8
  python load.py &
  sleep 60
  # -> "Fatal Python error: Segmentation fault" within ~15s of load starting
'

Performance

Stock = master merge base (e8efea4), patched = this branch (including the save/restore hook); both built from the same tree with identical flags (aarch64, CPython 3.13.14, best-of-3):

path stock patched delta
call_soon + cancel, non-debug 0.125 µs 0.129 µs noise (≤4 ns, sign flips between sessions)
asyncgen create→finalize, non-debug 1.491 µs 1.631 µs +140 ns (three thread-local ops in the hook)
call_soon + cancel, debug 3.337 µs 3.376 µs ~1% (within noise)
asyncgen create→finalize, debug 11.836 µs 7.368 µs −38% (skips the now-pointless capture)

Non-debug hot paths are untouched by construction (extract_stack is only reachable under loop._debug); the only non-debug cost is the thread-local save/set/restore per GC'd async generator, on a path whose real cost is dominated by uv_async_send plus the subsequently scheduled create_task/aclose().

Behavior change

In debug mode, the handle scheduling agen.aclose() from the finalizer hook no longer carries _source_traceback. Previously it captured the stack of whichever code happened to be running when GC fired — misleading rather than useful. Task-level debug tracebacks (create_task runs later in normal loop context) and firstiter capture are unaffected.

Validated with the reproducer plus targeted checks (normal handles still get _source_traceback in debug mode; asyncgen finalizer path executes cleanly); relying on CI for the full test matrix.

@david-runlayer

Copy link
Copy Markdown
Author

CI note: the test (3.14t, macos-latest) failure is the pre-existing flake family, not this change — Test_AIO_Process.test_process_streams_redirect (stdlib-asyncio variant; this patch isn't in that code path) failed because a debug-mode Executing <Task ...> took N seconds slow-callback warning leaked into the redirected stderr on the slow free-threaded runner. The same warning-pollution appears in master's latest Tests run (29349960216, 3.14t job, test_process_delayed_stdio*), which failed before this PR existed. Re-pushed to retrigger.

@david-runlayer
david-runlayer force-pushed the fix-asyncgen-finalizer-upstream branch 4 times, most recently from 474e3dd to 3f175ad Compare July 20, 2026 22:47
…frame teardown

The asyncgen finalizer hook can be invoked by a dealloc while CPython is
mid-way through clearing the current interpreter frame
(_PyEval_FrameClearAndPop -> PyObject_CallFinalizerFromDealloc ->
_asyncgen_finalizer_hook). In debug mode the hook's call_soon_threadsafe
creates a Handle whose source-traceback capture calls sys._getframe(),
which dereferences the half-cleared frame (garbage frame_obj) and
segfaults the interpreter.

Observed in production on CPython 3.13.14 under streaming HTTP load
(starlette request.stream() async generators being GC'd under
cancellation churn): worker SIGSEGV within seconds of enabling
loop.set_debug(True). Reproduces on 3.13 and 3.14, x86_64 and aarch64.

Fix: skip source-traceback capture (per-thread flag) for handles created
from the finalizer hook -- the traceback of a GC point is meaningless for
debugging anyway -- and make extract_stack() tolerate AttributeError/
ValueError/TypeError from walk_stack, which py3.14 can raise when
non-frame objects (e.g. _asyncio.TaskStepMethWrapper) appear in the
stack (MagicStack#715).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant