Repository navigation
Segmentation fault on 0.31.0 when inside a Temporal activity #1292
Description
I'm not sure if the root cause is with asyncpg or with temporal but this code works fine on 0.30.0, and crashes on 0.31.0 in await asyncpg.connect. Python 3.12.9.
#!/usr/bin/env python
import asyncio
import threading
from datetime import datetime, timedelta
import asyncpg
from temporalio import activity, workflow
from temporalio.client import Client
from temporalio.worker import Worker
TASK_QUEUE = "asyncpg-temporal-test"
async def db_call(label: str) -> None:
"""Connect to Postgres and close."""
loop = asyncio.get_running_loop()
print(f"[{label}] thread={threading.get_ident()} loop_id={id(loop)} {loop!r}")
conn = await asyncpg.connect(
host="localhost",
port=5432,
database="database",
user="andrei",
)
print(f"[{label}] connected to Postgres")
await conn.close()
@activity.defn
async def db_activity() -> None:
return await db_call("activity")
@workflow.defn
class TestWorkflow:
@workflow.run
async def run(self) -> None:
print("[workflow] starting db_activity")
return await workflow.execute_activity(
db_activity,
start_to_close_timeout=timedelta(seconds=30),
task_queue=TASK_QUEUE,
)
async def main() -> None:
# 1) Baseline: asyncpg on a fresh loop, no Temporal
print("=== 1) Direct asyncpg on fresh loop (no Temporal) ===")
await db_call("direct")
# 2) Bring up Temporal client + worker and run a workflow that calls db_activity
print("\n=== 2) Temporal worker + workflow calling async activity ===")
client = await Client.connect("localhost:7233")
worker = Worker(
client,
task_queue=TASK_QUEUE,
workflows=[TestWorkflow],
activities=[db_activity],
)
async with worker:
# This executes TestWorkflow.run, which in turn executes db_activity
result = await client.execute_workflow(
TestWorkflow.run,
id=datetime.now().isoformat(),
task_queue=TASK_QUEUE,
)
print(f"[main] workflow result: {result}")
if __name__ == "__main__":
asyncio.run(main())Repro steps:
- Start a temporal server:
temporal server start-dev - In a separate terminal, run the code above:
% python test_asyncpg.py
=== 1) Direct asyncpg on fresh loop (no Temporal) ===
[direct] thread=8654430720 loop_id=4353195520 <_UnixSelectorEventLoop running=True closed=False debug=False>
[direct] connected to Postgres
=== 2) Temporal worker + workflow calling async activity ===
[workflow] starting db_activity
[activity] thread=8654430720 loop_id=4353195520 <_UnixSelectorEventLoop running=True closed=False debug=False>
zsh: segmentation fault python test_asyncpg.pyMetadata
Metadata
Assignees
Labels
No labels
Activity
idevelop commentedon Dec 2, 2025
@elprans this may be of interest, I notice you made many of the heavy changes in this version
elprans commentedon Dec 3, 2025
@idevelop any chance for a traceback or coredump?
idevelop commentedon Dec 3, 2025
idevelop commentedon Dec 3, 2025
fantix commentedon Jan 5, 2026
I can reproduce this, looking into the reason now.
fantix commentedon Jan 5, 2026
Quick update: it seems that the Temporal client is doing a lot of sys module manipulation to create its Python sandbox. If I change your test code not to actually do
asyncpg.connectwithin the activity, but rather 2await db_call("direct")before and after running a no-op Temporal workflow, the secondasyncpg.connect()will also segfault. I'm looking closer into what is hacked by Temporal and why that fails asyncpg.fantix commentedon Jan 5, 2026
Moving
import asyncpginto thedb_callfunction fixed the crash. So Temporal client sandbox must be re-importing the top imports (including asyncpg) which led to some global state issues. Let me create a simplified repro.fantix commentedon Jan 5, 2026
Segmentation fault: 11 python -u test_minimal_segfault.py
idevelop commentedon Jan 14, 2026
@fantix what would explain it working in
0.30and only crashing in0.31?fantix commentedon Jan 14, 2026
Maybe it's because of a Cython update, but both
0.30and0.31have an open far-end version constraint on Cython<4.0, so I'm not sure yet.Jakan-Kink commentedon May 14, 2026
This bug is causing SegFaults when I perform pytests, I can reproduce it with both Python 3.13.13 and 3.14.4.
I am using pytest-xdist with -n8 on a M3 Pro 6p/6e, macOS 26.4.1. Each test in the file is loading a fixture which creates a database named f"test_{uuid.uuid4().hex[:8]}" per test and performs Alembic migrations. When I roll back to 0.30.0, the issue goes away.
Jakan-Kink commentedon Jul 6, 2026
Following up on my earlier comment — I dug into why my pytest runs were segfaulting and I can tie it to @fantix's sys.modules repro now, plus answer the "why does 0.30 work" question.
fantix's minimal repro didn't crash for me on Python 3.13.14 / macOS arm64 until I added one ingredient: generation 1 has to actually be used (a connect) before the modules get dropped. This crashes every time on 0.31.0:
On 0.30.0 the same script doesn't crash, but it's silently wrong instead: the re-imported generation's exception classes no longer match what the driver raises, so an
except asyncpg.PostgresError:around a failing query inuse_gen2doesn't catch — the raisedUndefinedTableErroris the generation-1 class. Re-import-after-use was never really sound; 0.31.0 just upgraded the failure from silent exception-matching breakage to a crash.As for why: I think it's the Cython floor, not the <4.0 ceiling. 0.31.0's build requirements moved from Cython(>=0.29.24) to Cython(>=3.2.1), and Cython 3.2 builds the extension with PEP 489 multi-phase init and per-module state (which the subinterpreter/freethreading work in #1279 needs). Drop-and-reimport creates a new module object and tears down the old one's state, but the generation-1 C types still resolve their defining module's state. The macOS crash report for the repro above bears that out — same faulting frame as the lldb capture earlier in this thread (
__pyx_pw_7asyncpg_8protocol_8protocol_12BaseProtocol_1__init__ + 428), the faulting instruction is a refcount increment with the object register at 0x0, andPyModule_Type/PyModule_GetStateare sitting in the register file. That's a NULL module-state lookup in the middle ofBaseProtocol.__init__. 0.30.0's single-phase build keeps its C globals static across re-import, which is why it survives (with the silent exception split instead).In my case the thing doing the drop-and-reimport wasn't Temporal, it was coverage.py. When a
--cov/source entry is a dotted module name,coverage.inorout.file_and_path_for_module()resolves it withimportlib.util.find_spec(), which imports the package chain (pulling in asyncpg transitively), then un-imports what it just imported so the modules get measured on the real import later. Any pytest run with--cov=yourpkg.modulewhereyourpkgtransitively imports asyncpg produces exactly the sequence above. After switching to path-form--covsources my whole suite (2790 tests, pytest-xdist -n8) runs clean on 0.31.0.idevelop commentedon Jul 31, 2026
@fantix any chance you could look into this?
SimplicityGuy commentedon Aug 17, 2026
Another data point on this, from a completely different trigger — no Temporal involved, which I think generalises the report usefully.
Summary
asyncpg.connect()segfaults duringProtocolconstruction when a coverage tracer is active. Reverting 0.31.0 → 0.30.0 fixes it, with nothing else changed — same command, same environment, same test.Versions
Traceback
Line 1078 is
pg_proto = protocol_factory()— i.e. it dies constructing the Cythonprotocol.Protocolobject, reached via_create_ssl_connection(the server does not offer SSL; this is the negotiation attempt).What makes it fire
It is 100% deterministic (verified 3/3) under this combination, and disappears if any one is removed:
--covpasses--cov=<pkg>(whole package) and--cov=<path/to/file.py>both pass; only--cov=<pkg>.<sub>.<module>crashesPoint 3 is the odd one and I can't explain it. It reproduces on all four coverage cores (
sysmon,ctrace,pytrace, default), so it isn't one tracer implementation.What I could not do
I was unable to minimise this into a standalone script, and I want to be upfront about that rather than imply otherwise. A bare
sys.settracetracer +asyncpg.connect()does not crash. Neither does a small pytest + SQLAlchemy async engine +coverage run --source=<narrow>reproduction, with or withoutssl=prefer/require. Something about the larger suite's fixture setup is still required, and I haven't isolated it.So this is offered as a corroborating data point rather than a ready-made repro: the value is that the crash site and the 0.30 → 0.31 boundary match this issue exactly, while the trigger here is coverage rather than Temporal. Both Temporal's sandbox and coverage install tracing/instrumentation, so "a tracer is installed during connect" looks like the common factor rather than anything Temporal-specific.
Happy to run further diagnostics (coredump,
PYTHONFAULTHANDLER, a debug build, bisect across 0.31.0 commits) if that would help narrow it.Jakan-Kink commentedon Aug 17, 2026
@SimplicityGuy That matches exactly what I was saying a few comments back, only yours expanded the coverage cores to include the other 3, not just base pytest-coverage. See if you can use the script I provided as a replication on your system.
SimplicityGuy commentedon Aug 18, 2026
Ran @Jakan-Kink's script, then bisected. Summary up front:
9e42642— "Add Python 3.14 support, experimental subinterpreter/freethreading support" (Add Python 3.14 support, experimental subinterpreter/freethreading support #1279). Its parent6fe1c49is clean.-DCYTHON_USE_MODULE_STATE. Removing just that from v0.31.0 fixes it; removing the other two new flags does not.Py_INCREF(NULL)on__pyx_mstate_global->__pyx_ptype_..._CoreProtocol— a module-state slot that has been NULLed while__pyx_mstate_globalstill points at it.Environment: Python 3.14.5 and 3.13, macOS arm64 (also reproduced on 3.13, so not 3.14-specific).
1. The script reproduces
@Jakan-Kink's script: SIGSEGV 3/3 on 0.31.0, clean 3/3 on 0.30.0, same faulting frame as the lldb capture earlier in the thread (
__pyx_pw_7asyncpg_8protocol_8protocol_12BaseProtocol_1__init__). One difference from your report: in my environment generation 1 does not need to be used first — a bare import/drop/re-import/connect crashes just as reliably.2. Bisect:
9e42642(#1279)git bisectover the 22 commits inv0.30.0..v0.31.0, building each commit from source with Cython pinned to 3.2.9 throughout so the Cython version is not a free variable:Re-verified directly, 3 runs each:
6fe1c49clean 3/3,9e42642SIGSEGV 3/3.3. It is not the Cython floor
To answer @idevelop's "what would explain it working in 0.30 and only crashing in 0.31" and @fantix's Cython hypothesis: 0.30.0's constraint (
Cython>=0.29.24,<4.0.0) also admits 3.2.x, so both tags can be built with the same compiler.Same Cython, opposite outcomes — so the Cython bump in #1288 is not the cause. It is #1279's build configuration.
4. Ablation:
CYTHON_USE_MODULE_STATEis necessary and sufficient#1279 adds three
-Dflags insetup.py. Cython's own defaults in the generated C areCYTHON_USE_MODULE_STATE 0,CYTHON_PEP489_MULTI_PHASE_INIT 1,CYTHON_USE_TYPE_SPECS 0— so PEP 489 multi-phase init was already on before #1279; the genuinely new settings areMODULE_STATEandTYPE_SPECS.Rebuilding v0.31.0 with each subset, 3 runs each:
CYTHON_USE_MODULE_STATECYTHON_USE_TYPE_SPECS5. What actually faults
lldb on the release build:
protocol.c:60364is generated fromprotocol.pyx:79, the very first statement ofBaseProtocol.__init__:So
__pyx_mstate_globalis a live pointer but its__pyx_ptype_*_CoreProtocolslot is NULL — module state that has been cleared and not repopulated.6. The state is cleared without the module being re-executed
This part surprised me, and it may be the useful bit. After dropping the modules and re-importing:
The extension module object is returned from CPython's extension cache, is not re-executed — a
__STAMP__attribute set on generation 1 is still there — and yet its Cython module state has been cleared out from under it. Holding strong references to the old modules across the drop does not help, so this is not old-module deallocation.Minimal trigger: the pure-Python
asyncpgmodules andasyncpg.protocol.protocolmust both leavesys.modules, so that the extension is genuinely re-imported. Dropping the extension alone is harmless (nothing re-imports it); dropping only the Python modules is harmless (the extension stays cached).SIGSEGV on 0.31.0, clean on 0.30.0.
7. Correction to my earlier comment, and a standalone coverage repro
In my earlier comment I said the common factor with Temporal looked like "a tracer is installed during connect". That was wrong — coverage's tracer is irrelevant. The real mechanism is
coverage/inorout.py:313:file_and_path_for_module()callsimportlib.util.find_spec(), which imports the parent package chain, andsys_modules_saved()on exit doesdel sys.modules[m]for every module imported inside the block. If any parent package transitively imports asyncpg, that is exactly the drop-and-reimport sequence above — so this is the same bug as @fantix's repro, arriving by a different road. Same root cause as @Jakan-Kink described; this just pins the exact call site.It also explains the asymmetry I flagged as unexplained (only a narrow dotted
--covsource crashed):--cov=pkgimports onlypkg/__init__.py, which may not pull in asyncpg;--cov=pkg.sub.modimportspkgandpkg.subas parents, which does; and a path-form source never enters thesource_pkgsloop at all.Self-contained repro —
mypkg/sub/__init__.pycontaining justimport asyncpg, a trivialmypkg/sub/mod.py, and one test that connects:All three pass on 0.30.0. This is the ready-made reproduction I could not produce last time.
Workarounds for anyone hitting this now
asyncpg==0.30.0.--covsources rather than dotted module names.Jakan-Kink commentedon Aug 18, 2026
@SimplicityGuy's bisect prompted me to re-run my own matrix on Python 3.12.13 and 3.13.15 (macOS arm64, PG 18.4, coverage 7.15.4, stock wheels). Two of my earlier assumptions didn't survive that, and dropping them is what surfaced §4.
1. My Cython-floor hypothesis was wrong
The ablation settles it — same Cython 3.2.9, opposite outcomes.
CYTHON_USE_MODULE_STATEis the trigger.2. So was my "generation 1 must be used first" precondition
I'd built that into my repro after fantix's minimal script didn't crash for me, and it turns out to be an artifact of my environment, not a requirement. Bare import → drop → re-import → connect segfaults 3/3 on both 3.12.13 and 3.13.15 with no generation-1 use at all, matching @SimplicityGuy. Worth knowing for whoever writes the regression test: the drop-and-re-import alone is sufficient, and a test that sets up a connection first is testing more than it needs to.
Removing that assumption is what let me isolate the next part.
3.
asyncpg==0.30.0is not a safe workaroundprobe.py for how I tested this.
0.30.0 survives the re-import, but the cached extension still raises generation-1 exception classes while Python-level handlers hold generation-2 ones, so
except asyncpg.PostgresError:silently stops catching. Measured on both Pythons, 3/3, with and without prior generation-1 use:Here is the whole thing through real coverage, with no
sys.modulesmanipulation anywhere. Four files:mypkg/__init__.py— empty.mypkg/sub/__init__.py:mypkg/sub/mod.py:test_exc.py:Results, 3/3 deterministic in every cell on both 3.12.13 and 3.13.15:
The 0.30.0 failure, verbatim:
An
UndefinedTableErrorescaping anexcept asyncpg.PostgresError:block that is three lines away from it. Downgrading trades a loud crash for silently-swallowed database errors; for the coverage case the real fix is path-form--covsources.4. This is one bug with two faces, not two bugs
same_protocol_ext=Trueholds on both versions — the extension object is served from CPython's cache and never re-executed, exactly as @SimplicityGuy found, while the pure-Pythonasyncpg.exceptionsmodules do re-execute and mint fresh classes. The extension's stale references to generation-1 Python objects are the shared root cause;CYTHON_USE_MODULE_STATEonly changes when you find out, by NULLing the state before anything can be raised.That has a consequence for the fix: repopulating module state alone would stop the segfault and leave the exception-identity mismatch untouched — 0.30.0 is what that outcome looks like from the outside.
kb1ibt commentedon Oct 1, 2026
@elprans any chance this can be looked at soon?
HardMax71 commentedon Oct 13, 2026
It looks like I've got same segfault recently. The cause is in Cython, opened cython/cython#8048.
When asyncpg is imported a second time, Cython gives back the module that's already loaded. CPython then clears that module's state, and Cython never fills it again. So the next
connect()reads NULL and crashes. Still happens on 0.32.0.Building without
CYTHON_USE_MODULE_STATEstops the crash (tried on main), but thenimport asyncpgfails in a subinterpreter. Cython's maintainers also say module state "really isn't ready for any significant use yet" (cython#8006).