Forward host pytest arguments to in-worker pytest

AGENTS.md

@@ -16,6 +16,8 @@ This repository (`workers-py`) contains three Python packages that are used for
 - `packages/runtime-sdk` contains the runtime SDK for Python Workers, which provides a base class for Python Workers and utilities for working with Cloudflare's runtime.
 - `packages/django-cf` is the Django integration package, providing database backends for D1 and Durable Objects, an R2 storage backend, and Cloudflare Access middleware.
 
+There is also an unpublished `packages/testlib`, shared by the `runtime-sdk` and `django-cf` test suites for running pytest inside workerd and reporting the results as host-side tests.
+
 ### `packages/cli`
 
 For cli conventions, see `packages/cli/AGENTS.md`.
@@ -28,6 +30,10 @@ For runtime-sdk conventions, see `packages/runtime-sdk/AGENTS.md`.
 
 For django-cf conventions, see `packages/django-cf/AGENTS.md`.
 
+### `packages/testlib`
+
+For how in-worker test suites are exposed as host tests and how pytest arguments are forwarded into the worker, see `packages/testlib/AGENTS.md`.
+
 ## Build System & Commands
 
 This project uses the following tools to manage the build process:

packages/runtime-sdk/tests/conftest.py

@@ -101,9 +101,4 @@ def register_in_worker_suites(
     src_dir: Path,
     marks: dict[str, pytest.MarkDecorator] | None = None,
 ) -> None:
-    register_testlib_suites(
-        namespace,
-        src_dir,
-        marks=marks,
-        class_name=str.upper,
-    )
+    register_testlib_suites(namespace, src_dir, marks=marks)

packages/runtime-sdk/tests/web-frameworks-test/fastapi-tests/src/worker.py

@@ -32,7 +32,7 @@
 from pydantic import BaseModel
 from starlette.background import BackgroundTask
 from starlette.middleware.gzip import GZipMiddleware
-from testlib.entrypoint import TestRunner
+from testlib.entrypoint import RunSuiteRequest, TestRunner
 
 import asgi
 
@@ -594,7 +594,9 @@ def fastapi_app(self):
 @app.get("/run-tests/{suite_name:path}")
 async def run_suite(suite_name: str, request: Request):
     runner = TestRunner(request.scope["env"], extra_plugins=[FastAPIAppPlugin()])
-    result = runner.run_suite(suite_name)
+    result = runner.run_suite(
+        suite_name, RunSuiteRequest.from_query_params(request.query_params)
+    )
     return JSONResponse(result.payload, status_code=result.status)
 
 

packages/testlib/AGENTS.md

@@ -0,0 +1,54 @@
+# packages/testlib
+
+## Overview
+
+`testlib` holds shared helpers for tests that run pytest *inside* workerd. It is
+not published; `runtime-sdk` and `django-cf` vendor it into their test workers
+via `../packages/testlib` in `[tool.uv.sources]`.
+
+## Key modules
+
+| Module | Runs on | Purpose |
+|---|---|---|
+| `testlib/host.py` | host | `dev_server`, `pywrangler_sync`, `register_in_worker_suites`, arg/keyword forwarding |
+| `testlib/entrypoint.py` | worker | `TestRunner`, `TestRunnerEntrypoint` (`/run-tests/<suite>`, `/health`), `ResultCollector` |
+| `testlib/tracebacks.py` | both | Pickle worker exceptions and remap their frames onto host source roots |
+
+## How in-worker suites are exposed on the host
+
+- A worker project has `src/test_<suite>.py` modules. The worker serves
+  `GET /run-tests/<suite>`, runs that module with `pytest.main` and returns
+  per-test JSON results keyed by `ResultCollector._key` (`Class__name`).
+- `register_in_worker_suites(globals(), src_dir)` in a host test module parses
+  each `src/test_<suite>.py` with `ast` and generates one host test per
+  in-worker test. Host node IDs mirror the in-worker ones: a class registered
+  under the key `test_kv.py` (collected thanks to `__test__ = True`), with
+  in-worker classes mirrored as nested classes, e.g.
+  `tests/test_bindings.py::test_kv.py::TestFoo::test_bar[3.12]`.
+- The suite is run once per `dev_server` (`functools.cache` on
+  `get_suite_results`); each host test just looks up its result.
+
+## Forwarding from the outer to the inner pytest run
+
+- `worker_pytest_args(config)` forwards `config.invocation_params.args` minus
+  positional targets and `HOST_ONLY_OPTIONS` (currently `-m/--markexpr`).
+  `addopts` are never forwarded. Sent as repeated `?arg=` query params.
+- `host_only_keywords(item)` sends the `-k` keywords that exist only on the host
+  (ancestor names such as `test_bindings.py`, suite marks, the compat-config
+  param such as `3.12`) as repeated `?kw=` params; `ExtraKeywordsPlugin` adds
+  them to in-worker items so `-k` selects the same tests on both sides.
+- Workers that don't subclass `TestRunnerEntrypoint` (e.g. the FastAPI test
+  worker) must build a `RunSuiteRequest` from the query string themselves.
+
+## Conventions and pitfalls
+
+- Keep `host._result_key` and `entrypoint.ResultCollector._key` in sync.
+- Add host-only options to `HOST_ONLY_OPTIONS` as they turn up (plugin options
+  not installed in the worker, e.g. `-n`, `--cov`, `--lf`, produce an inner
+  usage error surfaced as a 500).
+- `-x`/`--maxfail` stops the inner session early; later host tests in that
+  suite then fail as "not found in results".
+- Python 3.12 (Pyodide 0.26.0a2) reports false passes for async in-worker
+  tests; verify failure behaviour on 3.13+.
+- `pywrangler sync` in tests may need `UV_NATIVE_TLS=1` on hosts with custom
+  CA certificates.

packages/testlib/testlib/entrypoint.py

@@ -6,7 +6,7 @@
 from dataclasses import dataclass
 from io import StringIO
 from typing import Any
-from urllib.parse import urlparse
+from urllib.parse import parse_qs, urlparse
 
 import pytest
 from pyodide.webloop import WebLoop
@@ -159,6 +159,48 @@ def run_pytest(pytest_args):
     assert exit_code == 0, f"pytest exit code {exit_code}"
 
 
+class ExtraKeywordsPlugin:
+    """Attach host-only ``-k`` keywords to every collected item.
+
+    The host's mirrored items carry keywords (host module name, compat-config
+    parameter, marks) that don't exist in the worker; adding them here makes a
+    forwarded ``-k`` expression select the same tests on both sides.
+    """
+
+    def __init__(self, keywords):
+        self.keywords = set(keywords)
+
+    def pytest_itemcollected(self, item):
+        item.extra_keyword_matches.update(self.keywords)
+
+
+@dataclass
+class RunSuiteRequest:
+    """Parameters the host sends along with a ``/run-tests/<suite>`` request."""
+
+    pytest_args: list
+    keywords: list
+
+    @classmethod
+    def from_url(cls, url):
+        """Parse a ``/run-tests`` URL (string or ``urlparse`` result).
+
+        The host sends pytest arguments as repeated ``arg`` and extra ``-k``
+        keywords as repeated ``kw`` query parameters.
+        """
+        query = urlparse(url).query if isinstance(url, str) else url.query
+        params = parse_qs(query, keep_blank_values=True)
+        return cls(pytest_args=params.get("arg", []), keywords=params.get("kw", []))
+
+    @classmethod
+    def from_query_params(cls, query_params):
+        """Build from a Starlette-style multi-dict with ``getlist``."""
+        return cls(
+            pytest_args=query_params.getlist("arg"),
+            keywords=query_params.getlist("kw"),
+        )
+
+
 @dataclass
 class TestRunnerResult:
     payload: Any
@@ -171,14 +213,22 @@ def __init__(self, env, extra_plugins=()):
         self.collector = ResultCollector()
         self.extra_plugins = list(extra_plugins)
 
-    def plugins(self):
+    def plugins(self, keywords=()):
         return [
             self.collector,
             EnvPlugin(self.env),
+            ExtraKeywordsPlugin(keywords),
             *self.extra_plugins,
         ]
 
-    def run_suite(self, suite_name):
+    def run_suite(self, suite_name, request=None):
+        """Run the ``test_<suite_name>`` module under pytest.
+
+        ``request`` carries the pytest arguments and extra ``-k`` keywords
+        forwarded from the host pytest invocation (see ``testlib.host``).
+        """
+        if request is None:
+            request = RunSuiteRequest([], [])
         module = f"test_{suite_name}"
         if importlib.util.find_spec(module) is None:
             return TestRunnerResult(
@@ -194,8 +244,8 @@ def run_suite(self, suite_name):
             redirect_stderr(output),
         ):
             exit_code = pytest.main(
-                ["--pyargs", module, "-p", "no:cacheprovider"],
-                plugins=self.plugins(),
+                ["--pyargs", module, "-p", "no:cacheprovider", *request.pytest_args],
+                plugins=self.plugins(request.keywords),
             )
         if exit_code != 0 and not self.collector.results:
             return TestRunnerResult(
@@ -221,11 +271,12 @@ def plugins(self):
         return []
 
     async def fetch(self, request):
-        path = urlparse(request.url).path
+        url = urlparse(request.url)
+        path = url.path
 
         if path.startswith("/run-tests/"):
             suite_name = path[len("/run-tests/") :]
-            result = self.runner.run_suite(suite_name)
+            result = self.runner.run_suite(suite_name, RunSuiteRequest.from_url(url))
             return Response.json(result.payload, result.status)
         if path == "/health":
             return Response.json({"ok": True})