Skip to content

Ensure store server serializes access per thread - #175

Merged
Yuge Zhang (ultmaster) merged 2 commits into
mainfrom
codex/fix-asyncio-event-loop-error
Oct 20, 2025
Merged

Yuge Zhang (ultmaster) merged 2 commits into
mainfrom
codex/fix-asyncio-event-loop-error

Conversation

@ultmaster

Copy link
Copy Markdown
Contributor

Summary

  • document that the loop-aware async lock is intentionally thread-unsafe while simplifying its per-event-loop bookkeeping
  • guard every LightningStoreServer delegate method with a server-level threading lock and route requests through those methods to keep access serialized within the server process

Testing

  • uv run pyright
  • uv run pytest tests/store/test_memory.py::test_store_lock_rebinds_to_new_event_loop
  • uv run pytest tests/store/test_client_server.py

https://chatgpt.com/codex/tasks/task_e_68f592757158832ea4368388b2b64c79

Copilot AI review requested due to automatic review settings October 20, 2025 02:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Adds an event-loop–aware async lock to the in‑memory store and serializes server delegate access with a server-level lock, updating API routes to funnel through wrapped methods.

  • Introduces _LoopAwareAsyncLock to allow reusing the in‑memory store across different event loops (thread-unsafe by design).
  • Wraps LightningStoreServer delegate calls with a new locking layer and routes FastAPI endpoints through the server methods.
  • Adds a test to verify store usability after switching to a new event loop.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
tests/store/test_memory.py Adds test ensuring the in-memory store lock rebinds correctly when used from a new event loop.
agentlightning/store/memory.py Implements _LoopAwareAsyncLock and replaces prior asyncio.Lock usage in the store.
agentlightning/store/client_server.py Adds per-server lock and routes HTTP handlers through serialized delegate methods.

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

# wait_for_rollouts can block for a long time; avoid holding the lock
# so other requests can make progress while we wait.
return await method(*args, **kwargs)
with self._lock:

Copilot AI Oct 20, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a blocking threading.Lock inside an async code path will block the entire event loop when contention occurs (another request waiting to acquire the lock will hard-block the loop), defeating concurrency and risking stalls. Replace threading.Lock with an asyncio.Lock (initialized as self._lock = asyncio.Lock()) and use async with self._lock: so awaiting the method does not block other tasks from running while they wait for the lock.

Suggested change
with self._lock:
async with self._lock:

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is intentional to make _call_store_method thread-safe.

@ultmaster Yuge Zhang (ultmaster) added the bug Something isn't working label Oct 20, 2025
@ultmaster
Yuge Zhang (ultmaster) merged commit 55284f8 into main Oct 20, 2025
17 checks passed
@ultmaster
Yuge Zhang (ultmaster) deleted the codex/fix-asyncio-event-loop-error branch October 28, 2025 01:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working codex

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants