Skip to content

fix: prevent duplicate startup embedding jobs across workers#1286

Open
tarjan1 wants to merge 2 commits into
dataease:mainfrom
tarjan1:fix/prevent-duplicate-startup-embedding-jobs-across-workers
Open

fix: prevent duplicate startup embedding jobs across workers#1286
tarjan1 wants to merge 2 commits into
dataease:mainfrom
tarjan1:fix/prevent-duplicate-startup-embedding-jobs-across-workers

Conversation

@tarjan1

@tarjan1 tarjan1 commented Jul 22, 2026

Copy link
Copy Markdown

Summary

Prevent duplicate startup embedding initialization when SQLBot runs with
multiple Uvicorn worker processes.

Problem

Uvicorn executes the FastAPI lifespan independently in every worker process.
As a result, each worker could submit the same terminology, data-training,
table, and datasource embedding initialization jobs during application startup.

These jobs load embedding models and perform CPU-intensive calculations.
Running them repeatedly across workers can consume significant CPU and memory,
slow down application startup, and affect other services on the same host.

Process-local synchronization cannot prevent duplication across workers.

Solution

  • Use a PostgreSQL session-level advisory lock so that only one worker submits
    startup embedding jobs.
  • Reuse SQLBot's existing PostgreSQL dependency instead of introducing Redis,
    which is not provisioned by the current default deployment.
  • Release the lock and its dedicated database connection during application
    shutdown.

Known limitation

PostgreSQL advisory locks are local to a database. This solution coordinates
workers only when they connect to the same PostgreSQL database.

If SQLBot workers use different databases or PostgreSQL instances, they will
not share the same lock. Such deployments would require a shared external
coordinator, such as Redis.

Verification

Automated tests

python -m pytest tests/test_distributed_lock.py -v

9 passed

The tests cover:

  • Stable advisory-key generation
  • Successful lock acquisition
  • Lock contention between workers
  • SQLAlchemy acquisition errors
  • Lock release and connection cleanup
  • Cleanup when unlocking fails
  • Single acquisition attempt per worker
  • Startup task execution only in the lock-holding worker
  • Worker state reset during release

The changed files also pass Ruff lint, Ruff format, and git diff --check.

Multi-worker verification

The application was started against PostgreSQL 17.10 with:

python -m uvicorn main:app --host 127.0.0.1 --port 8000 --workers 2

Observed results:

  • Both worker processes completed application startup.
  • Exactly one worker acquired the startup lock.
  • The other worker skipped startup embedding initialization.
  • An independent PostgreSQL session could not acquire the same lock while
    SQLBot was running.
  • The lock became available again after the worker processes stopped.

Scope

This change does not modify API contracts or database schemas.

Closes #1285

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Duplicate startup embedding jobs when using multiple Uvicorn workers

1 participant