Skip to content

Claim a job before running it, so two runners never execute it at once - #660

Merged
daylioti merged 1 commit into
masterfrom
jobs-claim-lock
Nov 20, 2026
Merged

daylioti merged 1 commit into
masterfrom
jobs-claim-lock

Conversation

@daylioti

@daylioti daylioti commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

A queued job and a running job are the same thing on disk: a directory in _jobs_todo. A job stays there for its whole run and moves to _jobs_done only after the handler returns. Nothing in the jobs module marks a job as taken; there are only is_dir() checks, which are check-then-act. So any two runners can execute the same job side by side:

  • a request that queues a job and runs it inline with JobsFactory::runJob();
  • a scheduled processQueue() pass that lists _jobs_todo while that request is still running;
  • several queue passes, in different processes or containers sharing the job directories.

The second run repeats the handler's side effects: a second partner call, a second email, a second booking write. Each run also saves its own copy of the job, so one run's attempts entry can overwrite the other's.

Change

  • Job::claim() / Job::release(). The claim is an exclusive non-blocking flock() on <queue dir>/.locks/<job>.lock.
    • flock is per inode and held by the kernel, so it excludes other processes and other containers that mount the same directory.
    • It is dropped when the holder dies, however it dies (killed worker, OOM, timeout).
    • .locks is dot-prefixed, so the node index and the queue listings ignore it. It sits next to the jobs, not inside a job's directory, so taking a claim can never recreate a directory that a move (copy, then delete) is removing.
  • Job::run() runs the handler only under the claim. Every caller is covered: runJob(), processQueue(), and site code that loops over the queue itself.
    • A runner that cannot take the claim returns FALSE without recording an attempt. It logs Job {job} is claimed by another runner, skipped at info level on the jobs channel.
    • A job that was already claimed by the caller stays claimed after run() returns.
  • The claim re-reads the job from data.json on disk, not through the node index. The previous holder may have written it from another process or container, and the local index can still be catching up with that write. A job that is gone, or a directory without a document (half-way through a move), is not claimed.
  • Lock files are unlinked on release, while still locked. A runner that opened the file before the unlink and gets the lock afterwards notices that the path now names a different inode, and starts over on the current file.
  • .locks is created world-writable and sticky (01777), like /tmp. A root CLI run that happens to create it first cannot lock the site user out. A lock file the user cannot write is opened read-only, which flock() accepts.
    • If no lock file can be opened at all, the job runs unclaimed, as before this change, and a warning is logged. A permission problem doesn't stop the queue.
  • claim() and release() are public, so code that changes a queued job outside run() can hold the claim too. Merging new data into a job that has not started is one case.
  • __destruct() releases, so a claim never outlives its object.

Behaviour that does not change: the handler contract, the attempts limit, and the moves to _jobs_done / _jobs_unknown. A job whose runner died stays in _jobs_todo and runs on a later pass, still bounded by the attempts counter.

Testing

  • New parity test qdb/tests/quanta/11_job_claim.php (32 assertions), run in noext, fallback and daemon modes:
    • A held claim keeps a second runner out: no handler run, no attempt recorded. After release the job runs exactly once, moves to _jobs_done and leaves no lock file.
    • processQueue() skips a claimed job and runs the others. The next pass runs it once it is released.
    • A job completed on disk by another runner (move not done yet) is not run again on stale in-memory data, and its move is completed. A job moved away between listing and claim is not run.
    • A lock file left behind with no holder doesn't block the job. A claim is released when its object goes away.
    • A holder in another process blocks the job. After that process is killed with SIGKILL, the job runs.
  • Full parity suite on the image built from master, with this branch's src/modules/jobs mounted over it: all files pass in all three modes.
  • Race reproduction, outside the suite. Real Job.class.php with a stubbed node layer; 8 queue passes and 40 inline runs racing over 40 jobs in separate processes:
    • On master: 58 executions, 7 jobs run more than once, and lost attempts writes.
    • With this change: 40 executions, no duplicates, every job with exactly one attempt, no leftover lock files.
  • End to end in an application image, with qdbd running and _jobs_todo / _jobs_done on separate mounts: 6 concurrent processQueue() processes plus inline runJob() calls over 30 jobs. 30 executions, no duplicates, all moved to _jobs_done with one attempt, and no directory left without data.json.

A queued job and a running job are the same thing on disk, a directory in
_jobs_todo, so nothing stopped a request that runs its job inline and a
queue pass started meanwhile from executing the same job side by side.

Job::run() now runs the handler only while holding the job's claim: an
exclusive non-blocking flock() on _jobs_todo/.locks/<job>.lock. A runner
that cannot take it skips the job. Job::claim() / Job::release() are public
so callers that change a queued job outside run() can hold the claim too.

After taking the claim the job is re-read from data.json on disk, since the
previous holder may have written it from another process or container that
the local index has not caught up with. A job that is gone, or a directory
without a document (mid-move), is skipped.
@daylioti
daylioti merged commit 7c00f86 into master Nov 20, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant