Repository navigation
Claim a job before running it, so two runners never execute it at once - #660
Merged
Merged
Conversation
A queued job and a running job are the same thing on disk, a directory in _jobs_todo, so nothing stopped a request that runs its job inline and a queue pass started meanwhile from executing the same job side by side. Job::run() now runs the handler only while holding the job's claim: an exclusive non-blocking flock() on _jobs_todo/.locks/<job>.lock. A runner that cannot take it skips the job. Job::claim() / Job::release() are public so callers that change a queued job outside run() can hold the claim too. After taking the claim the job is re-read from data.json on disk, since the previous holder may have written it from another process or container that the local index has not caught up with. A job that is gone, or a directory without a document (mid-move), is skipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A queued job and a running job are the same thing on disk: a directory in
_jobs_todo. A job stays there for its whole run and moves to_jobs_doneonly after the handler returns. Nothing in the jobs module marks a job as taken; there are onlyis_dir()checks, which are check-then-act. So any two runners can execute the same job side by side:JobsFactory::runJob();processQueue()pass that lists_jobs_todowhile that request is still running;The second run repeats the handler's side effects: a second partner call, a second email, a second booking write. Each run also saves its own copy of the job, so one run's
attemptsentry can overwrite the other's.Change
Job::claim()/Job::release(). The claim is an exclusive non-blockingflock()on<queue dir>/.locks/<job>.lock..locksis dot-prefixed, so the node index and the queue listings ignore it. It sits next to the jobs, not inside a job's directory, so taking a claim can never recreate a directory that a move (copy, then delete) is removing.Job::run()runs the handler only under the claim. Every caller is covered:runJob(),processQueue(), and site code that loops over the queue itself.FALSEwithout recording an attempt. It logsJob {job} is claimed by another runner, skippedat info level on thejobschannel.run()returns.data.jsonon disk, not through the node index. The previous holder may have written it from another process or container, and the local index can still be catching up with that write. A job that is gone, or a directory without a document (half-way through a move), is not claimed..locksis created world-writable and sticky (01777), like/tmp. A root CLI run that happens to create it first cannot lock the site user out. A lock file the user cannot write is opened read-only, whichflock()accepts.claim()andrelease()are public, so code that changes a queued job outsiderun()can hold the claim too. Merging new data into a job that has not started is one case.__destruct()releases, so a claim never outlives its object.Behaviour that does not change: the handler contract, the attempts limit, and the moves to
_jobs_done/_jobs_unknown. A job whose runner died stays in_jobs_todoand runs on a later pass, still bounded by the attempts counter.Testing
qdb/tests/quanta/11_job_claim.php(32 assertions), run in noext, fallback and daemon modes:_jobs_doneand leaves no lock file.processQueue()skips a claimed job and runs the others. The next pass runs it once it is released.src/modules/jobsmounted over it: all files pass in all three modes.Job.class.phpwith a stubbed node layer; 8 queue passes and 40 inline runs racing over 40 jobs in separate processes:attemptswrites._jobs_todo/_jobs_doneon separate mounts: 6 concurrentprocessQueue()processes plus inlinerunJob()calls over 30 jobs. 30 executions, no duplicates, all moved to_jobs_donewith one attempt, and no directory left withoutdata.json.