A running session on a sush server may have a number of jobs in its queue which have not yet launched, either because their predecessing jobs have not yet been gossiped to it (required for hash chaining), or merely because their predecessing jobs are still in progress and the concurrency of the jobs were not set to allow them to launch yet.
At any time, we may learn through gossip that our session has been ended by a proactive operator action, or superseded by another new session whose start causally after ours — or, most rarely, mutually annihilated by a causally incomparable new session — and we must determine what to do with our queue of jobs from our old session.
The un-executed queue is definitively causally prior to a superseding session, and causally incomparable to an annihilating one, so it would in principle be defensible for us to rule any of these ways:
- All jobs in a session run to completion, even if we know before even launching them that the session has already ended.
- All queued jobs for a session get dropped from the queue as soon as we learn that the session has ended.
- (We could also, orthogonally, rule that running jobs are actively killed at session termination, but I think we should not do this for principle of least surprise.)
I believe that the right choice is option 2: as soon as we know that the session has ended for any reason (termination by operator, supersession by a causally posterior new session, or annihilation by an incomparable one), we should cease starting new jobs for the session but we should allow its already-started jobs to finish normally, and report their status as usual.
Mechanically, this would look like setting a "skip" flag and fast-forwarding through the remainder of the job start messages in the causal queue, until we hit the start of the next session. We might choose to report a special "skipped" status for such jobs, so that there's a record that we received the job start message but declined to execute due to the end of the session.
A running session on a sush server may have a number of jobs in its queue which have not yet launched, either because their predecessing jobs have not yet been gossiped to it (required for hash chaining), or merely because their predecessing jobs are still in progress and the concurrency of the jobs were not set to allow them to launch yet.
At any time, we may learn through gossip that our session has been ended by a proactive operator action, or superseded by another new session whose start causally after ours — or, most rarely, mutually annihilated by a causally incomparable new session — and we must determine what to do with our queue of jobs from our old session.
The un-executed queue is definitively causally prior to a superseding session, and causally incomparable to an annihilating one, so it would in principle be defensible for us to rule any of these ways:
I believe that the right choice is option 2: as soon as we know that the session has ended for any reason (termination by operator, supersession by a causally posterior new session, or annihilation by an incomparable one), we should cease starting new jobs for the session but we should allow its already-started jobs to finish normally, and report their status as usual.
Mechanically, this would look like setting a "skip" flag and fast-forwarding through the remainder of the job start messages in the causal queue, until we hit the start of the next session. We might choose to report a special "skipped" status for such jobs, so that there's a record that we received the job start message but declined to execute due to the end of the session.