I believe the following is true, by my own conceptual model, but I have not fully verified it in the code. Caveat lector: one of the obligations intended by this ticket is to verify these statements and their consequences. All below should be read as implicitly holding this qualification unless otherwise noted.
Currently, a job may specify its targets by wildcard (all sleds), cubby number (sled 16), or baseboard ID. These specifications are part of the job request and the signed job. However, at the time the signed job is submitted to the rack via a sush server, the former two (wildcard and cubby) are resolved via DDM to baseboard IDs. The gossip network then transmits jobs with fully specified baseboards, and a sled's own sush server determines whether to execute a given job by mere string-equality matching on the specified baseboard ID (it never consults its own cubby number, or matches against a wildcard).
This "middle binding" has the desirable property that — for all jobs so-specified — a sled added to the rack after the job is submitted will never run jobs submitted prior to its addition to the rack. Additionally, if the sled is unreachable by DDM, it will also never run those jobs (since DDM could not resolve the baseboard ID at job submission time, it will be omitted from the concrete baseboard list in the gossip set). This holds even under arbitrary restart of the sled, regardless of any other persistent state.
However: a job addressed (partially or fully) by exact baseboard ID does not get the happy-accidental benefit of this! If you sign a job that addresses a particular baseboard ID, DDM is not invoked to resolve that baseboard ID at job submission time, which means it enters the gossip network regardless of whether the sled was reachable at the time the job was submitted — and even regardless of whether the sled was ever added to the rack. If the sled ever joins gossip while the session containing that job is unterminated (which is an indefinite period of time, since sessions do not time out), it will "catch up" by executing every single job in the most-recent session. In principle, this could happen hours, days, or weeks after the support session is "finished" from the human perspective, provided that the session is not affirmatively terminated by human action.
I believe that we should make this behavior uniform and precise, so that an operator can rely on it as a general rule: "a job only runs on the sleds which are resolvable by DDM on the sled to which it was submitted, at the time it was submitted". This does not mean that sleds always run jobs immediately, or do not "catch up" on un-executed jobs after a reboot. It merely prevents jobs from being initially addressed to a sled that is not resolvable at time of submission. To restate the above, this is already true of jobs addressed by wildcard or cubby number (I believe!); we should cause it to be true of jobs addressed by baseboard ID, by requiring DDM to resolve those sleds as well, and filtering them out from the gossiped job if they cannot be reached. As part of this work, we should also ensure that if the sled in a given cubby cannot be resolved by DDM, we filter it out from the list of addressed baseboard IDs and proceed, rather than, say, throwing an error. This would make behavior consistent across all ways of addressing sleds.
Why does this matter so much? The footgun seems at first like you have to be trying to fire it, but I disagree: we already know that customers, support, or manufacturing may want to script sush across many racks. It would be a natural thing to write a script which gathers all the baseboard IDs and addresses sush jobs directly to those baseboard IDs. It would be the furthest thing from the operator's mind that doing this introduces the hazard that those jobs would linger, unexecuted, indefinitely, then suddenly execute all at once on boot for all sleds that were not reachable at the time the job was submitted — doubly so if they previously had experience with the by-cubby addressing functioning in precisely the opposite manner. We can prevent this footgun from existing, so we should.
I believe the following is true, by my own conceptual model, but I have not fully verified it in the code. Caveat lector: one of the obligations intended by this ticket is to verify these statements and their consequences. All below should be read as implicitly holding this qualification unless otherwise noted.
Currently, a job may specify its targets by wildcard (all sleds), cubby number (sled 16), or baseboard ID. These specifications are part of the job request and the signed job. However, at the time the signed job is submitted to the rack via a sush server, the former two (wildcard and cubby) are resolved via DDM to baseboard IDs. The gossip network then transmits jobs with fully specified baseboards, and a sled's own sush server determines whether to execute a given job by mere string-equality matching on the specified baseboard ID (it never consults its own cubby number, or matches against a wildcard).
This "middle binding" has the desirable property that — for all jobs so-specified — a sled added to the rack after the job is submitted will never run jobs submitted prior to its addition to the rack. Additionally, if the sled is unreachable by DDM, it will also never run those jobs (since DDM could not resolve the baseboard ID at job submission time, it will be omitted from the concrete baseboard list in the gossip set). This holds even under arbitrary restart of the sled, regardless of any other persistent state.
However: a job addressed (partially or fully) by exact baseboard ID does not get the happy-accidental benefit of this! If you sign a job that addresses a particular baseboard ID, DDM is not invoked to resolve that baseboard ID at job submission time, which means it enters the gossip network regardless of whether the sled was reachable at the time the job was submitted — and even regardless of whether the sled was ever added to the rack. If the sled ever joins gossip while the session containing that job is unterminated (which is an indefinite period of time, since sessions do not time out), it will "catch up" by executing every single job in the most-recent session. In principle, this could happen hours, days, or weeks after the support session is "finished" from the human perspective, provided that the session is not affirmatively terminated by human action.
I believe that we should make this behavior uniform and precise, so that an operator can rely on it as a general rule: "a job only runs on the sleds which are resolvable by DDM on the sled to which it was submitted, at the time it was submitted". This does not mean that sleds always run jobs immediately, or do not "catch up" on un-executed jobs after a reboot. It merely prevents jobs from being initially addressed to a sled that is not resolvable at time of submission. To restate the above, this is already true of jobs addressed by wildcard or cubby number (I believe!); we should cause it to be true of jobs addressed by baseboard ID, by requiring DDM to resolve those sleds as well, and filtering them out from the gossiped job if they cannot be reached. As part of this work, we should also ensure that if the sled in a given cubby cannot be resolved by DDM, we filter it out from the list of addressed baseboard IDs and proceed, rather than, say, throwing an error. This would make behavior consistent across all ways of addressing sleds.
Why does this matter so much? The footgun seems at first like you have to be trying to fire it, but I disagree: we already know that customers, support, or manufacturing may want to script sush across many racks. It would be a natural thing to write a script which gathers all the baseboard IDs and addresses sush jobs directly to those baseboard IDs. It would be the furthest thing from the operator's mind that doing this introduces the hazard that those jobs would linger, unexecuted, indefinitely, then suddenly execute all at once on boot for all sleds that were not reachable at the time the job was submitted — doubly so if they previously had experience with the by-cubby addressing functioning in precisely the opposite manner. We can prevent this footgun from existing, so we should.