docs(examples): add marker-driven instrument folder uploader - #582
Closed
Caushi wants to merge 1 commit into
Closed
Conversation
A cron-friendly uploader for instrument output: each finished run folder becomes one workunit, uploaded once its marker file appears. Three details are load-bearing: - The folder -> workunit-id memory is a sidecar state file outside the watched tree. Keeping it inside the run folder would upload it as a resource, and since its content changes after every upload its md5 would change too, re-uploading it on every scan. A config guard rejects a state_dir inside watch_dir. - The state file is written atomically (temp + os.replace). A bare write_text truncates first, so a crash mid-write would leave an empty file, which reads as "not yet uploaded" and creates a duplicate workunit for the run. - A new run uploads with force=True. B-Fabric's duplicate check is container-wide, so an unrelated run that produced byte-identical content would otherwise suppress this run's copy and leave it with no workunit at all -- and with no id to remember, stranded on the create path forever. Instrument runs are events, not content. The reuse path keeps dedup, which is what makes repeated scans cheap. The operator's marker stays in the run folder (that is where the operator is when the run ends) and is excluded via exclude_names.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A cron-friendly uploader for instrument output: each subfolder of a watched directory becomes one workunit, uploaded once its marker file appears (operator presses "done"). Configuration is one YAML file per machine, so a fleet shares the script and differs only in config.
Three load-bearing details
Sidecar state file. The folder → workunit-id memory lives outside the watched tree. Inside the run folder it would be uploaded as a resource — and since its content changes after every upload, its md5 would change too, re-uploading it on every single scan and defeating the dedup the design depends on. A config guard rejects a
state_dirinsidewatch_dir(resolved, so symlinks and..cannot smuggle it back in).Atomic state write.
write_texttruncates before writing, so a crash or full disk mid-write leaves an empty file — which reads back as "not yet uploaded" and creates a duplicate workunit for the run, exactly the failure the state file exists to prevent. Written temp-then-os.replaceinstead.A new run uploads with
force=True. B-Fabric's duplicate check is container-wide, not per-run. An unrelated run that happened to produce byte-identical content (a calibration file, a blank) would otherwise suppress this run's copy, leaving the folder with no workunit of its own — and with no id to remember, stranded on the create path for every future scan. Instrument runs are events, not content: two runs producing identical bytes are still two runs, and the operator placing the marker is the deliberate assertion that this is a genuine acquisition. The reuse path keeps dedup, which is what makes repeated scans cheap and is whyforceis not simply on everywhere —force=Trueon an unchanged folder 409s on the server's per-workunit path uniqueness.Verified end to end
Against a live B-Fabric + tus server (application 588, container 403), two-level nested folder:
AVAILABLEResources kept their relative names (
sub/deeper/deep.raw), the marker never appeared as a resource, and exactly one workunit existed per run. The duplicate-content case was checked separately: a folder whose content already existed elsewhere in the container now gets its own workunit and a persisted id, where previously it was stranded.Open question for reviewers
The marker stays inside the run folder — that is where the operator is when the run ends, and a done-flag in a distant directory is one that gets skipped or misspelled. It is excluded via
exclude_names, so it is never uploaded. Say the word if you would rather it lived instate_dirtoo.