Skip to content

docs(examples): add marker-driven instrument folder uploader - #582

Closed
Caushi wants to merge 1 commit into
feat/upload-exclude-namesfrom
feat/instrument-folder-uploader
Closed

docs(examples): add marker-driven instrument folder uploader#582
Caushi wants to merge 1 commit into
feat/upload-exclude-namesfrom
feat/instrument-folder-uploader

Conversation

@Caushi

@Caushi Caushi commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

A cron-friendly uploader for instrument output: each subfolder of a watched directory becomes one workunit, uploaded once its marker file appears (operator presses "done"). Configuration is one YAML file per machine, so a fleet shares the script and differs only in config.

Stacked on #581 (needs its exclude_names). Review that one first; this PR retargets to main once it merges.

Three load-bearing details

Sidecar state file. The folder → workunit-id memory lives outside the watched tree. Inside the run folder it would be uploaded as a resource — and since its content changes after every upload, its md5 would change too, re-uploading it on every single scan and defeating the dedup the design depends on. A config guard rejects a state_dir inside watch_dir (resolved, so symlinks and .. cannot smuggle it back in).

Atomic state write. write_text truncates before writing, so a crash or full disk mid-write leaves an empty file — which reads back as "not yet uploaded" and creates a duplicate workunit for the run, exactly the failure the state file exists to prevent. Written temp-then-os.replace instead.

A new run uploads with force=True. B-Fabric's duplicate check is container-wide, not per-run. An unrelated run that happened to produce byte-identical content (a calibration file, a blank) would otherwise suppress this run's copy, leaving the folder with no workunit of its own — and with no id to remember, stranded on the create path for every future scan. Instrument runs are events, not content: two runs producing identical bytes are still two runs, and the operator placing the marker is the deliberate assertion that this is a genuine acquisition. The reuse path keeps dedup, which is what makes repeated scans cheap and is why force is not simply on everywhere — force=True on an unchanged folder 409s on the server's per-workunit path uniqueness.

Verified end to end

Against a live B-Fabric + tus server (application 588, container 403), two-level nested folder:

Scan Result Workunit
initial uploaded 3, skipped 0, failed 0 created, AVAILABLE
unchanged re-scan uploaded 0, skipped 3 reused
new file in a new subdir uploaded 1, skipped 3 reused

Resources kept their relative names (sub/deeper/deep.raw), the marker never appeared as a resource, and exactly one workunit existed per run. The duplicate-content case was checked separately: a folder whose content already existed elsewhere in the container now gets its own workunit and a persisted id, where previously it was stranded.

Open question for reviewers

The marker stays inside the run folder — that is where the operator is when the run ends, and a done-flag in a distant directory is one that gets skipped or misspelled. It is excluded via exclude_names, so it is never uploaded. Say the word if you would rather it lived in state_dir too.

A cron-friendly uploader for instrument output: each finished run folder
becomes one workunit, uploaded once its marker file appears.

Three details are load-bearing:

- The folder -> workunit-id memory is a sidecar state file outside the watched
  tree. Keeping it inside the run folder would upload it as a resource, and
  since its content changes after every upload its md5 would change too,
  re-uploading it on every scan. A config guard rejects a state_dir inside
  watch_dir.

- The state file is written atomically (temp + os.replace). A bare write_text
  truncates first, so a crash mid-write would leave an empty file, which reads
  as "not yet uploaded" and creates a duplicate workunit for the run.

- A new run uploads with force=True. B-Fabric's duplicate check is
  container-wide, so an unrelated run that produced byte-identical content
  would otherwise suppress this run's copy and leave it with no workunit at
  all -- and with no id to remember, stranded on the create path forever.
  Instrument runs are events, not content. The reuse path keeps dedup, which
  is what makes repeated scans cheap.

The operator's marker stays in the run folder (that is where the operator is
when the run ends) and is excluded via exclude_names.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant