osl-pathfinder

A tiny two-way mapping between a short id and the files it owns.

pip install osl-pathfinder==1.0.0

You write the filename of each kind of file once, as a template with {field} placeholders. Pathfinder then gives you clean conversions in both directions — and groups every kind under one short id:

fields  <->  id (a short label)  <->  path (per file kind)

So you stop hand-writing the code that slices an id out of a filename, or rebuilds a filename from an id, in every script. One source of truth.

from osl_pathfinder import Pathfinder

pf = Pathfinder(templates={
    "eeg":  "/data/sub-{subject}/ses-{session}/sub-{subject}_ses-{session}_run-{run}_block-{block}_eeg.fif",
    "fmri": "/data/sub-{subject}/ses-{session}/sub-{subject}_ses-{session}_run-{run}_block-{block}_bold.nii.gz",
    "t1w":  "/data/sub-{subject}/anat/sub-{subject}_T1w.nii.gz",
}, anchor="eeg")

pf.field2id(subject="007", session="1", run="2", block="3")  # -> "007-1-2-3"
pf.id2path("007-1-2-3", "eeg")                              # -> Path(".../_eeg.fif")  (must exist)
pf.id2path("007-1-2-3", "t1w")                              # -> shared T1w (subject only)
pf.path2id(some_path, "eeg")                               # -> "007-1-2-3"
pf.id2path("007-1-2-3")                                    # all kinds -> {"eeg": Path, "fmri": Path, "t1w": Path}

Or, if you are as lazy as I am, use a compact ID and {foo} for filename text you do not care about:

from osl_pathfinder import Pathfinder

pf = Pathfinder(
    id="{subject:d}{session:1d}{run:1d}{block:1d}",  # -> 7111, 10121
    templates={
        "eeg": (
            "/data/sub-{subject}/ses-{session}/"
            "{foo}_run-{run}_block-{block}_eeg.fif"
        ),
        # The EEG and fMRI filename prefixes are intentionally untracked.
        "fmri": (
            "/data/sub-{subject}/ses-{session}/"
            "{foo}un-{run}_block-{block}_bold.nii.gz"
        ),
        "t1w": "/data/sub-{subject}/anat/sub-{subject}_T1w.nii.gz",
    },
    anchor="eeg",
)

pf.field2id(subject="7", session="1", run="2", block="3")  # -> "7123"
pf.id2path("7123", "eeg")     # existing EEG Path
pf.id2path("7123", "t1w")     # shared subject-level T1w
pf.path2id(some_path, "eeg")  # -> "7123"
pf.id2path("7123")             # -> {"eeg": Path, "fmri": Path, "t1w": Path}

Run python example.py for a complete, self-contained tour.

Why an ID?

The id is a compact, readable label for one entity (subject/session/run/block). Use it as a plot title, a dict key, a CSV column — without dragging a full path around, and without re-deriving it from a filename each time.

Custom and compact IDs

Pass id="..." to control exactly how IDs look. The compact quick start above uses id="{subject:d}{session:1d}{run:1d}{block:1d}": subject 7/session 1/run 1/block 1 becomes "7111", while subject 10/session 1/run 2/block 1 becomes "10121". The trailing fixed-width :1d fields make the separator-free ID unambiguous, so it still round-trips through id2field().

Numeric fields and canonical values

Numeric path format specifications also define how a real path is parsed. For example, {subject:03d} parses the text "007" as the integer 7, after which Pathfinder exposes the value as the string "7". The complete conversion is:

path component "sub-007"
    -> parse {subject:03d}
integer 7
    -> Pathfinder string field
{"subject": "7"}
    -> render id="{subject:d}{session:1d}{run:1d}{block:1d}"
ID "7111"

Thus a path ending in sub-007_ses-01_run-01_block-01_eeg.fif maps to "7111". This behavior is specific to numeric specifications such as :03d; a string-alignment specification such as :0>3 preserves the parsed text "007" instead. All values returned by id2field() and path2field() are strings, and IDs such as "1111" are strings rather than integers.

Use the same kind of format specification for a field everywhere it appears, both within one template and across file kinds. For example, do not casually mix numeric {subject:03d} with string-alignment {subject:0>3}: the former parses "007" to the canonical field value "7", while the latter preserves "007". Pathfinder does not reject this difference across kinds. Mix field specifications only when you deliberately understand and want the resulting canonical values.

ID ambiguity protection

The bare default joins fields with - ("007-1-2-3") only because, without a separator or fixed widths, an id can't be split back unambiguously — so supply an explicit id= (as above) whenever you want the compact form.

Ambiguity check. The constructor rejects an id template where two variable-width fields touch with no separator, because id2field couldn't recover them — e.g. "{subject:d}{session:d}{run:1d}{block:1d}" raises ValueError (subject/session are both variable-width and adjacent). Fix it by adding a separator or giving all but one a fixed width ({session:1d}).

Mixed granularity (one T1w for many blocks)

A template only uses the fields it names. t1w mentions just {subject}, so every block of a subject resolves to the same T1w file. No special-casing.

Independent path wildcards: {foo}

Ordinary placeholders are consistent fields: if {subject} appears twice in a pattern, both occurrences must match the same string. Use the reserved placeholder {foo} for unimportant text that is only a garbage bin, not an entity field. Each {foo} occurrence is independent, so repeated occurrences may match different strings:

pf = Pathfinder(
    templates={
        "raw": "/data/{foo}b-{subject:02d}/{foo}s-{session:02d}/"
               "eeg/{foo}-checker_eeg.set",
    },
    id="{subject:d}{session:1d}",
)
pf.id2path("12", "raw")  # each {foo} becomes its own `*` glob

{foo} is omitted from path2field and never contributes to the default ID. path2id and scan therefore work even when its occurrences match different strings. A {foo} kind can only resolve an existing file: require_existence=False raises because inventing a wildcard would make an output path non-reproducible. Define build-output kinds with explicit fields.

Keep dataset fields consistently named

Literal path labels may differ while the underlying Pathfinder field stays the same. For example, if fMRI calls run a scan and block a part, keep using the same {run} and {block} fields:

templates = {
    "fmri": (
        "/data/sub-{subject:03d}_ses-{session:02d}_"
        "scan-{run:02d}_part-{block:02d}/bold.nii.gz"
    ),
    "eeg": (
        "/data/sub-{subject:03d}_ses-{session:02d}_"
        "run-{run:02d}_block-{block:02d}/eeg.fif"
    ),
}

If modalities genuinely use different entity values—for example, EEG run2block1 corresponds to fMRI run1block3—Pathfinder does not hide the mismatch. Correct the dataset names, or create a soft-link layer with consistent names when the source cannot be changed.

API at a glance

Constructor

API	purpose	filesystem behavior
Pathfinder(templates, id=None, anchor=None, sep="-")	define every file-kind template and the ID format	discovers and freezes the ID cohort from existing anchor files

• templates is the required {kind: template} mapping. Path templates may be strings or PathLike values and are stored as strings; every concrete path returned by Pathfinder is a pathlib.Path.
• id selects the ID fields and formatting; if omitted, the anchor fields are joined with sep.
• anchor selects the file kind used to discover IDs and defaults to the first kind in templates insertion order. Use the finest-grained file kind as the anchor.

Properties

These are attributes, not methods: use pf.ids, not pf.ids().

Property	value	lifecycle
pf.templates	read-only {kind: template} mapping	frozen at construction
pf.anchor	kind used for ID discovery	frozen at construction
pf.id_template	format template used to build and parse IDs	frozen at construction
pf.id_fields	ordered tuple of fields encoded by the ID	frozen at construction
pf.ids	list copy of the anchor-derived ID cohort	cohort frozen at construction
pf.full_table	latest table returned by scan()	None before scanning; replaced by each scan

Conversion and lookup methods pf.A2B(A, ...)

The defaults are fuzzy=False and require_existence=True.

Every field2B() method rejects foo in its supplied fields: {foo} is only an independent wildcard written inside a path template; it is never a field value that a caller can bind.

Method	input → output	filesystem behavior
pf.field2id(fields, *, fuzzy=False) (default)	complete fields → one ID	no disk access
pf.field2id(constraints, *, fuzzy=True)	partial fields → matching IDs	filters fixed pf.ids; no rescan
pf.id2field(id)	one ID → field dictionary	no disk access
pf.id2path(id, kind, *, require_existence=True) (default)	one ID → one existing kind's Path	globs missing non-ID fields; exactly one match required
pf.id2path(id, kind, *, require_existence=False)	one ID → one output Path	no disk access; ID plus extras must supply every template field
pf.id2path(id, *, require_existence=...)	one ID → {kind: Path} for every kind	applies the selected existence mode to every kind
pf.field2path(partial_fields, kind, *, require_existence=True) (default)	partial or complete fields → one existing Path	missing fields become *; exactly one match required
pf.field2path(complete_fields, kind, *, require_existence=False)	fields complete for this kind → one output Path	no disk access; every ordinary placeholder in this kind required
pf.path2field(path, kind)	supplied path → field dictionary	parses the path; does not test existence
pf.path2id(path, kind)	supplied path → ID	parses the path; does not test existence

Exact field2id(..., fuzzy=False) requires one scalar value per field and rejects lists or other non-string iterables. Iterable values express OR matching only in fuzzy mode: session=["1", "2"] means session 1 or session 2 when fuzzy=True.

Unlike field2id(..., fuzzy=True), a partial field2path() lookup never returns a list. It returns the single matching path or raises if the pattern is missing or ambiguous.

“Complete fields” means complete for the selected kind, not complete for the whole Pathfinder: a subject-level t1w template containing only {subject} does not require session, run, or block fields.

Path lookup modes

id2path and field2path take a keyword-only require_existence boolean:

• require_existence=True (default) finds an existing file. Missing fields become glob wildcards. The pattern must match exactly one path; zero or multiple matches raise. The all-kinds id2path(id) form checks every kind and reports all missing kinds together.

• require_existence=False builds an output path. It does not inspect the filesystem. Every ordinary placeholder used by the selected kind must be supplied, and the returned Path need not exist. A template containing {foo} is rejected because an independent wildcard has no build-time value.

The all-kinds build form succeeds only when every kind can be rendered exactly and no selected template contains {foo}.

id2path accepts extra keyword fields for ordinary placeholders outside the ID, but they cannot override a field encoded by the ID. field2path instead takes every constraint in its required first mapping argument. Neither method accepts foo; unknown fields and misspelled flags such as req_exist=False raise TypeError.

Validation method

Method	input → output	filesystem behavior
pf.scan(csv_path=None)	fixed IDs × kinds → validation table	checks current disk state; prints the table and optionally writes CSV

scan() validates every configured kind for the fixed IDs discovered during construction; it does not discover new IDs or change the anchor. Missing files are normal validation rows, while patterns matching multiple files are collected and raised together. Existing rows contain a concrete Path; missing or ambiguous rows contain path=None and retain the expected pattern with unresolved portions shown as *.

The latest table is available as pf.full_table. It is only a scan cache: path-lookup methods always inspect the filesystem directly.

Module-level split helpers

These functions are not methods on pf, but functions using it.

Function	input → output	filesystem behavior
split_pf(pf, proportions, unit=None, seed=42, json_path=None)	fixed IDs → reproducible subset lists	optionally saves an atomic JSON replay record
split_pf_from_file(pf, json_path)	saved record + current fixed cohort → the same subset lists	reads and validates the JSON record

Pipelines: define once, read and write

Construction discovers the fixed ID cohort from the anchor, so create the Pathfinder after the anchor files are available. Define every kind once in one shared module and import that pf in each pipeline step. Read inputs with the default existence check; build outputs with require_existence=False:

# pathfinder.py
pf = Pathfinder(templates={"raw": ..., "src": ..., "power": ...}, id="...")

# 1.py: src does not exist yet
for file_id in pf.ids:
    raw_path = pf.id2path(file_id, "raw")
    src_path = pf.id2path(file_id, "src", require_existence=False)
    save(transform(raw_path), src_path)

# 2.py: src now exists
for file_id in pf.ids:
    raw_path = pf.id2path(file_id, "raw")
    src_path = pf.id2path(file_id, "src")

Reproducible development/evaluation subsets

The helper returns one sorted ID list per requested proportion and does not modify pf. Pass json_path= when you also want a replayable JSON split record:

from osl_pathfinder import split_pf, split_pf_from_file

split_path = "results/natview_dev_eval_split.json"
dev_ids, eval_ids = split_pf(
    pf,
    proportions=[0.8, 0.2],
    unit=["subject"],       # keep every session/run/block of a subject together
    seed=42,                # fixed by default; record it with your results
    json_path=split_path,
)

# Later, or on another machine with the same Pathfinder cohort:
dev_ids, eval_ids = split_pf_from_file(pf, split_path)

• Per-file splitting: unit=None or unit=[] makes every discovered file ID independently splittable.

• Grouped splitting: unit=["subject", "block"] keeps every session and run belonging to the same subject/block together. Unit names must exactly match fields in pf.id_fields.

• Percentages apply to units: an indivisible group is assigned as a whole, so grouped subsets may contain slightly different file counts when groups have different sizes.

• Random sampling is isolated: IDs and grouping keys are sorted before a local NumPy RandomState is used. Splitting does not change NumPy's global random state.

• Exact replay needs the saved result: a seed and unchanged sorted IDs are normally repeatable, but NumPy documents machine/build caveats. Pin NumPy and save the JSON split record or returned ID lists for cross-machine replay. NumPy is installed as a normal Pathfinder dependency.

The saved JSON is a small replay record, not a second path manifest. Version 1.0 freezes the following split-record schema:

key	required value
format	string, exactly "osl-pathfinder.split"
version	integer, exactly 1
id_fields	non-empty array of unique strings
unit	array of unique strings; empty means per-ID splitting
proportions	non-empty array of finite, non-negative numbers summing to 1
seed	integer from 0 to 2**32 - 1, or null
input_ids	sorted array of unique string IDs in the input cohort
shuffled_keys	array of unique, non-empty string arrays
counts	non-empty array of non-negative integers, one per subset

Replay uses input_ids, shuffled_keys, and counts directly, then checks the current Pathfinder's cohort and grouping fields. It does not depend on rerunning the random generator. A changed cohort or incompatible ID definition raises an error instead of silently changing the split. Required keys and meanings are strict. Unknown additional keys are allowed and ignored during replay, so a project or later compatible writer can attach optional provenance metadata.

Split records are written through a temporary file in the target directory and atomically replaced into place. The parent directory must already exist.

For developers

pip install -e '.[test]'  # editable checkout, for local development + pytest
pytest -q