A content-addressed, cached, parallel pipeline runner for reproducible ML workflows, in the spirit of DVC and Bazel. You declare stages (in YAML or with Python decorators), and dagcache hashes each stage's code, parameters and input data into a cache key. It then runs only what changed, restores everything else from a local or S3 cache, and can tell you why a stage is stale and what produced any artifact.
$ dagcache status # after editing train.C in params.yaml
load up-to-date
...
train stale param train.C changed: 1.0 → 0.05
evaluate pending upstream stage 'train' is stale
Re-running a whole ML pipeline after every small change wastes time, and timestamps-based tools cannot tell you why something is stale. dagcache keys every stage on what it actually depends on, so results are reused whenever the same inputs reappear.
ML engineers and data scientists whose pipeline is a handful to a few hundred scripts (load → split → featurize → train → evaluate) and who want:
- no recomputation after a revert, a branch switch, or a fresh clone (results come from a shared cache);
- an honest answer to "is this model stale, and why?";
- provenance for every artifact: which code, which params, which data;
- local-first operation: a single
pip install, no server, and optionally S3.
It is a compact implementation (about 4,600 lines of typed Python), not a replacement for DVC in production. The Limitations section says where it falls short.
| Stages | shell commands in dagcache.yaml, or Python functions with @pipeline.stage(...); the DAG is inferred from deps/outs (files or directories) |
| Cache keys | SHA-256 over code, declared params, input content, output declaration, declared env vars, and a salt. See docs/HASHING.md |
| Code hashing | .py files and functions hashed by normalised AST, so comments and formatting are ignored. Decorated functions include, transitively, the local helpers and constants they reference (in their bodies or as default-argument values) and the versions of the libraries they use |
| Transitive invalidation + early cutoff | a change propagates downstream through content, so an upstream that re-runs but writes identical bytes stops the cascade |
| Caches | local content-addressed store; remote on a shared directory or any S3-compatible service; push/pull/autopush; copy or hard-link restore |
| Parallelism | independent stages run concurrently (-j N), bounded, fail-fast or --keep-going |
| Concurrency safety | flock-based repo/stage/lock-file locks: concurrent repro processes run each stage once; gc waits for runs |
| status | per-stage state (up-to-date, cached, stale, pending, always-run, error) plus the reason: which param, file, function, library or dependency changed |
| lineage | trace any artifact back to source data, as a text tree, JSON, mermaid or DOT |
| gc | mark-and-sweep from the lock file (+ extra lock files, + recently used) |
| Integrity | every object verified on write and read; manifests self-verify; cache verify |
flowchart LR
def["dagcache.yaml / pipeline.py"] --> pipe["Pipeline<br/>(DAG from deps/outs)"]
params["params.yaml"] --> keys
pipe --> keys["KeyBuilder<br/>code + params + input digests"]
hash["Hasher + stat cache"] --> keys
keys --> plan["status: plan()"]
keys --> run["Runner + DagExecutor<br/>(-j N, per-stage flock)"]
run -->|hit| restore["restore outputs"]
run -->|miss| exec["run stage"] --> store["store outputs + manifest"]
restore & store --> lock["dagcache.lock"]
store --> local[(".dagcache/cache<br/>objects + actions")]
local <-->|push / pull / autopush| remote[("remote: dir or S3")]
lock --> gc["gc (mark & sweep)"]
local --> lineage["lineage"]
Each stage execution writes an action manifest, addressed by its cache key.
The manifest records the key components, the content digests of the outputs,
and the keys of the upstream producers. dagcache.lock (committed with your
code) holds the current manifest for each stage. Details, algorithms and
complexity are in docs/ARCHITECTURE.md.
Requires Python 3.12 and uv on Linux or macOS.
git clone https://github.com/ishaan1732/dagcache.git && cd dagcache
make check # format, lint, mypy --strict, 165 tests + coverage gate
make demo # the walkthrough below, offline, in build/demo/
A pipeline in YAML (examples/wine/dagcache.yaml, abridged):
params: params.yaml
stages:
split:
cmd: >-
python src/split.py --data data/raw.csv --out-dir data/split
--test-size ${split.test_size} --seed ${split.seed}
code: [src/split.py, src/common.py]
deps: [data/raw.csv]
outs: [data/split/train.csv, data/split/test.csv]
train:
cmd: python src/train.py --features data/features --out models/model.pkl --C ${train.C}
code: [src/train.py]
deps: [data/features] # a directory output of `featurize`
outs: [models/model.pkl]
The same idea with decorators (examples/decorators/pipeline.py):
from dagcache import Pipeline, StageContext
from helpers import zscore # a local module: edits to it invalidate `normalize`
pipeline = Pipeline("decorators")
@pipeline.stage(deps=["data/samples.json"], outs=["data/normalized.json"])
def normalize(ctx: StageContext) -> None:
values = json.loads(ctx.path("data/samples.json").read_text())
ctx.path("data/normalized.json").write_text(json.dumps(zscore(values)))
Commands:
dagcache repro [STAGE...] [-j N] [--force] [--keep-going] [--dry-run]
dagcache status [--json] [--check] # --check: exit 1 if anything needs work (CI)
dagcache dag [--format text|mermaid|dot] [--status]
dagcache lineage PATH [--format text|json|mermaid|dot]
dagcache gc [--dry-run] [--keep-days N] [--keep-lock OTHER.lock] [--remote]
dagcache push|pull [STAGE...]
dagcache cache stats|verify
dagcache -f pipeline.py:pipeline <command> # Python-defined pipelines
Remote cache: set cache.remote: s3://bucket/prefix (plus cache.s3.endpoint_url
for MinIO/SeaweedFS/R2) or a directory path, or export DAGCACHE_REMOTE. S3 needs
pip install 'dagcache[s3]' and the standard AWS credential chain. See .env.example.
Output of make demo, pasted from a real run (some sections trimmed, marked ...).
The pipeline is a scikit-learn classifier on the UCI Wine data bundled with
scikit-learn (see DATA.md):
=== 3. First run (4 parallel workers): everything runs ====================
$ dagcache repro -j 4
repro: 7 stage(s), jobs=4
| load: wrote 178 rows x 13 features to data/raw.csv
ran load 1.16s 6d5f61c530 (never run)
...
| evaluate: held-out metrics: {'accuracy': 0.963, 'macro_f1': 0.9619, 'n_test': 54}
ran evaluate 0.79s 692f1d80cc (never run)
done in 4.38s: 7 ran, 0 restored, 0 up-to-date, 0 failed, 0 not run (max parallel 2)
=== 4. Second run: nothing to do ==========================================
$ dagcache repro
...
done in 0.01s: 0 ran, 0 restored, 7 up-to-date, 0 failed, 0 not run (max parallel 2)
=== 5. Change a hyper-parameter: only train + evaluate are stale ==========
# edit wine/params.yaml: 'C: 1.0' -> 'C: 0.05'
$ dagcache status
load up-to-date
profile up-to-date
split up-to-date
featurize up-to-date
train stale param train.C changed: 1.0 → 0.05
baseline up-to-date
evaluate pending upstream stage 'train' is stale
2 of 7 stage(s) need work
=== 6. Revert it: results come back from the cache, nothing recomputes ====
# edit wine/params.yaml: 'C: 0.05' -> 'C: 1.0'
$ dagcache repro --no-output
...
restored train 0.00s 2e986e92ee (param train.C changed: 0.05 → 1.0)
restored evaluate 0.00s 692f1d80cc (dependency models/model.pkl changed (0057f627 → 4219e2bf))
done in 0.01s: 0 ran, 2 restored, 5 up-to-date, 0 failed, 0 not run (max parallel 2)
=== 7. Comment-only code edit and an unused param: still up to date =======
...
0 of 7 stage(s) need work
=== 8. Early cutoff: load re-runs but writes identical bytes ==============
# edit wine/src/load.py: 'print(f"wrote' -> 'print(f"exported'
$ dagcache repro --no-output
ran load 0.74s 86602c67ca (code file src/load.py changed (fffd8f5b → e304ad0b))
up-to-date profile 5fb5fdde62
...
done in 0.81s: 1 ran, 0 restored, 6 up-to-date, 0 failed, 0 not run (max parallel 2)
=== 9. Lineage of the metrics file ========================================
$ dagcache lineage metrics/metrics.json
metrics/metrics.json @8e9b581d47
⮑ stage evaluate key 692f1d80cc at 2026-10-25T12:30:27+00:00
cmd: python src/evaluate.py --features data/features --model models/model.pkl --out metrics/metrics.json
code: src/evaluate.py@748c0a6e
├── data/features @a45f8e3312
│ ⮑ stage featurize key 57cb37d3ef at 2026-10-25T12:30:25+00:00
│ params: features.scaler="standard"
│ ├── data/split/test.csv @f4e1d38ebf
│ │ ⮑ stage split key 07820c09d2 at 2026-10-25T12:30:24+00:00
│ │ params: split.seed=7, split.test_size=0.3
│ │ └── data/raw.csv @bbb5d1482d
│ │ ⮑ stage load key 86602c67ca at 2026-10-25T12:30:30+00:00
...
└── models/model.pkl @4219e2bf96
⮑ stage train key 2e986e92ee at 2026-10-25T12:30:26+00:00
params: train.C=1.0, train.max_iter=500
...
=== 12. Share through a remote: a fresh clone runs nothing ================
$ dagcache push
pushed 7 action(s), 11 new object(s) (45.5 KiB) to ../shared-remote
# cloned pipeline + params + lock file + src (no data, no cache)
$ dagcache repro --no-output
...
done in 0.08s: 0 ran, 7 restored, 0 up-to-date, 0 failed, 0 not run (max parallel 2)
=== 13. Two concurrent runs on a cold workspace share the work ============
[process 1] exit 0: done in 3.99s: 0 ran, 0 restored, 7 up-to-date, 0 failed, 0 not run (max parallel 2)
[process 2] exit 0: done in 3.97s: 7 ran, 0 restored, 0 up-to-date, 0 failed, 0 not run (max parallel 2)
stage executions across both processes: 7 (pipeline has 7 stages)
=== 14. Python-decorator pipeline: function-level code hashing ============
# edit decorators/helpers.py: 'sd = statistics.pstdev(values) or 1.0' -> 'sd = statistics.pstdev(values) or 1e-9'
$ dagcache -f pipeline.py status
generate up-to-date
normalize stale function helpers.zscore changed (749a30ba → c7930593)
summarize pending upstream stage 'normalize' is stale
$ dagcache -f pipeline.py repro
up-to-date generate ecd9c05e3e
ran normalize 0.00s 91d9045d2f (function helpers.zscore changed (749a30ba → c7930593))
up-to-date summarize f92a3556fb
In the last step summarize stays up to date although its upstream re-ran: the
helper change does not alter the normalised output (no sample has zero spread),
so early cutoff stops the cascade.
The held-out accuracy (0.963 at C=1.0, 1.0 at C=0.05, majority baseline 0.389)
is measured on a 54-row stratified test split that no stage fits on. The dataset
is tiny, so treat these numbers as a smoke test of the pipeline, not as a model
result. The demo changes C to show caching, not to choose a model.
The full list, with costs, is in DECISIONS.md. The ones that shape behaviour:
- Content, not upstream keys, flows downstream. This gives early cutoff. The
price is that
statusreportspending(notstale) for stages behind one that must run, because their inputs are unknown until it runs. - Command templates are keyed, not expanded commands. A param change then reads as a param change, not as "command changed".
- AST-normalised code hashing. Reformatting, comments and decorator arguments never trigger re-runs. Docstrings and library upgrades do.
- Locks, not a daemon. Three
flocklevels and a re-check under the stage lock make concurrentreproruns safe without a server process. - JSON lock file. The first YAML version spent about 33 s of a 100-stage run in the YAML parser (cProfile); JSON with memoised reads brought that run to about 0.9 s.
- The remote is optional and failures degrade. Lookup errors count as misses,
and autopush errors are warnings with a hint to run
dagcache push.
Every number below was produced by the command next to it, on the development machine (macOS arm64, 8 cores, Python 3.12.13).
| what | command | result |
|---|---|---|
| full gate | make check |
ruff format + lint clean, mypy --strict clean, 165 passed (2 integration tests deselected) |
coverage (line + branch, src/dagcache) |
make check (pytest --cov, gate fail_under = 85) |
95.09 % |
| real S3 server | make integration (SeaweedFS 3.97 in Docker) |
2 passed |
| source size | wc -l src/dagcache/*.py src/dagcache/backends/*.py |
4,616 lines (tests: 2,575 in tests/*.py, 2,639 with tests/integration) |
The tests include property tests (Hypothesis: canonical-JSON order independence,
tree-digest order independence, scheduler correctness on random DAGs, topological
order). There is a multi-process test in which three repro processes race on a
cold workspace and every stage must run exactly once, a backend contract suite
that runs against both the filesystem store and S3 (moto), and an end-to-end test
of the scikit-learn example.
make bench (synthetic workloads generated by bench/bench.py):
| benchmark | result | detail |
|---|---|---|
| SHA-256, 256 MiB file, no stat cache (in page cache) | 2.04 GB/s | (0.132s) |
| same file, stat-cache hit | 12 µs | per lookup |
| first run, 300-stage DAG (trivial stages) | 1.67 s | (5.6 ms/stage) |
| no-op repro, 300 stages | 0.20 s | (0.66 ms/stage) |
| status, 300 stages | 0.06 s | (0.19 ms/stage) |
| 8 independent 0.5 s stages, jobs=1 | 4.30 s | (max parallel 1) |
| 8 independent 0.5 s stages, jobs=4 | 1.17 s | (max parallel 4) |
| 8 independent 0.5 s stages, jobs=8 | 0.76 s | (max parallel 8) |
Machine load varies (other jobs were running), so treat these as orders of magnitude. The "first run" row measures dagcache's per-stage overhead (hashing, storing, manifest and lock-file writes) with stages that do almost nothing.
- Undeclared inputs are invisible. If a script reads a file that is not in
depsorcode, changes to it do not invalidate anything. There is no sandboxing or syscall tracing (Bazel sandboxes, and some tools trace syscalls). - POSIX only. Locking uses
flock, so Windows is not supported, andflockon NFS is unreliable. Use a local disk for.dagcache/, and use a directory remote on NFS only for push/pull. - Concurrent runs with different inputs in one workspace are not isolated. Concurrent runs of the same state are safe and share work. Two people running different params in the same checkout at the same time can race on outputs.
- Python stages share the GIL. Decorated pure-Python stages do not speed up
with
-j(command stages and NumPy/scikit-learn code do). - Python fingerprints are heuristic. They follow references to local
functions, classes, modules and constants (including default-argument
values), and record library versions. Behaviour driven by an object's runtime
state (including a default argument that is an arbitrary object), dynamic
imports, or
getattrstrings is not seen. Thesaltfield is the escape hatch. - Env var digests are unsalted SHA-256 in a committed lock file. Do not track low-entropy secrets.
- Remote GC is not coordinated across machines.
gc --remoteuses the local lock file as roots and can delete objects a teammate just pushed. It is opt-in. - Lock-file writes are O(stages) each, so a full run writes O(V²) bytes. That is fine to a few thousand stages but would need batching beyond that.
- Validation scope. The S3 backend is unit-tested against moto and
integration-tested against SeaweedFS, not against AWS S3 itself. CI is defined
in
.github/workflows/ci.yml(Linux and macOS). Nothing here has been used in production. - Small example. The ML example uses a 178-row dataset, so its metrics say nothing about modelling.
- Batch lock-file writes, and a SQLite-backed lock index for very large DAGs.
- Optional sandboxed execution (read-only inputs, private temp dirs) to catch undeclared dependencies.
foreach/matrix stages for parameter sweeps, each cached separately.- Leases for coordinated GC on shared remotes; GCS/Azure backends.
- Process-pool execution for CPU-bound Python stages.
src/dagcache/ library + CLI (typed, mypy --strict)
backends/ filesystem and S3 stores
tests/ 165 tests (unit, property, multi-process, end-to-end), integration/
examples/wine/ scikit-learn pipeline in YAML
examples/decorators/ the same ideas with Python decorators
scripts/demo.py `make demo`; scripts/integration.sh `make integration`
bench/bench.py `make bench`
docs/ ARCHITECTURE.md, HASHING.md
MIT, see LICENSE.
Maintained by Ishaan Sachdeva, Technology Strategist and Software Engineer, with experience across enterprise technology and consumer-facing platforms. His work focuses on Python, machine learning, generative AI, and data-driven workflows.
For project inquiries: Email · LinkedIn.
dagcache was originally authored by Ishaan Sachdeva.