Reclaim the tagged derivative subtrees - #537
Conversation
Most of a full cache is inside worktrees that have to stand: 18 of the 72 measured on the reference host carried a whole `.pixi/envs/default`, about 94.5 GB of the 104.5, and a site that is dirty, unpushed or unprovable keeps every byte of it. What makes reaching inside one legitimate is what the standing verdict is a statement about. Every reason a site stands, except a claimant's, is an answer about git's account of that site's content, and nothing under `.pixi/envs/` has ever been in that account: pixi's own `.pixi/.gitignore` of `*` and `!config.toml` puts it outside every index, every status and every commit. The two sets of bytes are disjoint, and a file the installer wrote is what separates them. The gate is the Cache Directory Tagging Specification's published 43 byte signature, and it never reads a directory's name: rattler, cargo, uv and pytest write one, `python -m venv` and npm write none, so the same name lands on both sides. The walk stops at the outermost tag and at every site the forest holds. A tag says regenerable and not by what, so one reader answers with the thing that re-derives it: `conda-meta/pixi` for the environment name, then a `pixi.lock` found by walking up inside the site. `manifest_path` is never stored, because it is a container path on every host environment. An absent lock, a lock that no longer names the environment, and a tag no reader recognises all stand and are named with their bytes. No new flag. It rides `--prune`'s own plan and its own y/N, and the acting pass re-reads both records under the lock before anything goes. What goes is the tagged directory alone, never `.pixi`. Closes #472.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Reviewer's GuideThis PR extends --prune with a separately accounted, tag-gated derivative sweep inside standing agent worktrees: it discovers outermost CACHEDIR.TAG directories, proves pixi recipes from local metadata and lockfiles, excludes claimant-pinned or unrecipeable trees, presents them in the existing plan and confirmation, revalidates them under lock before deletion, and leaves --ls behavior unchanged. Sequence diagram for pruning regenerable subtreessequenceDiagram
participant User
participant Prune as dl --prune
participant Sweep as WorktreeSweep
participant Gate as CACHEDIR.TAG gate
participant Reader as Pixi recipe reader
participant Plan as Prune plan
participant FS as Filesystem
User->>Prune: run --prune
Prune->>Sweep: weigh standing worktrees
Sweep->>Gate: declared_regenerable(directory)
Gate-->>Sweep: outermost tagged directories
Sweep->>Reader: pixi_recipe(tag, site)
Reader-->>Sweep: Recipe or NoRecipe
Sweep-->>Plan: reclaimable and withheld derivatives
Plan-->>User: show paths, sizes, recipes, and y/N
User->>Prune: confirm y
Prune->>Sweep: re-weigh under lock
Sweep->>Gate: re-read CACHEDIR.TAG
Sweep->>Reader: re-read metadata and pixi.lock
Reader-->>Sweep: derivable or withheld
Sweep->>FS: remove_tree_as_far_as_it_goes(tagged directory)
FS-->>User: report reclaimed or withheld
File-Level Changes
Assessment against linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Codecov Report❌ Patch coverage is Additional details and impacted files
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
blooop
left a comment
blooop
left a comment
There was a problem hiding this comment.
This was generated by AI during review.
Fresh context, adversarial, at a884c7f1b1aae660fe1d74e6cc64e7bf48afb0d3. The reviewer did not write this code. Every finding below was proved by a probe that was run; four mutants were killed against the existing tests.
Note on why this review exists: this PR's review check was green for a bad reason. Sourcery posted a quota refusal in a wording scripts/review_verdict.sh does not recognise, so a refusal counted as a review (filed as #538). This is the only review this PR has had.
Standards
Axis: pass. No path was found that destroys non-regenerable data in a realistic scenario. The gate, the walk, the reader and the claimant fold hold up under attack, and the tests are not vacuous.
Measured against the real host, read-only, using the same predicate the code uses: 9 sites, 12 tagged directories, 9 derivable, 20.54 GiB reclaimable, and zero files carrying a link outside their tree — so the byte figure is not inflated by shared inodes. New walk costs 423 directories in 0.01 s plus 0.72 s of exclusive_usage warm.
Spec
Axis: fail — 7 findings, none blocking, but F1/F2/F3 are on the surface the user reads and say false things.
F1. A refused second listing drops the approved reclaim silently: nothing done, nothing said, exit 0
When ClonePicture::of fails in the acting pass, reclaim extends report.withheld from plan.going only. plan.derivatives is dropped with no withheld_derivatives entry, so a directory the user was told would be reclaimed is neither reclaimed nor mentioned.
reclaimed=[] withheld_derivatives=[] withheld=[] nothing_to_say=true
PROBE FAILED: the user approved reclaiming a subtree and the run reports nothing at all about it
Probe planned one derivable subtree with going empty, then renamed clone/.git between plan and act. WorktreeReport::nothing_to_say() is true, so nothing is emitted, and PruneReport::finished() reads only refused/forget_refused, so dl exits 0.
Fix: give the early-return arm the derivative half its sibling already has — extend withheld_derivatives with a NotDerivableNow arm meaning this pass could not re-take the classification.
F2. The report says the cache tag is gone while it is still on disk
NotDerivableNow has NoTagThere and Answered, and no arm for the third real cause: the containing site became collectable between plan and act, so weigh returned it as removable with an empty derivatives list, and reclaim_derivatives reads that absence as NoTagThere.
withheld … /agent-one/.pixi/envs/default -> there is no longer a cache tag at that place,
so nothing there declares itself regenerable
PROBE FAILED: the report says the tag is gone when it is right there
The tag, conda-meta/pixi and pixi.lock were all asserted present in the same probe. Reached by deleting an uncommitted NOTES.md while reading the plan.
Fix: distinguish the causes at the source — a new arm for its site is no longer standing, so this pass did not weigh it, or re-read the tag before choosing NoTagThere. The arm's own doc says "it was removed, or its whole site was", and neither is true here.
F3. The pointer the plan prints is not the command the measurement was made with, and it fails outright without a manifest
Recipe::describe prints pixi install -e {environment}. Every measurement this module and its docs rest on is pixi install --frozen --offline. Two failures of the plan's own footer ("one command and no network beyond the shared package cache"):
- With a stale lock — the case the module explicitly reclaims — plain
pixi installcompares the lock hash against the manifest and re-solves, which is network. Only--frozengives the measured behaviour. - With the lock present and the manifest absent, the printed command errors outright:
$ pixi install -e default # pixi.lock only, no manifest
Error: × could not find pixi.toml or pyproject.toml with tool.pixi at directory /tmp/lockonly
Fix: print pixi install --frozen -e {environment}, and make lockfile_above also require a manifest beside the lock — existence only, since reading it would break the deliberate "the manifest is never read" rule.
F4. The plan prints "devlaunch will never reclaim it" one line above reclaiming it
Unaccountable::RegisteredElsewhere::describe() says "devlaunch will never reclaim it, and only the repository that registered it can", while Reason::subject() maps NotThisClonesToAccountFor to GitsAccountOfContent, so the tagged subtree inside it is reclaimed on the same plan.
STANDING …/worktrees/theirs: … devlaunch will never reclaim it, and only the repository that registered it can
DERIV …/worktrees/theirs/.pixi/envs/default: standing=None <- will be reclaimed
The behaviour is decided on #468 §6 and is not in question; the words are now false. This is the same "shipped contradiction" argument the PR body makes in its own favour, pointing the other way.
F5. Blank::ASiteSitsInside is narrower than its own sentence
classify's nested-site check joins against forest, which walk_sites builds by descending only the .claude/worktrees/ spine. So the arm can fire only when the tag sits at <site>/.claude or <site>/.claude/worktrees. A registered worktree anywhere else under the tag is invisible to the fold, and Blank::describe() ("a git worktree sits inside it") claims a guarantee the check cannot make.
Both placements are absurd in the wild, so this is words rather than risk. On the new-public-surface question: the arm is justified as an exhaustiveness answer, is produced only at derivatives.rs:458 as claimed, and Subject::AClaim never enters a Standing, so first_claim never sees it.
F6. "Never .pixi" is not structural, only accidental
Nothing excludes .pixi; it survives because real pixi puts conda-meta/pixi one level down.
derivatives=[(".claude/worktrees/agent-one/.pixi", None)]
PROBE FAILED: `.pixi` went and took config.toml with it
Confirmed benign on this host: .pixi carries no tag and .pixi/envs/default/CACHEDIR.TAG is the 43 published bytes at offset 0. But the PR body, the CHANGELOG and docs/cleanup.md all state "never .pixi" as a property. Either make it one, or soften the three sentences to "the tag pixi writes is one level below .pixi, so .pixi is not the unit".
F7. Content created between the plan and the y goes unnamed
Unlike plan.going, which has grew_past, the derivative path has no blast-radius check: the re-read confirms derivable, not unchanged. A file written into the tag after the plan, or a git init'd repository at .pixi/envs/default/src/someones-repo, is removed without having been named.
Not blocking — the module is explicit that the tag is a claim about purpose and not a proof about contents, and the pre-plan half is tested. But #525 was blocked on exactly the post-plan half of this shape for going units, and the post-plan half here is neither tested nor named in docs/cleanup.md. Worth one test row and one sentence so the asymmetry with grew_past is a recorded decision rather than a gap.
Claims verified
| Claim | Verdict |
|---|---|
declared_regenerable is the one expression of "what counts as a derivative" |
Holds — one production callsite; plan, act and listing all reach it through the single weigh_clone |
first_claim is the one expression of "what counts as a claimant" |
Holds — one callsite; Reason::subject() is a full wildcard-free match |
| Plan and act reach the same function, not two copies | Holds |
the_gate_never_reads_a_directorys_name is not vacuous |
Holds — mutant returning true by name failed that test and the walk test |
| Other empty-collection tests are not vacuously empty | Holds — three further mutants killed (symlink-following, nested-site check, first_claim) |
| The approved set can shrink and never grow | Holds — reclaim_derivatives iterates planned only |
| Symlinks never followed, at or inside the tag | Holds — probed with a symlink out of the cache and one to the site's own file; DirEntry::file_type is lstat |
| 43 bytes at offset 0, compared as bytes | Holds — near-miss table covers wrong hex, leading space, non-zero offset, truncation, empty |
dl --ls does not cost derivatives, nothing lost |
Holds — bytes stay visible in --ls --size because bytes_in walks the whole tree |
manifest_path never stored |
Holds |
No public constructor for Derivative |
Holds |
public-api green at this SHA, so the hand-derived snapshot is CI-verified |
Holds — run 33282488106 |
| No new promised-tier row naming an undefined type | Holds — api.txt untouched by this diff. For the record the pre-existing #531 gap is 37 types named in api.txt and defined only in rest.txt |
1 participant
Closes #472, implementing the decision on #468. It is the last substantive ticket of map #444 and its largest single payoff: about 94.5 GB of the measured 104.5.
What it does
dl --prunealready sweeps the agent git worktrees inside the clones it keeps (#454). Most of the bytes are in the worktrees that have to stand: 18 of the 72 on the reference host carried a whole.pixi/envs/default, and a site that is dirty, unpushed or unprovable keeps every one of them. This reclaims those environments while the worktree stands.Why that is not the wedge principle 1 exists to stop
Because of what the standing verdict is a statement about. Every reason a site stands, except a claimant's, is an answer about git's account of that site's content, and nothing under
.pixi/envs/has ever been in that account: pixi writes a.pixi/.gitignoreof*and!config.toml, which puts it outside every index, every status and every commit in every clone. The two sets of bytes are disjoint by construction, and the construction is a file the installer wrote.The shipped contradiction settles it: the dirt probe already reports nothing about
.pixi/envs, so refusing would print this site holds work that exists nowhere else, 0 bytes and decline to give back 5 GB in the same breath.The gate never reads a directory's name
The published 43 byte
CACHEDIR.TAGsignature at offset 0, compared as bytes. Measured on #468: rattler, cargo, uv and pytest write one;python -m venvwrites none and npm writes none anywhere beneathnode_modules. The same name lands on both sides, so no rule keyed on.venvcould express this..pixiand.pixi/envsappear nowhere in the predicate.The walk stops at the outermost tag (the outer declaration covers what is inside it, and descending would bill the same bytes twice under R3), at every site the forest holds, and at every symlink. It never puts the question to a site itself.
A recipe, or it stands
One reader is implemented and it reaches the whole 94.5 GB. Three reads: the tag;
conda-meta/pixiforenvironment_name, withmanifest_pathnever stored because it is a container path on every host environment; and apixi.lockfound by walking up from the tag, inside the site, whoseenvironments:map names it. An absent lock, a lock that no longer names the environment (withpixi clean -eas the pointer), and a tag no reader recognises all stand and are named with their bytes.The claimant fold
A
git worktree lockor a repository lockdlcould not take pins the subtree. Dirty, unpushed and unprovable do not, because they were never statements about these bytes. A worktree of another repository is in the second group, per #468 §6. It is one call toReason::subject(), which is wildcard free, so a newBlankarm has to answer the question where it is added. There is no second list of arms anywhere.No new flag
It rides
--prune's own plan and its own y/N. The plan names each directory and its size before the question, and the acting pass re-derives the whole clone under the lock and acts only where the re-read also says derivable. Hanging it on--force-worktreeswould have made one flag carry two consents, which #459 refused. What goes is the tagged directory alone, never.pixi, which holdsconfig.toml.What the plan looks like
Run against a scratch cache with a real registered worktree, a real tag, a real lockfile, a
node_modulesand a stdlib.venv:Answered
y, the tagged directory went and.pixi/config.toml,.pixi,node_modules,.venv,pixi.lockand the human'sNOTES.mdall stayed.Types
Derivativehas private fields, noDefaultand no constructor: the only thing entitled to say this is regenerable and its recipe is on disk is a read that answered.Option<Derivative>is refused, because itsNonewould mean both nothing tagged here and tagged but not costable.Taggedhas no arm for "no tag": an untagged directory is not one.public_api_snapshots::nothing_but_a_read_mints_a_derivativeasserts the absence, anchored on the type's own row so it cannot pass vacuously.Tests
30 new rows. 18 unit rows over the gate, the walk, the reader and the fold in
derivatives/tests.rs; 12 whole-run rows over real git, real registrations and real directories inagent_worktrees/tests.rs. Every recipe case from the issue is a row, including the two fixture rows that prove the predicate never reads a name:the_gate_never_reads_a_directorys_nameranges over the predicate itself for eight names on both sides, anda_node_modules_and_a_stdlib_venv_beside_an_environment_are_untouchedranges over a whole run.Notes for review
cargo-public-apion the machine this was built on, so the 107 newpublic-api.rest.txtrows were written by hand against the file's own conventions. CI'spublic-apijob regenerates and diffs, so its verdict is the real one; if it disagrees, its output is what to apply.Blankgains one arm,ASiteSitsInside, for the case where a site sits inside a tagged directory.Reason::subject()answersAClaimfor it, which is #468 §6's sentence made literal. It is produced by the derivative fold alone and never by a site's own verdict.dl --lsdeliberately does not cost these:Derivatives::NotAskedon the listing path, with a test asserting the going and standing answers are identical either way.🤖 Generated with Claude Code
Summary by Sourcery
Reclaim safely regenerable tagged environments from agent worktrees that remain after pruning.
New Features:
dl --pruneto reclaim tagged, regenerable subtrees inside agent worktrees that must remain.Enhancements:
dl --lsand avoid double-counting nested tagged directories or derivatives inside worktrees being removed.Documentation:
Tests: