Skip to content

perf: one listing instead of two questions a file, and one collection instead of two - #55

Merged
donislawdev merged 1 commit into
mainfrom
perf/one-collection-and-one-listing
Sep 5, 2026
Merged

perf: one listing instead of two questions a file, and one collection instead of two#55
donislawdev merged 1 commit into
mainfrom
perf/one-collection-and-one-listing

Conversation

@donislawdev

Copy link
Copy Markdown
Owner

Three findings from the performance report - P6, P11 and the small half of P12. Two of them came back with the report's own suggestion measured and refused.

P11 - the largest item in the whole report

preflight asked the filesystem twice for every planned file. It reads the output directory once.

100 000 files, --dry-run, order alternated
before  15718 / 15817 / 19691 / 19421 ms
after     493 /  520 /   551 /   558 ms

About 30x - the check was 97% of that run. Ten thousand names measured on their own: 1.937 s of stat calls against 8.9 ms for one listing.

It is also a fix, not only a speedup. os.Stat follows a link, so a link pointing at nothing answered "no name here" and the run replaced it without a word. A directory entry is what a taken name is, whatever it points at. With the old question put back, the new guard reports that the run went ahead over a name somebody else's link was holding.

A directory that cannot be listed is still asked about file by file - both systems allow write permission without read, and reading nothing there and calling it empty would let a run write over what is inside. That fallback has its own guard, which skips on Windows because denying a listing there needs an ACL.

P6 - and the metric the report asked for is wrong

Every run forced two collections, however small - GODEBUG=gctrace=1, exactly two on --count 1 --size 1kb. It now asks /gc/heap/allocs:bytes first, at 251 ns against 519 µs, and collects only when that says it might be over. The shortcut is sound by an inequality, not an estimate: the live heap cannot have grown by more than has been allocated.

🔴 The report asked for /gc/heap/live:bytes. Measured: with it, all four existing ceiling guards stay green - 25 MB of allocation makes the collector run on its own and the lagging reading catches up by luck. Switch the collector off and the luck goes: a plan six times the ceiling is accepted. The new guard turns the collector off for exactly that reason.

⚠️ On a large plan the gain is invisible (ranges overlap) - it is one collection per run, and the guard package went 355 s to 311 s.

P12 - two done, one refused by measurement, two false

🔴 io.CopyBuffer with a bigger buffer does nothing: os.File implements io.WriterTo, so it takes the whole job and throws the buffer away. 128 KiB, 256 KiB and 1 MiB with a plain file all take the same 167-172 ms that io.Copy takes. Hidden behind a reader offering only Read, 256 KiB takes 161 ms against 199. The size is measured too - 64 KiB is 182 ms and nothing above a quarter of a megabyte can be told apart, so sixteen workers cost 4 MiB, not 64.

🔴 TotalBytes walked twice is not done, and that is a measurement: one walk over 100 000 planned files has a median of 0 s, maximum 541 µs.

planChildren is sized up front.

Checks

Full suite green, preflight --quick 12 of 12, byte stability green on all 54 pinned cases, every mutation caught (one unjudged on Windows because its guard skips there and the entry says so), staleness.py clean at 794 patterns.

engine.go went past the length ceiling, and that guard asks for a split by what the parts do rather than a bigger number - so preflight and the name questions are their own file. Two ceilings came down.

⚠️ tools/linux-check.py is red on five stored screens - and it is red identically on main, with CI's ubuntu green on the same commit. Not from this branch, cause not established, written up as O182.

🤖 Generated with Claude Code

… instead of two

Three findings from the performance report, and two of them came back with the
report's own suggestion measured and refused.

preflight asked the filesystem twice for every planned file. It reads the
output directory once. A run of 100 000 files with --dry-run goes from
15.7-19.7 s to 0.49-0.56 s, order alternated - the check was 97% of it. Ten
thousand names measured on their own: 1.937 s of stat calls against 8.9 ms for
one listing.

That is also a fix rather than only a speedup, and the guard for it says so.
os.Stat follows a link, so a link pointing at nothing answered "no name here"
and the run replaced it without a word. A directory ENTRY is what a taken name
is, whatever it points at. With the old question put back, the new guard
reports that the run went ahead over a name somebody else's link was holding.

A directory that cannot be LISTED is still asked about file by file. Both
systems allow write permission without read, a run into such a directory has
always worked, and reading nothing there and calling it empty would let the run
write over what is inside. That fallback has its own guard, which skips on
Windows because denying a listing there needs an ACL.

The plan ceiling forced a collection to take every reading, so a run of one
kilobyte paid for two of them - measured with GODEBUG=gctrace=1, exactly two on
every run however small. It asks /gc/heap/allocs:bytes first, at 251 ns against
519 us, and only collects when that says it might be over. The shortcut is
sound by an inequality rather than by an estimate: the live heap cannot have
grown by more than has been allocated.

The report asked for /gc/heap/live:bytes and that metric is WRONG here. It
reports the heap as of the last collection, and measured on 2026-09-05 all four
existing ceiling guards stay green with it, because 25 MB of allocation makes
the collector run on its own and the lagging reading catches up by luck. With
the collector switched off the luck goes: a plan six times the ceiling is
accepted. The new guard turns the collector off for exactly that reason.

hashFile reads in 256 KiB pieces. The report asked for a 1 MB buffer through
io.CopyBuffer and that does nothing at all: os.File implements io.WriterTo, so
CopyBuffer hands it the whole job and throws the buffer away - 128 KiB, 256 KiB
and 1 MiB with a plain file all take the same 167-172 ms that io.Copy takes.
Hidden behind a reader that offers only Read, 256 KiB takes 161 ms against 199.
The size is measured too: 64 KiB is 182 ms and nothing above a quarter of a
megabyte can be told apart, so sixteen workers cost four megabytes.

planChildren is sized up front, since the total is known from the groups.

TotalBytes being walked twice is NOT done, and that is a measurement rather
than an oversight: one walk over 100 000 planned files has a median of 0 s and
a maximum of 541 us, so two of them cost half a millisecond of a nineteen
second run. Widening a signature for that would be a change nothing can see.

engine.go went past the length ceiling, and the guard asks for a split by what
the parts do rather than for a bigger number - so preflight and the questions
it asks about names are their own file now. Two ceilings came down with it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@donislawdev
donislawdev merged commit 479c494 into main Sep 5, 2026
18 checks passed
@donislawdev
donislawdev deleted the perf/one-collection-and-one-listing branch September 5, 2026 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant