perf: one listing instead of two questions a file, and one collection instead of two - #55
Merged
Merged
Conversation
… instead of two Three findings from the performance report, and two of them came back with the report's own suggestion measured and refused. preflight asked the filesystem twice for every planned file. It reads the output directory once. A run of 100 000 files with --dry-run goes from 15.7-19.7 s to 0.49-0.56 s, order alternated - the check was 97% of it. Ten thousand names measured on their own: 1.937 s of stat calls against 8.9 ms for one listing. That is also a fix rather than only a speedup, and the guard for it says so. os.Stat follows a link, so a link pointing at nothing answered "no name here" and the run replaced it without a word. A directory ENTRY is what a taken name is, whatever it points at. With the old question put back, the new guard reports that the run went ahead over a name somebody else's link was holding. A directory that cannot be LISTED is still asked about file by file. Both systems allow write permission without read, a run into such a directory has always worked, and reading nothing there and calling it empty would let the run write over what is inside. That fallback has its own guard, which skips on Windows because denying a listing there needs an ACL. The plan ceiling forced a collection to take every reading, so a run of one kilobyte paid for two of them - measured with GODEBUG=gctrace=1, exactly two on every run however small. It asks /gc/heap/allocs:bytes first, at 251 ns against 519 us, and only collects when that says it might be over. The shortcut is sound by an inequality rather than by an estimate: the live heap cannot have grown by more than has been allocated. The report asked for /gc/heap/live:bytes and that metric is WRONG here. It reports the heap as of the last collection, and measured on 2026-09-05 all four existing ceiling guards stay green with it, because 25 MB of allocation makes the collector run on its own and the lagging reading catches up by luck. With the collector switched off the luck goes: a plan six times the ceiling is accepted. The new guard turns the collector off for exactly that reason. hashFile reads in 256 KiB pieces. The report asked for a 1 MB buffer through io.CopyBuffer and that does nothing at all: os.File implements io.WriterTo, so CopyBuffer hands it the whole job and throws the buffer away - 128 KiB, 256 KiB and 1 MiB with a plain file all take the same 167-172 ms that io.Copy takes. Hidden behind a reader that offers only Read, 256 KiB takes 161 ms against 199. The size is measured too: 64 KiB is 182 ms and nothing above a quarter of a megabyte can be told apart, so sixteen workers cost four megabytes. planChildren is sized up front, since the total is known from the groups. TotalBytes being walked twice is NOT done, and that is a measurement rather than an oversight: one walk over 100 000 planned files has a median of 0 s and a maximum of 541 us, so two of them cost half a millisecond of a nineteen second run. Widening a signature for that would be a change nothing can see. engine.go went past the length ceiling, and the guard asks for a split by what the parts do rather than for a bigger number - so preflight and the questions it asks about names are their own file now. Two ceilings came down with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three findings from the performance report -
P6,P11and the small half ofP12. Two of them came back with the report's own suggestion measured and refused.P11 - the largest item in the whole report
preflightasked the filesystem twice for every planned file. It reads the output directory once.About 30x - the check was 97% of that run. Ten thousand names measured on their own: 1.937 s of stat calls against 8.9 ms for one listing.
It is also a fix, not only a speedup.
os.Statfollows a link, so a link pointing at nothing answered "no name here" and the run replaced it without a word. A directory entry is what a taken name is, whatever it points at. With the old question put back, the new guard reports that the run went ahead over a name somebody else's link was holding.A directory that cannot be listed is still asked about file by file - both systems allow write permission without read, and reading nothing there and calling it empty would let a run write over what is inside. That fallback has its own guard, which skips on Windows because denying a listing there needs an ACL.
P6 - and the metric the report asked for is wrong
Every run forced two collections, however small -
GODEBUG=gctrace=1, exactly two on--count 1 --size 1kb. It now asks/gc/heap/allocs:bytesfirst, at 251 ns against 519 µs, and collects only when that says it might be over. The shortcut is sound by an inequality, not an estimate: the live heap cannot have grown by more than has been allocated.🔴 The report asked for
/gc/heap/live:bytes. Measured: with it, all four existing ceiling guards stay green - 25 MB of allocation makes the collector run on its own and the lagging reading catches up by luck. Switch the collector off and the luck goes: a plan six times the ceiling is accepted. The new guard turns the collector off for exactly that reason.P12 - two done, one refused by measurement, two false
🔴
io.CopyBufferwith a bigger buffer does nothing:os.Fileimplementsio.WriterTo, so it takes the whole job and throws the buffer away. 128 KiB, 256 KiB and 1 MiB with a plain file all take the same 167-172 ms thatio.Copytakes. Hidden behind a reader offering onlyRead, 256 KiB takes 161 ms against 199. The size is measured too - 64 KiB is 182 ms and nothing above a quarter of a megabyte can be told apart, so sixteen workers cost 4 MiB, not 64.🔴
TotalByteswalked twice is not done, and that is a measurement: one walk over 100 000 planned files has a median of 0 s, maximum 541 µs.planChildrenis sized up front.Checks
Full suite green,
preflight --quick12 of 12, byte stability green on all 54 pinned cases, every mutation caught (one unjudged on Windows because its guard skips there and the entry says so),staleness.pyclean at 794 patterns.engine.gowent past the length ceiling, and that guard asks for a split by what the parts do rather than a bigger number - sopreflightand the name questions are their own file. Two ceilings came down.tools/linux-check.pyis red on five stored screens - and it is red identically onmain, with CI's ubuntu green on the same commit. Not from this branch, cause not established, written up asO182.🤖 Generated with Claude Code