Repository navigation
perf(archives): faster CRC-32 (System.IO.Hashing) and LHA CRC-16 - #1531
Merged
Merged
Conversation
Crc32 now hands the work to System.IO.Hashing.Crc32 (carry-less multiply or CRC instructions on x64 and ARM64, tables elsewhere) instead of one table lookup per byte; its API and results stay the same. 7z, ZIP, xz (CRC-32 check), lzip, ALZ, EGG and ARJ all check through it. LHA's CRC-16 (Crc16Stream) reads 16 bytes per step with sixteen lookup tables (slicing by 16) instead of one lookup per byte. System.IO.Hashing 10.0.14 (MIT) is pinned in Directory.Packages.props, listed in THIRD-PARTY-NOTICES and kept out of ReadyToRun like the other Microsoft-signed libraries so it keeps its signature. ChecksumTests compares both with the old byte-table loops for random lengths up to 1 MB, unaligned starts and data fed in random pieces. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Archives/Crc32.csnow wrapsSystem.IO.Hashing.Crc32instead of one table lookup per byte. Same class, sameAppend/Value/Compute, same values. On x64 and ARM64 it uses the CPU's carry-less multiply / CRC instructions; elsewhere it falls back to tables. Every reader and writer that checks CRC-32 goes through it: 7z, ZIP, xz (CRC-32 check), lzip, ALZ, EGG and ARJ, plus the APNG and HWP writers.Crc16StreaminLhaReader.cs) now handles 16 bytes per step with sixteen lookup tables (slicing by 16). Still our own code.System.IO.Hashing10.0.14 (MIT, dotnet/runtime) is pinned inDirectory.Packages.propsand listed inTHIRD-PARTY-NOTICES.md. It is also added toPublishReadyToRunExclude, like the other Microsoft-signed DLLs, so it keeps its Authenticode signature.Looked at, left alone
Lzma/Crc64.cs(xz's default check): still one table lookup per byte. It is the next candidate for .xz, but it was left out becauseLzma/is in flight in Built-in engines: faster LZMA / LZMA2 decoding (1.2–1.5×) #1527.Numbers
Shared PC under load from other jobs. Each value is the median of 3 runs, before and after alternated, extracting with the real readers to a temp folder (wall-clock MB/s of unpacked data).
Unsafeinstead of the array was no faster.Tests
New
ChecksumTestscompares CRC-32 and CRC-16 against the old byte-table loops (kept in the test as the reference):Crc16Streamread through a stream that returns short reads, at unaligned buffer offsetsTo check the test itself, I broke one slicing table index and the CRC-16 test failed.
Related suites, Release, filter
Archive|SevenZip|Lzma|DiskImage|Bzip2|Checksum|Apng|HwpWriterTests|Nsis:FILEE_HIDE_7ZIP=1: 399 passed, 48 skippedIn both runs, the two
MediaTestsAPNG cases that theApngfilter also matches timed out waiting for ffprobe (60 s limit) while the PC was loaded. Rerun alone, they andApngTestspass 9/9. The APNG they wrote has correct chunk CRCs (checked against zlib), and ffprobe reads it in under a second.dotnet buildhas 0 errors and no warnings in the touched files.dotnet format Filee.slnx --verify-no-changes --severity warnexits 0.Nothing here depends on OS or architecture:
System.IO.Hashingpicks its own code path at run time, and the slicing code reads little-endian explicitly.🤖 Generated with Claude Code