Skip to content

perf(archives): faster CRC-32 (System.IO.Hashing) and LHA CRC-16 - #1531

Merged
KnifeLemon merged 1 commit into
builtin-enginesfrom
p4/b14-crc-speed
Nov 14, 2026
Merged

KnifeLemon merged 1 commit into
builtin-enginesfrom
p4/b14-crc-speed

Conversation

@KnifeLemon

Copy link
Copy Markdown
Owner

What changed

  • Archives/Crc32.cs now wraps System.IO.Hashing.Crc32 instead of one table lookup per byte. Same class, same Append / Value / Compute, same values. On x64 and ARM64 it uses the CPU's carry-less multiply / CRC instructions; elsewhere it falls back to tables. Every reader and writer that checks CRC-32 goes through it: 7z, ZIP, xz (CRC-32 check), lzip, ALZ, EGG and ARJ, plus the APNG and HWP writers.
  • LHA's CRC-16 (Crc16Stream in LhaReader.cs) now handles 16 bytes per step with sixteen lookup tables (slicing by 16). Still our own code.
  • System.IO.Hashing 10.0.14 (MIT, dotnet/runtime) is pinned in Directory.Packages.props and listed in THIRD-PARTY-NOTICES.md. It is also added to PublishReadyToRunExclude, like the other Microsoft-signed DLLs, so it keeps its Authenticode signature.

Looked at, left alone

  • bzip2 block CRC: I tried moving it out of the inverse-BWT loop into one slicing-by-16 pass per 64 KB flush. Decoding got slightly slower (0.95–0.97×, 8 runs each). The per-byte CRC was already hidden behind the walk's cache misses, so I dropped the change.
  • cpio byte sum and CAB XOR checksum: they already run at 1.0–1.6 GB/s, which isn't hot next to their decoders.
  • NSIS CRC: the NSIS reader isn't on this branch yet.
  • Lzma/Crc64.cs (xz's default check): still one table lookup per byte. It is the next candidate for .xz, but it was left out because Lzma/ is in flight in Built-in engines: faster LZMA / LZMA2 decoding (1.2–1.5×) #1527.

Numbers

Shared PC under load from other jobs. Each value is the median of 3 runs, before and after alternated, extracting with the real readers to a temp folder (wall-clock MB/s of unpacked data).

Input Before After
7z stored, 273 MB 180 359 2.0×
7z LZMA, 250 MB mixed 26.2 29.2 1.11×
7z LZMA2, 36 MB text (8 runs) 18.6 20.3 1.09×
LHA -lh0-, 67 MB 182 310 1.70×
LHA -lh5-, 36 MB text 51.6 60.7 1.18×
ARJ method 1, 36 MB text 51.6 58.6 1.14×
  • With the faster LZMA decoder from Built-in engines: faster LZMA / LZMA2 decoding (1.2–1.5×) #1527 underneath (its head plus this change): 7z LZMA mixed 20.2 → 21.6 MB/s wall, 1.13× by CPU time; 7z LZMA2 text 24.3 → 26.3 MB/s, 1.12× by CPU time. The absolute numbers are lower because the PC was busier during that run.
  • Checksum alone, 64 KB buffers: CRC-32 went from ~220 MB/s to 2–13 GB/s. CRC-16 went from ~195 MB/s to ~640–710 MB/s; slicing by 8 reached ~530–600 MB/s, and indexing through Unsafe instead of the array was no faster.

Tests

  • New ChecksumTests compares CRC-32 and CRC-16 against the old byte-table loops (kept in the test as the reference):

    • lengths 0–39, then 40 random lengths up to 1 MB
    • start offsets 0–63 into the buffer
    • data appended in random pieces from 1 byte to the whole length, with the running value checked after every piece
    • Crc16Stream read through a stream that returns short reads, at unaligned buffer offsets
    • the catalogue check values for "123456789"

    To check the test itself, I broke one slicing table index and the CRC-16 test failed.

  • Related suites, Release, filter Archive|SevenZip|Lzma|DiskImage|Bzip2|Checksum|Apng|HwpWriterTests|Nsis:

    • with 7-Zip: 436 passed, 11 skipped
    • with FILEE_HIDE_7ZIP=1: 399 passed, 48 skipped

    In both runs, the two MediaTests APNG cases that the Apng filter also matches timed out waiting for ffprobe (60 s limit) while the PC was loaded. Rerun alone, they and ApngTests pass 9/9. The APNG they wrote has correct chunk CRCs (checked against zlib), and ffprobe reads it in under a second.

  • dotnet build has 0 errors and no warnings in the touched files. dotnet format Filee.slnx --verify-no-changes --severity warn exits 0.

  • Nothing here depends on OS or architecture: System.IO.Hashing picks its own code path at run time, and the slicing code reads little-endian explicitly.

🤖 Generated with Claude Code

Crc32 now hands the work to System.IO.Hashing.Crc32 (carry-less multiply
or CRC instructions on x64 and ARM64, tables elsewhere) instead of one
table lookup per byte; its API and results stay the same. 7z, ZIP, xz
(CRC-32 check), lzip, ALZ, EGG and ARJ all check through it.

LHA's CRC-16 (Crc16Stream) reads 16 bytes per step with sixteen lookup
tables (slicing by 16) instead of one lookup per byte.

System.IO.Hashing 10.0.14 (MIT) is pinned in Directory.Packages.props,
listed in THIRD-PARTY-NOTICES and kept out of ReadyToRun like the other
Microsoft-signed libraries so it keeps its signature.

ChecksumTests compares both with the old byte-table loops for random
lengths up to 1 MB, unaligned starts and data fed in random pieces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@KnifeLemon KnifeLemon added the built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet label Nov 14, 2026
@KnifeLemon
KnifeLemon merged commit 1092ca5 into builtin-engines Nov 14, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant