Improve pipeline I/O performance

[msettles]

Summary

• add pigz-backed gzip read/write paths with conservative default pigz threading and compression-level controls
• force stdout pipeline output to remain uncompressed, including -t stdout
• replace dynamic_bitset read keys with packed BitKey encoding and add faster SeqScreener kmer handling
• batch FASTQ/tab/SAM writes and reduce selected hot-path string/vector copies
• update regression fixtures for the new compression-level option and make JSON fixture comparisons key-order stable

Benchmarks

Sample: /Users/biomsettles/projects/Archive/Alliance_MiSeq_Bacterial_Data/00-Original/sample1 with 805,327 paired fragments.

• baseline hts_LengthFilter: 65.40s
• pigz hts_LengthFilter: 18.22s
• streamed pipeline with --compression-level 1 and default pigz threads: 16.89s, 338M output
• streamed pipeline with final -u: 16.83s, 1.0G output
• HTSTREAM_PIGZ_THREADS=2: 17.03s; HTSTREAM_PIGZ_THREADS=4: 16.97s, effectively flat on this sample

Validation

• cmake --build build -j8
• ctest --test-dir build --output-on-failure (13/13 passed)
• cd regression && ./regression.sh ../build test
• stdout byte checks confirmed pipeline output is plain tab text, not gzip
• --compression-level 10 is rejected
• gzip -t passed for benchmark compressed output

[joe-angell]

Very nice, I'm surprised bitkey is faster than boost dynamic bitset, but it looks good.

[msettles]

All of these requests were for speed improvements as my biggest issues have been speed, full flow cells were taking a day or more to process. I think the total speed up, was between 15-20x and I wired in a prototype in hts_Stats for parallelism. New bottleneck seems to be decompression of a single file. ALL AI enhanced improvement. On July 6, 2026 at 10:46:00 AM, Joe Angell ***@***.***) wrote: ***@***.**** approved this pull request. Very nice, I'm surprised bitkey is faster than boost dynamic bitset, but it looks good. — Reply to this email directly, view it on GitHub <#272?email_source=notifications&email_token=AAE6RZT2HTD3ZC4HOWMHSVL5DPQVRA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINRTHAZTOMBWGE22M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#pullrequestreview-4638370615>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE6RZX42DHCOXITXBFYMRD5DPQVRAVCNFSNUABEKJSXA33TNF2G64TZHM3DGNZQGMZDCOB3JFZXG5LFHM2DOOBYGAZDSMBUGKQXMAQ> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AAE6RZWJPXFVTSKP6M6ZOOL5DPQVRA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINRTHAZTOMBWGE22M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/AAE6RZVUCHA5SEHPMCVQMDT5DPQVRA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINRTHAZTOMBWGE22M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>