pileup: worker panic (base_mods_adapter.rs:80 "entered unreachable code") is silently swallowed in whole-genome mode — presents as an infinite zero-CPU hang

[eelkebos]

Summary

modkit pileup 0.6.4 on a publicly available mod-BAM panics in a worker thread:

thread '<unnamed>' panicked at modkit-core/src/pileup/base_mods_adapter.rs:80:30:
internal error: entered unreachable code

The failure presents in two completely different ways depending on invocation:

• With --region: the panic propagates and modkit exits rc=101 in ~2 s. Annoying but honest.
• Without --region (whole-genome): the panic is never surfaced. The main thread blocks
  forever on a channel recv from the dead worker: zero CPU, no output growth, no error, no exit.
  In an unattended pipeline this reads as a hang and costs hours before anyone intervenes
  (it cost us three multi-hour attempts before we bisected it).

Even if the panic itself is a data edge case, the swallowed-join behaviour seems worth fixing
independently: a dead worker should fail the run, not freeze it.

Reproduction (public data)

The input is public: ENA run SRR18936814-24 series (GSE185307 / PRJNA768481); the file below is
SRR18936820 (ISPRO.bc01.meg242.remora1.edgefix.mod_mappings.sorted.hg38.bam,
https://ftp.sra.ebi.ac.uk/vol1/run/SRR189/SRR18936820/ — 2021-era megalodon/remora mod tags).
Reference: GRCh38 no-alt analysis set.

Fast reproduction (panic, rc=101, ~2 s):

modkit pileup ISPRO.bc01.…hg38.bam out.cpg.bed \
  --ref GCA_000001405.15_GRCh38_no_alt_analysis_set.fna \
  --cpg --combine-strands --modified-bases 5mC 5hmC --combine-mods \
  --region chr1:248750000-248757812 --threads 8

Hang reproduction: same command without --region — runs until chr1 is mostly done, then
freezes permanently (last flushed row is an unrelated position ~1.25 Mb before the fault,
which misdirects debugging).

Bisection

Full trail in our repo (docs/MODKIT_BC01_HANG.md); highlights:

• chr1:248750000-248753906 (3,906 bp): fails
• chr1:248753906-248757812 (3,906 bp): also fails — at least two independent trigger
  sites in chr1:248,750,000-248,757,812
• Windows centred on the last-flushed row (chr1:247,998,886) from 1 bp to 1.5 Mb all pass —
  that position is a red herring produced by the swallowed-panic presentation.
• A second sample from the same series (ISPRO.bc05/SRR18936816) freezes identically in
  whole-genome mode near the chr1 centromere; likely the same class of trigger.

Version regression range (tested 2026-08-16, bioconda osx-64 builds)

version	fault interval	control interval	notes
0.5.0	rc=2	rc=2	CLI-incompatible with these flags
0.6.0–0.6.2	exit 0, 0 rows	exit 0, 0 rows	all ~1.28M reads rejected ("failed processing") — these legacy megalodon tags were simply not parsed
0.6.3–0.6.4	panic rc=101	ok (real rows)	legacy-tag parsing evidently added between 0.6.2 and 0.6.3, with the unreachable-code panic on some reads

So the bug is confined to the parsing support introduced in v0.6.3 (2026-05-21): before it,
silent rejection; after it, a panic that whole-genome mode swallows into an infinite hang.

Trigger confirmed read-level (workaround exists)

Excluding the 51 reads overlapping chr1:248,750,000-248,757,812 (ids available on request)
allows 0.6.4 to complete all of chr1 cleanly — 1,381,526 pileup rows. The panic trigger is
carried by specific reads' base-modification records, not by pileup state. Happy to attach the
samtools view records for those reads.

Update — the same defect on unplaced scaffolds, and what it cost (2026-08-23)

Since drafting, we worked the bug from the other end and found the two behaviours are one
problem with a much wider blast radius than "a data edge case in one region".

Per-chromosome verification is not sufficient. We built tooling that bisects each primary
chromosome to a minimal fault interval and excludes the offending reads: on this BAM it isolated
faults across 11 of 24 primary chromosomes (4 panicking, 7 presenting as the silent hang) and
excluded 3,316 reads of ~1.28 M, verifying every primary chromosome individually.

The very next whole-genome pileup still hung for 24 hours — 1 day elapsed against 2.5 minutes
of CPU, the stack inside modkit's own frames — with output stopping at
chr17_GL000205v2_random:185453, 6.38 M rows in. The cause is that whole-genome mode also walks
the unplaced/random scaffolds, which our per-chromosome probing had never touched. So the
silent-hang mode does not just make the panic hard to see; it makes exhaustive verification of
the primary chromosomes insufficient, because the survivors hide on contigs a reasonable person
does not think to check.

Our fix was to stop using whole-genome mode. The pipeline now iterates chr1..22/X/Y with
--region and concatenates, logging per-chromosome row counts. The same BAM that hung for a day
then completed in ~2 minutes, 16,130,208 CpG rows across all 24 chromosomes. We are no longer
blocked, and we are filing because the swallowed-join behaviour cost us several multi-hour
attempts before we understood it, and would cost anyone else the same.

The two asks, in priority order:

1. A dead worker should fail the run, not freeze it — independent of the panic's cause. This
   is the expensive half: an honest rc=101 in seconds is recoverable, a silent zero-CPU hang in
   an unattended pipeline is not.
2. The panic itself (base_mods_adapter.rs:80), which our matrix places in the 0.6.2 → 0.6.3
   range and ties to specific reads' base-modification records — see the reproduction above.

Environment

• modkit 0.6.4, macOS 26.5 (Apple Silicon M1), conda-installed
• Happy to run patched builds / provide samtools view extracts of the failing intervals.