296 MISMATCH snippets across 78 records: the real evidence backlog, now that it is visible

[realmarcin]

Follow-up from #593. Filed so the backlog that PR exposes is tracked rather than left implicit in a diff.

What the number is

scripts/evidence_snippet_audit.py classifies every one of the 5271 evidence snippets. After #593:

bucket	count	meaning
MATCH	4765	literal substring of real source text
RENDERING	66	faithful quote, cache carries a PDF/XML extraction artefact
WEAK	3
MISMATCH	296	real source text exists and the snippet is not in it
NOCONTENT	141	no source on disk to check against — that is #591

MISMATCH is the bucket that matters. Every one of these is a claim where the paper is cached and the quote is not in it.

Why it grew, and why that is good

On main MISMATCH was 190 and NOCONTENT was 379. #593 gave 238 previously-unverifiable snippets a real source; 132 turned out faithful and 106 did not. The 106 were always wrong — they were simply unexaminable.

The worklist

296 snippets across 78 records. Concentrated:

16  BioAsteroid_ISS_Chondrite_Biomining_Consortium
15  Lunar_Simulant_Phosphate_Solubilizing_Bacteria_Nicotiana
15  Microcoleus_Massilia_Cyanosphere_Urea_Mutualism
13  Bacteroides_Eubacterium_Gnotobiotic_Gut_Model
13  Ostreococcus_Dinoroseobacter_BVitamin_Mutualism
11  Thalassiosira_Ruegeria_Phycosphere_Coculture
 9  Anammox_Granule_Metabolic_Interaction_Community
 8  Drosophila_FiveSpecies_Gnotobiotic_Gut_Microbiota

The top three are worth looking at first not because they are largest but because two of them (BioAsteroid, Lunar_Simulant) come from the space/regolith curation cohort, which suggests a systematic problem with one curation pass rather than 31 independent errors.

How to work it — and one rule that is easy to get wrong

uv run python scripts/evidence_snippet_audit.py --list-mismatch prints them with a diff against the closest cached text.

A MISMATCH is not automatically a fabrication. Before rewriting anything, check in this order:

1. Is the cache abstract-only while the snippet is from Methods/Results? Then it is a retrieval gap, not a curation error — run cache_fulltext.py first. This is the mistake corrected in Two records I curated from abstracts were wrong, not just thin (#183) #577/The partial record was not partial, I searched badly (#183) #578 and it accounted for several apparent "misattributions" there.
2. Is it a rendering artefact the normaliser did not catch? e.g. B S _PETase for BsPETase. Those belong in RENDERING; if one is landing in MISMATCH the normaliser needs the case, and the snippet must not be edited to match the cache (the justfile is explicit).
3. Is the reference simply the wrong paper for the claim? Seen twice recently — a genome announcement cited for cultivation conditions.
4. Only then treat it as a paraphrase to be replaced with a real quote, or a claim to be withdrawn.

Doing (4) first would "fix" the corpus by editing quotes to match whatever text happens to be cached, which is worse than the current state.

[realmarcin]

Correcting this issue's emphasis — I filed it wrong

I opened this describing 296 MISMATCH snippets as "the real evidence backlog" and "fabrication suspects", and pointed at the space/regolith cohort as evidence of "one bad curation pass". I then ran the check I had told everyone else to run first, and the framing does not survive it.

290 of the 296 (98%) sit on an abstract-only cache. Only 6 are mismatches against a paper whose full text is actually on disk.

ABSTRACT-ONLY cache: 290  (98%)
full text cached:      6  (2%)

The two records I singled out are the clearest cases. All 16 of BioAsteroid's mismatches cite one reference (PMID:41617698, 3873-byte abstract-only cache); all 15 of Lunar_Simulant's cite one (PMID:37945659, 2967 bytes). Their snippets are Methods-level detail —

"The medium used for this experiment was a solution of 50% v/v R2A medium"
"Samples grew for 19 days at 19.5"
"yeast extract 0.25; peptone 0.25; casamino acids 0.25; glucose 0.25"

— quoted from a Methods section against a cache that only ever held the abstract. That is step 1 of the triage order in the issue body, and it is the #577/#578 error again: a property of my retrieval mistaken for a property of the curation. There is no evidence of a bad curation pass; there is a retrieval gap behind 78 references.

So #591 is the fix for 98% of this issue, not curation triage. This should be worked after the re-sweep, not before, or the sweep will be re-deriving the same 290.

The 6 that are real, characterised

These are exactly the snippets that were validating circularly before #593 — they passed only because they were listed in the cache file. Now that the listing is gone they are checked against real full text, and they are three different things, none of them fabrication:

1 — a rendering artefact the normaliser does not cover. PET ... P. putida KT2440 (ATCC(R) 47054). The cache says (ATCC® 47054). The curator wrote (R) for ®. This belongs in RENDERING, not MISMATCH, and per the justfile must not be fixed by editing the snippet.

2 — table flattening. OMM12 ... "Lactobacillus reuteri I49, Enterococcus faecalis KB1, Blautia coccoides YL58". All three names are in the cache, but as separate rows of a strain table: ... Lactobacillus reuteri I49 DSM 32035 1 Bifidobacterium .... The curator joined three cells with commas. Faithful in content, not a contiguous quote.

3 — a quote stitched from two places. PET ... "R. jostii was added to reduce the inhibition caused by terephthalic acid". The paper has R. jostii was added to the existing consortium to break down TPA at one point and adding R. jostii to reduce the inhibition caused by terephthalic acid (TPA) at another. The snippet welds the opening of one to the tail of the other. This is the only one of the six that is a genuine curation defect, and it is a paraphrase presented as a quote — the #347 class.

Revised recommendation

1. Do Re-sweep the abstract-only caches: some "needs access" verdicts predate the #586 fix #591 first. It removes ~290 of these.
2. Teach the audit normaliser ®/(R) and the table-flattening case, so they land in RENDERING where they belong rather than inflating the suspect count.
3. Re-run, and treat whatever survives as the actual curation backlog. On today's evidence that is a single-digit number, not 296.

Leaving the original text above intact rather than editing it, since the wrong framing is the more useful record.