Re-sweep the abstract-only caches: some "needs access" verdicts predate the #586 fix

[realmarcin]

Follow-up promised in #587, kept out of that PR because the deliverable here is data, and it should not ride in on a tooling diff.

Why

189 PMID caches are still abstract-only. Some fraction of them are false negatives:

• before cache_fulltext.py: a transient 5xx aborts the batch and is indistinguishable from "not available" #586, a transient 5xx aborted the whole batch, so every reference after the failure was never attempted — and the output gave no way to tell that from "attempted and unavailable";
• before cache_fulltext.py: "OA but no full-text XML" is a verdict, and #587 reports it as an outage #590, a paper Europe PMC reports as OA but serves no full-text XML for was reported as an error rather than as the verdict it is;
• before cache_fulltext.py: an Unpaywall outage is reported as "this paper has no OA location" #589, an Unpaywall outage was reported as "no OA copy exists".

Sweeps run under those conditions produced the current "needs access" set. I do not know how much of it is real. That set is what #183 uses to decide which records cannot be enriched, so the error propagates into curation decisions.

Evidence that the yield is real

A 14-reference canary batch on #587, restricted to PMIDs > 33000000, returned 7 full texts and wrote 464 K. An earlier single fetch grew one cache from 485 bytes to 270,006. The strong predictor is publication year — a 14-reference batch of 2006–2009 papers returned zero, all genuinely closed.

Proposed pass

1. Re-run cache_fulltext.py over all 189 abstract-only caches, in batches, newest-first since that is where the yield is.
2. Record the outcome distribution: [cached] / not open-access / OA but no full-text XML / [error]. The last two are now distinguishable, which is the point.
3. For anything that comes back [error] after retries, re-run once more before believing it.
4. Report which records in Enrich records lacking members/growth conditions once full text is accessible #183's needs-access set are now reachable, and hand those to the normal fetch-first curation loop.

Not to be done blind: this is a batch, so it gets a canary first, and the per-batch counts get logged rather than summarised — a sweep that silently drops its tail is exactly what got us here.

Note on scope

This does not touch DOI-only references, which have their own retrieval gap (#259). Worth doing after, with the same method.

[realmarcin]

Measured yield, so this can be scheduled on evidence rather than hope

I probed every reference behind the MISMATCH backlog (#596) against Europe PMC — metadata + an actual full-text fetch attempt, nothing written to disk. This is the concrete answer to "is the re-sweep worth running".

78 references sit behind 290 MISMATCH snippets. Of those:

	references	retrievable	snippets	recoverable
PMIDs	58	38 (66%)	209	125
DOIs	20	15 (75%)	81	66
total	78	53 (68%)	290	191 (66%)

Two thirds of the entire MISMATCH backlog is recoverable by fetching alone, with no curation judgement required. Non-retrievable breakdown: 24 genuinely closed-access, 1 "OA but Europe PMC serves no full-text XML" (the #590 state).

Two things worth noting

The DOI path outperforms the PMID path — 75% of references and 81% of their snippets, against 66% and 60%. That is the opposite of what I expected, and it means DOI-cited records are not the poor relation here. Anything that treats DOIs as second-class in this sweep would be leaving the better yield on the table.

Retrievability tracks recency, hard. The closed set is dominated by older work (10.1128/aem.67.12.5520-5525.2001, 10.1016/j.cell.2014.05.041); the retrievable set is dominated by recent open-access venues (fmicb, ismejo, spectrum, fcimb). Consistent with the earlier observation that a 14-reference batch of 2006–2009 PMIDs returned zero while a batch of post-2033000000 PMIDs returned 7 of 14. Sweep newest-first — the yield curve is steep.

Suggested scope when this runs

Do these 53 references first rather than all 189 abstract-only caches. They are the ones with a known consumer: each one currently makes specific snippets unverifiable, so the benefit is measurable the moment the fetch lands, rather than speculative. The remaining abstract-only caches can follow.

Expected outcome, stated in advance so it can be checked rather than assumed: MISMATCH should fall from 296 to roughly 105, and MATCH should rise by most of the difference. If it does not, the assumption that these snippets are faithful Methods quotes is wrong and that is worth knowing.

Probe was read-only — git status clean throughout, no cache entries written.