Refresh deletes a cached abstract when PubMed stores it in <OtherAbstract> (PIP/NASA/KIE/AIDS)

[cmungall]

Refreshing a cached PubMed record can silently delete the abstract it already held. For old records whose abstract PubMed stores in <OtherAbstract> rather than <Abstract>, the extractor finds nothing, writes content_type: unavailable, and overwrites the cache file with an empty ## Content section. Any snippet validated against that text then fails with "No content available for reference".

PMID:5697815 is a worked example. Its cache entry held a 967-character abstract; after a refresh the file went from 1787 to 850 bytes with the abstract gone. PubMed does serve the text — it is just in a different element:

$ curl -s "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pubmed&id=5697815&retmode=xml" \
    | grep -oE "<(Abstract|OtherAbstract)[^>]*>"
<OtherAbstract Type="PIP" Language="eng">
<AbstractText>

There is no <Abstract> element at all, so this line in _parse_abstract returns None:

https://github.com/linkml/linkml-reference-validator/blob/main/src/linkml_reference_validator/etl/sources/pmid.py#L233

abstract = soup.find("Abstract")

if not abstract:
    return None

PMID:6680124 fails the same way (1695 → 650 bytes, a 1175-character PIP abstract dropped).

Suggested fix

Fall back to <OtherAbstract> when <Abstract> is absent. Type is one of PIP, KIE, NASA, or AIDS — all genuine abstracts contributed by other indexing programs, mostly on pre-1990 records. Prefer Language="eng", since OtherAbstract is also where PubMed puts translated abstracts, and a French translation should not become the text that English snippets are matched against.

Worth noting this is a data-loss bug rather than a coverage gap: the entry is not merely left un-enriched, the text it already had is deleted. That puts it in the same family as #85, which stopped a refresh demoting a full-text entry — the same protection does not cover abstract_only, because an abstract-only entry returning no abstract looks like an honest no-op.

How I hit it

Running validate over one dismech KB entry refreshed 148 cache files. Four went abstract_only → unavailable; two of those (the ones above) were this bug. The other two are correct — PMID:4869291 and PMID:37769103 genuinely have no abstract in PubMed, and the previous cache had stored the MEDLINE citation header as though it were content, so unavailable is the more honest label there.

Across the dismech cache, 15 files currently hold an abstract that lives in OtherAbstract (11 PIP, 3 AIDS, 2 NASA) and would be scrubbed on their next refresh.

[github-actions]

Verification

Independently re-fetched both records from efetch just now — the report is accurate:

PMID	<Abstract>	<OtherAbstract>
5697815	absent	Type="PIP" Language="eng" — "A study was undertaken to determine whether the absorption of glucose from the normal small bowel also involves increased absorption of sodium and water..."
6680124	absent	Type="PIP" Language="eng" — "A rapid test to identify Vibrio cholerae in stools has been developed. The test depends on the ability of the vibrios to multiply in a specially designed medium..."

And _parse_abstract in pmid.py:233 (src/linkml_reference_validator/etl/sources/pmid.py) does exactly soup.find("Abstract") → None on both, confirming the mechanism.

Why #85's guard doesn't catch this

Traced _preserve_cached_full_text in reference_fetcher.py:420: it opens with if cached is None or self.needs_full_text(cached): return None — i.e. it bails out (allows the overwrite) whenever the cached entry itself still needs full text. An abstract_only entry always needs full text, so this guard exits on its very first line for exactly the case in this issue. It was built to stop full_text_* → abstract_only demotion, not abstract_only → unavailable — the issue's own framing ("the same protection does not cover abstract_only") is correct and precisely locates the gap.

Pattern

This is the third report in this family in under a week: #85 (full-text silently demoted to abstract on refresh, merged 2026-09-18) and #67 (XML extractor discarding a full body over a substring match, closed 2026-09-15) are both "refresh silently destroys previously-good cached content because a narrower/failed fetch looks indistinguishable from an honest negative." Worth considering, as a follow-up rather than part of this fix, whether _preserve_cached_full_text's underlying principle ("a refresh may improve an entry; it may never demote one") should generalize to a content-type ordering (unavailable < abstract_only < full_text_*) rather than three separate special-cased guards accreting one per source.

On the suggested fix

Falling back to OtherAbstract with Language="eng" preferred looks right and matches what's in the live records above. Worth deciding explicitly in the PR: if only a non-English OtherAbstract exists (no eng copy), should it be used anyway (better than nothing, but validating English snippets against it would legitimately fail) or should that still resolve to unavailable? The issue doesn't say, and it's the one edge the two worked examples don't exercise since both are Language="eng".

Given how precisely scoped this already is — exact line, two verified worked examples, suggested fix, and the known-affected set (15 dismech cache entries: 11 PIP, 3 AIDS, 2 NASA) — this looks ready to hand off:

@dragon-ai-agent please fix _parse_abstract in src/linkml_reference_validator/etl/sources/pmid.py to fall back to <OtherAbstract> when <Abstract> is absent, preferring Language="eng" when multiple OtherAbstract elements exist (falling back to unavailable if no English-language abstract or other-abstract exists — do not serve a non-English abstract for matching against English snippets). Write tests first per this repo's TDD convention, covering both PMID:5697815 and PMID:6680124 as worked examples, plus a case with a non-English-only OtherAbstract. Follow TDD and doctest conventions from CLAUDE.md.