The cache-file format has no owner: consumers must mirror the emitter, so every new frontmatter field is a downstream break

[cmungall]

linkml-reference-validator writes the files in references_cache/, but offers no way to validate one. There is an emitter (ReferenceFetcher._save_to_disk) and a deliberately lenient reader (_load_markdown_format, which reads field by field with frontmatter.get(...) and ignores anything it does not recognise). Both are private, and there is no model, schema, or checker for the format.

So a consumer that wants to gate on cache-file integrity — "is this file well-formed, does its name match its reference_id, has somebody hand-edited it" — has to reimplement the format. dismech did: ReferenceCacheFrontmatter, a Pydantic model with extra="forbid" listing every field it had observed this library write.

That mirror inverts your own leniency into strictness, and the result is that any field you add is a breaking change downstream. It just happened. Upgrading from v0.2.1 to current main added five fields:

extractor_version        145
xml_extraction_version    44
full_text_declined        32
full_text_access_type      3

Validating a single KB entry rewrote 146 cache files, and all 146 then failed dismech's check-reference-cache-frontmatter gate with Extra inputs are not permitted. Nothing was wrong with the files; the downstream schema simply did not know about fields the upstream emitter had started writing.

The generalisation is the problem, not this instance. Every consumer that gates on these files has to mirror your emitter, and every field you add obliges you to negotiate a migration with each of them — or they go red. That is a lot of churn for a library whose reader does not care about unknown fields at all.

What I think should happen

The library that writes the format should own its contract. Concretely:

1. Publish the frontmatter model as public API, versioned with the library.
2. Publish a checker — validate_cache_file(path) and scan_cache_dir(dir) returning structured findings — so a consumer gets "is this a valid cache file" without reimplementing it.
3. Make it forward-compatible by construction: unknown keys must be allowed. This matters in both directions. Your reader already ignores them, so strictness would be a new and stricter claim than the library itself makes. And consumers legitimately add their own — dismech writes a database: field naming the structured source (Orphanet, ClinGen, ICEES) a record came from, which your loader silently and correctly ignores today.

A validator that rejects unknown keys would break that, so the sensible contract is: these fields must be present and well-typed; anything else is none of the library's business. If you would rather extensions be namespaced (x_database:, or a nested extensions: map) that works too, as long as it is declared rather than left for each consumer to guess.

Then a downstream gate becomes: call your checker for format validity, and layer only genuinely local policy on top. dismech's remaining rules are things you should not care about — for example that a PMID: record must carry authors or a journal, with a carve-out for NCBI Bookshelf and agency-guideline monographs.

Interim

dismech has added the five fields to its own model so its gate passes, including html_full_text_version, which none of the 146 files carried but the emitter can write. That is a patch over the symptom and it will need doing again on the next field you add — which is the argument for this issue.

Related: this is the same coupling that #88 shows from the other side. A consumer cannot tell "this record genuinely has no abstract" from "this version of the extractor could not find one", because the file format carries no statement about what the writer was able to determine.

[github-actions]

Summary

This is a well-scoped API-design request, not a bug report, and it checks out against current main: ReferenceFetcher._save_to_disk (reference_fetcher.py:1279) and _load_markdown_format (reference_fetcher.py:1646) are both still private, _save_to_disk currently emits at least extractor_version, xml_extraction_version, full_text_declined, and full_text_access_type as frontmatter keys, and there is no ReferenceContent-derived schema, Pydantic model, or validate_cache_file/scan_cache_dir entry point anywhere in src/. So a downstream consumer really does have only two options today: reimplement the format strictly (and break every time a field is added, as dismech's extra="forbid" model did), or reimplement it loosely (and silently miss real corruption).

Pattern: this is the third time private-cache internals have leaked into a downstream dependency

Looking at the closed-issue history, this repo has hit variants of "the cache format/helpers are private but a consumer needs them" repeatedly:

• Make _split_query public: downstream snippet reporting has to reimplement it #74 — _split_query had to be made public because downstream snippet reporting needed to reimplement it. Direct precedent for "publish the private helper as API" being the accepted fix here.
• Frontmatter is split on the first --- substring, not the first --- line, so titles containing --- break the record #71 — frontmatter was split on the first --- substring rather than the first --- line, corrupting records whose title contained ---. A structural frontmatter-parsing bug, fixed, but it's exactly the class of defect a public validate_cache_file checker (as proposed here) would have caught immediately in CI instead of via a downstream KB run.
• _save_to_disk writes unquoted author values, so an author containing a colon does not survive a round trip #70 — _save_to_disk wrote unquoted YAML author values, so an author name containing a colon didn't round-trip. Same theme: the emitter has no schema/contract holding it to a serialization guarantee, so hand-rolled string formatting has repeatedly produced subtly invalid frontmatter.
• Landing-page fallback is stamped full_text_html: a cache entry can claim full text while holding only the abstract #65 — a landing-page fallback got mislabeled full_text_html even though only the abstract was stored — i.e., the frontmatter can assert something about content quality/completeness that isn't true. This is the same "the format carries no statement about what the writer was able to determine" gap called out in the issue's link to Refresh deletes a cached abstract when PubMed stores it in <OtherAbstract> (PIP/NASA/KIE/AIDS) #88.

None of these were caused by a schema-less format, but a public model + checker (as this issue proposes) would have surfaced #71 and #70 as a test failure in this repo rather than as cache corruption discovered downstream, and would let #65/#88's "was the writer even able to determine this" ambiguity be resolved as an explicit field instead of an inferred one.

Related: #88

Confirmed connection, as noted in the issue body: #88 (refresh silently drops abstracts that live in PubMed's <OtherAbstract>) is the same root coupling from the content side — a consumer can't distinguish "genuinely no abstract" from "this extractor version couldn't find one" because the frontmatter has no field for extractor confidence/completeness. If frontmatter gets a public schema per this issue, that's a natural place to add such a field (e.g., an explicit abstract_status: absent | not_attempted | extraction_failed) rather than overloading full_text_declined/content_type.

Suggested framing for whoever picks this up

The three concrete asks in the issue are independently shippable and roughly increasing in cost:

1. Publish the frontmatter model (e.g., a CacheFrontmatter TypedDict/dataclass/Pydantic model in models.py, versioned alongside EXTRACTOR_CACHE_VERSION/XML_EXTRACTION_CACHE_VERSION) — cheapest, mostly a rename/export of what _save_to_disk already constructs field-by-field.
2. Publish validate_cache_file(path) / scan_cache_dir(dir) built on that model, returning structured findings — natural fit for cli/cache.py, which already has cache-inspection commands.
3. Decide and document the unknown-key policy (allow-and-ignore vs. namespaced x_*/extensions: map) so it's a declared contract instead of an emergent property of a lenient parser. Given _load_markdown_format already ignores unrecognized keys today, the honest move is to make that permissiveness the documented contract (option 3 in the issue) rather than switching to strict validation, which would just relocate the breakage this issue is about.

No PR opened per instructions — flagging the concrete precedents (#74, #71, #70, #65) and the #88 linkage for whoever scopes the implementation.

[cmungall]

A concrete instance of this, from the consumer side — and the sharpest part is that I caused it myself.

Retiring dismech's monkeypatches required upgrading its pin to 0.3.0rc1. Its ReferenceCacheFrontmatter uses extra="forbid", so every field this library added since its previous pin rejected every cache file the new version touched. That took three CI failures to fully surface:

1. Five fields at once (extractor_version, xml_extraction_version, html_full_text_version, full_text_declined, full_text_access_type) — 146 of 146 refreshed files rejected.
2. absent_content_version — 3 files.
3. full_text_source_item_id — not yet triggered, found only by going looking.

Item 2 is the instructive one. absent_content_version is the stamp I added here in #90, after writing the downstream model fix. So an upstream change I made myself broke a downstream mirror I maintain myself, and I still did not notice until CI told me. That is not a discipline problem that better attention fixes — the consumer has no signal that the emitter grew a field.

What I ended up doing downstream is a workaround for the missing contract, not a fix:

emitted = set(re.findall(r'lines\.append\(\s*\n?\s*f?"([a-z_]+):', inspect.getsource(reference_fetcher)))
missing = sorted(emitted - set(ReferenceCacheFrontmatter.model_fields))
assert not missing

A regex over your source, in a consumer's test suite. It works, and it caught both remaining fields — but it depends on the exact spelling of lines.append(f"key: ...") in a private method, so it breaks the moment the emitter is refactored. It also has to special-case two strings that look like frontmatter keys and are not: access_type: (a value prefix inside full_text_declined) and clinicaltrials: (a CURIE prefix).

Nobody should have to write that. It is the strongest argument I can give for the ask in this issue: publish the model and a checker, and make unknown keys permitted so a consumer's own extension field (dismech writes database:, which your loader already ignores correctly) does not become a breaking change in the other direction.