Allow annotating excerpts with spans indicating different aspect of claims

[cmungall]

Current excerpts might be used to support complex claims involving many aspects. LRV is used in projects that go beyond simple triple assertions.

E.g. the excerpt

"bob eats pepperoni pizza at weekends"
=>

subject: bob
predicate: likes
object: pizza
object_subtype: meat
temporality: sat

the relationship between the excerpt and the claim is complex. Parts of the except are more specific, parts involve a leap of inference, and parts overclaim. Additionally, a claim may be made from multiple excerpts.

The proposal is to allow annotation of an excerpt with spans:

<span role="subject">bob</span> <span role="predicate">eats</span> <span role="object_subtype">pepperoni</span> <span role="object">pizza</span> at <span role="temporality">weekends</span>

LRV would not dictate a vocabulary like "subject", "predicate", this is 100% under the control of the domain schema and none of LRV's business. LRV would remove span markers to derive the text that would be used for validation in exactly the same way.

Domain schemas could choose to also add other qualifiers beyond role, e.g. to score congruence, e.g. allowing them to derive a final claim-evidence score. Again, this is purely the business of the domain schema (but LRV could provide some minimal checking support).

The title field could also be annotated this way. Sometimes a claim support might span title and excerpt.

There would need to be a flag (off by default) that indicates whether spans are to be stripped. We don't want to break existing snippets that might intentionally include html/sgml markup

[github-actions]

Summary

This is a design proposal to let excerpts (and titles) carry inline span markup — <span role="...">...</span> — so that a single quoted excerpt can be decomposed into the parts of a structured claim it supports (subject / predicate / object / qualifiers), with the vocabulary of role values left entirely to the domain schema. LRV's job would stay narrow: strip the span markers to recover the plain text used for substring/quote validation, exactly as today, gated behind an off-by-default flag so existing snippets containing legitimate HTML/SGML aren't silently mangled.

Related issues in this repo

This slots into a cluster of open issues that are all pulling on the same thread — "an excerpt is not just a flat string, it's evidence for a structured claim, and the file format/data model needs to grow up to reflect that":

• Feature: Validate titles against fetched DOI metadata #10 – Validate titles against fetched DOI metadata (closed). This proposal's aside — "the title field could also be annotated this way" — directly extends Feature: Validate titles against fetched DOI metadata #10. If title validation and excerpt validation eventually share a code path, spans in titles vs. spans in excerpts should probably reuse the same parsing/stripping logic rather than diverging.
• Support multiple --text-group values in a single validation pass #40 – Support multiple --text-group values in a single validation pass (open). Same underlying need at the CLI layer: today you validate title and excerpt as two separate passes over two separate capture groups. Spans are the in-schema analog of that — multiple "named" sub-texts inside one excerpt. Worth checking whether a unified internal representation ("one excerpt → N named text fragments, whether the fragments come from regex groups or from spans") could serve both issues.
• Allow image snippet checking #63 – Allow image snippet checking (open). Conceptually the same generalization one level up: "excerpt" stops being a bare string and becomes a richer evidence object (text with role-tagged spans, or a cropped image region). If both land, a shared "Evidence" abstraction (as opposed to two ad hoc extensions) would avoid duplicated validation logic.
• The cache-file format has no owner: consumers must mirror the emitter, so every new frontmatter field is a downstream break #89 – The cache-file format has no owner (open). Not about excerpts directly, but the same lesson applies here in advance: if spans become a real LRV feature, the span/role data model should be published as versioned public API (a LinkML class, presumably) rather than left implicit in string-parsing logic, so downstream projects (dismech etc.) aren't left reverse-engineering the markup grammar the way The cache-file format has no owner: consumers must mirror the emitter, so every new frontmatter field is a downstream break #89 describes them reverse-engineering the cache frontmatter.

A design consideration worth flagging before implementation

Inline markup (<span role="x">...</span>) has a structural limitation that stand-off/offset-based annotation avoids: spans can't overlap or nest awkwardly without extra escaping machinery, and this proposal's own example is close to hitting that — temporality covers "weekends" but a real claim might need a span that overlaps with predicate or spans a discontinuous range (e.g., "eats ... at weekends" as one temporal-scope span with object nested inside it). This is a well-trodden problem in biomedical/NLP text annotation, and the established prior art is to keep the text untouched and store spans out-of-band as (start_offset, end_offset, role) triples referencing it:

• The W3C Web Annotation Data Model (TextQuoteSelector / TextPositionSelector) is the general-purpose standard for this — annotations as separate objects pointing at exact/prefix/suffix-anchored text ranges, rather than markup embedded in the text itself.
• BioC, PubAnnotation, and the brat rapid annotation format are the biomedical-NLP-specific versions of the same idea — all "stand-off" rather than inline, precisely because inline tagging can't cleanly represent overlapping or discontinuous spans and because it risks colliding with markup that's part of the actual source text (the exact worry raised in this issue's last paragraph about not breaking snippets with legitimate HTML/SGML).
• The general research framing for "a span of source text as evidence for a structured judgement" is what NLP calls extractive rationales (e.g. the ERASER benchmark line of work, DeYoung et al. 2020) — not directly reusable code, but useful vocabulary/prior art if a "congruence/evidence score" (as floated in the issue) gets designed later.

None of this necessarily changes the proposal — inline spans are simpler to author by hand in a markdown/YAML excerpt, which may matter more here than in a document-annotation tool — but it's worth deciding deliberately rather than defaulting to inline, since retrofitting offset-based spans later would be a breaking format change. Worth a line in the eventual design doc/PR about why inline-vs-stand-off was chosen either way.

(Note: I wasn't able to run a live literature search in this session — the references above are from general knowledge rather than a fresh lookup, so it's worth independently confirming exact citations before relying on them.)

---

🤖 Generated with Claude Code

[cmungall]

Agent disclosure: I am Codex, an AI agent, posting at @cmungall’s request.

I think this is a good fit for aspect-level claim/evidence evaluation. A span identifies the words offered as evidence for an aspect; a review records whether those words justify that aspect. Leaving role meanings and congruence judgments to the domain schema gives LRV a clear responsibility: validating the quoted text after removing the annotation wrappers.

A domain schema could use claim-relative paths as roles. For the claim object in this issue, for example:

<span role="/predicate">eats</span>
<span role="/temporality">weekends</span>

These paths would identify the claimed values likes and sat, respectively. The annotation makes each proposed relationship inspectable: the evaluator still needs to assess the inference from eating to liking, and whether “weekends” establishes the specifically claimed Saturday timing. A domain convention could use / for the whole claim; LRV would not need to interpret any of these paths.

A few details seem worth specifying:

• Allow multiple spans per aspect and multiple aspects per span. One phrase can bear on both a term and its qualifier. Contextual implications may require selecting an entire sentence. Absence of a narrowly corresponding phrase should not automatically become a mismatch: a qualifier can be reasonably implied without appearing verbatim.
• Keep the surrounding excerpt available to the evaluator. A highlighted phrase can depend on nearby negation, uncertainty, population, or experimental scope. The span helps locate the evidence while the excerpt preserves its meaning.
• Keep reviewer judgments separate from evaluated inputs. Producer-supplied role annotations could be useful inputs to an evidence evaluator. Curator-supplied congruence scores would reveal the expected answer and should stay in review metadata when benchmarking that evaluator.
• Make title participation explicit. Validating annotated titles fits naturally. Using the title as additional evidence changes what the evaluator receives, so an evaluation configuration should record whether the task uses the excerpt alone or title plus excerpt.

For hand-curated YAML, I favor starting with the proposed opt-in inline markup. The earlier comment’s concern about overlapping spans is real, but a general annotation framework need not be a prerequisite. Define precisely which wrappers are removed, preserve their enclosed text and existing source markup, and check interaction with ellipses and editorial brackets.

A useful downstream interface would let a curator select a claim aspect and see its proposed evidence highlighted alongside the assessment and rationale. That would make partial support much easier to inspect than a single whole-claim verdict.