Follow-ups to the absence guard, found by running it

docs/adr/absence-requires-evidence.md

@@ -0,0 +1,88 @@
+# Absence requires evidence
+
+Status: accepted
+Date: 2026-08-10
+
+## Context
+
+`lib/coverage.mjs` has stated the rule since the four-state model was
+introduced: of RUN, EMPTY, FAILED and UNRUN, only EMPTY licenses an absence.
+The synthesis prompt repeats it three times in capitals.
+
+Measured against production v4.2.1 — thirteen questions, three repetitions each,
+answers judged blind — **fourteen of thirty-nine answers asserted that VFB holds
+nothing, and not one of them had a query behind it that had run and come back
+empty.** Three of those denials were about `γ Kenyon cells`, `α/β Kenyon cells`
+and `MBON-γ1pedc>α/β`, which are FBbt_00100247, FBbt_00100248 and
+FBbt_00100246 — classes the ontology has held for years.
+
+An instruction violated at that rate is not a control. This is the same lesson
+the project already learned for counts, where the answer was not a firmer
+instruction but `repairMistranscribedCounts`: state the rule, then enforce it
+deterministically after the model has had its say.
+
+## Decision
+
+Three layers, in the order they should bite.
+
+**1. Make the name findable.** `lib/nameNormalise.mjs` transliterates Greek
+letters to the spellings FlyBase and FBbt actually store, and normalises primes,
+arrows, dashes and non-breaking spaces. It is the first rung of `nameVariants`,
+and only when there is something to transliterate — an ASCII name still pays
+exactly one extra search for its plural, as before.
+
+**2. Go and look before conceding.** If the drafted answer claims an absence and
+nothing came back empty, `maybeEscalateBeforeAbsence` injects the failed and the
+most relevant unrun queries and writes the answer again. Failed queries go
+first: a lookup that fell over is the one case with positive evidence that the
+absence is an artefact.
+
+**3. Do not ship what was not earned.** `gateAbsence` removes absence claims the
+ledger does not license. It is the floor, not the fix — by the time it fires,
+escalation has already had its chance.
+
+`renderShelf` also gained a NAMES THAT DID NOT MATCH block, for the
+half-resolved question: one term matches, another does not, the shelf is
+non-empty so the no-coverage floor never fires, and nothing in the prompt
+distinguished "never looked at" from "checked and found wanting".
+
+## Consequences
+
+The absence rate went from 14/39 to 1/39 on the same questions, with no errors
+and a median cost of 66 s → 77 s. The full 64-task battery stays at 64/64.
+
+Three deliberate costs, recorded because each one is a place a future reader may
+reasonably disagree:
+
+- **Relevance orders escalation attempts; it does not veto them.** "Are any
+  mushroom body output neurons cholinergic?" scores every query the class
+  advertises at zero, because no label contains the word "cholinergic" — while
+  `SubClasses`, scoring zero, is the query that answers it. Filtering here would
+  escalate nothing in the case the guard exists for.
+
+- **A query the escalation chose cannot license the claim it was chosen to
+  check.** An early version ran `ListAllAvailableImages` for a question about
+  hemisphere symmetry, got an empty, and licensed the symmetry denial with it.
+  The guard was manufacturing its own permission, which is worse than the bug it
+  replaces. So a run where escalation genuinely proves an absence now hedges it
+  rather than stating it flatly. Hedging a true absence is a worse answer;
+  asserting a false one is a wrong answer.
+
+- **Escalation stops at `ABSENCE_ESCALATION_DEADLINE_MS` (120 s, env-tunable).**
+  It buys a second synthesis costing 60–90 s, and one started too late takes the
+  first answer down with it — which it did, once, on the first CI run of this
+  branch. A false absence is worse than a hedge; it is not worse than no answer.
+
+## Known limits
+
+- A licence is per-RUN, not per-claim: one genuinely empty query entitles every
+  absence sentence in that answer. Narrowing it to the subject of each sentence
+  needs claim-to-query matching, which is not reliably decidable from the text.
+- The detector is a pattern list built from sentences production actually wrote.
+  It will miss constructions nobody has written yet. Add them from observed
+  output rather than by imagination, and keep a negative case beside each one —
+  "serotonin is not present in these neurons" is a claim about the world and
+  must survive.
+- `T3.8` runs at 190–205 s against the workflow's `TASK_BATTERY_TIMEOUT_MS` of
+  240 s and has been marginal for several releases. It is unrelated to this
+  change and wants that ceiling raised to 360000.

lib/absence.mjs

@@ -175,7 +175,25 @@ const ABSENCE_PATTERNS = [
   // is not fixed, so the list has to cover the family rather than the instance.
   new RegExp(`\\b(?:is|are|were|was)\\s+not\\s+(?:present|available|found|included|recorded|annotated|provided|listed|given|shown|reported|captured|specified|detailed)\\b${GAP}\\b(?:in|for|within|by)\\s+${DB}`, 'i'),
   // "no such records exist" / "there is no record of X" with VFB nearby
-  new RegExp(`\\bthere (?:is|are) no\\b${GAP}\\b(?:record|data|entry|entries|image)s?\\b`, 'i')
+  new RegExp(`\\bthere (?:is|are) no\\b${GAP}\\b(?:record|data|entry|entries|image)s?\\b`, 'i'),
+  // THE EVIDENCE BLOCK, TALKING ABOUT ITSELF, IN VFB'S NAME.
+  //
+  // harnessFraming rewrites "the provided evidence" to "VFB evidence", which
+  // reads to a user as a statement about the database. Its author deliberately
+  // declined to rewrite the NEGATIVE forms — turning "the evidence does not
+  // provide X" into "VFB does not record X" would upgrade a limit of this answer
+  // into a claim about VFB, which is the invention that module exists to stop.
+  // Correct, and it leaves the sentence in place:
+  //
+  //   "VFB evidence lists queries for subclasses ... but it does not include a
+  //    completed query or result set identifying specific types lacking images."
+  //
+  // Deleting it is the move neither module was making. It is not a claim about
+  // fly anatomy that a reader loses, it is a description of this program's
+  // working set, and the gap between subject and negation is long here — the
+  // sentence recites a whole catalogue first — so this pattern gets a long window
+  // that the specificity of its subject pays for.
+  new RegExp('\\b(?:VFB|the|this)\\s+evidence\\b[^.!?\\n]{0,160}?\\b(?:does not|doesn\'t|do not)\\s+(?:include|contain|provide|list|show|specify|report)\\b', 'i')
 ]
 
 /** Split into sentences, keeping each sentence's offset so a repair can be spliced back. */

lib/harnessFraming.mjs

@@ -106,7 +106,25 @@ const INPUT_NOUN_RULES = [
 // X" is an admission of a gap, and rewriting it to "VFB does not record X"
 // would upgrade a statement about this answer's reach into a claim about the
 // database — exactly the invention the INPUT_SENTENCE rule exists to avoid.
-const EVIDENCE_SUBJECT = '(?:the|this|that)\\s+(?:VFB\\s+)?evidence'
+//
+// THE DETERMINER IS OPTIONAL, AND THAT IS A BUG FIX RATHER THAN A LOOSENING.
+//
+// INPUT_NOUN matches "(the )?provided evidence" INCLUDING the article, so by the
+// time these rules run the text no longer says "the evidence" — it says "VFB
+// evidence", with nothing in front of it. Every rule below then failed to match
+// this module's own output. Production, S46:
+//
+//   "VFB evidence lists queries for subclasses, scRNAseq data, transgene
+//    expression, and connectivity for mushroom body output neuron, but it does
+//    not include a completed query or result set identifying specific types
+//    lacking images."
+//
+// That began life as "the provided evidence lists …", should have become "VFB
+// records …", and instead shipped a sentence about this program's working set
+// wearing VFB's name. It is the exact failure this module exists to prevent,
+// produced by this module, because two of its passes disagreed about whether the
+// article was part of the noun.
+const EVIDENCE_SUBJECT = '(?:(?:the|this|that)\\s+(?:VFB\\s+)?|VFB\\s+)evidence'
 
 const EVIDENCE_SUBJECT_RULES = [
   // "the evidence does provide information on X" -> "VFB records X"
@@ -184,6 +202,22 @@ export function stripHarnessFraming(text = '') {
   for (const re of CLAUSE_RULES) out = out.replace(re, '')
   for (const [re, to] of EVIDENCE_SUBJECT_RULES) out = out.replace(re, to)
   for (const [re, to] of INPUT_NOUN_RULES) out = out.replace(re, to)
+  // AND AGAIN, BECAUSE THE PASS ABOVE CREATES WORK FOR THE PASS BEFORE IT.
+  //
+  // EVIDENCE_SUBJECT needs "the evidence"; INPUT_NOUN matches "the provided
+  // evidence" as a unit and rewrites it to "VFB evidence". So a sentence written
+  // as "the provided evidence lists queries for …" is invisible to the subject
+  // rules on the first pass — the word "provided" is in the way — and by the
+  // time it has been normalised into the shape those rules recognise, they have
+  // already run. It shipped as "VFB evidence lists queries for subclasses,
+  // scRNAseq data, transgene expression … but it does not include a completed
+  // query", which is this program's working set wearing VFB's name.
+  //
+  // Two passes, not a merged rule: the rules are correct, their ORDER was wrong,
+  // and re-running them is cheaper to read and to verify than teaching every
+  // subject rule to also spell out INPUT_NOUN. It terminates because the second
+  // pass has no output the first pass has not already seen.
+  for (const [re, to] of EVIDENCE_SUBJECT_RULES) out = out.replace(re, to)
 
   // Only after something was actually removed or rewritten. An answer that
   // opens with "However," of its own accord is the model's prose, not our

lib/ledger.mjs

@@ -21,6 +21,12 @@ export function createLedger(question, { maxToolRounds = 24 } = {}) {
     resources: [],               // data_resource handles (lightweight)
     retrieval: [],               // [{ kind:'doc'|'literature', reasons:[], ... }]
     budget: { toolRoundsLeft: maxToolRounds, tokenEstimate: 0 },
+    // Wall clock, not rounds. The round budget bounds how much WORK a question
+    // may do and says nothing about how long that work has taken — and the two
+    // come apart badly when the backends are slow. Anything deciding whether to
+    // buy one more model call late in a run has to know how much of the run
+    // deadline is already gone.
+    startedAt: Date.now(),
     openQuestions: [],           // sub-questions still uncovered (derived from plan)
     registry: {}                 // authoritative normalised-label -> id, from VFB data only
   }

lib/orchestrator.mjs

@@ -396,6 +396,39 @@ export async function runHarness(question, deps) {
   return { answer: gateAbsence(ledger, answer, log), ledger, trace, guardTripped: true }
 }
 
+/**
+ * How far into a run the escalation may still buy a second synthesis.
+ *
+ * THE NUMBER IS DERIVED, NOT PICKED.
+ *
+ * The escalation costs up to three MCP rounds plus a full synthesis, and the
+ * measured cost of that second pass is 60-90 s. So the only question is what it
+ * has to finish inside of, and there are two ceilings, of which the tighter one
+ * governs:
+ *
+ *   600 s  DEFAULT_RUN_DEADLINE_MS, the product's real ceiling.
+ *   240 s  TASK_BATTERY_TIMEOUT_MS, which is this project's own recorded opinion
+ *          about when an answer has taken too long to be worth having.
+ *
+ * Half the run deadline was the first attempt, and it is wrong against the
+ * second: an escalation starting at 290 s cannot possibly finish before 240 s,
+ * so the guard was guaranteed to turn the slowest answers into no answer at all.
+ * It did precisely that on the first CI run of this branch — 63 of 64 tasks
+ * passed and the 64th was "Timed out after 240000 ms".
+ *
+ * 120 s + 90 s = 210 s leaves real margin under the tighter ceiling. Measured
+ * across the 39-rep tranche, the first synthesis lands inside 120 s on the large
+ * majority of questions (median 63 s), so this trims the tail rather than the
+ * guard.
+ *
+ * Env-overridable because how fast the backends are on the day is a deployment
+ * fact rather than a code one.
+ */
+export const ABSENCE_ESCALATION_DEADLINE_MS = Math.max(
+  10000,
+  Number(process.env.ABSENCE_ESCALATION_DEADLINE_MS) || 120000
+)
+
 /**
  * Go back and look, rather than tell the reader there is nothing to find.
  *
@@ -426,6 +459,25 @@ export async function runHarness(question, deps) {
  */
 export async function maybeEscalateBeforeAbsence(ledger, answer, deps, log = () => {}) {
   if (!ledger || ledger._absenceEscalated) return false
+  // TIME, WHICH IS NOT THE SAME THING AS ROUNDS.
+  //
+  // The escalation buys a second synthesis, and a synthesis is the most
+  // expensive call in the pipeline. Measured on the first full run of this
+  // branch: S14 escalated at around three minutes, went round again, and the
+  // second synthesis timed out — so a question that had produced a usable if
+  // hedged answer produced "the language service did not respond" instead. One
+  // rep in thirty-nine, and entirely self-inflicted.
+  //
+  // A false absence is worse than a hedge, which is why this guard exists. It is
+  // NOT worse than no answer at all, so the guard has to stop wanting a second
+  // opinion once there is no longer time to form one. The gate below still runs
+  // — it costs nothing — so a run that skips escalation still cannot ship a
+  // denial it did not earn. It simply has to hedge instead of going and looking.
+  const elapsed = Date.now() - (ledger.startedAt || Date.now())
+  if (elapsed > ABSENCE_ESCALATION_DEADLINE_MS) {
+    log({ step: 'absence-escalation', injected: 0, reason: 'too-late', elapsed })
+    return false
+  }
   const licence = absenceLicence(ledger)
   // A run holding a genuine empty result is entitled to its absence. So is an
   // answer that never made one.
@@ -921,6 +973,43 @@ async function resolveTerms(ledger, names, deps, models, log, speculative = new
     }
     log({ resolve: name, id: effectiveId, refs: publications.length, queries: digest?.queries?.length || 0, superseded: superseded ? superseded.fromId : undefined })
 
+    // A DEFINITION IS ALREADY WRITTEN. DO NOT ASK A MODEL TO WRITE IT AGAIN.
+    //
+    // "What is the ellipsoid body?" returned the same 254 characters on all
+    // three reps: "a doughnut shaped synaptic neuropil domain of the central
+    // complex of the adult brain that lies just anterior to the fan-shaped
+    // body." VFB's own Description does not stop there:
+    //
+    //   "… Its hole (the ellipsoid body canal) points anteriorly and has an axon
+    //    tract (the anterior bundle) running through it. It is divided into
+    //    concentric layers and into 16 radial segments, 8 per hemisphere,
+    //    numbered 1-8 from superior medial to inferior medial (Ito et al.,
+    //    2014)."
+    //
+    // Two thirds of a curated, cited definition, gone — and not to a truncation
+    // bug. The extractor below is a weak model summarising the whole term-info
+    // record into one `claim`, so a definitional question sends authoritative
+    // prose through two paraphrase hops, extract and then synthesise, before it
+    // reaches the reader. Summarising a summary is what costs the last two
+    // sentences, and it costs them identically every time, which is why three
+    // reps agreed to the character.
+    //
+    // So for a definitional question the description is carried VERBATIM as its
+    // own evidence row. This is the same principle the rest of the pipeline
+    // already runs on — the model narrates, the deterministic layer carries what
+    // is already exact — applied to text rather than to counts. The extractor
+    // still runs, because it finds things the description does not say; this row
+    // only guarantees the definition itself survives the trip.
+    if (info && digest?.description && isDefinitionalQuestion(ledger.question)) {
+      addEvidence(ledger, buildEvidenceRow({
+        source: 'vfb',
+        claim: `VFB's full definition of ${digest.name || name}: ${digest.description}`,
+        verbatim: digest.description,
+        locator: { term: name, id: effectiveId, field: 'description' }
+      }))
+      log({ definition_verbatim: name, id: effectiveId, chars: digest.description.length })
+    }
+
     // VFB-first: let the extractor decide what in the FULL term-info answers the
     // question (Description, Relationships, Queries, counts, …) — no field is
     // pre-selected. Skip only when there is no term-info at all.