Skip to content

Follow-ups to the absence guard, found by running it - #18

Merged
Robbie1977 merged 4 commits into
mainfrom
fix/absence-followups
Aug 10, 2026
Merged

Follow-ups to the absence guard, found by running it#18
Robbie1977 merged 4 commits into
mainfrom
fix/absence-followups

Conversation

@Robbie1977

Copy link
Copy Markdown
Contributor

Follow-up to #17. Everything here was found by running the merged guard and
measuring it, not by reading it — three defects it exposed, one it introduced,
and the decision record.

The one this branch introduced

Escalation had no sense of time. It buys a second synthesis, the most
expensive call in the pipeline. S14 escalated at around three minutes, went
round again, and timed out — turning a usable if hedged answer into "the
language service did not respond". One rep in thirty-nine, entirely
self-inflicted.

Half the run deadline was the wrong derivation. There are two ceilings, and the
tighter one governs: 600 s DEFAULT_RUN_DEADLINE_MS, and 240 s
TASK_BATTERY_TIMEOUT_MS, which is this project's own recorded opinion about
when an answer has taken too long to be worth having. An escalation starting at
290 s cannot finish a 60–90 s second pass before 240 s, so the guard was
guaranteed to turn the slowest answers into no answer at all. 120 s + 90 s =
210 s leaves margin; median first synthesis across the tranche is 63 s, so this
trims the tail rather than the guard. Env-overridable.

A false absence is worse than a hedge. It is not worse than no answer.

Three the measurement exposed

harnessFraming was producing the framing it exists to remove.
EVIDENCE_SUBJECT needs "the evidence"; INPUT_NOUN matches "the provided
evidence" as a unit and rewrites it to "VFB evidence". The subject rules ran
first, found "provided" in the way, and did nothing — and the noun rule then
produced exactly the subject they would have handled, one pass too late. S46
shipped:

VFB evidence lists queries for subclasses, scRNAseq data, transgene
expression … but it does not include a completed query or result set
identifying specific types lacking images.

This program's working set, wearing VFB's name. The rules were right and their
order was wrong, so the subject pass now runs again after the noun pass. The
existing test asserted the weaker outcome ("VFB evidence"); it now asserts the
one the module was always aiming at ("VFB records three connectome datasets").

Definitions were being summarised twice. "What is the ellipsoid body?"
returned the same 254 characters on all three reps. VFB's curated definition
does not stop there — the ellipsoid body canal, the anterior bundle, the 16
radial segments, the Ito et al. citation. Not a truncation: the extractor is a
weak model summarising the whole term-info record into one claim, so
authoritative prose went through two paraphrase hops before reaching the reader.
Summarising a summary costs the last two sentences identically every time, which
is why three reps agreed to the character. A definitional question now carries
the description verbatim as its own evidence row — the principle the pipeline
already runs on for counts, applied to text. 254 → 471 characters, all three
sentences and the citation intact.

The absence verb list was too narrow. It had present/available/found and not
provided/listed/given, and a live answer walked through the gap with "specific
counts … are not provided in the current data" about counts VFB advertises
queries for. Negative cases added alongside: "serotonin is not present in these
neurons" is a claim about the world and must survive.

Result

Same 13 questions, three reps each, against a local build of this branch:

absence claims errors median max
production v4.2.1 18 / 39 reps 0 66 s 429 s
this branch 1 / 39 reps 0 77 s 246 s

Task battery 64/64, 0 errors, on the tip. 1188 unit tests, 0 failures.

Also

docs/adr/absence-requires-evidence.md records the decision, and specifically
the three costs that are places a future reader may reasonably disagree —
relevance orders escalation rather than vetoing it; a query the guard chose
cannot license the claim it was chosen to check; escalation stops at 120 s — each
with the measurement that settled it. It also records the known limits: a licence
is per-run rather than per-claim, and the detector is a pattern list that should
grow from observed output rather than from imagination.

One thing left for you: T3.8 runs at 190–205 s against the workflow's
TASK_BATTERY_TIMEOUT_MS of 240 s and has been marginal for several releases —
it timed out once here without escalating at all. .github/workflows/task-battery.yml
wants 240000360000; my token has no workflow scope.

claude added 3 commits August 10, 2026 09:56
Escalation had no sense of time. It buys a second synthesis, which is
the most expensive call in the pipeline, and S14 escalated at around
three minutes and then timed out — turning a usable if hedged answer
into "the language service did not respond". One rep in thirty-nine and
entirely self-inflicted. A false absence is worse than a hedge; it is not
worse than no answer, so the escalation now stops wanting a second
opinion past half the run deadline. The gate still runs, so a run that
skips escalation still cannot ship a denial it did not earn.

harnessFraming was producing the framing it exists to remove.
EVIDENCE_SUBJECT needs "the evidence"; INPUT_NOUN matches "the provided
evidence" as a unit and rewrites it to "VFB evidence". The subject rules
ran first, found "provided" in the way, did nothing — and the noun rule
then produced exactly the subject they would have handled, one pass too
late. S46 shipped "VFB evidence lists queries for subclasses, scRNAseq
data ... but it does not include a completed query": this program's
working set wearing VFB's name. The rules were right and their order was
wrong, so the subject pass now runs again after the noun pass.

Definitions were being summarised twice. "What is the ellipsoid body?"
returned the same 254 characters on all three reps, dropping two thirds
of VFB's curated definition — the ellipsoid body canal, the anterior
bundle, the 16 radial segments and the Ito et al. citation. Not a
truncation: the extractor is a weak model summarising the whole record
into one claim, so authoritative prose went through two paraphrase hops
before reaching the reader. A definitional question now carries the
description verbatim as its own evidence row. 254 characters to 471, all
three sentences and the citation intact.
First CI run of this branch: 63 of 64 tasks passed and the 64th was
"Timed out after 240000 ms". Half the run deadline was the wrong
derivation. There are two ceilings — 600 s DEFAULT_RUN_DEADLINE_MS, and
240 s TASK_BATTERY_TIMEOUT_MS, which is this project's own recorded
opinion about when an answer has taken too long to be worth having — and
the tighter one governs. An escalation starting at 290 s cannot finish a
60-90 s second pass before 240 s, so the guard was guaranteed to turn the
slowest answers into no answer at all.

120 s + 90 s = 210 s leaves margin under the tighter ceiling. Median
first synthesis across the 39-rep tranche is 63 s, so this trims the tail
rather than the guard. Env-overridable: how fast the backends are on the
day is a deployment fact, not a code one.
The three deliberate costs are the part worth writing down: relevance
orders escalation rather than vetoing it, a query the guard chose cannot
license the claim it was chosen to check, and escalation stops at 120 s
because a second opinion that arrives after the answer is due is not a
second opinion. Each is a place a future reader may reasonably disagree,
so each has the measurement that settled it.
@Robbie1977
Robbie1977 force-pushed the fix/absence-followups branch from 03d495c to e02f542 Compare August 10, 2026 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants