Follow-ups to the absence guard, found by running it - #18
Merged
Conversation
Escalation had no sense of time. It buys a second synthesis, which is the most expensive call in the pipeline, and S14 escalated at around three minutes and then timed out — turning a usable if hedged answer into "the language service did not respond". One rep in thirty-nine and entirely self-inflicted. A false absence is worse than a hedge; it is not worse than no answer, so the escalation now stops wanting a second opinion past half the run deadline. The gate still runs, so a run that skips escalation still cannot ship a denial it did not earn. harnessFraming was producing the framing it exists to remove. EVIDENCE_SUBJECT needs "the evidence"; INPUT_NOUN matches "the provided evidence" as a unit and rewrites it to "VFB evidence". The subject rules ran first, found "provided" in the way, did nothing — and the noun rule then produced exactly the subject they would have handled, one pass too late. S46 shipped "VFB evidence lists queries for subclasses, scRNAseq data ... but it does not include a completed query": this program's working set wearing VFB's name. The rules were right and their order was wrong, so the subject pass now runs again after the noun pass. Definitions were being summarised twice. "What is the ellipsoid body?" returned the same 254 characters on all three reps, dropping two thirds of VFB's curated definition — the ellipsoid body canal, the anterior bundle, the 16 radial segments and the Ito et al. citation. Not a truncation: the extractor is a weak model summarising the whole record into one claim, so authoritative prose went through two paraphrase hops before reaching the reader. A definitional question now carries the description verbatim as its own evidence row. 254 characters to 471, all three sentences and the citation intact.
First CI run of this branch: 63 of 64 tasks passed and the 64th was "Timed out after 240000 ms". Half the run deadline was the wrong derivation. There are two ceilings — 600 s DEFAULT_RUN_DEADLINE_MS, and 240 s TASK_BATTERY_TIMEOUT_MS, which is this project's own recorded opinion about when an answer has taken too long to be worth having — and the tighter one governs. An escalation starting at 290 s cannot finish a 60-90 s second pass before 240 s, so the guard was guaranteed to turn the slowest answers into no answer at all. 120 s + 90 s = 210 s leaves margin under the tighter ceiling. Median first synthesis across the 39-rep tranche is 63 s, so this trims the tail rather than the guard. Env-overridable: how fast the backends are on the day is a deployment fact, not a code one.
The three deliberate costs are the part worth writing down: relevance orders escalation rather than vetoing it, a query the guard chose cannot license the claim it was chosen to check, and escalation stops at 120 s because a second opinion that arrives after the answer is due is not a second opinion. Each is a place a future reader may reasonably disagree, so each has the measurement that settled it.
Robbie1977
force-pushed
the
fix/absence-followups
branch
from
August 10, 2026 09:56
03d495c to
e02f542
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #17. Everything here was found by running the merged guard and
measuring it, not by reading it — three defects it exposed, one it introduced,
and the decision record.
The one this branch introduced
Escalation had no sense of time. It buys a second synthesis, the most
expensive call in the pipeline. S14 escalated at around three minutes, went
round again, and timed out — turning a usable if hedged answer into "the
language service did not respond". One rep in thirty-nine, entirely
self-inflicted.
Half the run deadline was the wrong derivation. There are two ceilings, and the
tighter one governs: 600 s
DEFAULT_RUN_DEADLINE_MS, and 240 sTASK_BATTERY_TIMEOUT_MS, which is this project's own recorded opinion aboutwhen an answer has taken too long to be worth having. An escalation starting at
290 s cannot finish a 60–90 s second pass before 240 s, so the guard was
guaranteed to turn the slowest answers into no answer at all. 120 s + 90 s =
210 s leaves margin; median first synthesis across the tranche is 63 s, so this
trims the tail rather than the guard. Env-overridable.
A false absence is worse than a hedge. It is not worse than no answer.
Three the measurement exposed
harnessFramingwas producing the framing it exists to remove.EVIDENCE_SUBJECTneeds "the evidence";INPUT_NOUNmatches "the providedevidence" as a unit and rewrites it to "VFB evidence". The subject rules ran
first, found "provided" in the way, and did nothing — and the noun rule then
produced exactly the subject they would have handled, one pass too late. S46
shipped:
This program's working set, wearing VFB's name. The rules were right and their
order was wrong, so the subject pass now runs again after the noun pass. The
existing test asserted the weaker outcome ("VFB evidence"); it now asserts the
one the module was always aiming at ("VFB records three connectome datasets").
Definitions were being summarised twice. "What is the ellipsoid body?"
returned the same 254 characters on all three reps. VFB's curated definition
does not stop there — the ellipsoid body canal, the anterior bundle, the 16
radial segments, the Ito et al. citation. Not a truncation: the extractor is a
weak model summarising the whole term-info record into one
claim, soauthoritative prose went through two paraphrase hops before reaching the reader.
Summarising a summary costs the last two sentences identically every time, which
is why three reps agreed to the character. A definitional question now carries
the description verbatim as its own evidence row — the principle the pipeline
already runs on for counts, applied to text. 254 → 471 characters, all three
sentences and the citation intact.
The absence verb list was too narrow. It had present/available/found and not
provided/listed/given, and a live answer walked through the gap with "specific
counts … are not provided in the current data" about counts VFB advertises
queries for. Negative cases added alongside: "serotonin is not present in these
neurons" is a claim about the world and must survive.
Result
Same 13 questions, three reps each, against a local build of this branch:
Task battery 64/64, 0 errors, on the tip. 1188 unit tests, 0 failures.
Also
docs/adr/absence-requires-evidence.mdrecords the decision, and specificallythe three costs that are places a future reader may reasonably disagree —
relevance orders escalation rather than vetoing it; a query the guard chose
cannot license the claim it was chosen to check; escalation stops at 120 s — each
with the measurement that settled it. It also records the known limits: a licence
is per-run rather than per-claim, and the detector is a pattern list that should
grow from observed output rather than from imagination.
One thing left for you:
T3.8runs at 190–205 s against the workflow'sTASK_BATTERY_TIMEOUT_MSof 240 s and has been marginal for several releases —it timed out once here without escalating at all.
.github/workflows/task-battery.ymlwants
240000→360000; my token has noworkflowscope.