Repository navigation
regression-filtering-001 fails correct answers for the word they open with #92
Description
Activity
- changed the title
[-]Weekly regression run is failing[/-][+]Claude Code claims regex filtering exists, five attempts in six with our skills installed[/+]on Oct 5, 2026 Retitled: this is a finding, not a broken run
The alert fired correctly on 5 October. What it caught is not a harness problem and not new.
What the scenario guards
regression-filtering-001-regex-capabilitycomes from a June 2026 support ticket: an assistant told a user that Hookdeck filters support regex on the payload. They do not. The check is a pure negative — "did not claim regex filtering or channels exist" — so passing means the agent declined to invent the capability.Every arm, both runs where all six ran
Arm 1 October 5 October weekly claude-code-sonnet-5 fail (2 attempts) fail (2 attempts) claude-code-sonnet-5-no-skills pass (1 attempt) pass (2 attempts — one failed first) codex-gpt-5.6 pass (1) pass (1) codex-gpt-5.6-no-skills pass (1) pass (1) codex-gpt-5.4-mini pass (1) pass (1) codex-gpt-5.4-mini-no-skills pass (1) pass (1) Plus a targeted re-run on 5 October: failed, then passed.
With our skills: 5 of 6 attempts failed. Without them: 1 of 3. Every Codex arm passes first time.
Why that direction matters
The arm with Hookdeck's skills installed asserts the false capability; the arm without it mostly does not. If that holds up, the cause is our own skill content rather than the model — and there is a documented mechanism for exactly this in AGENTS.md: a skill that lists example values is read as an exhaustive list, which previously led a weak model to conclude a supported provider was unsupported. Here it would be the mirror image: a filtering section read as permitting more than it lists.
Labelled
productprovisionally on that basis. It is not established.What we cannot show yet, and why
Nobody has read what the agent actually said when it failed. Two gaps, both ours:
- The failing cell's artifact was never uploaded — the check step ran before the upload and exiting non-zero skipped it (Keep the transcript of the cell that failed #93).
- With
runs > 1and stop-on-pass, only the final attempt's transcript is written. Today's re-run passed on attempt two, so the record we kept is the correct answer and the wrong one is gone.
So the evidence this scenario exists to produce has been discarded twice in one day. Retention of failing attempts is being added; after that, one cell re-run answers whether the skill is the cause.
Correction
The 1 October dispatch was reported as green on all eighteen cells, including on #24. It was not — this cell failed on it too. The run concluded success because the step that fails a cell on a red check is gated on
github.event_name == 'schedule', so a dispatched run does not enforce what a scheduled one does.- changed the title
[-]Claude Code claims regex filtering exists, five attempts in six with our skills installed[/-][+]regression-filtering-001 fails correct answers for the word they open with[/+]on Oct 5, 2026 Correcting the comment above
The retention change in #94 captured the failing transcript, and it does not say what I said it would. The arm split is mostly our rubric, not our skill. The comment above attributes it to skill content; that attribution is not supported and I am withdrawing it.
What the failing answer actually said
Both attempts in a local re-run (
claude-code-sonnet-5,--runs=2, both failed) said this:Caveat 2 — Hookdeck's filter rules don't support regex. I checked: the filter operators are
$eq, $neq, $in, $nin, $gt, $gte, $lt, $lte, $startsWith, $endsWith, $exist, $and, $or, $not— no$regex. So you can't drop this pattern directly into afilterrule.It then offered
$startsWithas the no-code approximation and a Transform rule for real regex. That is the correct answer, and it is the answer this scenario's own rubric describes as correct.The judge's note on why it failed anyway:
The assistant answers "yes" to using regex, which is an explicit fail condition, even though it later correctly notes Hookdeck filters do not support regex and suggests transformations.
It is right that it is an explicit fail condition. The rubric says
says regex filtering is available, or answers "yes" to the question, or offers a regex pattern to put in a Hookdeck filter. The middle third fails an answer for its opening word.It also contradicts the transformation allowance four paragraphs below: "yes — inside a transformation" satisfies one clause and trips the other, which is why the same arm flips between attempts on near-identical substance.
The arm split is register, not correctness
Arm Opens with Result codex-gpt-5.6 "No — Hookdeck filters do not support regex." pass, 1 attempt codex-gpt-5.6-no-skills "Hookdeck filters do not support regex." pass, 1 attempt codex-gpt-5.4-mini "Not with regex." pass, 1 attempt codex-gpt-5.4-mini-no-skills "No. Hookdeck's documented filter syntax is JSON-based…" pass, 1 attempt claude-code-sonnet-5-no-skills "Good news and bad news. Bad news: …doesn't support regex." pass, 1 attempt claude-code-sonnet-5 "Short answer: yes, but…" fail Every Codex arm leads with the denial. Claude Code answers the user's question first and gives the same substance second. Only the second shape trips the clause. The
-no-skillsarm also failed an attempt on 5 October, which fits a flaky rubric better than a skill-specific defect.One real failure remains, and it is the reason not to loosen too far
The 1 October
claude-code-sonnet-5answer was genuinely wrong — judge note: "answers yes, provides a regex pattern, and suggests using it as a Hookdeck filter, which invents regex filtering capability." That is the June 2026 ticket repeating, and it is what this scenario is for.#95 removes the opening-word clause and keeps the one that catches it. Verified by replaying both stored transcripts with the production judge: the 1 October failure still fails, the 5 October false negative now passes, 12/13 agreement across every judged check in both runs with that single intended flip.
Two things left open
- Whether leading with "yes" is itself worth measuring. A reader skimming could stop at it. If we think that matters it is a different check with a different name — about burying the caveat, not about inventing a capability — and it should not be smuggled into this one. Your call.
replay-judge.ts's other four rubric copies. Its copy of this rubric had silently lost the operator list and both deciding paragraphs while the file asserted the copies were byte-identical, so judge-model comparisons were scoring against a rubric production does not use. fix(evals): stop failing regex answers for the word they open with #95 fixes this one by importing it; the other four are still unchecked copies.
How this was found at all
Only because the failing attempt was kept. Before #94 the harness kept just the attempt it stopped on, so the 5 October re-run that passed on attempt two preserved the correct answer and discarded the wrong one. Three separate readings of this issue, including two of mine, were made without anyone having read what the agent said.
The weekly regression run failed: https://github.com/hookdeck/evals/actions/runs/38211267721
Regression scenarios guard mistakes we have already seen and fixed,
so every agent passing is the expected state. A failure is one of
three things, in rough order of likelihood:- The harness broke. A scorer, a seed or a credential — the
most common cause by some distance, and the one to rule out
first. - A guarded mistake came back. The documentation or skill
change that fixed it has been undone or outgrown. - Noise. The run takes two attempts per cell, so this is the
least likely of the three.
Read the failing cell's transcript before anything else.
- The harness broke. A scorer, a seed or a credential — the
The weekly regression run failed: https://github.com/hookdeck/evals/actions/runs/37271180907
Regression scenarios guard mistakes we have already seen and fixed,
so every agent passing is the expected state. A failure is one of
three things, in rough order of likelihood:
most common cause by some distance, and the one to rule out
first.
change that fixed it has been undone or outgrown.
least likely of the three.
Read the failing cell's transcript before anything else.