Repository navigation
fix(agent): let wizard_ask re-collect a field the downstream call rejected - #1367
Merged
Merged
Conversation
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a Context Mill branch:
Add Results will be posted here when complete. |
johncwaters
approved these changes
Oct 5, 2026
johncwaters
left a comment
johncwaters
left a comment
Contributor
There was a problem hiding this comment.
Approved by hand after checking the automated review.
Rebase the correction carve-out onto the current main branch and preserve the updated commandments tests.
Gilbert09
force-pushed
the
posthog/ask-correction-carve-out
branch
from
October 5, 2026 10:29
19de647 to
a502b2a
Compare
Gilbert09
enabled auto-merge (squash)
October 5, 2026 10:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The seeded warehouse task is the one step of a wizard run that stops to collect live credentials, and a credential the user mistypes — or one PostHog's API rejects for a permission the key lacks — is the normal case, not the edge case. The connection forms behind it are unforgiving: several have a field whose valid value looks very much like a different value the same provider hands out, so a plausible-looking answer that fails validation is routine.
Two pieces of wizard-owned prompt text disagree about what happens next:
WIZARD_ASK_SUBJECT_DESCRIPTIONtells the agent to reuse the samesubjectwhen it is "re-asking after a validation failure", and the ask caps are built around that: a cancelled or timed-out call is refunded, and the adjacency nudge is counted per subject precisely so a legitimate second call about one thing is not blocked.wizard_askcommandment said, with no exception: "do not re-ask in text or callwizard_askagain for the same fields."The commandment ships in the system prompt, so it is the one that wins. Telemetry for this flow carries an agent reporting exactly that conflict — it read the prohibition as absolute after a downstream permission error and abandoned the source for the browser fallback rather than letting the user correct one field. That is the loss the in-CLI source setup exists to avoid, and it costs the run a source the user had already agreed to connect and supplied most of the values for.
Changes
src/agent/commandments.ts— split the final rule of thewizard_askblock in two:wizard_askagain to confirm a field the user already answered.subject, and name what was rejected and what a valid value looks like. Bounded ("stop after a couple of attempts on one subject") so the carve-out cannot turn into a retry loop, and explicitly not extended to a dismissed or timed-out ask — the result already names those fields and their reason, and that stays a decline the agent falls back from.No behaviour change to the caps, the bridge, or either tool facade. This aligns the commandment with the contract
subjectalready advertises.Test plan
vitest run— 3436 tests pass across 204 files, including the 5 updated per-axis prompt snapshots.tsc --noEmit,eslint, andprettier --checkclean on the touched files.src/agent/__tests__/commandments.test.tspin both halves: the carve-out survives (the agent may re-ask rejected fields, reusing the subject), and a dismissed or timed-out ask is still not a correction. The existing "do not re-ask" assertion is unchanged and still passes.Created with PostHog Desktop
🤖 Generated with Claude Code