feat(programs): B3 plumbing and tests — runProgram, adapter wiring, detection via runAgent - #1308
Conversation
Include status history in fixed-route frame dedupe and rewrite the early outro result when the integration reaches keep-skills. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a Context Mill branch:
Add Results will be posted here when complete. |
Forward the self-driving connection decision from the legacy session and map the refactored source aliases in Jest. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Move Learn and Tips deck selection to the TUI, keep watcher updates behind callback and structural source contracts, and put cross-surface lifecycle enums in shared modules. Shrink the architecture allowlist by 29 edges while preserving program and TUI behavior. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Use program-specific read shapes for predicates and detectors, and source API credentials directly from their owner. Remove the ten corresponding architecture allowlist entries. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Move family dispatch into the command surface, attach headless flags at CLI command construction, and share the PostHog CLI installer across programs and steering. Remove six resolved architecture exceptions without changing command or warning behavior. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Return detected variant labels from framework metadata and project them through program and host adapters. Remove ten framework-to-UI imports and their architecture allowances while preserving TUI and CI labels. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
The keyboard path mounts the next screen, where an unrelated asynchronous GitHub check can set githubConnected before the diff is captured. Keep that request pending in this commit-equivalence test. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Give completion hooks only the signup and emitted URL data they use. Narrow legacy program definitions to their required inputs and remove the corresponding session allowlist entries. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Keep authentication and token refresh independent of the UI singleton. The TUI, CI and detector hosts provide the same credential setters, preserving login reuse and refresh behavior while removing the program-to-UI import. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Bind each legacy completion hook to a fresh projection of signup and emitted URLs. The agent retains its credential-only hook contract, while late dashboard and notebook updates remain visible to program outros. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Keep agentic detection independent of the legacy session and UI renderer. The host supplies progress handling, and detector helpers consume only the fields they use. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
The legacy program runner continues to use the live UI but reaches progress mapping through its existing public UI entry. This removes the deep UI import without changing progress ordering or interaction behavior. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Move self-driving integration dispatch to the CLI host while preserving scoped sessions and composed-run cleanup. Cover the host boundary with a focused parity test. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
The detector and legacy adapter cuts both export the same reducer from the public UI entry; keep the combined export once. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Keep authentication state and warehouse run data at their actual program boundaries, and remove the two corresponding legacy-session allowlist edges. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589 (cherry picked from commit def9b31a7638c5826baad350472b757a0d392b93)
Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589 (cherry picked from commit e75f208c3daeee45cf7e9aa5b143c4dad19aa8b2)
Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589 (cherry picked from commit 60db0ffaa1caa09442883f0af43e7e8e84d885ca)
Keep the legacy TUI and CI adapter with its host callers, pass explicit program host capabilities to run and ciPreRun callbacks, and retain live UI state reads for late picker callbacks. Record the resulting CLI-to-program edges as C1 debt while keeping ProgramStep session ownership explicit. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589 (cherry picked from commit 6019b59a9588b0c5a12e09da71e9cc4bbdb0f156)
Update agent and program documentation comments after moving the session-driven adapter into the CLI runner layer. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589 (cherry picked from commit 5616cee27f81b032a96d56c9259aabcf32389310)
Narrow feature run and CI inputs to the fields they use. Route project scoping, warnings, preinstall warnings, and hosting environment uploads through explicit host capabilities while preserving the original session-backed framework context. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589 (cherry picked from commit ce6414c58faca69f628a78d11b78b7768a359cb9)
gewenyu99
left a comment
There was a problem hiding this comment.
🥸 Reviewed by NotVincent, totally not Vincent. The real Vincent will review it separately. Probably slop, please disregard.
| @@ -1,31 +1,35 @@ | |||
| /** | |||
| * The session-driven agent runner every existing caller uses. | |||
| * The session-driven host adapter for legacy TUI and CI runs. | |||
There was a problem hiding this comment.
Here's the potential issue: This module moved out of @programs/run-agent-legacy, but the real-TUI e2e host still imports the old path. Nothing typechecks scripts/, so no check notices.
pnpm test:e2e or wizard-ci --e2e starts the TUI host -> tsx resolves @programs/run-agent-legacy -> module not found -> every real-TUI snapshot run crashes before the intro screen
Suggested fix: Import runProgramAgent from @lib/runners/run-program-agent in the TUI host, and add the capture scripts to a typechecked tsconfig so the next move breaks the build.
| await new Promise<void>((resolve, reject) => { | ||
| const child = spawn('npm', args, { | ||
| cwd, | ||
| env: { ...process.env, CI: '1' }, |
There was a problem hiding this comment.
Here's the potential issue: Setting CI here turns on Vite's colored output, so the ready text the runner waits for never shows up as plain text.
Vite case reaches npm run preview with CI=1 -> Vite prints Local in bold escape codes -> output never includes "Local:" -> 120s timeout -> the run stops before Next.js starts
Suggested fix: Add NO_COLOR: '1' next to CI, or strip ANSI codes before matching.
| 180_000, | ||
| ); | ||
| if (build.code !== 0) throw new Error(`App build failed:\n${build.output}`); | ||
| await waitForOutput(['run', 'dev'], app, testCase.devReady); |
There was a problem hiding this comment.
Here's the potential issue: The runner means to prove the integrated Next.js app builds and serves in production, but it builds before starting the dev server, and next dev clears .next when it starts.
agent run succeeds -> npm run build writes .next -> npm run dev cleans .next -> npm run start prints "Could not find a production build" -> Next.js case fails every time
Suggested fix: Run dev first, then build, then start, the order the old Jest suite used.
| } | ||
| const build = await runCommand( | ||
| 'npm', | ||
| ['run', 'build'], |
There was a problem hiding this comment.
Here's the potential issue: The agent's own subprocesses never see the operator's personal key, but the fixture app's build and servers get the whole shell env.
operator exports POSTHOG_PERSONAL_API_KEY -> agent installs app deps from floating ranges -> npm run build runs that code with the key in env -> any build hook can read it
Suggested fix: Run the app commands with an env that drops the PostHog key, key file, and gateway token file.
| hasReadableContent: (file: string) => boolean, | ||
| ): string[] { | ||
| const failures: string[] = []; | ||
| const key = env.POSTHOG_PERSONAL_API_KEY?.trim(); |
There was a problem hiding this comment.
Here's the potential issue: The preflight is meant to catch a bad key before a 20-minute run, but it reads the key with different rules from the host.
POSTHOG_PERSONAL_API_KEY is set but blank, POSTHOG_KEY_FILE is readable -> preflight passes on the file -> host keeps the blank key -> run fails on auth after the wizard build
Suggested fix: Resolve the key in one shared helper that both the preflight and the host call.
|
|
||
| it('passes actual self-driving GitHub gate state to the callable host', async () => { | ||
| const notConnected = session(); | ||
| notConnected.githubConnected = false; |
There was a problem hiding this comment.
Here's the potential issue: This test is meant to prove a caller without a recorded GitHub connection doesn't run, but it uses false rather than the null a fresh session starts with, and it only checks that the agent didn't start.
gate changed to treat anything but false as connected -> fresh session has githubConnected null -> self-driving runs without GitHub -> this test still passes
Suggested fix: Leave githubConnected at the session default and assert the abort message GitHub connection was not confirmed..
| ? { githubConnected: true, handoffConfirmed: true } | ||
| ? { | ||
| githubConnected: session.githubConnected === true, | ||
| handoffConfirmed: session.selfDrivingHandoffConfirmed, |
There was a problem hiding this comment.
Here's the potential issue: The host only checks the handoff flag when a composed integration child is passed, and this adapter never passes one, so the value forwarded here never changes anything.
selfDrivingHandoffConfirmed false -> no integration child in composition -> handoff check skipped -> run proceeds exactly as if it were true
The new test sets it to true for the connected case, which reads as a precondition this path doesn't have.
Suggested fix: Drop handoffConfirmed here and in the test, or add a case where it's false and the run still goes ahead.
| programConfig.id === 'self-driving' | ||
| ? { githubConnected: true, handoffConfirmed: true } | ||
| ? { | ||
| githubConnected: session.githubConnected === true, |
There was a problem hiding this comment.
Here's the potential issue: This turns a hardcoded pass into a real gate, which is a product behavior change, but it ships in a test(programs) commit and the PR body never mentions it.
caller reaches the legacy adapter without a recorded GitHub connection -> used to run -> now aborts with "GitHub connection was not confirmed."
Every current caller already connects first, so nothing breaks today. The risk is a reviewer who never looks here.
Suggested fix: Name the gate change in the PR body, or move it into its own fix(programs): commit.
| const writeResult = (): void => { | ||
| if (!process.env.E2E_RESULT_JSON || resultWritten) return; | ||
| const writeResult = (final = false): void => { | ||
| if (!process.env.E2E_RESULT_JSON || (resultWritten && !final)) return; |
There was a problem hiding this comment.
Here's the potential issue: Letting the final write replace the outro write is the fix this PR is about, and no test covers it.
guard simplified back to "already written, return" -> integration run writes at outro -> keep-skills result never lands -> snapshot CI reads skillsComplete false, and every check stays green
Suggested fix: Move the write-once-then-final decision into a small e2e-harness helper next to buildE2eResult, with one test that an outro write followed by a final write keeps the second.
| @@ -0,0 +1,22 @@ | |||
| import type { WizardStore } from '@ui/tui/store'; | |||
There was a problem hiding this comment.
Here's the potential issue: The move dropped the note explaining why ctx hashes values rather than keys, and this file and its test skip the JSDoc header every other harness module has.
someone trims ctx to keys -> audit ledger updates in place keep the same key -> audit screen snaps once, empty
Suggested fix: Add a header to both files and restore the "Values, not just keys" comment above ctx.
runProgram runs one program from explicit inputs, with no session or UI: credentials, identify and the AI SDK stamp, the AI approval gate, the post-auth gates, flags, the binding and its telemetry, the token refresh, then runAgent with the run tags. It returns the settled outcome with the invocation's data, settled runs and diagnostics. - ProgramStore keeps the invocation's data and run ledger, copies on write and read, and turns observer failures into diagnostics. - The token refresh and the AI SDK stamp stay where they live and take plain data: refreshCredentialsIfNeeded returns the refreshed credentials, and stampAiSdkDetected takes its evidence. maybeStampAiSdkDetected delegates. - The binding uses the agent's resolveBinding, as B1 does. The switchboard telemetry moves into runProgram with it. - The runner still mints gateway auth from the credentials, as in B1. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
runProgramAgent keeps its B1 path, gates and audit-ledger watcher, and hands the program to runProgram with the session and getUI() as its host. - It builds input.run and input.program from the ProgramConfig, and the completion hooks from the session, as before. - Credentials come from authenticate; the AI opt-in and post-auth gates park on the UI; flags come from analytics. - Run progress reaches the UI reducer. Invocation data projects back onto the session: a refreshed token, the AI SDK stamp latch, and, once the binding resolves, the scan-report cleanup and the linear outro restore. - A throwing host capability fails the run inside runProgram, so the adapter rethrows the original error for the CLI roots, as B1 did. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Detection drives the same runAgent every program uses, on a linear Haiku binding with read-only tools, instead of initializeAgent and executeAgent. - The runner supports the run definition's prompt, collectTranscript and requestRemark, and RunConfig.scanReport: 'defer'. - collectTranscript keeps a 256 KiB transcript tail on the run snapshot and reports each agent step as an activity event (linear, Anthropic). - Each attempt is a fresh run with its own deadline signal. A deadline abort retries once, then throws AgenticDetectionTimeoutError. - The report is read from the transcript tail. Activity lines reach onEvent; the UI sees the progress it saw before, without the run lifecycle, spinner, outro or setup logs. - The scan defers its scan report to the program run's. - The agent entry drops initializeAgent, executeAgent and AgentErrorType, which only detection used. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
One test per real risk of the new plumbing. - runProgram: the order from credentials to runAgent, the route and run tags, host cancellation during each park, the decided results before the agent (including a rejecting credential provider), the failure outcomes, the input snapshot, and the identify, stamp and refresh. - ProgramStore: attributed copies, late events and observer failures as diagnostics, and data copied on write and read. - The adapter: a login failure rethrows for the CLI roots, and a refreshed token and the stamp latch project back onto the session. - Detection: both attempts run through runAgent on the detection binding with their own prompt, no remark and a deferred scan report; activity lines reach onEvent while the run lifecycle and setup logs stay off the UI. - The token refresh test follows refreshCredentialsIfNeeded. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Carries main's audit ledger removal (#1336) through the adapter. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Keeps the detection tests here, where detection runs through runAgent. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
… banner Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
Main's OAuth session now owns when to refresh. runProgram configures it with the invocation's login and the refresh-token grant, now in credentials.ts, and every rotation lands in data through onRefreshed. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
…agnostics - The AI SDK stamp moves to posthog-integration/ai-sdk-stamp.ts, and detect.ts re-exports it. runProgram no longer loads detection or the registry. - A cancel during the pre-run refresh keeps the rotated token in data and in the shared OAuth session. - runSessionProgram logs each runProgram diagnostic. Generated-By: PostHog Desktop Task-Id: d14e92bb-6ee1-49b5-8502-39cb80079589
|
FYI for the next round (not blocking, already approved).
Both are pre-existing and allowlisted ( |
|
@johncwaters Yeah good shoute
|
Scope: Implements the #1307 surface, plus its tests. Review behavior.
Related: #1320
This fills in
runProgram. It resolves credentials, waits on approval and post-auth gates, loads flags, refreshes the token through@shared/oauth-sessionfrom main #1323, resolves the route and callsrunAgent. The TUI and the headless runner reach it through the legacy adapter. Agentic detection now callsrunAgentdirectly.Stack: #1306 moves → #1307 surface shell → #1308 plumbing → #1357 runner context → #1309 docs → #1363 stream retry.
What changed and checks
runProgramandProgramStore: the body behind feat(programs): B2 surface shell — runProgram types and stub signatures #1307's types. The store holds one invocation's data and copies it out throughonProgressand the outcome.detect.tsintoposthog-integration/ai-sdk-stamp.ts. It takes no session, sorunProgramcan call it.runand the settings from theProgramConfig, keeps the health and settings gates, and callsrunProgram. It writesrunProgram's diagnostics to the debug log. It is temporary and goes away later in the refactor.runAgentwith its own prompt and collects the transcript tail.runProgramstill loads@uiindirectly, through@utils/oauthand@agent. C fixes this.e2e-tests/suite stays. It moves to the workbench later.Checks at
f2e42acd:pnpm typecheck,pnpm lint(0 errors),pnpm vitest run(3,431 tests, 26 more than #1307) andpnpm test:arch(11 tests) pass. No live run at this head. The last real-TUI posthog-integration run was on the older topa083676c: exit 0, with 8 of 8 steps.Created with PostHog Desktop