Repository navigation
Add a Postgres event log, live agent streams, and channel-agnostic threads - #85
jacobprall wants to merge 41 commits into
Conversation
Adds factory-db to the Blueprint, linked to both services as DATABASE_URL, and app/db: an append-only events table with per-stream versions, append and appendMany with expected versions and idempotency keys, ordered reads by stream version and by (tx, seq) below the snapshot watermark, NOTIFY on commit, a projection framework with db:rebuild, and migrations applied at startup under an advisory lock. Event types are declared with defineEvent in app/events/catalog.ts. Tests run on PGlite; concurrency tests run on real Postgres when TEST_DATABASE_URL is set, which CI now provides. doctor and local preflight check that DATABASE_URL connects and migrations are current.
Every runAgent call now records its agent run as an agent_run:<id> stream: agent_run.started, assistant.message, tool.call, tool.result, usage, and agent_run.finished, with the outcome, turns, usage, and cost taken from the host's own summary. The agent_runs projection answers cost per issue, agent, and model. Token deltas from the Pi driver (protocol version 3) travel over NOTIFY on the live channel and are never stored. Until the runner reports live, the host feeds the recorder the downloaded output after the run. The gateway holds one LISTEN connection per instance and serves two SSE endpoints behind EVENTS_READ_TOKEN: one agent run, or everything on an issue or pull request. Each sends the backlog after Last-Event-ID, then partial messages, then live events and deltas. setup:render generates the token. Vitest runs *-postgres.test.ts files one at a time, since a running transaction anywhere on the server holds back subject readers.
Claims move to the subject stream: a run appends claim.acquired at the version it read, so concurrent runs are fenced and the loser rereads. Each task run's lifecycle (started, rejected, progress, sandbox, stopped, finished) is logged on its run stream, with runs and claims projections. Ownership checks read the claims row; takeover and /factory cancel stop the previous trace from the runs projection. The status comment and factory:working are now a best-effort view synced from Postgres. The gateway records delivery.received before starting a task and returns 502 when Postgres is unavailable. run.finished moves from log lines to the event log.
Requeueing after a failed write trimmed the queue to its bound, which could cut agent_run.finished, so the run's outcome and cost never reached the log. Only display events are trimmed now, and they count as dropped.
Both plans' events share one id schema in the catalog. The claim now passes the event subject straight to runAgent, and the unused emit() telemetry is gone, since runs and agent runs are both recorded in the log.
run-agent.sh now tees the runner's JSONL to its own stdout as well as the root-owned <runDir>.jsonl, and writes its PID to <runDir>.pid. tee -p keeps the file complete if the host's stream goes away. The exit status is still the runner's (PIPESTATUS[1]; tee is last). runAgent starts it with the new Sandbox.execStream and feeds each complete stdout line to the recorder as it arrives. LineRelay counts the bytes of the lines it passed on; if the command stream drops, followRun follows the file from that offset with `tail -c +N -f --pid`, retrying every 5 seconds until the deadline, so no line is skipped or repeated. After the command ends the host downloads the file, relays any lines the live copy missed, and takes the result, outcome, and cost from the file alone. SandboxClient.exec now collects output through execStream.
Upgrade Pi from 0.83 to 1.5.2 (pi-coding-agent, pi-ai, and pi-agent-core pinned to the same exact version) and rewrite the Pi driver on the coding agent SDK's session API. The Driver interface and the kernel are unchanged. - Lock the session down: no context files, skills, prompt templates, themes, or extensions are loaded from the checkout or anywhere else; settings and the session live in memory, and agentDir is an empty scratch directory. - Enforce the turn limit through a ModelRuntime proxy whose streamSimple asks the kernel first, so retries and compaction count as turns. - Use Pi's own transient-error retry (same classifier, two retries, 2 s base backoff) instead of the driver's loop. Turn cache warming off: it re-sends requests on a timer. - Port the tools to Pi 1.5.2's factories; bash runs through local operations that pin the allowlisted env. - Add tests that a checkout's AGENTS.md, skill, and extension change no request and the extension never runs, and that the request after max_turns is never sent. - Document the new security invariant: no repository file is loaded as a prompt, skill, prompt template, or extension.
The event layer now names what work runs on and events are about a thread: a channel's unit of conversation, keyed <channel>:<id>. GitHub is the first channel, with keys github:owner/name#issue-12 and github:owner/name#pr-34. - app/threads owns the key format (ThreadKey schema and type), the GitHubThread type, and githubThread(); subjectKey is gone. append rejects a thread key outside the format. - The events, runs, claims, and agent_runs columns are now thread; the claim stream is thread:<thread>. - Deliveries are recorded on delivery:github:<id>, which is also the event's idempotency key. The Workflows idempotency key stays the raw GitHub delivery ID. - The gateway serves GET /threads/:thread/stream for a URL-encoded key and GET /threads/github/:owner/:repo/:kind/:number/stream as the GitHub alias, which the status comment shows. - Catalog wording, docs, and tests describe threads.
A Blueprint can't reference a database from a workflow service: every sync failed with "envVarsToCreate.DATABASE_URL env var depends on non-existent DB", and fromService can't copy the gateway's reference value. render.yaml now links factory-db only to the gateway. `npm run setup:render -- --link-database`, run after the Blueprint is applied, finds the workspace's one factory-gateway web service (allowing a Render name suffix), reads the DATABASE_URL the Blueprint gave it, and writes it to the software-factory-workflows env group. It is idempotent, never prints the value, and fails with "apply the Blueprint first" when the gateway or its DATABASE_URL doesn't exist yet. Workflows picks the value up on its next deploy. doctor's fix for a missing Workflows DATABASE_URL points at the new step. Local connection advice uses sslmode=verify-full, which pg recommends to keep today's behavior as sslmode=require changes meaning in pg v9. The setup runbook, configuration reference, quickstart, and README describe the new post-Blueprint step and redeploy.
GitHub delivery IDs now exceed Number.MAX_SAFE_INTEGER, so doctor rejected the delivery list and would have looked up a rounded ID.
The recorder clipped fields by character count, but append() caps a payload at 256 KB of JSON-encoded UTF-8. Escape-heavy text, such as ANSI output where ESC takes six bytes, passed the clip and then failed every later write, and finish() discarded the run's history. Fields are now clipped to byte budgets at whole characters, and an event append() would reject (too large, or a malformed thread key) is dropped at enqueue, so only database errors reach the retry. Closes #86
A run now owns its issue or pull request by starting on it: run.started, carrying the run and trace IDs, is appended to the thread:<key> stream, and the trace whose run started last owns the thread until its runs finish. There is no separate claim: claim.acquired, claim.released, acquire(), release(), cancelClaim(), and the claims projection are gone, and a threads projection records each thread's latest trace and run. - withRun checks permission before starting, so an unauthorized trigger displaces nothing. It is recorded as run.rejected on its own stream and shown at most once, only on a thread with no runs. - A root run takes over and stops the displaced trace: Workflows cancel, Sandbox termination, and run.stopped. A child run starts only while its own trace is the thread's latest, so a stale child can't take it back. - /factory cancel is a thread run like any other. Starting it stops what was running, and the status comment shows what it cancelled or that nothing was running; it posts no other comment. - An error after the thread moved on ends the run as superseded instead of reporting a failure and rethrowing. - The status comment ID is recorded only if unset; a sync that loses the race deletes its duplicate and updates the winner. - publish() checks ownership immediately before moving the branch. - The status view shows the run that started last while it runs, and otherwise the trace's root, by explicit order. - app/claims is now app/runs: Claim -> Run, ClaimOptions -> RunOptions, withClaim -> withRun. Closes #87 Closes #89
Biome's noUnsafeOptionalChaining failed npm run lint on three payload reads.
Trigger labels did two jobs: a command to start a task and the state of the issue's conversation. Nothing kept that state accurate, so a comment on an approved spec restarted Triage and stopped Spec-to-PR, a declined issue kept re-running Triage, and a skipped reply dropped an issue out of Autopilot. A factory draft pull request was also its own thread, so /factory cancel there found nothing and a manual review ran beside Autopilot's. - run.finished records waiting: "reply", projected to runs.waiting. Triage waits after asking, drafting, revising, or keeping a spec, and after skipping a reply; Autopilot waits when Triage asks or skips. - route() stays pure: deliveries that depend on thread state return a lookup, which the gateway decides from Postgres. A comment resumes the latest trace's root task only when it finished waiting. - Trigger labels are commands: defineGitHubTask removes a root run's label once it has started. Autopilot's label re-adding and Triage's removal of factory:spec-approved are gone. - Spec-to-PR links its draft pull request's thread to the issue's (thread.linked, projected to thread_links). Cancel, PR review, and the event stream on a linked pull request act on the issue's thread. PrReviewInput.workflowIssue becomes issue and is no longer parent-only. Closes #88 Closes #90
- The runner escapes every non-ASCII character as \uXXXX, so each JSONL line is 7-bit ASCII. A chunk boundary can no longer split a character, and the byte offset the host counts from decoded output matches <runDir>.jsonl exactly. - run-agent.sh creates <runDir>.jsonl before writing <runDir>.pid. The follow command exits with a dedicated status while either is missing and run-agent.sh hasn't written <runDir>.exit, and followRun retries it until the deadline, so older snapshots are covered too. - The PR Reviewer prompt renders verify commands with verbatimCodeSpan, so a command with backticks stays in one code span. Closes #91
… serving - publish() uploads blobs 8 at a time and starts no upload after the first failure or SHA mismatch. - SandboxClient no longer paces requests client-side; it still retries 429 and 503 responses with backoff. - The read-only shell policy computes written paths in one helper, skips the values of wrapper flags (timeout -s/-k, nice -n, env -u/-C, xargs -I and friends), honors cp/mv/ln -t and --target-directory, rejects attached git -c<cfg> among git's global options, and no longer throws on command names that are Object.prototype keys. - serve() is split into a serialized write queue, drainBacklog, and emitPartials ahead of the live phase, with unchanged SSE behavior; a test covers skipping the partial of a message the backlog already sent. Closes #92
…pers Route host modules and setup scripts through each module's index.ts: github now exports githubToken, appJwt, githubApiUrl, and normalizePrivateKey; sandbox exports SAFE_REF; triage exports TriageHandoff; and a new host-only app/runner/index.ts exposes the protocol types and bundleRunner. Code bundled into the Sandbox runner keeps importing app/runner/core directly, since the barrel loads esbuild. app/db/agent-runs.ts keeps importing the events catalog: the barrel would put it in the db/events cycle and fail when loaded first. readBody() now returns the raw Buffer, verifySignature() computes the HMAC over those bytes, and the gateway decodes UTF-8 and parses JSON only after the signature checks out. Add a dependency-free app/util/ with the one errorMessage and the SHA regex, replacing the copies in the runner, sandbox, scripts, and inline ternaries. Sandbox log lines use msg instead of message, and githubToken prunes stale tokenCache entries when inserting. Closes #93
Three modules each held pure definitions alongside code with heavy dependencies, which made barrel imports cycle or need exceptions. - app/events is the event catalog only. AgentRunRecorder moves to the new app/agent-runs, so app/db imports the events barrel without a cycle. - app/agents is agent definitions only. runAgent and the relay move to app/agent-runs, so app/sandbox reads the agent registry without a cycle that depended on export order. - app/runner/index.ts is the runner's Sandbox-safe public API, used by agent definitions, harnesses, and the host alike. The esbuild bundler moves to the new app/runner-bundle. tests/agents-catalog.test.ts keeps app/agent-runs and app/runner-bundle out of the Sandbox bundle.
Collapse matrices of malformed responses and duplicate HTTP statuses to one representative case each, and drop tests of request headers, token cache keys, exact write sequences, and constants. Doctor, preflight, setup, and GitHub suites keep their checks of exit codes, actionable messages, env groups and .env contents, GitHub and Render request contracts, and secret scrubbing.
Drop tests of upload concurrency, task registration, and reviewer prompt formatting. Fold repository matrices, thread linking, and duplicate supersede and nothing-running cases into the tests they repeat. Trim assertions on exact commands, agent prompts, mock call shapes, and event stream contents, keeping GitHub writes, run rows, and publish outcomes.
Delete tests/routes.test.ts, which unit-tested route() and duplicated the gateway's HTTP tests; one table in gateway.test.ts now sends signed webhooks through createGateway and asserts which task starts or why the delivery is ignored. Merge overlapping tests in the gateway, streams, db, format, and markers suites, and drop schema tests that only exercised Zod defaults or happy-path parses.
Keep tests of trust boundaries and observable outcomes; drop assertions on exact command strings, call order that is not a security contract, timing, task registration, and third-party internals. - Merge overlapping sandbox, kernel, agents, and recorder cases into table-driven or combined tests. - Run the harness conformance suite on Pi only and delete the scripted harness helper. - Fold the read-only shell policy checks into one allowed/rejected table. - Remove fake-sandbox options and FakeApi parts no test uses.
Six suites each built their own fetch router or HTTP world. Replace them with tests/helpers/fake-fetch.ts: a route table keyed by method and path (with :param segments) that replies with JSON, a Response, or a thrown Error and records each request's method, URL, headers, and JSON body. Each suite now passes its world as route data, overriding only the routes a scenario changes. Repeated preflight scenarios that assert the same outcome are table-driven. Every scenario and its contract checks are kept.
Workflow tests each defined their own in-memory GitHub, sandbox, and agent fakes. Replace them with tests/helpers/fake-github.ts (plain state plus failNext for injected failures), FakeOpenedSandbox in fake-sandbox.ts, and fake-agents.ts, and assert on GitHub's end state instead of recorded calls. publish.test.ts keeps its own fake: it serves the Git Data HTTP API under the real gitData client and hashes trees like git, which its security tests need.
Removes every comment from TypeScript, shell, SQL, and the Blueprint. Kept: shebangs, shellcheck directives, and biome-ignore suppressions, which tools read.
Sonnet sometimes writes its XML tool-call syntax into a long string argument, burying later fields or leaving closing tags at the end. Submit now cuts each string at the first tool-call tag before validating and names the markup when the cut leaves fields missing. Empty arguments go to Pi's own validation, so arguments Pi couldn't parse reach the model with the parse error and the text it sent. Output schemas list short fields before long text, so a leak in the last field no longer takes the others with it.
Render reports a parent task waiting on a subtask as paused. A takeover treated only pending and running as active, so it never cancelled an Autopilot root mid-stage; the root resumed once its child failed. Paused now counts as active.
Describes each component and how a label flows through them, lists the factory's labels and how triggers, replies, cancellation, and factory pull requests behave, and adds channels to the extension list.
A factory draft pull request no longer borrows its issue's thread through a stored link. Autopilot runs Triage and Spec-to-PR, then starts PR Reviewer on the draft pull request as an independent root run through the Render API, finding its own workflow's slug from its task run. The review's status, cancellation, and takeover all happen on the pull request, as they would for any pull request. Removes thread.linked, the thread_links projection, linked-PR routing in the gateway, the stream alias, and PR Reviewer's issue input.
None of these migrations has reached main, so they become a single 001_event_log.sql. Drops updated_seq from the projections, which nothing reads, the default on events.version, which the store always sets, and the last SQL comments.
starmorph
left a comment
There was a problem hiding this comment.
I found six items I’d address before merging. Each inline comment explains the impact and suggests a fix.
| tools: ToolDefinition[], | ||
| ): Promise<AgentSession> { | ||
| const agentDir = await mkdtemp(join(ctx.scratchDir, "pi-agent-")); | ||
| const settingsManager = SettingsManager.inMemory(SETTINGS); |
There was a problem hiding this comment.
[P1] Keep repository content out of the agent's system prompt
Pi 1.5.2 still discovers project-level appended system prompts when appendSystemPrompt is omitted, and SettingsManager.inMemory(SETTINGS) trusts the project by default. This lets repository-controlled text cross into the privileged system prompt, including during reviews of fork PRs.
Set appendSystemPrompt: [] on the resource loader and pass { projectTrusted: false } to SettingsManager.inMemory. Extend the harness test to assert that the model receives only the configured system instructions regardless of project prompt configuration.
| if (Value.Check(this.schema as TSchema, args)) return { ok: true, checked: structuredClone(args) }; | ||
| const found = [...Value.Errors(this.schema as TSchema, args)].map((e) => `${e.instancePath || "(root)"}: ${e.message}`); | ||
| const problems = found.length > 0 ? [...new Set(found)] : ["(root): does not match the output schema"]; | ||
| const cleaned = cutAtMarkup(this.#markup, args); |
There was a problem hiding this comment.
[P1] Preserve valid submission text before attempting recovery
The unconditional cleanup silently drops valid content. For example, a review summary with GitHub's <details><summary>Results</summary>…</details> markup is accepted as just <details><summary>Results. Inline findings that quote XML can be shortened too.
Validate the original object first and preserve it when valid. If validation fails, attempt recovery only for clearly identifiable leaked tool-call syntax, then revalidate or return an error so the model can retry. Add coverage that preserves HTML and quoted XML while retaining recovery for a genuinely malformed submission.
|
|
||
| const port = Number(process.env.PORT ?? 3000); | ||
| const webhookSecret = requireEnv("GITHUB_WEBHOOK_SECRET"); | ||
| const eventsReadToken = requireEnv("EVENTS_READ_TOKEN"); |
There was a problem hiding this comment.
[P1] Provision the reader token before upgrading the gateway
The PR's upgrade instructions skip plain npm run setup:render. The documented --link-database path only adds DATABASE_URL to Workflows, so a gateway upgraded from the base revision is missing EVENTS_READ_TOKEN and exits before serving webhooks.
Document an in-repo upgrade sequence that provisions the gateway token before the deploy, links the database for Workflows, verifies both services, and refreshes snapshots. Include how to resume existing issues waiting for a reply, since the new database has no record of those threads.
| environment groups that the setup runbook below creates, and your fork's | ||
| `factory.config.ts` must first list the repositories the factory may access. | ||
| The runbook has you apply the Blueprint once both are ready. | ||
| ## Setup |
There was a problem hiding this comment.
[P1] Restore the setup warning beside Deploy to Render
The button appears before the prerequisites, and this removes the instruction to wait before applying the Blueprint. Users can hit missing env groups or provision a paid database and end up with a failing Workflows deploy until they link the database and redeploy.
Restore a short warning beside the button. Point people to the runbook to configure the fork and create env groups before applying. Call out the paid Postgres resource and the required database-link and Workflows redeploy steps there.
| await runs.terminateSandbox(sandboxId).catch((error) => warn("could not terminate the stopped run's sandbox", error, { sandboxId })); | ||
| } | ||
| const payload = { ...stop, active }; | ||
| await db.withTx(async (tx) => { |
There was a problem hiding this comment.
[P1] Keep failed cancellation and sandbox cleanup retryable
The Render cancel and sandbox terminate calls can both fail, yet this still records run.stopped and tells the maintainer the run was cancelled. A follow-up /factory cancel reports nothing running while the original provider resources can still be active.
Distinguish revoking thread ownership from confirming provider cleanup. Persist pending or failed cleanup with the original run and sandbox IDs, retry it, and report success only after the resources are confirmed stopped. Add a test where both provider operations fail and a follow-up cancel retries them.
| title: spec.title, | ||
| actor: input.actor, | ||
| mode: ctx.metadata.parentTaskRunId ? "continue" : "replace", | ||
| mode: root ? "replace" : "continue", |
There was a problem hiding this comment.
[P1] Bind reply-triggered tasks to the run waiting for an answer
The gateway checks whether a thread is waiting, but a queued reply later starts in unconditional replace mode. If a maintainer starts Spec-to-PR in between, the stale reply can cancel the newer implementation. A redelivery also re-evaluates an earlier ignored decision against current state, while the event log keeps the original decision.
Pass the expected waiting run ID and consume its reply state atomically when the task starts. Persist and reuse the routing decision for a delivery ID, including an ignored decision. Cover delayed and duplicate replies, and preserve a reply that arrives after questions are posted but before the waiting state is committed.
Leads with what the factory does, then how it works (stages, components, and trust model), how work arrives, deploying with the agent-guided setup and a configuration table, watching agents work, local development, extending, limits, and resources.
Says what the template handles and what you bring, gives security its own heading again, corrects the Sandbox description to one Sandbox per stage, and trims the configuration table to per-repository settings with a Required column, pointing to the configuration doc for other defaults.
Closes #84, closes #86, closes #87, closes #88, closes #89, closes #90, closes #91, closes #92, closes #93.
Summary
The factory gets one source of truth in Postgres, agents you can watch live, and run coordination that no longer depends on GitHub labels or comments.
Event log and live streams
factory-dbholds an append-onlyeventstable with per-stream versions and ordered reads that never skip a late commit. Projections (runs,threads,thread_links,agent_runs) update in the same transaction as the append, anddb:rebuildreplays them from the log. Both services migrate at startup.NOTIFYwithout being stored. The Workflows host relays the runner's output as it arrives and resumes from a byte offset if the stream drops. The runner writes ASCII-only JSONL, so the offset always matches the root-owned output file, which stays authoritative. The gateway serves SSE at/threads/:thread/stream(plus a GitHub alias) and/agent-runs/:id/stream, behindEVENTS_READ_TOKEN.<channel>:<id>; GitHub is the first channel (github:owner/name#issue-12).Run coordination
run.startedis appended to the thread's stream at the version it read, so of two concurrent starts only one wins. A root run takes over: it stops the displaced trace, cancels its Workflows run (including a parent paused on its child), and terminates its Sandboxes. A child run continues only while its trace is the thread's latest. There is no claim to acquire or release./factory cancelis an ordinary run that does no work. Every task goes through onewithRunlifecycle; a run that fails after it was displaced ends as superseded.Agents
createAgentSession, with the kernel's turn limit enforced through aModelRuntimewrapper, Pi's discovery off (nothing loads from the checkout), and cache warming off.submitcuts each string at the first tool-call tag before validating, names the markup when fields are missing, and lets Pi report arguments it couldn't parse together with the text the model sent. Output schemas list short fields before long text.Code and tests
app/agentsholds agent definitions,app/agent-runsruns them from the host,app/eventsis the event catalog,app/runneris the Sandbox runner's public API, andapp/runner-bundlebuilds the bundle. Modules import each other only through theirindex.ts, with no cycles.AGENTS.mdnow asks for none, apart from tool directives.Setup and docs
setup:rendergeneratesEVENTS_READ_TOKEN. After the Blueprint is applied,setup:render -- --link-databasegives the Workflows service itsDATABASE_URL, because Blueprint sync can't reference a database from aworkflowservice.doctorchecks the database and keeps webhook delivery IDs exact.Nothing in a Sandbox gains a credential, an endpoint, or an allow-list entry. Runner output only becomes display events on its own agent run.
Upgrading an existing deployment
Let running factory tasks finish first. Apply the Blueprint (it creates the paid
factory-db), runnpm run setup:render -- --link-database, releasefactory-workflows, then refresh repository snapshots (sandbox/run-agent.shchanged). Trigger labels are now removed when their run starts, so labels left on open issues no longer mean anything.Test plan
npm run lint,npm run typecheck, and 261 unit tests (PGlite)npx vitest run --project postgres): late-commit gap, version races, concurrent root starts, a child racing a takeover, two gateway instances, concurrent migrationsrender blueprints validate render.yamlandnpm run doctoragainst the deployment (41 checks)/factory cancelstops a running review with no review posted