Skip to content

feat(evals): add workflow evaluation suites#5780

Draft
TheodoreSpeaks wants to merge 1 commit into
stagingfrom
codex/workflow-evals-draft
Draft

feat(evals): add workflow evaluation suites#5780
TheodoreSpeaks wants to merge 1 commit into
stagingfrom
codex/workflow-evals-draft

Conversation

@TheodoreSpeaks

@TheodoreSpeaks TheodoreSpeaks commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add workflow-scoped Eval suites with code, LLM, and workflow judges, selected block outputs, block mocks, and immutable draft snapshots
  • run suites and individual tests with parallel execution, SSE reconciliation, retries, durable cancellation, and compact terminal status/details UI
  • expose Eval authoring, discovery, and run controls to Mothership and the workflow VFS
  • add one backward-compatible database migration, seed data, contracts, and coverage across execution and UI boundaries

Testing

  • bun run lint
  • bun run lint:check
  • bun run --filter sim type-check
  • bun run check:api-validation:strict
  • bun run check:migrations origin/staging
  • bun run mship-tools:check
  • bun run vfs-snapshot-contract:check
  • 464 focused Eval, API, execution, SSE, VFS, and UI tests

Notes

  • Reflection work is intentionally excluded from this PR.
  • The full monorepo typecheck currently fails in the unchanged Docs app on its existing Zod/tool typing mismatch; the Sim package typecheck passes.

@vercel

vercel Bot commented Jul 20, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview, Comment Jul 21, 2026 12:39am

Request Review

@gitguardian

gitguardian Bot commented Jul 20, 2026

Copy link
Copy Markdown

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant