Skip to content

feat: add visual Agent Eval system - #68

Merged
leecyang merged 5 commits into
LingXi-Org:mainfrom
lyyzka:codex/agent-eval-dashboard
Aug 26, 2026
Merged

feat: add visual Agent Eval system#68
leecyang merged 5 commits into
LingXi-Org:mainfrom
lyyzka:codex/agent-eval-dashboard

Conversation

@lyyzka

@lyyzka lyyzka commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add versioned synthetic/golden Eval suites, deterministic evaluators, immutable reports, baseline gates, JSON/Job Summary artifacts, and an Admin pipeline Dashboard
  • cover answer, teaching, RAG, tools, Approval/safety, task completion, multi-Agent collaboration, latency, Token/model/IPython/tool efficiency, and cost
  • hydrate real Agent OS input, model, IPython, Host Bridge, Approval, Canvas, and final-answer traces using Agent-side durations
  • sanitize Host Action arguments/results before Eval persistence; RAG keeps only source/chunk/marker/title metadata and merges automatic plus dynamic knowledge.search citations
  • add Commit/Prompt/model comparison, per-capability and per-Case deltas, failure-category clusters, resource KPIs, and clickable Trace drill-down

Review follow-up

  • add $lingxiloop-verify-change Eval classification and $lingxiloop-eval-change guidance for suites, baselines, deterministic/model Eval, Trace sanitization, comparison, and verification
  • retain frozen observation replay as eval:harness, and add eval:runtime: three deterministic, network-free Cases run the current AgentOSRuntime through MemoryHostAdapter, ScriptedModelDriver, IPython/Host Bridge, dynamic RAG, and Approval before evaluation
  • add focused Eval persistence integration through npm run test:integration:eval
  • make evalFocused fail closed: it is true only when every changed path is Eval-owned; shared Agent OS, migration, API/Admin, and integration infrastructure paths restore their owning tests
  • force fullMatrix=true for package manifests, workflows, and the classifier itself, so CI/dependency selector changes are exercised before their focused result is trusted
  • add positive focused coverage plus negative fixtures for Eval combined with runtime.ts, migrate.ts, admin-router.ts, package manifests, workflow/classifier paths, and integration infrastructure
  • keep ordinary Eval-only PRs fast while intentionally expanding this selector-changing PR to the complete CI matrix

Verification

  • npm run lint
  • npm run typecheck
  • npm run server:typecheck
  • npm run guard:brand
  • npm run guard:agent-os
  • npm run guard:llm-tracked
  • npm run version:check
  • npm run test:eval — 14 passed
  • npm run eval:check — frozen harness and real Agent OS runtime gates both passed at 100%
  • npm test — 283 passed
  • npm run build
  • classifier tests — 15 passed; pure Eval fixtures remain focused, while the current selector/workflow diff resolves to evalFocused=false, fullMatrix=true, and integration=full
  • Open Notebook native scope — 7 passed
  • lingxiloop-eval-change Skill validation passed
  • integration runner verified to accept eval.test.ts and reject unknown files; database-backed execution is owned by PR CI because this machine has no configured PostgreSQL/Redis
  • desktop and 390px mobile browser QA from the Dashboard implementation, including version comparison and sanitized Trace drill-down; no console warnings or errors

Closes #67

@lyyzka
lyyzka force-pushed the codex/agent-eval-dashboard branch from 720bd32 to 1bfcbd4 Compare August 26, 2026 10:06
@lyyzka

lyyzka commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

@leecyang 第二轮 review 的 3 个阻塞项已在 1bfcbd4 补齐,并已 rebase 到包含 #69 的当前 main:PR CI 现由 change classifier 选择范围;Eval PR 只跑 focused static/Eval/build 与单文件 eval.test.ts integration,Compose/desktop 已正确 skipped;eval:check 同时运行 frozen harness self-test 和 3 Case 的真实 deterministic Agent OS runtime gate;新增 .agents/skills/lingxiloop-eval-change 并通过 Skill 校验。新一轮 CI 全部通过,PR 为 MERGEABLE/CLEAN,烦请最终复审。

@lyyzka

lyyzka commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

已按本轮 review 将 Eval CI selector 改为 fail-closed,并推送 commit ea58069

  • evalFocused 现在只允许全部为 Eval-owned 路径;runtime.tsmigrate.tsadmin-router.ts、Admin/shared integration infrastructure 等共享路径会恢复 owning tests。
  • package.json / package-lock.json.github/workflows/**lingxiloop-verify-change 自身会设置 fullMatrix=true
  • 增加正向 focused fixture 及 reviewer 点名组合的反例测试;classifier 15/15 通过。
  • 当前 PR 因修改 selector/workflow 被分类为 evalFocused=false, fullMatrix=true, integration=full,本轮远端会完整跑 Open Notebook、全量 unit/integration、Compose 与 Windows/macOS desktop。

本地验证已完成:lint、前后端 typecheck、guards、version check、Eval unit/runtime gates、全量 283 unit、build、Open Notebook 7/7、Skill validation。正在等待完整 PR CI。

@lyyzka

lyyzka commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

@leecyang 本轮唯一阻塞项已修复并完成全矩阵验证,麻烦进行最终复审。

最终 commits:

  • ea58069:将 evalFocused 改为严格 Eval-owned paths.every(isEvalPath);共享 runtime/DB/API/integration 路径恢复 owning checks;package manifests、workflows、classifier 自身设置 fullMatrix=true;补齐点名反例测试。
  • 4914d81:完整矩阵首次运行暴露了 main 既有 integration fixture 失败后遗留 WebSocket、掩盖断言直到 20 分钟 timeout;修复 fixture/握手/cleanup,并给 runner 增加 Node 22 原生 --test-force-exit 失败保障,未修改产品逻辑。

最终 CI run 32967761875 全绿:

  • classifier 5s
  • static/full unit/Eval/build 1m43s
  • full server integration 2m37s
  • Compose smoke 4m25s
  • macOS 1m6s / Windows 2m12s

当前 PR 状态 mergeStateStatus=CLEAN,工作区干净。

@leecyang
leecyang merged commit 6698cd4 into LingXi-Org:main Aug 26, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant