Skip to content

Add Volcengine smoke and full Provider replay - #89

Merged
2002yy merged 7 commits into
mainfrom
p0-e3/volcengine-smoke-replay
Jul 29, 2026
Merged

Add Volcengine smoke and full Provider replay#89
2002yy merged 7 commits into
mainfrom
p0-e3/volcengine-smoke-replay

Conversation

@2002yy

@2002yy 2002yy commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Scope

Continue P0-E3 with a replay-only Volcengine Ark adapter and bounded smoke/full workflow modes. This measures compatibility before any production Provider expansion.

Changes

  • add VolcengineArkReplayProvider as a subclass of the provenance-approved real-provider replay adapter
  • use the official OpenAI-compatible Ark base URL https://ark.cn-beijing.volces.com/api/v3
  • accept VOLCENGINE_API_KEY with ARK_API_KEY fallback and an exact Model ID / Endpoint ID
  • keep Volcengine out of the production chat Provider owner
  • add explicit --model-name and --base-url replay arguments
  • add a default smoke mode that runs only clean_requests_session
  • keep AnswerClaim evaluation exclusive to full, which runs all 10 gold cases
  • require provider identity, model name, endpoint fingerprint, exact smoke/full scope and record-only boundaries
  • reject Agent Plan /api/plan/v3 and Coding Plan /api/coding/v3 in adapter code, tests and documentation
  • add adapter, routing, identity, JSON mode, usage, scope and workflow regression tests

Boundaries

  • no production ChatTurn or SQLite write
  • no committed learning truth change
  • no claim UI or production claim producer
  • Volcengine is replay-only and does not appear in the production chat Provider owner
  • no automatic Provider spend in PR CI
  • smoke is connectivity/schema evidence only and cannot produce a model-quality conclusion
  • a real workflow still requires a configured GitHub secret and manual dispatch

Validation

Final head CI #1696 passed the complete repository gate: pytest, RAG K1, Ruff, packaging, detect-secrets, expanded mypy, frontend tests/build, 35/35 fixture+narrow browser journeys and 12/12 real-stack cases.

No real Volcengine request was made. After merge, the next operation is one manually dispatched Volcengine smoke using the exact Model/Endpoint ID. Only a green smoke may proceed to two full 10-case runs.

@2002yy
2002yy marked this pull request as ready for review July 29, 2026 12:37
@2002yy
2002yy merged commit b1fac06 into main Jul 29, 2026
2 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 898cb0599e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +233 to +235
if replay_scope == "smoke":
if provider_report.get("cases") != 1:
raise SystemExit("Smoke replay must execute exactly one case.")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject smoke replays with unparsable provider output

When Volcengine returns plain text or malformed JSON, parse_provider_answer() records a fallback candidate with a nonempty results[].parse_error, while run_provider_answer_replay() still reports status: completed and produces an answer_quality dictionary. This smoke branch checks only counts and scope, so that response passes the workflow even though the smoke is intended to establish JSON/schema compatibility before a full paid run; inspect the selected result's parse error and fail the smoke when it is nonempty.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant