fix(ci): bound anonymous review provider budget - #722
Conversation
|
Warning Review limit reached
Next review available in: 19 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
Comment |
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current head601fb6410f800e893597a42719bc73509a86f0e3. -
Head SHA:
601fb6410f800e893597a42719bc73509a86f0e3 -
Workflow run: 30832817120
-
Workflow attempt: 1
Coverage evidence
Coverage Decision
- Result: FAIL
- Test evidence: not proven passing
- Docstring evidence: not proven passing when configured
- Failure count: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: opencode-review-dispatch.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: opencode-review-dispatch.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["CI script (2 files)"]
S2 --> I2["review and security gate shell path"]
I2 --> R2["Review risk: CI script (2 files)"]
R2 --> V2["bash -n plus Strix self-test"]
Evidence --> S3["Test (2 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (2 files)"]
R3 --> V3["targeted test run"]
OpenCode Review Overview
--> Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow (2 files)"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow (2 files)"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["CI script (2 files)"]
S2 --> I2["review and security gate shell path"]
I2 --> R2["Review risk: CI script (2 files)"]
R2 --> V2["bash -n plus Strix self-test"]
Evidence --> S3["Test (2 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (2 files)"]
R3 --> V3["targeted test run"]
|
|
Current-head follow-up: the previous central coverage run failed because |
seonghobae
left a comment
seonghobae
left a comment
There was a problem hiding this comment.
Re-evaluate the exact current head after the base coverage fix and current security checks.
|
@coderabbitai review exact current head 851e686 |
|
✅ Action performedReview finished.
|
|
Current-head follow-up for The completed trusted-default-branch dispatch for prior head
The branch now maps every observed repository-owned category without replaying provider text, and the bounded schema-repair prompt requires the exact sentinel/identity, changed-file path, all verification posture labels, coverage/docstring evidence, and receipt-bound probes. It also routes same-repository status publication through the job-scoped Exact-head local verification: 75 focused contract/model-pool tests passed, Ruff check passed, runner |
There was a problem hiding this comment.
Pull request overview
OpenCode reviewed the current-head bounded evidence and found no blocking issues.
Findings
No blocking findings.
Summary
Approval sufficiency: positive current-head evidence — Coverage execution evidence reports PASS ('supported repository test suites passed'), Failed GitHub Check evidence reports no completed failed checks, Other unresolved review thread evidence reports no unresolved threads, and the focused hunks match PR intent. Verification posture: Coverage execution evidence reports supported repository test suites passed (test command python3 -m pytest tests) and configured repository docstring gates passed or advisory; no OPENCODE_EXECUTION_RECEIPT lines exist in bounded evidence, so runtime behavior is established only from trusted current-head diff traces, CodeGraph verbatim source, and the coverage PASS. Linter/static: no separate lint receipt beyond the coverage gate; shellcheck results were not part of the bounded evidence (source limitation, not a repository claim). TDD/regression: new parametrized test_control_rejection_is_reduced_to_canonical_validator_category (tests/test_opencode_model_pool_runner.py:1045-1078) plus 122 added runner-test lines; Coverage execution evidence PASS. Coverage: PASS per Coverage execution evidence — 'supported repository test suites passed'. Docstring coverage: 'configured repository docstring gates passed or docstring coverage was advisory' per Coverage execution evidence. DAG: head-flow Mermaid flowchart: flowchart LR; A["opencode-review-dispatch.yml:4345 free-budget env"]-->B["run_opencode_review_model_pool.sh budget accounting"]-->C["keyed Terra/OpenAI/OpenRouter/GitHub Models fallbacks"]; D["opencode-review-dispatch.yml:7840 token-source select"]-->E["Publish repository_dispatch status step"]; B-->F["opencode_review_approve_gate.sh diagnostics"]-->G["classify_control_rejection() canonical category"]-->H["write_schema_repair_prompt() repair retry"]; F-->I["tests/test_opencode_model_pool_runner.py:1045-1078"]-->J["python3 -m pytest tests (coverage PASS)"] — reflects head flow with the base-to-head delta (free-budget env, token-source, classifier). PoC/execution: no execution receipts in bounded evidence; validation rests on the trusted diff traces plus coverage PASS. DDD/domain: CI review-pool domain — classifier keeps untrusted provider output out of repair prompts (canonical repository-owned categories only). CDD/context: OPENCODE_FREE_TOTAL_BUDGET_SECONDS contract propagates workflow env -> pool runner -> regression tests. Similar issues: PR splits #674; prior free-budget retry-blowout fixes #683/#662; CodeQL mixed-SHA regression from #545/#719 is fixed by aligning init/analyze to the existing v4.37.4 upload-sarif SHA (branch history commit 14baf54). Claim/concept check: PR claims a 900s combined anonymous cap — matches the added env var at opencode-review-dispatch.yml:4348 and the runner/test pairing. Standards search: GitHub Actions expression precedence (&& binds tighter than ||) is respected in the token-source chain; repository pinned-action convention (full SHA plus version comment) followed. Compatibility/convention: no breaking contract; new OPENCODE_*_SECONDS config uses multi-word snake_case; CodeQL init/analyze now share the v4.37.4 SHA f205ea1c3313d32999d8d6a48b4f6530d4437b38 with upload-sarif. Breaking-change/backcompat: same-repo dispatch publish now prefers github.token (least-privilege reduction); the cross-repo secrets chain and the fail-closed run-step guard are preserved. Performance: caps anonymous burn at 900s vs the 11700s retry budget; no hot-path impact. Developer experience: pool operators receive canonical rejection categories instead of raw validator prose, and validator diagnostics are mirrored to the step stderr for triage. User experience: non-web — reviewed surface is workflow logs and PR status publication, not UI. Visual/DOM: non-web change; no DOM surface and no Playwright receipt exists in evidence. Accessibility/i18n: no UI surface; not applicable with reason. Supply-chain/license: github/codeql-action pinned to full SHA f205ea1c3313d32999d8d6a48b4f6530d4437b38 (# v4.37.4); no new dependencies introduced. Packaging: pyproject.toml present (requires-python >=3.10), tests under tests/; unpackaged_source_surfaces is empty. Security/privacy: cross-repo status publication fails closed when only github.token is available; the token-source expression cannot select github.token for a different repository; the classifier never replays provider prose; no secrets are printed. Changed-file evidence inspected: .github/workflows/opencode-review-dispatch.yml (hunks at lines 4345 and 7840), .github/workflows/scheduled-security-scan.yml (lines 93 and 99), scripts/ci/run_opencode_review_model_pool.sh (classifier/normalizer hunks), tests/test_opencode_model_pool_runner.py (CodeGraph current-head source lines 1045-1078).
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including .github/workflows/opencode-review-dispatch.yml, .github/workflows/scheduled-security-scan.yml, scripts/ci/run_opencode_review_model_pool.sh, scripts/ci/test_strix_quick_gate.sh, tests/test_opencode_agent_contract.py, and 1 more.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects .github/workflows/opencode-review-dispatch.yml to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source claims require trusted bounded source evidence prepared outside the isolated model process; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: deterministic repair does not infer browser runtime execution; source-backed DOM/UI evidence and trusted workflow receipts were reviewed when present, and non-web surfaces used API/CLI/log/docs/workflow evidence instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
Adversarial validation
{"status":"passed","probes":[{"path":".github/workflows/opencode-review-dispatch.yml","line":7840,"hypothesis":"The new token-source expression breaks cross-repo repository_dispatch: it could drop the privileged PR_REVIEW_MERGE_TOKEN/OPENCODE_APPROVE_TOKEN selection or misreport the token source, causing status publication to fail or to use a too-weak token on another repository.","attack_or_counterexample":"Cross-repo dispatch where target_repository != github.repository with PR_REVIEW_MERGE_TOKEN set, plus same-repo dispatch where target_repository == github.repository with no secrets configured.","evidence":"Trusted current-head focused hunk at .github/workflows/opencode-review-dispatch.yml:7840 traces the added OPENCODE_STATUS_TOKEN_SOURCE expression: the `target_repository == github.repository && 'github-token'` term short-circuits only for same-repo, while cross-repo falls through to the pre-existing secrets chain (`PR_REVIEW_MERGE_TOKEN != '' && 'PR_REVIEW_MERGE_TOKEN' || ...`), and the run-step guard (`GH_REPOSITORY != GITHUB_REPOSITORY && OPENCODE_STATUS_TOKEN_SOURCE == 'github-token'` -> error) still fails closed for cross-repo token-less publication; Coverage execution evidence reports supported repository test suites passed, including the updated tests/test_opencode_agent_contract.py status-publish contract coverage (receipt lines 1592/2045). source-line-sha256=d46c771bd1ad5d539c556f4a21677790cc8f73cf15dc8eb509f38d4886066b40","outcome":"falsified"},{"path":"tests/test_opencode_model_pool_runner.py","line":1077,"hypothesis":"classify_control_rejection misclassifies validator diagnostics so raw validator prose (untrusted provider-adjacent output) would be replayed into the repair prompt, or unmapped diagnostics produce a wrong category.","attack_or_counterexample":"Feed each canonical validator diagnostic string into the classifier via `bash -c 'source \"$1\"; classify_control_rejection \"$2\"; printf \"%s\" \"$LAST_CONTROL_REJECTION_KIND\"'` and feed unmapped text to check the fail-safe default.","evidence":"Trusted CodeGraph current-head source at tests/test_opencode_model_pool_runner.py:1077 shows `assert result.stdout == expected_category` inside the parametrized test_control_rejection_is_reduced_to_canonical_validator_category, which sources run_opencode_review_model_pool.sh, invokes classify_control_rejection, and asserts returncode 0 (line 1076); the classifier default branch sets unknown-control-contract for any unmapped text so no provider prose is ever reflected into write_schema_repair_prompt; Coverage execution evidence reports supported repository test suites passed. source-line-sha256=3a875423da3c8185888cd14f5f2f8bad3768cf3579b9506f3bc64e3cec02b188","outcome":"falsified"},{"path":".github/workflows/opencode-review-dispatch.yml","line":4345,"hypothesis":"OPENCODE_FREE_TOTAL_BUDGET_SECONDS=900 is a no-op: the pool runner ignores it, so a quota-starved anonymous catalog can still consume the full 11700s retry budget before keyed Terra/OpenAI/OpenRouter/GitHub Models fallbacks run.","attack_or_counterexample":"Quota-starved free catalog where every anonymous candidate fails fast within its per-candidate timeout; verify the pool hands off to keyed fallbacks after the 900s combined anonymous budget instead of retrying the free catalog for the whole retry budget.","evidence":"Trusted current-head focused hunk at .github/workflows/opencode-review-dispatch.yml:4345 adds the cap with the stated rationale (Anonymous free candidates share fifteen minutes total...), scripts/ci/run_opencode_review_model_pool.sh gains 122 lines pairing the budget accounting, and tests/test_opencode_model_pool_runner.py gains 122 lines of regression coverage (receipt lines 648 and 1077); Coverage execution evidence reports supported repository test suites passed. source-line-sha256=3c3eff942b0198dfa14f7f3ba475fa5b5519ded8598ba6158bc11c31e20d0b1f","outcome":"falsified"}],"residual_risk":"The pool-runner budget-accounting hunks between the visible classifier/normalizer diffs and line 906 were truncated in the evidence packet; enforcement is corroborated by the paired 122-line runner-test additions and coverage PASS, but the exact elapsed-time accumulation/rollover arithmetic was not directly inspected. Same-repo status publication now depends on the job GITHUB_TOKEN scopes (statuses/pull-requests write) in the workflow permissions block, which was not shown in the evidence; if those scopes are absent the publish step fails loudly (fail-closed) rather than silently mispublishing. No Strix vulnerability reports and no failed checks were present at head."}- Result: APPROVE
- Reason: No blocking issues at head 023db07: the 900s anonymous free-budget cap, canonical validator-rejection classification, CodeQL v4.37.4 pin alignment, and same-repo status-token selection match PR intent, are paired with regression tests, and are supported by passing coverage, no failed checks, and no unresolved threads.
- Head SHA:
023db07dafc138e2de0912cadc10e4e373a207de - Workflow run: 30848019631
- Workflow attempt: 2
Pull request was converted to draft
1 participant
Summary
Why this is split from #674
Two exact-head #674 review runs spent about 75 to 80 minutes across anonymous free candidates and reached the 30-attempt ceiling before later keyed fallbacks could run. This PR isolates only the bounded-provider fix from the unrelated base-lock changes already present in #674, so the central review workflow can adopt the queue bound independently.
Validation