fix(security): Strix provider 장애를 fail-closed로 유지 - #587
Conversation
|
검증 메모: 이 PR의 pull_request_target Strix 실행은 아직 기본 브랜치의 기존 래퍼를 사용하므로 녹색 결론 자체를 유효한 보안 증거로 보지 않았습니다.
이 PR은 바로 그 중화 분기를 제거하고 모든 비정상 gate 종료 코드를 그대로 반환합니다. 추가된 실행형 회귀 테스트는 quota/unavailable 출력을 내고 exit 1인 가짜 게이트를 실제 래퍼에 통과시켜, 래퍼도 exit 1이며 로그가 보존되는지 검증합니다. 로컬 전체 pytest 634개와 별도 보안 diff scan은 통과했습니다. |
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
This PR ensures Strix provider outages remain fail-closed by removing the workflow logic that downgraded certain provider/LLM failures to a successful check, while preserving always-run artifact/log collection and adding a regression test that executes the actual workflow run: block against a fake quota-exhaustion gate.
Changes:
- Remove the “neutral skip” downgrade path for provider/LLM outage signals in
.github/workflows/strix.ymland always propagate the gate exit code. - Add a regression test that extracts and executes the Strix workflow step script with a fake failing gate and asserts fail-closed behavior + log preservation.
- Update CI contract assertions to require the new fail-closed messaging and forbid the removed downgrade signals.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
tests/test_required_workflow_queue_contract.py |
Adds a regression test that runs the extracted Strix workflow script and asserts provider outage remains failing while preserving logs. |
scripts/ci/test_strix_quick_gate.sh |
Updates hardening assertions to require fail-closed messaging and forbid the removed neutral-skip mechanism. |
.github/workflows/strix.yml |
Removes provider-outage neutralization and prints a single error annotation before exiting with the gate’s code. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode could not approve from deterministic current-head evidence because GitHub Checks have failed.
Findings
1. HIGH Current-head GitHub Checks - Fix failed required checks before approval
- Problem: Failed same-head checks remain for
de5017263e6dd7632291e311f51271456acd0470. - Root cause: The model-unavailable evidence fallback is allowed only when peer GitHub Checks are complete and clean.
- Fix: Read and fix the failed check logs below, then rerun the current-head checks.
- Regression test: Keep the model-unavailable fallback gated on an empty failed-check rollup.
Failed checks:
- Python Security/pip-audit (Python dependency audit): FAILURE (https://github.com/ContextualWisdomLab/.github/actions/runs/29930526000/job/88958778202)
- pip-audit (Python dependency audit) check run: failure (https://github.com/ContextualWisdomLab/.github/actions/runs/29930526000/job/88958778202)
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: strix.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: strix.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["CI script: test_strix_quick_gate.sh"]
S2 --> I2["review and security gate shell path"]
I2 --> R2["Review risk: CI script: test_strix_quick_gate.sh"]
R2 --> V2["bash -n plus Strix self-test"]
Evidence --> S3["Test: test_required_workflow_queue_contract.py"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test: test_required_workflow_queue_contract.py"]
R3 --> V3["targeted test run"]
OpenCode Review Overview
Pull request overviewOpenCode cannot approve yet because required coverage evidence did not pass. Review outcome1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
Coverage evidenceCoverage Decision
Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: strix.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: strix.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["CI script: test_strix_quick_gate.sh"]
S2 --> I2["review and security gate shell path"]
I2 --> R2["Review risk: CI script: test_strix_quick_gate.sh"]
R2 --> V2["bash -n plus Strix self-test"]
Evidence --> S3["Test (3 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (3 files)"]
R3 --> V3["targeted test run"]
|
de50172 to
476527c
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
.github/workflows/strix.yml:767
- The step emits the same error annotation for all nonzero Strix exit codes, but the message text focuses on “provider outages without a vulnerability report”. When the gate fails due to real findings (also exit 1), this wording can be misread as implying an outage rather than a vulnerability signal. Consider explicitly stating that genuine findings also fail the check so the annotation stays accurate for both cases.
# target-app token that already validated and fetched that exact PR;
# preserve the target-repository token for pull_request_target runs.
There was a problem hiding this comment.
Pull request overview
OpenCode could not approve from deterministic current-head evidence because GitHub Checks have failed.
Findings
1. HIGH Current-head GitHub Checks - Fix failed required checks before approval
- Problem: Failed same-head checks remain for
476527cd0b37aecd9fcb0523c9373925a83f05f1. - Root cause: The model-unavailable evidence fallback is allowed only when peer GitHub Checks are complete and clean.
- Fix: Read and fix the failed check logs below, then rerun the current-head checks.
- Regression test: Keep the model-unavailable fallback gated on an empty failed-check rollup.
Failed checks:
- Strix Security Scan/strix: FAILURE (https://github.com/ContextualWisdomLab/.github/actions/runs/29992354086/job/89157847669)
- Strix Security Scan/strix: failure (https://github.com/ContextualWisdomLab/.github/actions/runs/29992354086/job/89157847669)
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: strix.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: strix.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["CI script: test_strix_quick_gate.sh"]
S2 --> I2["review and security gate shell path"]
I2 --> R2["Review risk: CI script: test_strix_quick_gate.sh"]
R2 --> V2["bash -n plus Strix self-test"]
Evidence --> S3["Test: test_required_workflow_queue_contract.py"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test: test_required_workflow_queue_contract.py"]
R3 --> V3["targeted test run"]
476527c to
db05652
Compare
db05652 to
6b8e9a5
Compare
6b8e9a5 to
5e5728c
Compare
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughStrix 게이트의 프로바이더 장애 중립 스킵 로직을 제거하고 종료 코드를 그대로 전파하도록 변경했습니다. CI 어서션과 가짜 게이트 실행 테스트도 실패-폐쇄 동작을 검증하도록 갱신되었습니다. ChangesStrix 게이트 실패 정책
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Workflow as Run Strix quick
participant Gate as TRUSTED_STRIX_GATE
participant ConsoleLog as strix_gate_console.log
participant Tests as CI tests
Workflow->>Gate: 게이트 실행
Gate-->>Workflow: RateLimitError 출력 및 exit 1
Workflow->>ConsoleLog: 게이트 콘솔 출력 저장
Workflow-->>Workflow: exit 1 전파
Tests->>Workflow: 실행 스크립트와 환경 변수 주입
Workflow-->>Tests: 실패 코드 및 로그 반환
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
4d2841c to
0949d2d
Compare
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current headb551e6267aa2657598dca8080a4d2f01286e564b. -
Head SHA:
b551e6267aa2657598dca8080a4d2f01286e564b -
Workflow run: 30801901307
-
Workflow attempt: 1
Coverage evidence
Coverage Decision
- Result: FAIL
- Test evidence: not proven passing
- Docstring evidence: not proven passing when configured
- Failure count: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: strix.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: strix.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["CI script: test_strix_quick_gate.sh"]
S2 --> I2["review and security gate shell path"]
I2 --> R2["Review risk: CI script: test_strix_quick_gate.sh"]
R2 --> V2["bash -n plus Strix self-test"]
Evidence --> S3["Test (3 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (3 files)"]
R3 --> V3["targeted test run"]
b551e62 to
123e70c
Compare
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current heade9e872c52925bbfa737e9fcf5fbee34f4fa466c2. -
Head SHA:
e9e872c52925bbfa737e9fcf5fbee34f4fa466c2 -
Workflow run: 30806086103
-
Workflow attempt: 1
Coverage evidence
Coverage Decision
- Result: FAIL
- Test evidence: not proven passing
- Docstring evidence: not proven passing when configured
- Failure count: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: strix.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: strix.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["CI script: test_strix_quick_gate.sh"]
S2 --> I2["review and security gate shell path"]
I2 --> R2["Review risk: CI script: test_strix_quick_gate.sh"]
R2 --> V2["bash -n plus Strix self-test"]
Evidence --> S3["Test (3 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (3 files)"]
R3 --> V3["targeted test run"]
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current heade9e872c52925bbfa737e9fcf5fbee34f4fa466c2. -
Head SHA:
e9e872c52925bbfa737e9fcf5fbee34f4fa466c2 -
Workflow run: 30806697451
-
Workflow attempt: 1
Coverage evidence
Coverage Decision
- Result: FAIL
- Test evidence: not proven passing
- Docstring evidence: not proven passing when configured
- Failure count: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: strix.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: strix.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["CI script: test_strix_quick_gate.sh"]
S2 --> I2["review and security gate shell path"]
I2 --> R2["Review risk: CI script: test_strix_quick_gate.sh"]
R2 --> V2["bash -n plus Strix self-test"]
Evidence --> S3["Test (3 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (3 files)"]
R3 --> V3["targeted test run"]
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current heade9e872c52925bbfa737e9fcf5fbee34f4fa466c2. -
Head SHA:
e9e872c52925bbfa737e9fcf5fbee34f4fa466c2 -
Workflow run: 30806697451
-
Workflow attempt: 2
Coverage evidence
Coverage Decision
- Result: FAIL
- Test evidence: not proven passing
- Docstring evidence: not proven passing when configured
- Failure count: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: strix.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: strix.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["CI script: test_strix_quick_gate.sh"]
S2 --> I2["review and security gate shell path"]
I2 --> R2["Review risk: CI script: test_strix_quick_gate.sh"]
R2 --> V2["bash -n plus Strix self-test"]
Evidence --> S3["Test (3 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (3 files)"]
R3 --> V3["targeted test run"]
e9e872c to
6dadc72
Compare
![pr-auto-rebase[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
6dadc72 to
d985c5d
Compare
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current headd985c5d02c613a3fdecb1d4744cd598a08921211. -
Head SHA:
d985c5d02c613a3fdecb1d4744cd598a08921211 -
Workflow run: 30836872678
-
Workflow attempt: 1
Coverage evidence
Coverage Decision
- Result: FAIL
- Test evidence: not proven passing
- Docstring evidence: not proven passing when configured
- Failure count: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: strix.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: strix.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["CI script: test_strix_quick_gate.sh"]
S2 --> I2["review and security gate shell path"]
I2 --> R2["Review risk: CI script: test_strix_quick_gate.sh"]
R2 --> V2["bash -n plus Strix self-test"]
Evidence --> S3["Test (3 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (3 files)"]
R3 --> V3["targeted test run"]
|
Closing as superseded by #734. The exact fail-closed workflow wrapper and executable regression contracts are now incorporated into #734 head |
|
Closing this pre-baseline branch to remove stale workflow pressure. The Strix provider-outage fail-closed regression remains in the consolidated post-#731 replay queue and should be replayed from repaired current main without neutralizing missing security evidence. |
요약
always()아티팩트 수집과gate-console.log보존은 유지합니다.보안 불변식
완전한 취약점 보고서가 없는 provider 장애는 통과가 아니라 불완전한 보안 증거입니다. 이 경우 required check와 repository_dispatch
strix상태는 실패로 남아야 합니다.검증
pytest -q tests/test_required_workflow_queue_contract.py -k strix_provider_outage_without_findings_fails_closed— 1 passedpytest -q— 634 passedactionlint -shellcheck= -pyflakes= .github/workflows/strix.yml— passedbash -n/git diff --check— passed기준선 참고
전체
scripts/ci/test_strix_quick_gate.sh실행에서는 이번 diff가 건드리지 않은 gate 시나리오들의 기존 기대값 불일치가 별도로 드러났습니다(nonrecoverable문구, explicit-empty Vertex provider, fallback baseline report 판정). 이번 PR은 false-success 종료 코드 전파만 좁게 수정하며 해당 별도 기준선 문제를 숨기거나 성공으로 표시하지 않습니다.Summary by CodeRabbit