feat(prism): dual-gate plagiarism with OpenRouter Grok 4.5 sole borderline verdict - #47
Conversation
…ole verdict Borderline/attach pairs no longer hard-reject from static scores alone. Exact source-hash clones still hard-reject; quarantine/attach outcomes call the OpenRouter adjudicator (httpx tool-calling) which alone allows or rejects. OpenRouter key loaded from /run/secrets/openrouter_api_key for durable prod use.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (3)
📝 WalkthroughWalkthroughPrism adds OpenRouter-backed plagiarism adjudication for borderline duplicate submissions. Exact source-hash matches remain deterministic rejects; borderline cases use structured tool-call verdicts, fail closed on errors, persist adjudication metadata, and are covered by unit and worker integration tests. ChangesPlagiarism adjudication flow
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant PrismWorker
participant adjudicate_plagiarism
participant OpenRouter
PrismWorker->>adjudicate_plagiarism: provide borderline comparison data
adjudicate_plagiarism->>OpenRouter: request SubmitPlagiarismVerdict tool call
OpenRouter-->>adjudicate_plagiarism: return structured verdict
adjudicate_plagiarism-->>PrismWorker: return normalized adjudication
PrismWorker->>PrismWorker: persist verdict and determine acceptance
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
… gate Make package-scoped prism-checks pass: collections.abc Mapping, import sort, drop unused test imports, wrap long getattr line.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (2)
packages/challenges/prism/tests/test_plagiarism_llm_adjudicator.py (1)
211-244: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider covering the exact-hash short-circuit at the worker level too.
This test asserts
classify_duplicatebehavior only; the "no LLM for exact hash" guarantee lives in_review_static_submission. A worker-level test that monkeypatchesadjudicate_plagiarismand asserts it is never called would pin the actual dual-gate contract.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/challenges/prism/tests/test_plagiarism_llm_adjudicator.py` around lines 211 - 244, Add a worker-level test for _review_static_submission that submits an exact source-hash match, monkeypatches adjudicate_plagiarism, and asserts the patched function is never called while the submission is rejected. Keep the existing classify_duplicate unit test unchanged and verify the worker’s dual-gate short-circuit directly.packages/challenges/prism/src/prism_challenge/evaluator/plagiarism_adjudicator.py (1)
360-392: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winRetries have no backoff; a 429 burst will be retried immediately and then fail closed.
Since a failed adjudication rejects a submission, an immediate retry on rate limiting is likely to waste the only retry. Consider a short sleep (with jitter) between attempts, and skipping retries for non-retryable 4xx.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/challenges/prism/src/prism_challenge/evaluator/plagiarism_adjudicator.py` around lines 360 - 392, Add retry backoff to the attempt loop surrounding the HTTP call and verdict extraction: after a failed attempt, sleep briefly with jitter before the next retry, especially for rate-limit responses. Detect non-retryable 4xx failures and stop retrying them immediately, while preserving the existing fail-closed behavior and final exception handling.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/challenges/prism/src/prism_challenge/evaluator/plagiarism_adjudicator.py`:
- Line 419: Fix the Ruff E501 violation in the configuration call by formatting
the max_source_chars argument across multiple lines while preserving its
existing value and fallback behavior.
- Line 25: Update the typing imports in plagiarism_adjudicator.py to remove
Mapping from typing and import it from collections.abc instead, while preserving
the existing Any and Protocol imports and all usage sites.
In `@packages/challenges/prism/src/prism_challenge/queue.py`:
- Around line 854-863: Update the adjudication call in the queue flow to run the
synchronous adjudicate_plagiarism function via await asyncio.to_thread,
forwarding the existing keyword arguments unchanged and preserving its returned
adjudication result.
In `@packages/challenges/prism/tests/test_plagiarism_llm_adjudicator.py`:
- Around line 251-309: Remove the dead ctx construction and its associated
PrismContext, component evaluator, and other unused imports in this test.
Preserve the PrismWorker import because it is required by the worker
instantiation, and keep the test’s actual database, repository, and worker setup
unchanged.
---
Nitpick comments:
In
`@packages/challenges/prism/src/prism_challenge/evaluator/plagiarism_adjudicator.py`:
- Around line 360-392: Add retry backoff to the attempt loop surrounding the
HTTP call and verdict extraction: after a failed attempt, sleep briefly with
jitter before the next retry, especially for rate-limit responses. Detect
non-retryable 4xx failures and stop retrying them immediately, while preserving
the existing fail-closed behavior and final exception handling.
In `@packages/challenges/prism/tests/test_plagiarism_llm_adjudicator.py`:
- Around line 211-244: Add a worker-level test for _review_static_submission
that submits an exact source-hash match, monkeypatches adjudicate_plagiarism,
and asserts the patched function is never called while the submission is
rejected. Keep the existing classify_duplicate unit test unchanged and verify
the worker’s dual-gate short-circuit directly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 1c36a061-d2a8-4a4d-9e5b-c55dfb3331b7
📒 Files selected for processing (5)
packages/challenges/prism/src/prism_challenge/config.pypackages/challenges/prism/src/prism_challenge/evaluator/plagiarism_adjudicator.pypackages/challenges/prism/src/prism_challenge/queue.pypackages/challenges/prism/tests/test_gateway_absence_and_deterministic_admission.pypackages/challenges/prism/tests/test_plagiarism_llm_adjudicator.py
| llm_cfg = plagiarism_llm_config_from_settings(self.settings) | ||
| adjudication = adjudicate_plagiarism( | ||
| current_code=current_code, | ||
| candidate_code=candidate_code, | ||
| comparison_report=pair_report, | ||
| deterministic_reason=duplicate.reason, | ||
| deterministic_outcome=duplicate.outcome, | ||
| candidate_submission_id=cand.submission_id, | ||
| config=llm_cfg, | ||
| ) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
Blocking HTTP call on the event loop: wrap adjudicate_plagiarism in asyncio.to_thread.
adjudicate_plagiarism is synchronous and issues an httpx.Client.post with timeout_seconds (default 90s) for up to max_retries + 1 attempts — up to ~180s of blocked event loop per borderline submission. In combined mode the API and worker share one loop (see combined_mode in packages/challenges/prism/src/prism_challenge/config.py Lines 352-360), so /health, /v1/registry, and /v1/weights/latest stall for the duration. asyncio is already imported in this module.
As per coding guidelines, "Keep the documented network surfaces available: /health, /v1/registry, /v1/weights/latest, ...".
🛠️ Proposed fix
llm_cfg = plagiarism_llm_config_from_settings(self.settings)
- adjudication = adjudicate_plagiarism(
- current_code=current_code,
- candidate_code=candidate_code,
- comparison_report=pair_report,
- deterministic_reason=duplicate.reason,
- deterministic_outcome=duplicate.outcome,
- candidate_submission_id=cand.submission_id,
- config=llm_cfg,
- )
+ adjudication = await asyncio.to_thread(
+ partial(
+ adjudicate_plagiarism,
+ current_code=current_code,
+ candidate_code=candidate_code,
+ comparison_report=pair_report,
+ deterministic_reason=duplicate.reason,
+ deterministic_outcome=duplicate.outcome,
+ candidate_submission_id=cand.submission_id,
+ config=llm_cfg,
+ )
+ )(asyncio.to_thread forwards kwargs directly, so partial is optional — await asyncio.to_thread(adjudicate_plagiarism, current_code=..., ...) works as well. The existing tests monkeypatch prism_challenge.queue.adjudicate_plagiarism, so they keep working.)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/challenges/prism/src/prism_challenge/queue.py` around lines 854 -
863, Update the adjudication call in the queue flow to run the synchronous
adjudicate_plagiarism function via await asyncio.to_thread, forwarding the
existing keyword arguments unchanged and preserving its returned adjudication
result.
Source: Coding guidelines
2 participants
Summary
quarantine/attachranks go only to an OpenRouter LLM adjudicator for the final verdict.x-ai/grok-4.5(OpenRouter slug matching opencodeopenrouter/x-ai/grok-4.5).PRISM_PLAGIARISM_LLM_*settings.Changes
plagiarism_adjudicator.py: structured OpenRouter tool-callSubmitPlagiarismVerdictvia httpx.config.py: LLM enable/required/model/key-file settings (PRISM_OPENROUTER_*,PRISM_PLAGIARISM_LLM_*).queue.py: wire dual-gate after deterministic rank; map LLM verdict to pass/reject + reasons.Test plan
uv run pytest packages/challenges/prism/tests/test_plagiarism_llm_adjudicator.py(and related plagiarism wiring) — 25 passedplagiarized=True used_llm=True model=x-ai/grok-4.5CI
Notes
PRISM_OPENROUTER_API_KEY_FILE(or env) andPRISM_PLAGIARISM_LLM_ENABLED=true/REQUIRED=trueas appropriate — no secrets in this PR.Summary by CodeRabbit