Skip to content

Add transparent adversarial Benchmark v1 - #38

Merged
LE0-Lin merged 1 commit into
mainfrom
feat/benchmark-v1
Sep 4, 2026
Merged

Add transparent adversarial Benchmark v1#38
LE0-Lin merged 1 commit into
mainfrom
feat/benchmark-v1

Conversation

@LE0-Lin

@LE0-Lin LE0-Lin commented Sep 4, 2026

Copy link
Copy Markdown
Owner

Why

AgentConfigScore had deterministic tests and a three-repository smoke benchmark, but no labeled mutation suite that quantified both expected detections and known misses. This made it too easy to over-interpret A 100 or a small set of demos.

What changed

  • add an offline labeled corpus expanded across six instruction formats
  • evaluate 74 deterministic contract cases, including 49 expected detections and 26 clean negative controls
  • add eight explicitly labeled, non-gating semantic and unmodeled danger challenges
  • report exact-case accuracy, precision, recall, F1, clean controls, and per-category results in JSON and Markdown
  • commit the generated report and test that it stays synchronized with scanner behavior
  • validate fixture paths so benchmark cases cannot write outside their temporary repository
  • document why closed-suite 100% metrics are not real-world accuracy estimates

Current result

  • deterministic contract: 74/74 exact matches
  • expected detections: 49, with 0 missing and 0 unexpected rule IDs
  • clean controls: 26/26
  • open challenge set: 0/8 detected and clearly reported

The challenge tier intentionally does not control the exit code. It is a visible research backlog, not a hidden failure or an inflated headline metric.

Verification

  • 122 unit and contract tests pass locally
  • repository self-scan: A 100, zero findings
  • pinned real-repository benchmark: all three repositories still match reviewed expectations
  • benchmark runs fully offline and uses no LLM/API tokens

@LE0-Lin
LE0-Lin merged commit 56c79c8 into main Sep 4, 2026
21 checks passed
@LE0-Lin
LE0-Lin deleted the feat/benchmark-v1 branch September 4, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant