Skip to content

feat(evals): a tool-selection benchmark for the inventory tools (0.16.3) - #30

Merged
mabry1985 merged 2 commits into
mainfrom
feat/tool-selection-eval
Nov 6, 2026
Merged

mabry1985 merged 2 commits into
mainfrom
feat/tool-selection-eval

Conversation

@mabry1985

Copy link
Copy Markdown
Member

Step 3 of making Inventory a first-class plugin: a tool-selection benchmark, mirroring careercoach-plugin's evals/tool_selection/.

What's added

  • evals/tool_selection/prompts.yaml: 53 fixed reseller prompts covering P&L, find, intake, price, sale (eBay and off-eBay), listing, bins, packing, review, CSV and delete. It also has questions that need no tool, and traps whose right answer is no tool: post to eBay, set sold by hand, message a buyer, end a live listing. The decisions behind each expectation are written at the top of the file. Ids, prices and URLs are invented.
  • run_eval.py:
    • registers the real plugin on a throwaway database and takes the schemas from registration;
    • adds four host decoys (web_search, fetch_url, memory_recall, ebay_price_check);
    • registers the turn-only scan, draft and sale tools too, but they are never correct for an operator prompt;
    • --dry-run checks the plumbing;
    • a live run uses EVAL_BASE_URL, EVAL_API_KEY and EVAL_MODEL, writes results/<label>.json (gitignored) and prints a per-capability table.
  • tests/test_tool_eval.py: pins the scoring rules and the corpus: unique ids, every expected tool registered, none turn-only, every operator-facing tool in at least two prompts, no operator data, and a complete dry run.
  • Docs: docs/development.md has a short section and a layout row. Version bumped to 0.16.3; there's no behavior change, so no upgrading note.

Baseline (protoLabs gateway, before any tuning)

Model Overall Weakest
protolabs/fast 43/53 (81.1%) sale 0/5, trap 2/4
protolabs/smart 48/53 (90.6%) sale 1/5, trap 3/4

Finding: both models pick the turn-only inventory_sale_candidates first for chat sale requests ("sold the X for $Y"), instead of inventory_mark_sold or a lookup. It's read-only, so it costs an extra step rather than causing a bad write, but its description should steer chat elsewhere. That's the next thing to tune and re-run; it isn't changed here.

Gates: 480 passed; ruff check and format clean.

🤖 Generated with Claude Code

evals/tool_selection/ mirrors careercoach-plugin's: 53 fixed reseller prompts across every
operator capability plus no-tool questions and traps, the real plugin registered on a throwaway
DB for its schemas, four host decoys, the turn-only tools registered but never correct for an
operator prompt. --dry-run checks the plumbing; a live run uses EVAL_BASE_URL / EVAL_API_KEY /
EVAL_MODEL and writes results/<label>.json (gitignored) with a per-capability table.

tests/test_tool_eval.py pins the scoring rules and the corpus: unique ids, every expected tool
registered, none turn-only, every operator-facing tool in at least two prompts, no operator
data, and a complete dry run.

Baseline through the protoLabs gateway: protolabs/fast 43/53, protolabs/smart 48/53. Both pick
the turn-only inventory_sale_candidates for chat sale requests, the next description to tune.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@protoreview protoreview Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QA panel review — WARN

code-review-structural · head bafe0cb25789 · formal

Low-risk PR: a standalone eval harness (new files only) plus a version bump. No behavioral changes to existing code. The one thing worth fixing before merge is the diagnostic label at run_eval.py:283 that claims to count "operator prompts" but actually sums across all prompts including chat/trap rows — a one-line filter fix. The panel disagreed on nothing; all findings are minor or nit. Verification confirmed all three findings with no changes. No gaps: the verifier annotated all 3 findings it was given.

Findings

Severity Location Finding Verified
🟡 minor evals/tool_selection/run_eval.py:279 The 'turn-only picks on operator prompts' diagnostic sums turn_only_pick over every prompt, including the chat/trap prompts that expect no tool, so the count i… confirmed
⚪ nit evals/tool_selection/run_eval.py:257 A single transient failure (network error, unexpected response shape) in live mode aborts the entire evaluation with no partial results, since the result JSON … confirmed
⚪ nit docs/development.md The new tests/test_tool_eval.py is not added to the 'Tests worth knowing about' table in docs/development.md, which the doc otherwise maintains as the list of … confirmed
findings JSON (machine-readable)
[
  {
    "file": "evals/tool_selection/run_eval.py",
    "line": 279,
    "severity": "minor",
    "category": "correctness",
    "claim": "The 'turn-only picks on operator prompts' diagnostic sums turn_only_pick over every prompt, including the chat/trap prompts that expect no tool, so the count is not restricted to operator prompts as the label claims.",
    "evidence": "turn_only = sum(r[\"turn_only_pick\"] for r in result[\"picks\"])\n    print(f\"turn-only picks on operator prompts: {turn_only}\")",
    "verdict": "confirmed",
    "note": "Lines 279\u2013280 sum turn_only_pick across all rows; the score() function sets turn_only_pick unconditionally for every prompt (line ~103: `\"turn_only_pick\": pick in TURN_ONLY`), including trap/chat prompts where expect==\"none\". The label says 'on operator prompts' but no filter restricts the sum to operator-only rows.",
    "line_original": 283,
    "line_corrected": true
  },
  {
    "file": "evals/tool_selection/run_eval.py",
    "line": 257,
    "severity": "nit",
    "category": "correctness",
    "claim": "A single transient failure (network error, unexpected response shape) in live mode aborts the entire evaluation with no partial results, since the result JSON is only written after all picks complete.",
    "evidence": "In live (non-dry-run) mode, model_pick is called sequentially for every prompt in a dict comprehension. If any single call raises (httpx.ConnectError, httpx.TimeoutException, KeyError from unexpected response shape, etc.), the exception propagates out of main() and the process exits with a traceback. All previously successful picks are discarded because the result JSON is only written after the en Fix: Wrap the per-prompt call in a try/except inside the comprehension (or a loop), collecting an error sentinel for failed prompts and reporting them in the output. Alternatively, add a simple retry with (protopatch confidence: medium)",
    "source": "protopatch",
    "verdict": "confirmed",
    "note": "Line 258: `picks = {p[\"id\"]: model_pick(client, base_url, api_key, model, p[\"prompt\"], tools) for p in prompts}` \u2014 no per-item try/except. The JSON write (line 275: `path.write_text(...)`) executes only after the comprehension completes. A single exception propagates out of main() with no partial output."
  },
  {
    "file": "docs/development.md",
    "line": 0,
    "severity": "nit",
    "category": "conventions",
    "claim": "The new tests/test_tool_eval.py is not added to the 'Tests worth knowing about' table in docs/development.md, which the doc otherwise maintains as the list of notable tests.",
    "evidence": "The table ends at \"| `tests/test_hardening.py` | Atomic writes, the sold guard, strict money, non-clobbering import |\" and the PR adds a separate \"### The tool-selection eval\" section (\"`tests/test_tool_eval.py` validates the corpus and runs the dry-run plumbing on every CI run\") instead of a row in that table.",
    "verdict": "confirmed",
    "note": "Diff confirms: the table's last row is `tests/test_hardening.py`; the new `tests/test_tool_eval.py` is mentioned only in the added `### The tool-selection eval` section, not as a row in the 'Tests worth knowing about' table."
  }
]

@protoreview protoreview Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Promoting the WARN verdict for head bafe0cb25789: all checks terminal-green, zero unresolved review threads. (approve-on-green)

Open findings carried by this approval — non-blocking, but they did not go away:

  • minor evals/tool_selection/run_eval.py:279 — The 'turn-only picks on operator prompts' diagnostic sums turn_only_pick over every prompt, including the chat/trap prompts that expect no tool, so the count is not restricted to operator prompts as t
  • nit evals/tool_selection/run_eval.py:257 — A single transient failure (network error, unexpected response shape) in live mode aborts the entire evaluation with no partial results, since the result JSON is only written after all picks complete.
  • nit docs/development.md:0 — The new tests/test_tool_eval.py is not added to the 'Tests worth knowing about' table in docs/development.md, which the doc otherwise maintains as the list of notable tests.

Approving a WARN does not resolve its findings (issue #22).

… the tests table

Review on #30: the label claimed 'operator prompts' while summing every row. It now prints the
total and the count on tool-expecting rows. docs/development.md lists tests/test_tool_eval.py,
and the README says a live run stops on the first error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mabry1985

Copy link
Copy Markdown
Member Author

Addressed in the latest commit: the turn-only count now prints the total and the count on tool-expecting rows (every prompt is an operator message, so a turn-only pick is wrong on any of them), and tests/test_tool_eval.py is in the docs tests table. Left as is: aborting a live run on the first transient error. Recording a failed call as a pick would change what's scored, so the README now says to re-run instead.

@protoreview protoreview Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QA panel review — PASS

code-review-structural · head 13a14f25c6b1 · formal

Clean PR. The single surviving finding (a protoPatch structural note claiming score(), load_prompts(), and fake_pick() lack dedicated unit tests) was refuted by the verifier, who pointed to test_score_acceptance (9 parametrized cases), test_score_rejects_incomplete_observations, test_committed_prompts_match_the_registered_surface, and test_dry_run_writes_results_without_a_network in tests/test_tool_eval.py — all directly exercising those functions. No findings remain. The three prior-round items (all minor/nit) were noted as resolved by the synthesis pass and do not carry forward. No verification gap: the verifier annotated all 1 finding it was given.

No findings — the review came back clean.

findings JSON (machine-readable)
[]

Addressed since the last round (3 minor/nit): evals/tool_selection/run_eval.py:279, evals/tool_selection/run_eval.py:257, docs/development.md:0 — no longer raised, and the PR changed the code around them since they were raised.

@protoreview protoreview Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Promoting the PASS verdict for head 13a14f25c6b1: all checks terminal-green, zero unresolved review threads. (approve-on-green)

@mabry1985
mabry1985 merged commit 55d7b1d into main Nov 6, 2026
3 checks passed
@mabry1985
mabry1985 deleted the feat/tool-selection-eval branch November 6, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant