diff --git a/.gitignore b/.gitignore index cc36313..86fb9b6 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,5 @@ __pycache__/ *.db *.db-wal *.db-shm + +evals/tool_selection/results/ diff --git a/docs/development.md b/docs/development.md index f1ee116..faa5040 100644 --- a/docs/development.md +++ b/docs/development.md @@ -25,6 +25,7 @@ Contributor notes: the file layout, the tests, checking the view by hand, and re | `skills/inventory-ops` | The agent's skill: the rules and the routines | | `docs/` | These pages, plus `design/` (the view redesign spec) | | `tests/` | The host-free suite | +| `evals/tool_selection` | The tool-selection benchmark: prompts, runner and its README | ## Tests @@ -43,9 +44,24 @@ Tests worth knowing about: |---|---| | `tests/test_view.py` | The view's four rules (one fetch shim, every call through `apiFetch`, no `--pl-*` definitions, no outside URLs), the public routes (exactly `/view`, `/b/`, `/i/` and `/sheet/`, all GET), and Copy list byte for byte | | `tests/test_docs_reference.py` | These docs: every route, tool and setting is documented, nothing removed is, defaults match, links resolve | +| `tests/test_tool_eval.py` | The tool-selection benchmark: scoring rules, every expected tool registered and none turn-only, every operator tool in two or more prompts, no operator data, a complete dry run | | `tests/test_plugin.py` | The manifest: the version matches `pyproject.toml`, it ships disabled, the skill is found, and runtime dependencies come from the host | | `tests/test_hardening.py` | Atomic writes, the sold guard, strict money, non-clobbering import | +### The tool-selection eval + +`evals/tool_selection/` scores the first tool a model picks for about 50 reseller prompts, without +running any tool, so a renamed tool, a new tool or a reworded description can be checked against a +fixed corpus. `tests/test_tool_eval.py` validates the corpus and runs the dry-run plumbing on every +CI run. A live run needs an OpenAI-compatible endpoint: + +```sh +EVAL_BASE_URL=https:///v1 EVAL_API_KEY=… EVAL_MODEL= \ + python evals/tool_selection/run_eval.py --label before +``` + +See [its README](../evals/tool_selection/README.md). + ## Checking the view by hand The view has no build step. To see a change, mount the real routers on a copy of a database and borrow the diff --git a/evals/__init__.py b/evals/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/evals/tool_selection/README.md b/evals/tool_selection/README.md new file mode 100644 index 0000000..a2a55d8 --- /dev/null +++ b/evals/tool_selection/README.md @@ -0,0 +1,49 @@ +# Tool-selection eval + +A fixed benchmark of the first tool a model picks for a reseller's request. It never runs a tool: +each prompt is an independent turn, and only the first call is scored. Mirrors +careercoach-plugin's `evals/tool_selection/`. + +- **`prompts.yaml`**: about 50 operator prompts across every capability (P&L, find, intake, price, + sale, listing, bins, packing, review, CSV, delete), plus questions that need no tool and traps + (post to eBay, set sold by hand, message a buyer, end a live listing), whose right answer is no + tool. The decisions behind the expectations are written at the top of the file. Ids, prices + and URLs are invented. +- **The surface** is every tool the plugin registers, schemas taken from registration on a + throwaway database, plus four host decoys a merchant agent's chat also has (`web_search`, + `fetch_url`, `memory_recall`, `ebay_price_check`). The plugin's own turn-only tools (scan, draft + and sale-paste) are registered too, and picking one for an operator prompt is always wrong. + +## Run it + +```sh +pip install -r requirements-dev.txt +python evals/tool_selection/run_eval.py --label smoke --dry-run # plumbing only, no network +EVAL_BASE_URL=https:///v1 EVAL_API_KEY=… EVAL_MODEL= \ + python evals/tool_selection/run_eval.py --label baseline # a real model +``` + +A live run posts each prompt to `/chat/completions` with the tools and +`tool_choice: auto`, at temperature 0. Results go to `evals/tool_selection/results/