Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,5 @@ __pycache__/
*.db
*.db-wal
*.db-shm

evals/tool_selection/results/
16 changes: 16 additions & 0 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ Contributor notes: the file layout, the tests, checking the view by hand, and re
| `skills/inventory-ops` | The agent's skill: the rules and the routines |
| `docs/` | These pages, plus `design/` (the view redesign spec) |
| `tests/` | The host-free suite |
| `evals/tool_selection` | The tool-selection benchmark: prompts, runner and its README |

## Tests

Expand All @@ -43,9 +44,24 @@ Tests worth knowing about:
|---|---|
| `tests/test_view.py` | The view's four rules (one fetch shim, every call through `apiFetch`, no `--pl-*` definitions, no outside URLs), the public routes (exactly `/view`, `/b/`, `/i/` and `/sheet/`, all GET), and Copy list byte for byte |
| `tests/test_docs_reference.py` | These docs: every route, tool and setting is documented, nothing removed is, defaults match, links resolve |
| `tests/test_tool_eval.py` | The tool-selection benchmark: scoring rules, every expected tool registered and none turn-only, every operator tool in two or more prompts, no operator data, a complete dry run |
| `tests/test_plugin.py` | The manifest: the version matches `pyproject.toml`, it ships disabled, the skill is found, and runtime dependencies come from the host |
| `tests/test_hardening.py` | Atomic writes, the sold guard, strict money, non-clobbering import |

### The tool-selection eval

`evals/tool_selection/` scores the first tool a model picks for about 50 reseller prompts, without
running any tool, so a renamed tool, a new tool or a reworded description can be checked against a
fixed corpus. `tests/test_tool_eval.py` validates the corpus and runs the dry-run plumbing on every
CI run. A live run needs an OpenAI-compatible endpoint:

```sh
EVAL_BASE_URL=https://<gateway>/v1 EVAL_API_KEY=… EVAL_MODEL=<model> \
python evals/tool_selection/run_eval.py --label before
```

See [its README](../evals/tool_selection/README.md).

## Checking the view by hand

The view has no build step. To see a change, mount the real routers on a copy of a database and borrow the
Expand Down
Empty file added evals/__init__.py
Empty file.
49 changes: 49 additions & 0 deletions evals/tool_selection/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Tool-selection eval

A fixed benchmark of the first tool a model picks for a reseller's request. It never runs a tool:
each prompt is an independent turn, and only the first call is scored. Mirrors
careercoach-plugin's `evals/tool_selection/`.

- **`prompts.yaml`**: about 50 operator prompts across every capability (P&L, find, intake, price,
sale, listing, bins, packing, review, CSV, delete), plus questions that need no tool and traps
(post to eBay, set sold by hand, message a buyer, end a live listing), whose right answer is no
tool. The decisions behind the expectations are written at the top of the file. Ids, prices
and URLs are invented.
- **The surface** is every tool the plugin registers, schemas taken from registration on a
throwaway database, plus four host decoys a merchant agent's chat also has (`web_search`,
`fetch_url`, `memory_recall`, `ebay_price_check`). The plugin's own turn-only tools (scan, draft
and sale-paste) are registered too, and picking one for an operator prompt is always wrong.

## Run it

```sh
pip install -r requirements-dev.txt
python evals/tool_selection/run_eval.py --label smoke --dry-run # plumbing only, no network
EVAL_BASE_URL=https://<gateway>/v1 EVAL_API_KEY=… EVAL_MODEL=<model> \
python evals/tool_selection/run_eval.py --label baseline # a real model
```

A live run posts each prompt to `<EVAL_BASE_URL>/chat/completions` with the tools and
`tool_choice: auto`, at temperature 0. Results go to `evals/tool_selection/results/<label>.json`
(gitignored), and a per-capability table is printed. `--config KEY=VALUE` passes plugin settings;
only the keys are written to the results. A live run stops on the first network or response error and writes no partial results; re-run it. Dry-run picks are deterministic noise for checking the
plumbing, not a measure of anything.

`tests/test_tool_eval.py` keeps the corpus honest: every expected tool exists, no prompt expects a
turn-only tool, every operator-facing tool has at least two prompts, and the dry run produces a
complete result.

## Baseline (2026-11-06)

Through the protoLabs gateway, before any description tuning:

| Model | Overall | Weakest |
|---|---|---|
| `protolabs/fast` | 43/53 (81.1%) | sale 0/5, trap 2/4 |
| `protolabs/smart` | 48/53 (90.6%) | sale 1/5, trap 3/4 |

**Finding: the turn-only `inventory_sale_candidates` pulls operator sale requests.** Both models
picked it first for "sold the X for $Y" prompts, instead of `inventory_mark_sold` or a lookup with
`inventory_list` / `inventory_get`. It's read-only, so the harm is an extra step rather than a wrong
write, but its description should steer chat to `inventory_list` and `inventory_mark_sold`. That's
the next thing to tune and re-run.
Empty file.
94 changes: 94 additions & 0 deletions evals/tool_selection/prompts.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Fixed first-tool selection benchmark for the inventory plugin. Score ONLY the first call.
# Independent turns: no conversation history and no tool is ever executed. Item ids, lots, bins,
# prices and URLs are invented. Ids follow the plugin's style (BB-…, KT-…, B01) but match no real
# inventory.
#
# Surface: every tool the plugin registers (including the turn-only scan_*, draft_*, sale_* and
# scan_pricing_done/scan_sealed_price tools, which the plugin's own turns use) plus four host
# decoys an operator chat also has: web_search, fetch_url, memory_recall and ebay_price_check
# (the ebay plugin's sold-comps tool, which the inventory-ops skill names as the first step of
# a re-price). A turn-only pick on an operator prompt is always wrong.
#
# Decisions (from skills/inventory-ops/SKILL.md, documented so the corpus stays honest):
# - A sale or price named by item NAME (no id) may start with a lookup: inventory_list (query=)
# or inventory_get are accepted as well as the action tool. inventory_sale_candidates is a
# turn-only tool and is NOT accepted, even though it would find the item.
# - "Price it from comps" with no comps in the prompt starts with ebay_price_check (rule: comps
# first). With the comps given in the prompt it is inventory_set_price. A request that names the
# item only by id (BB-014) also accepts inventory_get first: a comps search needs the item's name,
# and "keep the current targets" needs the current targets.
# - "Set it to sold" with no sale price is a trap: rule 3 says never set sold by hand and the
# sale needs a price, so the right first move is to ask (none). The same for posting to eBay,
# messaging a buyer or ending a live eBay listing: there is no tool, so none.
# - An explicit "delete X, it's a duplicate" is inventory_delete_item (the operator asked; rule
# 5 is about asking before deleting on your own initiative).
# - Bin contents are inventory_list(bin_id=…); the bins themselves are inventory_bins.
aliases: {}

prompts:
- {id: report-01, prompt: "How's the Blood Bowl lot doing? Am I up yet?", expect: [inventory_summary], capability: report}
- {id: report-02, prompt: "Give me the P&L across all my lots.", expect: [inventory_summary], capability: report}
- {id: report-03, prompt: "What's lot KT-LOT-01 worth if the rest sells at target?", expect: [inventory_summary], capability: report}
- {id: report-04, prompt: "how much have I actually made so far, net", expect: [inventory_summary], capability: report}

- {id: find-01, prompt: "What Blood Bowl stuff do I still have for sale?", expect: [inventory_list], capability: find}
- {id: find-02, prompt: "list everything still planned from the combat patrol lot CSM-CP-01", expect: [inventory_list], capability: find}
- {id: find-03, prompt: "Show me item BB-014 with its comps and sales.", expect: [inventory_get], capability: find}
- {id: find-04, prompt: "do we have any ork boyz left?", expect: [inventory_list], capability: find}
- {id: find-05, prompt: "What's the price basis on KT-HC-01?", expect: [inventory_get], capability: find}

- {id: intake-01, prompt: "bought the chaos space marine combat patrol for 105, splitting it. Start a lot for it.", expect: [inventory_upsert_lot], capability: intake}
- {id: intake-02, prompt: "New lot: Kill Team starter, paid $90 at the local store on 2026-10-30.", expect: [inventory_upsert_lot], capability: intake}
- {id: intake-03, prompt: "Add a Griff Oberwald, new on sprue, to lot BB-2026-09.", expect: [inventory_upsert_item], capability: intake}
- {id: intake-04, prompt: "Add 10 Gretchin NoS to lot ARM-01 under Warhammer 40K.", expect: [inventory_upsert_item], capability: intake}
- {id: intake-05, prompt: "BB-014 is actually assembled, not NoS. Fix the condition.", expect: [inventory_upsert_item], capability: intake}

- {id: price-01, prompt: "Price BB-014 from eBay sold comps.", expect: [inventory_get, ebay_price_check], capability: price}
- {id: price-02, prompt: "Set KT-HC-01 to target 65, low 55, high 75: eBay sold comps, 9 sold, median 65, p25 55, p75 75.", expect: [inventory_set_price], capability: price}
- {id: price-03, prompt: "what are griff oberwald minis actually selling for on ebay", expect: [ebay_price_check], capability: price}
- {id: price-04, prompt: "Record a thin re-price on BB-003: only 2 sold comps at about $40, keep the current targets and say the comps are thin.", expect: [inventory_get, inventory_set_price], capability: price}

- {id: sale-01, prompt: "sold the griff oberwald for 34 + 5 shipping on ebay", expect: [inventory_list, inventory_get, inventory_mark_sold], capability: sale}
- {id: sale-02, prompt: "Record a sale: BB-014 sold on eBay for $46, buyer paid $6.20 shipping, fees $7.10, postage cost $5.80.", expect: [inventory_mark_sold], capability: sale}
- {id: sale-03, prompt: "KT-HC-01 went to a friend for 60 cash, no fees, no shipping.", expect: [inventory_mark_sold], capability: sale}
- {id: sale-04, prompt: "sold both lootas boxes as a lot for 75 to a friend from the club", expect: [inventory_list, inventory_get, inventory_mark_sold], capability: sale}

- {id: sale-05, prompt: "BB-003 sold on facebook marketplace for 40, local pickup.", expect: [inventory_mark_sold], capability: sale}

- {id: listing-01, prompt: "I listed BB-014 on eBay at $49: https://www.ebay.com/itm/100000000001", expect: [inventory_listing], capability: listing}
- {id: listing-02, prompt: "Which of my eBay listings are still active for KT-HC-01?", expect: [inventory_get, inventory_listing], capability: listing}
- {id: listing-03, prompt: "The eBay listing for BB-003 ended without selling. Mark that listing ended.", expect: [inventory_get, inventory_listing], capability: listing}

- {id: bins-01, prompt: "Make me five new bins.", expect: [inventory_bins], capability: bins}
- {id: bins-02, prompt: "Which bins do I have and how full are they?", expect: [inventory_bins], capability: bins}
- {id: bins-03, prompt: "What's in B02?", expect: [inventory_list], capability: bins}
- {id: bins-04, prompt: "I just put BB-014 and BB-015 in B03.", expect: [inventory_move_to_bin], capability: bins}
- {id: bins-05, prompt: "Rename bin B01 to 'Kill Team boxes'.", expect: [inventory_bins], capability: bins}
- {id: bins-07, prompt: "Move KT-HC-01 to bin B05, I reshelved it.", expect: [inventory_move_to_bin], capability: bins}
- {id: bins-06, prompt: "Which unsold items aren't in any bin yet?", expect: [inventory_list], capability: bins}

- {id: pack-01, prompt: "What do I need to pack tonight?", expect: [inventory_packing], capability: pack}
- {id: pack-02, prompt: "Sale 12 is packed and shipped.", expect: [inventory_packing], capability: pack}
- {id: pack-03, prompt: "Give me the pick list with bins.", expect: [inventory_packing], capability: pack}

- {id: review-01, prompt: "What's gone stale? Listings and old prices.", expect: [inventory_stale], capability: review}
- {id: review-02, prompt: "Give me the re-price worklist, oldest evidence first, max 10.", expect: [inventory_reprice_plan], capability: review}
- {id: review-03, prompt: "Which listings have been up too long without selling?", expect: [inventory_stale], capability: review}
- {id: review-04, prompt: "Line up the items whose price comps are out of date so we can re-price them.", expect: [inventory_reprice_plan, inventory_stale], capability: review}

- {id: csv-01, prompt: "Import my spreadsheet at stock/new-haul.csv.", expect: [inventory_import_csv], capability: csv}
- {id: csv-04, prompt: "here's the sheet from the show, load inventory/show-buys.csv into the inventory", expect: [inventory_import_csv], capability: csv}
- {id: csv-02, prompt: "Export all sales to sales-2026.csv so I can do taxes.", expect: [inventory_export_csv], capability: csv}
- {id: csv-03, prompt: "Dump the items in lot BB-2026-09 to a CSV for me.", expect: [inventory_export_csv], capability: csv}

- {id: delete-01, prompt: "Delete BB-099, it's a duplicate entry I made by mistake.", expect: [inventory_delete_item], capability: delete}
- {id: delete-02, prompt: "Remove item KT-TEST-1 entirely, it was a test row.", expect: [inventory_delete_item], capability: delete}

- {id: chat-01, prompt: "What does NoS mean on a listing?", expect: none, capability: chat}
- {id: chat-02, prompt: "Any tips for photographing a sealed box for eBay?", expect: none, capability: chat}
- {id: chat-03, prompt: "In general, is it better to sell a combat patrol sealed or split it into units?", expect: none, capability: chat}

- {id: trap-01, prompt: "Post the Space Marine starter box on eBay for me at $95.", expect: none, capability: trap}
- {id: trap-02, prompt: "Just set BB-014's status to sold.", expect: none, capability: trap}
- {id: trap-03, prompt: "Message the buyer of BB-014 that it shipped today.", expect: none, capability: trap}
- {id: trap-04, prompt: "End my eBay listing for KT-HC-01.", expect: none, capability: trap}
Loading
Loading