feat(rank): ToolRanker vocabulary, Bm25Ranker, and a Jev-backed tinytools-jev crate - #17
Conversation
Add the rank module: the ToolRanker trait a harness asks through without knowing which kind of ranker answers, the RankCandidate/RankHit/RankContext vocabulary, and Bm25Ranker built on the BM25 index moved here from the tinyagents harness's discovery module so both crates rank with one arithmetic. Tool gains a defaulted family() declaration so a search index can say which pack, toolkit or server a hit came from. Co-authored-by: Medulla <medulla@tinyhumans.ai>
Retrieve-then-decide: a retriever (Bm25Ranker by default) narrows the catalogue to retrieval_k candidates, and one Jev request — a Choice over the shortlist plus a none option and a needs_tool Noul — decides among them with calibrated probabilities. Kept out of the vocabulary crate because tinyjevclient carries an HTTP transport; consumed by pinned revision as that crate asks. Co-authored-by: Medulla <medulla@tinyhumans.ai>
|
Warning Review limit reached
This review includes 14 billable files and costs up to $3.50. Or wait 27 minutes for your next included review. View limit detailsLimit details: You’ve used the included review currently available. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (14)
Comment |
Tiny Sweeper review
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e2544bc844
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![tinysweeper[bot]](https://avatars.githubusercontent.com/in/4526585?s=60&v=4){kind=small}
There was a problem hiding this comment.
Requesting changes: 2 lane(s) blocking, worst finding is high.
Fix or reply to the findings below and push. The next review clears this automatically once they are gone — you should not need to dismiss anything by hand.
$0.0753 · 1,484,203 in / 53,764 out · 223,444 cached (15%) · ladder/vectors, gpt-5.6-luna, deepseek/deepseek-v4-flash, deepseek-v4-flash · 1,070 embedded
critique: $0.0436 · 809,649 in / 36,665 out · 108,364 cached (13%) · gpt-5.6-luna, deepseek/deepseek-v4-flash, deepseek-v4-flash
security: $0.0296 · 584,339 in / 15,580 out · 36,766 cached (6%) · gpt-5.6-luna
tests: $0.0006 · 36,444 in / 19 out · 32,476 cached (89%) · deepseek/deepseek-v4-flash
description: $0.0006 · 28,157 in / 195 out · 24,190 cached (86%) · deepseek/deepseek-v4-flash
….toml,crates/tinytools-jev/READ Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `decode` function now borrows the `JevDecision` instead of taking ownership, avoiding an unnecessary clone at the call site. This aligns the function's signature with its read-only usage and improves efficiency. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The test previously asserted exact equality for a floating-point value, which can be flaky due to rounding. It now checks that the value is within a small epsilon of the expected 0.05, making the test more robust. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add tests for the Jev ranker covering input validation, short-circuiting on empty candidate lists, retrieval behavior with large catalogues, and error forwarding from the evaluator. Also extend the rank error display test to cover backend and invalid input variants. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The variable `miss_ranker` was assigned but never used, so it has been renamed to `_unused_ranker` to suppress the unused variable warning and clarify its purpose in the test. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 03649ca200
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let p = *decision.probabilities.get(&c.key)?; | ||
| (p >= floor).then(|| RankHit { | ||
| key: c.key.clone(), | ||
| score: p, | ||
| confidence: Some(p), |
There was a problem hiding this comment.
When a host evaluator returns an out-of-range value such as 1.2 or infinity, this path exposes it directly as RankHit::confidence, violating the documented 0.0..=1.0 contract and potentially defeating downstream confidence gates. Validate every probability-bearing field in JevDecision as finite and within range, returning RankError for a malformed decision rather than emitting invalid ranking metadata.
Useful? React with 👍 / 👎.
| [dependencies] | ||
| async-trait = { workspace = true } | ||
| tinytools = { path = "../tinytools", version = "0.3.0" } | ||
| tracing = { workspace = true, optional = true } |
There was a problem hiding this comment.
The new tracing dependency is never referenced anywhere in tinytools-jev, so enabling the public tracing feature only adds a dependency without changing behavior. Remove the dependency and feature until instrumentation exists, or wire and document the intended events; leaving a no-op feature misleads consumers and needlessly expands their graph.
AGENTS.md reference: AGENTS.md:L150-L158
Useful? React with 👍 / 👎.
| second sibling: a `ToolRanker` backed by TypeSafe's Jev decision model through | ||
| `tinyjevclient`, kept out of the vocabulary crate because it carries an HTTP | ||
| transport. |
There was a problem hiding this comment.
This description no longer matches the added crate: its manifest has no tinyjevclient or HTTP dependency, and JevRanker::new instead requires a host-provided JevEvaluator. Update this section to describe that boundary so consumers are not incorrectly told that selecting this crate brings in a specific client and transport.
AGENTS.md reference: AGENTS.md:L204-L205
Useful? React with 👍 / 👎.
| //! Dependency-free Jev-backed tool ranking. | ||
| //! | ||
| //! A host supplies [`JevEvaluator`], retaining ownership of transport, | ||
| //! authentication, retry, and deadline policy. |
There was a problem hiding this comment.
The new crate root stops after a four-line description and provides no runnable example showing how to implement JevEvaluator and construct JevRanker, nor a complete explanation of what remains host-owned. Add a compiled crate-level example and explicit boundary description so the primary public workflow is checked by doctests as required for every crate root.
AGENTS.md reference: AGENTS.md:L197-L201
Useful? React with 👍 / 👎.
1 participant
Summary
Adds the vocabulary for ranking a catalogue of tools against an intent, and one Jev-backed implementation, so a harness can offer on-demand tool discovery without knowing which ranker answers.
tinytools::rank(new module):ToolRankertrait,RankCandidate/RankHit/RankContext/RankError, andBm25Ranker.Bm25Index+tokenizemove here from the tinyagents harness'stool::discoverso both crates rank with one arithmetic. The vocabulary crate stays HTTP-free (its dependency-list CI is unchanged).Tool::family()(defaulted,None): the pack / toolkit / server a tool belongs to, so a search index can say where a hit came from. Purely descriptive;ToolSpecis untouched.tinytools-jev(new sibling crate):JevRankerovertinyjevclient(pinned rev). Retrieve-then-decide: the retriever (BM25 by default, injectable) narrows the catalogue toretrieval_k(20), then one Jev request — aChoiceover the shortlist plus anoneoption, and aneeds_toolNoul— decides. Hits carry Jev's probability asconfidence;rank_detailedalso returns choice confidence,needs_tool,noneprobability, tokens, latency and attempts. When the lexical retriever finds nothing and the catalogue fits one Choice (≤255), Jev sees all of it — the paraphrase case a decision model exists for. Every failure is aRankErrorthe caller falls back from; the key never reaches an error or log.Consumers: tinyhumansai/tinyagents (pluggable ranker in
tool::discover) and tinyhumansai/openhuman (Jev-rankedtool_searchon the orchestrator). Part of a three-repo change; the follow-up PRs link back here.Test plan
cargo test --workspace— 350 passed (10 newranktests, 9 newtinytools-jevtests over a loopback HTTP server: shortlist shape,noneoption, retriever miss fallback, reserved/duplicate keys, 401 →Backendwithout the key, deadline)cargo clippy --workspace --all-targets -- -D warningscleancargo doc --workspace --no-depscleanCo-authored-by: Medulla medulla@tinyhumans.ai