-
Notifications
You must be signed in to change notification settings - Fork 1
feat: support OpenRouter System One inference #4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,7 +5,7 @@ mod test; | |
|
|
||
| mod types; | ||
|
|
||
| pub use types::{Client, ClientConfig, EvaluationFailure, EvaluationResult, RetryPolicy}; | ||
| pub use types::{Client, ClientConfig, EvaluationFailure, EvaluationResult, Provider, RetryPolicy}; | ||
|
|
||
| use std::time::{Duration, Instant}; | ||
|
|
||
|
|
@@ -71,8 +71,7 @@ impl Client { | |
| attempts = attempts.saturating_add(1); | ||
| match self.send_once(request).await { | ||
| Ok((response, request_id)) => { | ||
| response | ||
| .validate_for(request) | ||
| self.validate_response(&response, request) | ||
| .map_err(|error| EvaluationFailure { | ||
| error, | ||
| attempts, | ||
|
|
@@ -139,6 +138,17 @@ impl Client { | |
| .map_err(|source| Failure::Terminal(Error::Decode { source }))?; | ||
| Ok((decoded, request_id)) | ||
| } | ||
|
|
||
| fn validate_response( | ||
| &self, | ||
| response: &EvaluationResponse, | ||
| request: &EvaluationRequest, | ||
| ) -> Result<()> { | ||
| match self.config.provider { | ||
|
senamakel marked this conversation as resolved.
|
||
| Provider::TypeSafe => response.validate_for(request), | ||
| Provider::OpenRouter => response.validate_for_openrouter(request), | ||
|
senamakel marked this conversation as resolved.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Use the approved TypeSafe provider during evaluation This newly added branch preserves an OpenRouter evaluation path, allowing request data and credentials to be sent through an unapproved third-party provider. The repository rules require the TypeSafe provider and [RULE] unauthorized-provider · |
||
| } | ||
| } | ||
| } | ||
|
|
||
| impl ClientConfig { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,10 +22,44 @@ impl EvaluationResponse { | |
| /// Returns [`Error::InvalidResponse`] when answer ids or primitive types do | ||
| /// not match the request, or when a probability payload is inconsistent. | ||
| pub fn validate_for(&self, request: &EvaluationRequest) -> Result<()> { | ||
| self.validate_for_model(request, |response_model| response_model == request.model) | ||
| } | ||
|
|
||
| /// Check an `OpenRouter` System One response against its request. | ||
| /// | ||
| /// `OpenRouter` resolves bare Jev model IDs into the `typesafe/` namespace, | ||
| /// so a response can name a concrete release when the request used an | ||
| /// alias such as `jev-latest`. | ||
| /// | ||
| /// # Errors | ||
| /// | ||
| /// Returns [`Error::InvalidResponse`] when answer ids or primitive types do | ||
| /// not match the request, or when a probability payload is inconsistent. | ||
| pub fn validate_for_openrouter(&self, request: &EvaluationRequest) -> Result<()> { | ||
| let requested = request.model.trim_start_matches('~'); | ||
| let expected = if requested.contains('/') { | ||
| requested.to_owned() | ||
| } else { | ||
| format!("typesafe/{requested}") | ||
| }; | ||
| self.validate_for_model(request, |response_model| { | ||
| if matches!(requested, "jev-latest" | "typesafe/jev-latest") { | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Restrict resolved latest model identifiers For a request of [RULE] overly-permissive-validation · |
||
| response_model.starts_with("typesafe/jev-") | ||
| } else { | ||
| response_model == expected || response_model.starts_with(&format!("{expected}-")) | ||
| } | ||
| }) | ||
| } | ||
|
|
||
| fn validate_for_model( | ||
| &self, | ||
| request: &EvaluationRequest, | ||
| model_matches: impl FnOnce(&str) -> bool, | ||
| ) -> Result<()> { | ||
| if self.model.trim().is_empty() { | ||
| return Err(Error::invalid_response("response model must not be empty")); | ||
| } | ||
| if self.model != request.model { | ||
| if !model_matches(&self.model) { | ||
| return Err(Error::invalid_response( | ||
| "response model must match the requested model", | ||
| )); | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -35,7 +35,8 @@ classified `Error`, attempt count, and elapsed time. | |
| Response validation requires: | ||
|
|
||
| - exact question ids and primitive types; | ||
| - the exact requested model id; | ||
| - the exact requested model id for `TypeSafe`, or OpenRouter's resolved | ||
| `typesafe/` Jev release matching the requested Jev alias; | ||
| - finite probabilities in `[0, 1]`, with each distribution sum differing from | ||
| `1.0` by at most `0.000001`; | ||
| - Choice labels exactly matching criteria and the chosen label tying for the | ||
|
|
@@ -49,6 +50,14 @@ Remote base URLs require HTTPS, contain no credentials, query, or fragment, and | |
| automatic redirects are disabled. Plain HTTP is accepted only for literal | ||
| loopback IP addresses used by local test servers. | ||
|
|
||
| `ClientConfig::openrouter` uses OpenRouter's compatible System One base URL, | ||
| `https://openrouter.ai/api`. The first-party constructor and `Client::from_env` | ||
| retain the `TypeSafe` endpoint and `TYPESAFE_API_KEY` behavior. | ||
|
|
||
| `ClientConfig::tinyhumans_openrouter` uses Tiny Humans' OpenRouter proxy base | ||
| URL, `https://api.tinyhumans.ai/agent-integrations/openrouter`, and | ||
| accepts the key supplied explicitly to its constructor. | ||
|
|
||
| Authentication, request validation, response decoding, and non-connect | ||
| transport failures are terminal. Timeouts, connection-establishment failures, | ||
| 408, 429, 529, and server errors use the explicit retry policy. `max_retries` | ||
|
|
@@ -85,6 +94,8 @@ println!("{:?}", result.response.answers["violation"]); | |
| - Mock HTTP tests cover authentication, 408/429/529/5xx classification, timeout, | ||
| connection failure, redirects, decoding, retry exhaustion, Retry-After forms, | ||
| secret redaction, and failure metadata. | ||
| - OpenRouter configuration and resolved-Jev response validation have mock tests; | ||
| its paid live integration test is explicitly ignored by default. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When maintainers run the ignored tests to verify the production OpenRouter endpoint, this acceptance criterion cannot be satisfied: the resulting tree contains only AGENTS.md reference: AGENTS.md:L66-L68 Useful? React with 👍 / 👎. |
||
| - Every production source file has at least 90% line coverage. | ||
| - Format, clippy, build, tests, rustdoc, MSRV, cargo-deny, and coverage are green. | ||
|
|
||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.