Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,17 @@ real API call:
TYPESAFE_API_KEY='<key>' cargo run -p tinyjevclient --example basic
```

## OpenRouter

OpenRouter supports the same System One request and response format for Jev.
Construct the client explicitly with `ClientConfig::openrouter("<key>")`.
OpenRouter resolves `jev-latest` to a concrete `typesafe/jev-*` model ID in its
response.

For a Tiny Humans API key, use `ClientConfig::tinyhumans_openrouter("<key>")`,
which explicitly targets
`https://api.tinyhumans.ai/agent-integrations/openrouter/v1/systemone`.

Remote API roots must use HTTPS; HTTP is reserved for literal loopback IPs.
Failed evaluations retain their classified error, attempt count, and elapsed
time so reliability measurements do not lose unsuccessful work.
Expand Down
6 changes: 6 additions & 0 deletions crates/tinyjevclient/src/client/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,9 @@ transport failures use the same explicit bounded retry policy because the
transport error taxonomy cannot reliably distinguish transient DNS, TLS, and
connectivity failures from permanent ones. Other request/body/redirect errors
are terminal, and automatic redirects are disabled.

`ClientConfig::openrouter` targets OpenRouter's compatible System One API at
Comment thread
senamakel marked this conversation as resolved.
`https://openrouter.ai/api/v1/systemone`.

`ClientConfig::tinyhumans_openrouter` targets the Tiny Humans OpenRouter proxy
Comment thread
senamakel marked this conversation as resolved.
at `https://api.tinyhumans.ai/agent-integrations/openrouter/v1/systemone`.
16 changes: 13 additions & 3 deletions crates/tinyjevclient/src/client/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ mod test;

mod types;

pub use types::{Client, ClientConfig, EvaluationFailure, EvaluationResult, RetryPolicy};
pub use types::{Client, ClientConfig, EvaluationFailure, EvaluationResult, Provider, RetryPolicy};

use std::time::{Duration, Instant};

Expand Down Expand Up @@ -71,8 +71,7 @@ impl Client {
attempts = attempts.saturating_add(1);
match self.send_once(request).await {
Ok((response, request_id)) => {
response
.validate_for(request)
self.validate_response(&response, request)
.map_err(|error| EvaluationFailure {
error,
attempts,
Expand Down Expand Up @@ -139,6 +138,17 @@ impl Client {
.map_err(|source| Failure::Terminal(Error::Decode { source }))?;
Ok((decoded, request_id))
}

fn validate_response(
&self,
response: &EvaluationResponse,
request: &EvaluationRequest,
) -> Result<()> {
match self.config.provider {
Comment thread
senamakel marked this conversation as resolved.
Provider::TypeSafe => response.validate_for(request),
Provider::OpenRouter => response.validate_for_openrouter(request),
Comment thread
senamakel marked this conversation as resolved.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

priority high security confident

Use the approved TypeSafe provider during evaluation

This newly added branch preserves an OpenRouter evaluation path, allowing request data and credentials to be sent through an unapproved third-party provider. The repository rules require the TypeSafe provider and TYPESAFE_API_KEY; remove this provider branch and reject non-TypeSafe configurations instead.

[RULE] unauthorized-provider ·

}
}
}

impl ClientConfig {
Expand Down
57 changes: 57 additions & 0 deletions crates/tinyjevclient/src/client/test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,54 @@ async fn sends_the_documented_endpoint_and_bearer_header() {
assert!(sent.contains("\"model\":\"jev-latest\""));
}

#[tokio::test]
async fn openrouter_uses_system_one_and_accepts_a_resolved_jev_model() {
let (base_url, requests) = server(vec![response(
200,
&success().replace("jev-latest", "typesafe/jev-1.13-20260917"),
"",
)])
.await;
let mut config = ClientConfig::openrouter("secret-test-key");
config.base_url = base_url;
config.timeout = Duration::from_secs(1);
config.retry.max_retries = 0;
let result = Client::new(config)
.unwrap()
.evaluate(&request())
.await
.unwrap();
assert_eq!(result.response.model, "typesafe/jev-1.13-20260917");
let sent = requests.lock().await.join("");
assert!(sent.starts_with("POST /v1/systemone HTTP/1.1"));
}

#[tokio::test]
async fn tinyhumans_proxy_uses_the_compatibility_system_one_path() {
let (base_url, requests) = server(vec![response(
200,
&success().replace("jev-latest", "typesafe/jev-1.13-20260917"),
"",
)])
.await;
let mut config = ClientConfig::tinyhumans_openrouter("secret-test-key");
config.base_url = base_url;
config.timeout = Duration::from_secs(1);
config.retry.max_retries = 0;
Client::new(config)
.unwrap()
.evaluate(&request())
.await
.unwrap();
assert!(
requests
.lock()
.await
.join("")
.starts_with("POST /v1/systemone HTTP/1.1")
);
}

#[tokio::test]
async fn retries_rate_limits_and_reports_attempts() {
let (base_url, requests) =
Expand Down Expand Up @@ -220,6 +268,15 @@ fn validates_every_configuration_bound_and_redacted_key_replacement() {
let mut secure = ClientConfig::new("key");
secure.base_url = "https://example.com".into();
assert!(Client::new(secure).is_ok());
let openrouter = ClientConfig::openrouter("key");
assert_eq!(openrouter.base_url, "https://openrouter.ai/api");
assert_eq!(openrouter.provider, Provider::OpenRouter);
let tinyhumans = ClientConfig::tinyhumans_openrouter("key");
assert_eq!(
tinyhumans.base_url,
"https://api.tinyhumans.ai/agent-integrations/openrouter"
);
assert_eq!(tinyhumans.provider, Provider::OpenRouter);
let mut ipv6_loopback = ClientConfig::new("key");
ipv6_loopback.base_url = "http://[::1]:8080".into();
assert!(Client::new(ipv6_loopback).is_ok());
Expand Down
41 changes: 41 additions & 0 deletions crates/tinyjevclient/src/client/types.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,16 @@ use std::{fmt, time::Duration};

use crate::{Error, EvaluationResponse};

/// System One API provider.
#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
pub enum Provider {
/// `TypeSafe`'s first-party System One API.
#[default]
TypeSafe,
/// `OpenRouter`'s compatible System One API.
OpenRouter,
}

/// Async `TypeSafe` System One client.
#[derive(Clone)]
pub struct Client {
Expand All @@ -25,6 +35,8 @@ pub struct ClientConfig {
pub(super) api_key: ApiKey,
/// API root without the versioned endpoint path.
pub base_url: String,
/// Provider-specific response validation behavior.
pub provider: Provider,
/// Total timeout for one HTTP attempt.
pub timeout: Duration,
/// Transient failure retry policy.
Expand All @@ -38,6 +50,34 @@ impl ClientConfig {
Self {
api_key: ApiKey(api_key.into()),
base_url: "https://api.typesafe.ai".to_owned(),
provider: Provider::TypeSafe,
timeout: Duration::from_secs(30),
retry: RetryPolicy::default(),
}
}

/// Create configuration for `OpenRouter`'s System One API.
#[must_use]
pub fn openrouter(api_key: impl Into<String>) -> Self {
Self {
api_key: ApiKey(api_key.into()),
base_url: "https://openrouter.ai/api".to_owned(),
provider: Provider::OpenRouter,
Comment thread
senamakel marked this conversation as resolved.
Comment thread
senamakel marked this conversation as resolved.
timeout: Duration::from_secs(30),
retry: RetryPolicy::default(),
}
}

/// Create configuration for the `TinyHumans` `OpenRouter` System One proxy.
///
/// The proxy accepts a `TinyHumans` API key and forwards typed Jev requests
/// to `OpenRouter` while applying the caller's `TinyHumans` account limits.
#[must_use]
pub fn tinyhumans_openrouter(api_key: impl Into<String>) -> Self {
Self {
api_key: ApiKey(api_key.into()),
base_url: "https://api.tinyhumans.ai/agent-integrations/openrouter".to_owned(),
provider: Provider::OpenRouter,
timeout: Duration::from_secs(30),
retry: RetryPolicy::default(),
}
Expand All @@ -56,6 +96,7 @@ impl fmt::Debug for ClientConfig {
f.debug_struct("ClientConfig")
.field("api_key", &"[REDACTED]")
.field("base_url", &self.base_url)
.field("provider", &self.provider)
.field("timeout", &self.timeout)
.field("retry", &self.retry)
.finish()
Expand Down
18 changes: 9 additions & 9 deletions crates/tinyjevclient/src/error/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,35 +20,35 @@ pub enum Error {
reason: String,
},
/// Authentication was rejected.
#[error("TypeSafe authentication failed")]
#[error("provider authentication failed")]
Authentication,
/// The provider rejected the request shape.
#[error("TypeSafe rejected the request")]
#[error("provider rejected the request")]
Unprocessable,
/// The account or endpoint rate limit was reached.
#[error("TypeSafe rate limit exceeded")]
#[error("provider rate limit exceeded")]
RateLimited,
/// The `TypeSafe` service reported temporary overload.
#[error("TypeSafe service overloaded")]
/// The provider reported temporary overload.
#[error("provider service overloaded")]
Overloaded,
/// The endpoint returned another unsuccessful status.
#[error("TypeSafe request failed with status {status}")]
#[error("provider request failed with status {status}")]
HttpStatus {
/// Returned HTTP status code.
status: u16,
},
/// The request timed out.
#[error("TypeSafe request timed out")]
#[error("provider request timed out")]
Timeout,
/// The HTTP transport failed before a response was available.
#[error("TypeSafe transport failed")]
#[error("provider transport failed")]
Transport {
/// Underlying transport failure.
#[source]
source: reqwest::Error,
},
/// The response body was not valid JSON for the declared wire shape.
#[error("TypeSafe response could not be decoded")]
#[error("provider response could not be decoded")]
Decode {
/// Underlying JSON decoding failure.
#[source]
Expand Down
4 changes: 3 additions & 1 deletion crates/tinyjevclient/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,9 @@ mod error;
mod request;
mod response;

pub use client::{Client, ClientConfig, EvaluationFailure, EvaluationResult, RetryPolicy};
pub use client::{
Client, ClientConfig, EvaluationFailure, EvaluationResult, Provider, RetryPolicy,
};
pub use error::{Error, Result};
pub use request::{Choice, EvaluationRequest, Noul, NoulCriteria, Question, Score};
pub use response::{Answer, ChoiceAnswer, EvaluationResponse, NoulAnswer, ScoreAnswer, Usage};
36 changes: 35 additions & 1 deletion crates/tinyjevclient/src/response/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,44 @@ impl EvaluationResponse {
/// Returns [`Error::InvalidResponse`] when answer ids or primitive types do
/// not match the request, or when a probability payload is inconsistent.
pub fn validate_for(&self, request: &EvaluationRequest) -> Result<()> {
self.validate_for_model(request, |response_model| response_model == request.model)
}

/// Check an `OpenRouter` System One response against its request.
///
/// `OpenRouter` resolves bare Jev model IDs into the `typesafe/` namespace,
/// so a response can name a concrete release when the request used an
/// alias such as `jev-latest`.
///
/// # Errors
///
/// Returns [`Error::InvalidResponse`] when answer ids or primitive types do
/// not match the request, or when a probability payload is inconsistent.
pub fn validate_for_openrouter(&self, request: &EvaluationRequest) -> Result<()> {
let requested = request.model.trim_start_matches('~');
let expected = if requested.contains('/') {
requested.to_owned()
} else {
format!("typesafe/{requested}")
};
self.validate_for_model(request, |response_model| {
if matches!(requested, "jev-latest" | "typesafe/jev-latest") {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

priority medium critique confident

Restrict resolved latest model identifiers

For a request of jev-latest, this accepts any response beginning with typesafe/jev-, including typesafe/jev- itself and unrelated identifiers such as typesafe/jev-malicious. The documentation says the response may name a concrete release, so the matcher should require the provider's valid resolved-model format and a nonempty valid release suffix rather than accepting the entire prefix.

[RULE] overly-permissive-validation ·

response_model.starts_with("typesafe/jev-")
} else {
response_model == expected || response_model.starts_with(&format!("{expected}-"))
}
})
}

fn validate_for_model(
&self,
request: &EvaluationRequest,
model_matches: impl FnOnce(&str) -> bool,
) -> Result<()> {
if self.model.trim().is_empty() {
return Err(Error::invalid_response("response model must not be empty"));
}
if self.model != request.model {
if !model_matches(&self.model) {
return Err(Error::invalid_response(
"response model must match the requested model",
));
Expand Down
17 changes: 17 additions & 0 deletions crates/tinyjevclient/src/response/test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,23 @@ fn rejects_empty_model_extra_ids_and_nonmaximal_choice() {
assert!(nonmaximal.validate_for(&request()).is_err());
}

#[test]
fn openrouter_accepts_resolved_jev_models_only() {
let mut resolved = response();
resolved.model = "typesafe/jev-1.13-20260917".into();
resolved.validate_for_openrouter(&request()).unwrap();

let mut namespaced_latest = request();
namespaced_latest.model = "~typesafe/jev-latest".into();
resolved
.validate_for_openrouter(&namespaced_latest)
.unwrap();

let mut unrelated = resolved;
unrelated.model = "typesafe/other-1".into();
assert!(unrelated.validate_for_openrouter(&request()).is_err());
}

#[test]
fn rejects_out_of_range_empty_and_mismatched_probability_payloads() {
let mut confidence = response();
Expand Down
13 changes: 12 additions & 1 deletion docs/specs/system-one-client.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@ classified `Error`, attempt count, and elapsed time.
Response validation requires:

- exact question ids and primitive types;
- the exact requested model id;
- the exact requested model id for `TypeSafe`, or OpenRouter's resolved
`typesafe/` Jev release matching the requested Jev alias;
- finite probabilities in `[0, 1]`, with each distribution sum differing from
`1.0` by at most `0.000001`;
- Choice labels exactly matching criteria and the chosen label tying for the
Expand All @@ -49,6 +50,14 @@ Remote base URLs require HTTPS, contain no credentials, query, or fragment, and
automatic redirects are disabled. Plain HTTP is accepted only for literal
loopback IP addresses used by local test servers.

`ClientConfig::openrouter` uses OpenRouter's compatible System One base URL,
`https://openrouter.ai/api`. The first-party constructor and `Client::from_env`
retain the `TypeSafe` endpoint and `TYPESAFE_API_KEY` behavior.

`ClientConfig::tinyhumans_openrouter` uses Tiny Humans' OpenRouter proxy base
URL, `https://api.tinyhumans.ai/agent-integrations/openrouter`, and
accepts the key supplied explicitly to its constructor.

Authentication, request validation, response decoding, and non-connect
transport failures are terminal. Timeouts, connection-establishment failures,
408, 429, 529, and server errors use the explicit retry policy. `max_retries`
Expand Down Expand Up @@ -85,6 +94,8 @@ println!("{:?}", result.response.answers["violation"]);
- Mock HTTP tests cover authentication, 408/429/529/5xx classification, timeout,
connection failure, redirects, decoding, retry exhaustion, Retry-After forms,
secret redaction, and failure metadata.
- OpenRouter configuration and resolved-Jev response validation have mock tests;
its paid live integration test is explicitly ignored by default.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restore the claimed OpenRouter live test

When maintainers run the ignored tests to verify the production OpenRouter endpoint, this acceptance criterion cannot be satisfied: the resulting tree contains only crates/tinyjevclient/tests/public_api.rs, and a repo-wide search finds no OpenRouter live test. The added mock tests replace base_url with a loopback server, so they never exercise the advertised endpoint; restore an explicitly gated live test using the approved credential source or remove this criterion and update the implementation status.

AGENTS.md reference: AGENTS.md:L66-L68

Useful? React with 👍 / 👎.

- Every production source file has at least 90% line coverage.
- Format, clippy, build, tests, rustdoc, MSRV, cargo-deny, and coverage are green.

Expand Down
Loading