feat(tls): per-Provider-Key trust for the endpoint it declares - #862
Merged
Conversation
Outbound TLS trust had no configuration surface: no CA bundle, no client certificate, and no verification override anywhere in the config file or the resource schemas. An upstream behind a private or enterprise CA answered with a generic 502 `invalid peer certificate: UnknownIssuer`, and the only way out was `SSL_CERT_FILE` — process-wide, undocumented, and not honoured by several of the outbound paths at all. Adds `upstream.tls` (`ca_file`, `client_cert_file`, `client_key_file`, `verify`) and applies it to every stack the gateway dials out on: - the workspace reqwest, through `client_builder()` — the provider bridges, all seven guardrails, MCP OpenAPI/OAuth, A2A, passthrough, JWKS/OIDC discovery, and the OTLP exporter; - rmcp's own reqwest line (MCP streamable-http); - the Realtime WebSocket, which previously ignored `SSL_CERT_FILE` too since tokio-tungstenite builds its connector over webpki roots only; - the Bedrock SDK clients (provider bridge and guardrail); - the S3 / GCS / Azure log-export stores. Redis gets its own `tls` block on the `redis` config, since the cache / rate-limit backend is issued by a different authority than the model endpoints and has to be configurable apart. Three defects surfaced while wiring this up: - `SSL_CERT_FILE` worked only by accident. reqwest's `rustls-tls-native-roots` was never declared; it arrived through `object_store`'s `aws` feature, so dropping the log-export sink would have silently stopped the gateway trusting anything the OS trusts. Now declared on the workspace dependency. - `rediss://` could not connect at all. The `redis` dependency enabled no TLS feature, so the URL was rejected at parse time with "can't connect with TLS, the feature is not enabled" — the `TlsMode` handling in sentinel mode was unreachable. - `rustls::ClientConfig::builder()` panics when several crypto providers are compiled in, which the redis TLS feature makes true. The shared config now selects the provider explicitly instead of relying on the install order in `main`. Two knobs stop at the AWS SDK, whose stable HTTP surface exposes trust roots only: a client certificate cannot be presented and verification cannot be disabled. Both are logged when set rather than silently ignored. Sentinel mode likewise cannot take custom roots, and says so. Bad TLS material fails the boot with the file path in the message, instead of becoming a transport error on every upstream call. Ref #860
E2E: an HTTPS mock upstream whose leaf is signed by a throwaway CA (not self-signed, so trusting the issuer is what has to work). Five scenarios against a freshly spawned gateway: no config gives a 502 that names the certificate, `upstream.tls.ca_file` gives 200, `verify: false` gives 200 with no CA at all, `SSL_CERT_FILE` still works, and a `ca_file` holding no certificate fails the boot rather than quietly trusting nothing. The source-level guard in `upstream_http` grows a rule per non-reqwest outbound stack, since a client built without the shared trust material works against every public provider and fails only against the private CA the setting exists for. Each rule was mutation-checked: removing the wiring at any of the five sites fails the test. That guard also had a hole. It cut each file at the first `#[cfg(test)]`, which is commonly a struct-field attribute near the top — in `aisix-provider-bedrock/src/bridge.rs` it appears at line 84, so the whole file was excused from the pre-existing bare-reqwest-client scan too. It now cuts at the top-level test module. Config coverage for the new blocks: the `upstream.tls` shape from the issue, that an omitted block still verifies, that half a client identity is rejected at boot, and that `redis.tls` is independent of it. Ref #860
`upstream.tls` covers a deployment facing one private certificate authority. This covers the other half of the issue: a deployment facing more than one, where trust has to be declared where the endpoint is — on the Provider Key that carries `api_base`. Adds `provider_key.tls` with `ca_cert` (inline PEM, additive to the deployment's roots) and `verify`. The certificate is inline rather than a file path because the endpoint is declared on the resource, not in the gateway's own config file, and the operator configuring it has no way to place a file on the gateway's filesystem. reqwest attaches trust to a client, so an override cannot apply per request — it needs its own client and therefore its own connection pool. `upstream_tls` keys a cache on the settings themselves, so every key configured the same way shares one pool, and a key with no override keeps dispatching on the bridge's own client with no cache lookup at all. Wired through every surface that dispatches on behalf of a Provider Key, not just the ones behind a bridge: the four provider bridges plus `/v1/messages`, `/v1/responses`, `count_tokens`, rerank, audio transcription and speech, the videos surface, the jobs surface including the detached batch-usage download, and the raw passthrough tunnel. `VideoTarget` and the batch-attribution task needed the override threaded through, since neither carried the Provider Key value. Deliberately NOT applied to the Vertex and Azure token minters: those talk to the identity provider, and a private CA declared for a model endpoint says nothing about that host. A malformed `ca_cert` falls back to the shared client with a logged error, never to a client that trusts less than was asked for. Tests: E2E stands up one gateway with no `upstream.tls` at all and three keys against one endpoint — the endpoint's own CA reaches it, a different CA is still rejected, and `verify: false` reaches it with no CA. The middle one is the load-bearing case: it shows a CA on one key does not leak trust to another. Unit tests pin the pooling behaviour, and a new guard test fails any passthrough surface added on the shared client. Ref #860
# Conflicts: # crates/aisix-gateway/src/upstream_tls.rs
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 45 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (17)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
upstream.tls(#861) covers a gateway facing one private certificateauthority. This is the other half of #860: a deployment facing more than one,
where trust has to be declared where the endpoint is declared — on the Provider
Key that carries
api_base.What this adds
The certificate is inline rather than a file path because the endpoint is
declared on the resource, not in the gateway's own config file, and whoever
configures it has no way to place a file on the gateway's filesystem. For a CA
that applies to every upstream,
upstream.tls.ca_fileis still the rightplace.
Connection pooling
reqwest attaches trust to a client, so an override cannot be applied per
request — it needs its own client, and therefore its own connection pool.
upstream_tlskeys a cache on the settings themselves, so:lookup at all (the ordinary path is unchanged);
private CA is one extra pool, not N.
Wired through the whole family
Every surface that dispatches on behalf of a Provider Key, not only the ones
behind a bridge: the four provider bridges, plus
/v1/messages,/v1/responses,count_tokens, rerank, audio transcription and speech, thevideos surface, the jobs surface (including the detached batch-usage download),
and the raw passthrough tunnel.
VideoTargetand the batch-attribution taskneeded the override threaded through — neither carried the Provider Key value.
Deliberately not applied to the Vertex and Azure token minters: those talk
to the identity provider, and a private CA declared for a model endpoint says
nothing about
accounts.google.comorlogin.microsoftonline.com.Failure behaviour
A malformed
ca_certfalls back to the shared client with a logged error —never to a client that trusts less than was asked for. The request then fails
exactly as it would have with no configuration, which is the honest outcome.
Compatibility
Additive and optional. A stored Provider Key without
tlsloads unchanged anddispatches on the shared client.
verifydefaults to true, including for atls: {}block.Tests
E2E stands up one gateway with no
upstream.tlsat all and three ProviderKeys pointing at the same HTTPS endpoint: one carrying the endpoint's own CA
(200), one carrying a different CA (502), one with
verify: false(200). Themiddle case is the load-bearing one — it shows a CA configured on one key does
not leak trust to another, which is exactly what a single process-wide trust
store would have done.
Unit tests pin the pooling behaviour (no override → no dedicated client; equal
overrides → one client; malformed CA → fall back). A new guard test fails any
passthrough surface added on the shared client instead of
client_for, sincethat regression is invisible — it breaks one endpoint for one key while every
other endpoint keeps working.
Follow-up: the control-plane side (
cp-admin.yamlschema, Go model and etcdprojection, dashboard form) is a paired PR against api7/AISIX-Cloud; until it
lands this is reachable through the standalone
resources_filepath and theAdmin API, not through the dashboard.
Fixes #860