Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ In-process token-bucket limits on the Hosted API request boundary only. **Not di

**Default enablement:** on when `MUTINY_API_TOKEN` is set; off for tokenless loopback demo. Override with `MUTINY_RATE_LIMIT_ENABLED=0|1`.

**Other env knobs:** `MUTINY_RATE_LIMIT_{HEALTH,NORMAL,EXPENSIVE,INGEST}_{RPM,BURST}`, `MUTINY_RATE_LIMIT_MAX_KEYS` (default 4096), `MUTINY_RATE_LIMIT_IDLE_SECONDS` (stale bucket cleanup). Values `≤ 0` or non-integers **fail closed** at app start (do not mean “unlimited”).
**Other env knobs:** `MUTINY_RATE_LIMIT_{HEALTH,NORMAL,EXPENSIVE,INGEST}_{RPM,BURST}`, `MUTINY_RATE_LIMIT_MAX_KEYS` (default 4096), `MUTINY_RATE_LIMIT_IDLE_SECONDS` (default 300 seconds, stale bucket cleanup). RPM, burst, and key-count values require positive integers; idle seconds accepts positive fractional values. Non-positive or unparsable values **fail closed** at app start (do not mean “unlimited”). See the [rate-limit field guide](./docs/RATE_LIMITS.md) for all defaults, identity/category buckets, and Retry-After handling.

**Identity:** valid Bearer → single-tenant key `tenant` (token never logged). Otherwise → `request.client.host` only; `X-Forwarded-For` is **not** trusted.

Expand Down
121 changes: 121 additions & 0 deletions docs/RATE_LIMITS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# Hosted rate-limit field guide

Hosted uses in-process token buckets on `/api/*` requests. **Limits are per
process, not distributed or cluster-wide.** Workers/replicas have independent
buckets, and process restarts reset them. This is a single-tenant control-plane
guard, not a durable usage quota. Local CLI/Core campaign execution is unaffected;
HTTP uploads to Hosted are subject to the ingest limits.

Implementation: [rate_limit.py](../apps/api/src/mutiny_api/rate_limit.py) and the
HTTP middleware in [app.py](../apps/api/src/mutiny_api/app.py). Authentication and
observe-only boundaries are described in [SECURITY.md](../SECURITY.md).

## Configuration

Set these variables on the **API process** before startup. Configuration is read
when the app is created; restart the API to apply changes. Setting variables only
on the web process or in a customer's `mutiny.yaml` does not configure the API.

| Environment variable | Default | Meaning |
|---|---|---|
| `MUTINY_RATE_LIMIT_ENABLED` | Auth-dependent | Enabled when `MUTINY_API_TOKEN` is non-empty after trimming; otherwise disabled for local demos. Explicit `1`/`0` overrides this. |
| `MUTINY_RATE_LIMIT_HEALTH_RPM` | 1200 | Health-category refill rate, requests per minute. |
| `MUTINY_RATE_LIMIT_HEALTH_BURST` | 120 | Health bucket capacity, requests. |
| `MUTINY_RATE_LIMIT_NORMAL_RPM` | 600 | Normal-category refill rate, requests per minute. |
| `MUTINY_RATE_LIMIT_NORMAL_BURST` | 120 | Normal bucket capacity, requests. |
| `MUTINY_RATE_LIMIT_EXPENSIVE_RPM` | 60 | Expensive-category refill rate, requests per minute. |
| `MUTINY_RATE_LIMIT_EXPENSIVE_BURST` | 15 | Expensive bucket capacity, requests. |
| `MUTINY_RATE_LIMIT_INGEST_RPM` | 300 | Ingest-category refill rate, requests per minute. |
| `MUTINY_RATE_LIMIT_INGEST_BURST` | 60 | Ingest bucket capacity, requests. |
| `MUTINY_RATE_LIMIT_MAX_KEYS` | 4096 | Maximum number of identity/category buckets held in memory per process. |
| `MUTINY_RATE_LIMIT_IDLE_SECONDS` | 300 | Idle time before a bucket is eligible for cleanup, in seconds. |

Unset, empty, or whitespace-only settings use their defaults. The enable flag
also accepts `true`, `yes`, `on`, `enabled` and `false`, `no`, `off`, `disabled`
(case-insensitive). Unrecognized flag values fail app creation. RPM, burst, and
`MAX_KEYS` must be positive integers. `IDLE_SECONDS` is parsed as a positive
number and accepts fractional seconds. Non-positive values and unparsable
numbers raise a configuration error, even when limiting is disabled; zero does
not mean unlimited. Use `MUTINY_RATE_LIMIT_ENABLED=0` to disable explicitly.
Disabling limits does not disable authentication or the non-loopback token guard.

## Categories and token buckets

| Category | Requests |
|---|---|
| Health | `/api/health` and `/api/meta`. These are public, but still rate limited when the limiter is enabled. |
| Ingest | Paths starting with `/api/ingest/v1/`. Payload-size/redaction checks still apply separately. |
| Expensive | `POST /api/campaigns`, `/api/campaigns/{id}/start`, `/api/candidates/{id}/minimize`, `/api/candidates/{id}/regression`, and `/api/tests/run`. |
| Normal | Other `/api/*` requests, including list/poll requests and opening an SSE event stream. Streamed events are not additional HTTP requests. |

Each identity has a separate bucket for each category. A new bucket starts full
at `BURST`; each request consumes one token. Tokens refill continuously at
`RPM / 60` per second, up to the burst capacity. RPM is a refill rate, not a
fixed-minute request counter. For example, RPM 60 and burst 2 allow two immediate
requests, then refill one token per second. Exhausting normal traffic does not
consume health or ingest capacity.

Cleanup happens during requests, not on a background timer. Buckets idle longer
than `IDLE_SECONDS` are removed. At `MAX_KEYS`, the least recently used bucket is
evicted to make room. A later request for a removed key creates a fresh bucket;
these bounds control memory, not persistent per-user accounting.

## Identity and authentication order

- A valid configured Bearer token uses the shared `tenant` identity. All clients
holding that single-tenant token share capacity within each category.
- Missing/invalid Bearer credentials, or requests when no token is configured,
use `ip:<request.client.host>` (`ip:unknown` when unavailable). The limiter
does not read `X-Forwarded-For`; anonymous clients behind a proxy can share
a bucket when the ASGI client address is the proxy.
- Raw tokens are not bucket keys. Separate categories still apply to both
authenticated and anonymous identities.

Request order is request context/ID, rate limiter, authentication dependency,
then handler. Protected requests with bad credentials return 401 while under
limit; exhausted requests return 429 before authentication, including requests
with a valid token. A 429 does not prove that credentials were accepted.
Health/meta remain public. Rate limits do not restore removed customer execution
or filesystem access.

## 429 and Retry-After

An exhausted bucket returns HTTP 429 with `error.code=rate_limit_exceeded` and
`Retry-After`. The rejected request does not reach the route handler. Example
response for the normal category (delay and request ID vary):

```http
HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 1
X-Request-Id: example-request-id

{"error":{"code":"rate_limit_exceeded","message":"rate limit exceeded for normal operations; retry after the indicated delay","status":429,"request_id":"example-request-id"}}
```

The server emits a **whole-second delay**, not an HTTP date. It computes the
missing tokens divided by the category refill rate, then rounds positive delays
up to at least one second. This estimates when the bucket can admit one request;
other clients sharing it may consume that capacity first.

Clients should wait at least the indicated delay before retrying, bound their
attempts/total wait, and avoid retry storms. Retry only safe/idempotent operations;
ingest retries must preserve stable campaign/event IDs and sanitized payloads.
Keep redaction enabled. A 401 or payload-validation error needs its own handling,
not an unbounded rate-limit retry loop.

A Hosted sync failure can leave a successful local campaign with CLI exit 3;
local results remain authoritative. See [Hosted ingestion](./HOSTED_INGESTION.md)
for the current CLI sync and pending-file recovery contract. This HTTP guide
does not imply every client automatically retries.

## Operator checks

`GET /api/meta` reports `safety.rate_limits` as `in_process` or `disabled` and
`safety.rate_limits_distributed` as `false`. It does not expose effective numeric
limits; inspect API process configuration for those. If health probes receive
429, check the health bucket rather than treating it as a health-response payload;
see the [health field guide](./API_HEALTH.md).

Behavioral coverage lives in [unit tests](../tests/unit/test_rate_limit.py) and
[HTTP integration tests](../tests/integration/test_api_hosted_rate_limit.py).
2 changes: 2 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ with sandboxed mock tools for demos.
| CLI flags (`init` / `run` / `test`) | [CLI.md](./CLI.md) |
| Packaging / PyPI | [PUBLISHING.md](./PUBLISHING.md) |
| Clean-machine bootstrap | [COLD_START.md](./COLD_START.md) |
| Hosted rate-limit tuning and 429 responses | [RATE_LIMITS.md](./RATE_LIMITS.md) |
| Contribute / first PR | [CONTRIBUTING.md](../CONTRIBUTING.md) · [GOOD_FIRST_ISSUES.md](./GOOD_FIRST_ISSUES.md) |
| Architecture boundaries | [ARCHITECTURE.md](./ARCHITECTURE.md) |
| How the system runs | [SYSTEM_DESIGN.md](./SYSTEM_DESIGN.md) |
Expand All @@ -48,6 +49,7 @@ with sandboxed mock tools for demos.
| [DEMO_SCRIPT.md](./DEMO_SCRIPT.md) | Live demo scripts, backup, Q&A | System internals |
| [COLD_START.md](./COLD_START.md) | Clean-machine bootstrap + smoke gate | Product narrative |
| [API_HEALTH.md](./API_HEALTH.md) | Hosted health response fields and interpretation | External model connectivity or customer adapter validation |
| [RATE_LIMITS.md](./RATE_LIMITS.md) | Hosted limiter defaults, identities, and Retry-After behavior | Distributed quotas or client retry implementations |
| [GOOD_FIRST_ISSUES.md](./GOOD_FIRST_ISSUES.md) | Catalog of small contributor tasks | Execution ownership of Core redesigns |
| [CLI.md](./CLI.md) | `mutiny` commands and flags from `--help` | Hosted UI walkthrough |
| [../CHANGELOG.md](../CHANGELOG.md) | Unreleased + release notes | Live demo narration |
Expand Down
Loading