From c9e93e56d5ea53244fdf4119a00849c8019d1ace Mon Sep 17 00:00:00 2001 From: Basil Zafar Date: Sun, 15 Nov 2026 07:39:47 +0530 Subject: [PATCH] docs: document Hosted rate limits and Retry-After --- SECURITY.md | 2 +- docs/RATE_LIMITS.md | 121 ++++++++++++++++++++++++++++++++++++++++++++ docs/README.md | 2 + 3 files changed, 124 insertions(+), 1 deletion(-) create mode 100644 docs/RATE_LIMITS.md diff --git a/SECURITY.md b/SECURITY.md index 46cb7c8..fd71418 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -76,7 +76,7 @@ In-process token-bucket limits on the Hosted API request boundary only. **Not di **Default enablement:** on when `MUTINY_API_TOKEN` is set; off for tokenless loopback demo. Override with `MUTINY_RATE_LIMIT_ENABLED=0|1`. -**Other env knobs:** `MUTINY_RATE_LIMIT_{HEALTH,NORMAL,EXPENSIVE,INGEST}_{RPM,BURST}`, `MUTINY_RATE_LIMIT_MAX_KEYS` (default 4096), `MUTINY_RATE_LIMIT_IDLE_SECONDS` (stale bucket cleanup). Values `≤ 0` or non-integers **fail closed** at app start (do not mean “unlimited”). +**Other env knobs:** `MUTINY_RATE_LIMIT_{HEALTH,NORMAL,EXPENSIVE,INGEST}_{RPM,BURST}`, `MUTINY_RATE_LIMIT_MAX_KEYS` (default 4096), `MUTINY_RATE_LIMIT_IDLE_SECONDS` (default 300 seconds, stale bucket cleanup). RPM, burst, and key-count values require positive integers; idle seconds accepts positive fractional values. Non-positive or unparsable values **fail closed** at app start (do not mean “unlimited”). See the [rate-limit field guide](./docs/RATE_LIMITS.md) for all defaults, identity/category buckets, and Retry-After handling. **Identity:** valid Bearer → single-tenant key `tenant` (token never logged). Otherwise → `request.client.host` only; `X-Forwarded-For` is **not** trusted. diff --git a/docs/RATE_LIMITS.md b/docs/RATE_LIMITS.md new file mode 100644 index 0000000..747e539 --- /dev/null +++ b/docs/RATE_LIMITS.md @@ -0,0 +1,121 @@ +# Hosted rate-limit field guide + +Hosted uses in-process token buckets on `/api/*` requests. **Limits are per +process, not distributed or cluster-wide.** Workers/replicas have independent +buckets, and process restarts reset them. This is a single-tenant control-plane +guard, not a durable usage quota. Local CLI/Core campaign execution is unaffected; +HTTP uploads to Hosted are subject to the ingest limits. + +Implementation: [rate_limit.py](../apps/api/src/mutiny_api/rate_limit.py) and the +HTTP middleware in [app.py](../apps/api/src/mutiny_api/app.py). Authentication and +observe-only boundaries are described in [SECURITY.md](../SECURITY.md). + +## Configuration + +Set these variables on the **API process** before startup. Configuration is read +when the app is created; restart the API to apply changes. Setting variables only +on the web process or in a customer's `mutiny.yaml` does not configure the API. + +| Environment variable | Default | Meaning | +|---|---|---| +| `MUTINY_RATE_LIMIT_ENABLED` | Auth-dependent | Enabled when `MUTINY_API_TOKEN` is non-empty after trimming; otherwise disabled for local demos. Explicit `1`/`0` overrides this. | +| `MUTINY_RATE_LIMIT_HEALTH_RPM` | 1200 | Health-category refill rate, requests per minute. | +| `MUTINY_RATE_LIMIT_HEALTH_BURST` | 120 | Health bucket capacity, requests. | +| `MUTINY_RATE_LIMIT_NORMAL_RPM` | 600 | Normal-category refill rate, requests per minute. | +| `MUTINY_RATE_LIMIT_NORMAL_BURST` | 120 | Normal bucket capacity, requests. | +| `MUTINY_RATE_LIMIT_EXPENSIVE_RPM` | 60 | Expensive-category refill rate, requests per minute. | +| `MUTINY_RATE_LIMIT_EXPENSIVE_BURST` | 15 | Expensive bucket capacity, requests. | +| `MUTINY_RATE_LIMIT_INGEST_RPM` | 300 | Ingest-category refill rate, requests per minute. | +| `MUTINY_RATE_LIMIT_INGEST_BURST` | 60 | Ingest bucket capacity, requests. | +| `MUTINY_RATE_LIMIT_MAX_KEYS` | 4096 | Maximum number of identity/category buckets held in memory per process. | +| `MUTINY_RATE_LIMIT_IDLE_SECONDS` | 300 | Idle time before a bucket is eligible for cleanup, in seconds. | + +Unset, empty, or whitespace-only settings use their defaults. The enable flag +also accepts `true`, `yes`, `on`, `enabled` and `false`, `no`, `off`, `disabled` +(case-insensitive). Unrecognized flag values fail app creation. RPM, burst, and +`MAX_KEYS` must be positive integers. `IDLE_SECONDS` is parsed as a positive +number and accepts fractional seconds. Non-positive values and unparsable +numbers raise a configuration error, even when limiting is disabled; zero does +not mean unlimited. Use `MUTINY_RATE_LIMIT_ENABLED=0` to disable explicitly. +Disabling limits does not disable authentication or the non-loopback token guard. + +## Categories and token buckets + +| Category | Requests | +|---|---| +| Health | `/api/health` and `/api/meta`. These are public, but still rate limited when the limiter is enabled. | +| Ingest | Paths starting with `/api/ingest/v1/`. Payload-size/redaction checks still apply separately. | +| Expensive | `POST /api/campaigns`, `/api/campaigns/{id}/start`, `/api/candidates/{id}/minimize`, `/api/candidates/{id}/regression`, and `/api/tests/run`. | +| Normal | Other `/api/*` requests, including list/poll requests and opening an SSE event stream. Streamed events are not additional HTTP requests. | + +Each identity has a separate bucket for each category. A new bucket starts full +at `BURST`; each request consumes one token. Tokens refill continuously at +`RPM / 60` per second, up to the burst capacity. RPM is a refill rate, not a +fixed-minute request counter. For example, RPM 60 and burst 2 allow two immediate +requests, then refill one token per second. Exhausting normal traffic does not +consume health or ingest capacity. + +Cleanup happens during requests, not on a background timer. Buckets idle longer +than `IDLE_SECONDS` are removed. At `MAX_KEYS`, the least recently used bucket is +evicted to make room. A later request for a removed key creates a fresh bucket; +these bounds control memory, not persistent per-user accounting. + +## Identity and authentication order + +- A valid configured Bearer token uses the shared `tenant` identity. All clients + holding that single-tenant token share capacity within each category. +- Missing/invalid Bearer credentials, or requests when no token is configured, + use `ip:` (`ip:unknown` when unavailable). The limiter + does not read `X-Forwarded-For`; anonymous clients behind a proxy can share + a bucket when the ASGI client address is the proxy. +- Raw tokens are not bucket keys. Separate categories still apply to both + authenticated and anonymous identities. + +Request order is request context/ID, rate limiter, authentication dependency, +then handler. Protected requests with bad credentials return 401 while under +limit; exhausted requests return 429 before authentication, including requests +with a valid token. A 429 does not prove that credentials were accepted. +Health/meta remain public. Rate limits do not restore removed customer execution +or filesystem access. + +## 429 and Retry-After + +An exhausted bucket returns HTTP 429 with `error.code=rate_limit_exceeded` and +`Retry-After`. The rejected request does not reach the route handler. Example +response for the normal category (delay and request ID vary): + +```http +HTTP/1.1 429 Too Many Requests +Content-Type: application/json +Retry-After: 1 +X-Request-Id: example-request-id + +{"error":{"code":"rate_limit_exceeded","message":"rate limit exceeded for normal operations; retry after the indicated delay","status":429,"request_id":"example-request-id"}} +``` + +The server emits a **whole-second delay**, not an HTTP date. It computes the +missing tokens divided by the category refill rate, then rounds positive delays +up to at least one second. This estimates when the bucket can admit one request; +other clients sharing it may consume that capacity first. + +Clients should wait at least the indicated delay before retrying, bound their +attempts/total wait, and avoid retry storms. Retry only safe/idempotent operations; +ingest retries must preserve stable campaign/event IDs and sanitized payloads. +Keep redaction enabled. A 401 or payload-validation error needs its own handling, +not an unbounded rate-limit retry loop. + +A Hosted sync failure can leave a successful local campaign with CLI exit 3; +local results remain authoritative. See [Hosted ingestion](./HOSTED_INGESTION.md) +for the current CLI sync and pending-file recovery contract. This HTTP guide +does not imply every client automatically retries. + +## Operator checks + +`GET /api/meta` reports `safety.rate_limits` as `in_process` or `disabled` and +`safety.rate_limits_distributed` as `false`. It does not expose effective numeric +limits; inspect API process configuration for those. If health probes receive +429, check the health bucket rather than treating it as a health-response payload; +see the [health field guide](./API_HEALTH.md). + +Behavioral coverage lives in [unit tests](../tests/unit/test_rate_limit.py) and +[HTTP integration tests](../tests/integration/test_api_hosted_rate_limit.py). diff --git a/docs/README.md b/docs/README.md index a7594ab..596d4d1 100644 --- a/docs/README.md +++ b/docs/README.md @@ -27,6 +27,7 @@ with sandboxed mock tools for demos. | CLI flags (`init` / `run` / `test`) | [CLI.md](./CLI.md) | | Packaging / PyPI | [PUBLISHING.md](./PUBLISHING.md) | | Clean-machine bootstrap | [COLD_START.md](./COLD_START.md) | +| Hosted rate-limit tuning and 429 responses | [RATE_LIMITS.md](./RATE_LIMITS.md) | | Contribute / first PR | [CONTRIBUTING.md](../CONTRIBUTING.md) · [GOOD_FIRST_ISSUES.md](./GOOD_FIRST_ISSUES.md) | | Architecture boundaries | [ARCHITECTURE.md](./ARCHITECTURE.md) | | How the system runs | [SYSTEM_DESIGN.md](./SYSTEM_DESIGN.md) | @@ -48,6 +49,7 @@ with sandboxed mock tools for demos. | [DEMO_SCRIPT.md](./DEMO_SCRIPT.md) | Live demo scripts, backup, Q&A | System internals | | [COLD_START.md](./COLD_START.md) | Clean-machine bootstrap + smoke gate | Product narrative | | [API_HEALTH.md](./API_HEALTH.md) | Hosted health response fields and interpretation | External model connectivity or customer adapter validation | +| [RATE_LIMITS.md](./RATE_LIMITS.md) | Hosted limiter defaults, identities, and Retry-After behavior | Distributed quotas or client retry implementations | | [GOOD_FIRST_ISSUES.md](./GOOD_FIRST_ISSUES.md) | Catalog of small contributor tasks | Execution ownership of Core redesigns | | [CLI.md](./CLI.md) | `mutiny` commands and flags from `--help` | Hosted UI walkthrough | | [../CHANGELOG.md](../CHANGELOG.md) | Unreleased + release notes | Live demo narration |