From 9034e85309a65b847a95b2667e1d63936fef43f7 Mon Sep 17 00:00:00 2001 From: bibi45c <70627109+bibi45c@users.noreply.github.com> Date: Mon, 9 Nov 2026 03:30:29 +0800 Subject: [PATCH] docs(docs): redraw the as-built views at v0.2.0 Brings docs/diagrams/as-built.md from ea0f9b5 (16 Oct) to v0.2.0: the console, plan limits with Redis, held-reservation recovery, the access plans and console tables, the accounting adjustments table, and the cfg-D cloud configuration. Adds a domain derivation with aggregates, class diagrams for two use cases (billing one streamed call, admitting one call), the Redis key layout and the delivery pipeline. Refs #78 --- docs/diagrams/as-built.md | 1081 ++++++++++++++++++++++++++----------- 1 file changed, 771 insertions(+), 310 deletions(-) diff --git a/docs/diagrams/as-built.md b/docs/diagrams/as-built.md index 149db61..b4cc5cc 100644 --- a/docs/diagrams/as-built.md +++ b/docs/diagrams/as-built.md @@ -1,97 +1,105 @@ # As-built architecture views -These views show FlowGate as it is built on `main`, not as the proposal planned it. -They are drawn from the code, and every diagram names the files it was checked -against, so a reviewer can compare the two. Where the build differs from the -proposal, the difference is listed under [Deviations](#7-deviations-from-the-proposal). - -- **Snapshot:** `main` at `ea0f9b5` (16 October 2026), plus the cloud runs of - 17 October from the operator's record, which is not in the repository. Earlier - versions were drawn at `31d5c4a` (14 October), `de208ff` (15 October) and - `467167c` (16 October). The measured results and the report are not published - yet, so no view shows a measured value. The views are checked again at the - freeze on 19 October, before they go into the slides (#17). -- **Scope:** S1 and S2 as built: `POST /v1/messages` and `POST /v1/responses` through - the gateway, key verification by model and protocol, cost hold and settlement, and - Prometheus metrics on every service. S3 as built: the Helm chart, KEDA autoscaling, - Prometheus, in-cluster k6, `deploy.py` and the opt-in Grafana dashboard, all run - on kind. On 17 October the Terraform was applied to AWS twice, and the chart, - Prometheus, KEDA and the AWS Load Balancer Controller ran on EKS, with k6 inside - the cluster (operator record, 17 Oct). The `deploy/aws` scripts have no recorded - run against EKS or RDS; of them, `reset_db.py` has run on kind. +These views show FlowGate as it is built at release `v0.2.0`, not as the proposal +planned it. They are drawn from the code, and every diagram names the files it was +checked against, so a reviewer can compare the two. Where the build differs from the +proposal, the difference is listed under [Deviations](#9-deviations-from-the-proposal). + +- **Snapshot:** tag `v0.2.0`, commit `1d259c1` (8 November 2026). The chart's + default images are those of `c9b5bbf`, the last code change. Earlier versions of + this page were drawn at `31d5c4a`, `de208ff`, `467167c` and `ea0f9b5` + (14–16 October). +- **Scope:** everything on `main` at the tag: + - the gateway's `POST /v1/messages` and `POST /v1/responses`, with key + verification, per-tenant plan limits, cost hold and settlement; + - the access and metering services, and the recovery of held reservations; + - the web console, with its user portal, admin area and playground; + - the Helm chart, KEDA autoscaling, Prometheus, the optional Grafana dashboard + and the Terraform for AWS; + - the CI and image pipelines. +- **Where things ran:** + - **Cloud:** the cloud days of 17 and 18 October, and the demo recording of + 19 October. Their evidence is in the release assets + [evidence-day1](https://github.com/bibi45c/gateflow/releases/tag/evidence-day1), + [evidence-day2](https://github.com/bibi45c/gateflow/releases/tag/evidence-day2) + and [evidence-day3](https://github.com/bibi45c/gateflow/releases/tag/evidence-day3). + - **kind only:** plan limits with Redis (#157) and the recovery job (#154). Both + ran their acceptance on kind on 8 November (operator record, 8 Nov; results in + the issue comments). - **Legend:** - - Solid box: built on `main` and run: locally, in CI, on kind or, in the cloud - view, on 17 October. - - Dotted box (`stroke-dasharray:2 2`): on `main`, but no run is recorded where it - is drawn. Only the cloud view uses it. - - Dashed box (`stroke-dasharray:5 5`): planned, optional, or present but unused. - - Solid arrow: the call runs; in the cloud view, it ran on 17 October. Dotted - arrow (`-.->`): it does not run yet, because it is planned or optional or, in - the cloud view, no run of it is recorded. - - Tags, in the cloud view only: - **[ran 17 Oct]** ran on or against AWS on 17 October 2026; the environment is - torn down daily (operator record, 17 Oct). On a group, the tag covers every - box inside without a tag of its own; - **[optional]** built and verified on kind; optional in the cloud; - **[kind]** run on kind; no run on EKS is recorded; - **[AWS only]** tested without AWS, but only the AWS path uses it, and no run - against EKS or RDS is recorded. + - Solid box or arrow: built and run, locally, in CI, on kind or in the cloud. + - Dashed box (`stroke-dasharray:5 5`) or dotted arrow (`-.->`): optional and off + by default, or present but not used in that view. + - Tags in the cloud view: **[EKS]** ran on EKS between 17 and 19 October; + **[kind]** run on kind, with no run on EKS; **[optional]** off by default. - The earlier diagrams in [index.html](index.html) were drawn from the plans before the code existed. Where the two disagree, this page follows the code. Contents: 1. [Logical view](#1-logical-view) -2. [Context map](#2-context-map) +2. [Domain design](#2-domain-design) 3. [Physical view: local stack](#3-physical-view-local-stack) 4. [Physical view: cloud](#4-physical-view-cloud) 5. [Persistence view](#5-persistence-view) -6. [Use case design: reserve and settle one call](#6-use-case-design-reserve-and-settle-one-call) -7. [Deviations from the proposal](#7-deviations-from-the-proposal) +6. [Use case design: bill one streamed call](#6-use-case-design-bill-one-streamed-call) +7. [Use case design: admit one call](#7-use-case-design-admit-one-call) +8. [Delivery pipeline](#8-delivery-pipeline) +9. [Deviations from the proposal](#9-deviations-from-the-proposal) ## 1. Logical view -Three business services, each with its own process, configuration and data. Coding -tools call only the gateway. The gateway calls access and metering through their -internal HTTP APIs, and metering calls access for prices. No service reads another -service's tables. +FlowGate has three business services and a web console. Each runs in its own +process, with its own configuration and data: -In the cluster, Prometheus scrapes the three services, and a KEDA ScaledObject scales -the gateway on two Prometheus signals. The chart turns autoscaling on with -`values-aws.yaml` and in the kind autoscale check; otherwise the gateway runs a -fixed replica count. An optional Grafana release shows one load and scaling -dashboard from Prometheus; the chart ships the dashboard only with -`dashboard.enabled`. Operator scripts and k6 are not shown here: they are tools for -operating and testing, not product components (see the cloud view). +- **Coding tools** call only the gateway. +- **The gateway** calls access and metering through their internal HTTP APIs. When + plan limits are on, it also keeps the shared limit state in Redis. +- **Metering** calls access for prices. +- **The console** calls the internal APIs of access and metering, the gateway's + public `/v1/messages` (for the playground, with the user's own key) and + Prometheus (for the admin overview). +- **A scheduled run of the metering binary**, `metering recover`, settles + reservations whose caller never settled them. + +No service reads another service's tables. ```mermaid flowchart TB - subgraph clients["Coding tools"] + subgraph clients["Consumers"] OC["OpenCode
Anthropic provider"] CX["Codex CLI
Responses provider"] + BR["Browser
console users and admins"] + end + + subgraph con["console :8070"] + CW["web
user portal, admin area,
playground; signed session cookie"] end subgraph gw["gateway :8080"] RL["relay
Messages and Responses:
request checks, SSE relay,
usage and outcome"] - AD["admission
verify, channel check,
reserve, settle"] + AD["admission
verify, channel check,
plan limits, reserve, settle"] + LM["ratelimit
one Lua script per call"] RL --> AD + AD --> LM end subgraph ac["access :8081"] - AAPI["httpapi
keys/verify, price-versions"] + AAPI["httpapi
keys/verify, price-versions,
tenants, keys, channels,
console login"] end subgraph me["metering :8082"] - MAPI["api
reservations, settle"] + MAPI["api
reservations, settle,
wallets, usage, adjustments"] BL["billing
max cost, charge"] PC["price cache
(in process)"] + REC["metering recover
scheduled run, same image"] MAPI --> BL MAPI --> PC end - subgraph up["Model upstreams"] + subgraph up["Producers: model channels"] MU["mock-upstream :8090"] - OL["Ollama :11434"] + OL["Ollama :11434
or vLLM behind it"] end subgraph stores["Data stores"] @@ -99,67 +107,104 @@ flowchart TB SA[("schema access")] SC[("schema accounting")] end - RD[("Redis
runs in compose,
no service uses it yet")] + RD[("Redis
plan limit state,
no durable data")] + end + + subgraph plat["Cluster platform (Helm on kind and EKS)"] + PR["Prometheus
scrapes every 10 s"] + SO["KEDA ScaledObject
two Prometheus triggers"] end OC -->|"POST /v1/messages
gateway key"| RL CX -->|"POST /v1/responses
gateway key"| RL + BR -->|"HTTP, session cookie"| CW + CW -->|"playground: POST /v1/messages
the user's own key"| RL + CW -->|"console login, tenants,
keys, channels, prices"| AAPI + CW -->|"wallets, usage,
adjustments"| MAPI + CW -.->|"admin overview, PromQL"| PR AD -->|"POST /internal/v1/keys/verify
key, model, protocol"| AAPI AD -->|"POST /internal/v1/reservations
and .../settle"| MAPI + LM -.->|"EVALSHA, when limits are on"| RD PC -->|"GET /internal/v1/price-versions/{id}"| AAPI + REC -->|"settle held reservations
older than the threshold"| SC RL -->|"same endpoint as the client
body unchanged"| MU - RL -->|"same, for the ollama channel"| OL + RL -->|"same, for the ollama channels"| OL AAPI --> SA MAPI --> SC - - subgraph plat["Cluster platform (Helm on kind and EKS; not in compose)"] - PR["Prometheus
scrapes every 10 s"] - SO["KEDA ScaledObject
two Prometheus triggers"] - GF["Grafana, optional
one load and scaling dashboard"] - end - PR -->|"GET /metrics"| gw PR -->|"GET /metrics"| ac PR -->|"GET /metrics"| me SO -->|"PromQL: active admitted,
arrival rate × D"| PR - SO -->|"scales replicas 2 to 4,
through its HPA"| gw - GF -.->|"PromQL"| PR + SO -->|"scales the gateway
through its HPA"| gw - classDef planned stroke-dasharray:5 5 - class RD,GF planned + classDef optional stroke-dasharray:5 5 + class RD,REC optional ``` | Call | Authentication | Timeout and retries | |---|---|---| | tool → gateway, `/v1/messages` or `/v1/responses` | gateway key in `x-api-key` or `Authorization: Bearer` | none on the reply; upstream connect 10 s | | gateway → access, verify | `Bearer ACCESS_INTERNAL_TOKEN`; the body names the protocol for Responses, and a missing protocol means Messages | 2 s, no retry | +| gateway → Redis, plan limits | the password in the Redis URL (`RATELIMIT_REDIS_URL`) | one bounded call per admitted call; an error or timeout refuses the call with 503. Tenants on an unlimited plan never call Redis | | gateway → metering, reserve | `Bearer METERING_INTERNAL_TOKEN` | 2 s, retried once on a timeout, 5xx or connection error | | gateway → metering, settle | `Bearer METERING_INTERNAL_TOKEN` | 2 s per attempt, up to 2 retries, with a fresh context | | metering → access, price version | `Bearer ACCESS_INTERNAL_TOKEN` | 1 s; each version is cached for the life of the process | +| `metering recover` → accounting schema | the metering database role | one run at most 360 s (`activeDeadlineSeconds`); no overlapping runs | +| console → access, metering | the same internal tokens as the gateway and metering | as in `services/console/internal/backend` | +| console → gateway, playground | the signed-in user's own gateway key | as for any tool | | gateway → upstream, Messages | none for the mock and Ollama; otherwise the channel's `credential_env` value as `x-api-key` | connect 10 s; forwards only `content-type`, `anthropic-version` and `anthropic-beta` | | gateway → upstream, Responses | the same credential as `Authorization: Bearer` | connect 10 s; forwards only `content-type` | -| Prometheus → gateway, access, metering, `GET /metrics` | none; for scrapes inside the cluster only, never routed by the ingress | scrape every 10 s, timeout 5 s | +| Prometheus → services, `GET /metrics` | none; scrapes inside the cluster only, never routed by the ingress | every 10 s, timeout 5 s | | KEDA → Prometheus, two queries | none; inside the cluster | a query is empty unless every gateway target is up and fresh; KEDA then reports an error, never a zero, and the HPA does not scale down on it | -| Grafana → Prometheus, dashboard queries | none; inside the cluster. Grafana has no Ingress; the operator opens it through a localhost port-forward and signs in with the admin password from a Secret | not set in `grafana-values.yaml`, so Grafana's defaults; POST queries with a 10 s minimum interval | Every internal call carries `X-Request-Id`, the gateway's `req_`, and every internal error uses the body in [api/errors.md](../../api/errors.md). -Checked against: `services/gateway/main.go`, -`services/gateway/internal/admission/admission.go`, `.../admission/settle.go`, -`.../internalapi/caller.go`, `.../relay/relay.go`, `.../relay/protocol.go`, -`.../relay/messages.go`, `.../relay/responses.go`, `.../relay/upstream.go`, -`.../accessclient/client.go`, `services/access/main.go`, -`services/metering/main.go`, `services/metering/internal/accessclient/accessclient.go`, -`services/access/internal/httpapi/httpapi.go`, -`deploy/helm/gateflow/templates/scaledobject.yaml`, `deploy/helm/gateflow/values.yaml`, -`deploy/helm/values-aws.yaml`, `deploy/helm/monitoring/prometheus-values.yaml`, -`deploy/helm/monitoring/grafana-values.yaml`, `deploy/helm/gateflow/templates/dashboard.yaml`, -`deploy/helm/README.md` (Autoscaling, Load and scaling dashboard). - -## 2. Context map - -Five logical contexts map to three deployed services, as proposal section 5.3 planned. +Checked against: `services/gateway/main.go`, `services/gateway/internal/config/config.go`, +`.../admission/admission.go`, `.../admission/limit.go`, `.../admission/settle.go`, +`.../ratelimit/ratelimit.go`, `.../relay/relay.go`, `.../relay/protocol.go`, +`.../accessclient/client.go`, `.../meteringclient/client.go`, +`services/access/internal/httpapi/`, `services/metering/main.go`, +`services/metering/recover.go`, `services/metering/internal/accessclient/`, +`services/console/internal/config/config.go`, `services/console/internal/backend/backend.go`, +`services/console/internal/web/playground.go`, [api/access.yaml](../../api/access.yaml), +[api/metering.yaml](../../api/metering.yaml), +`deploy/helm/gateflow/templates/scaledobject.yaml`, `.../templates/recovery.yaml`, +`.../templates/redis.yaml`, `deploy/helm/gateflow/values.yaml`. + +## 2. Domain design + +### 2.1 From the domain to the services + +The domain was analysed through the events of one coding session: an operator +registers channels and prices, an admin issues a key, the tool's calls are admitted, +held, relayed and settled, and the tenant reads its usage. Grouping those events by +the rules that change together gives three kinds of subdomain, five bounded +contexts and three services: + +| Domain events | Subdomain | Kind | Bounded context | Service | +|---|---|---|---|---| +| CallAdmitted, CallRefused (429, 503), CallRelayed, OutcomeDecided | Model access and relay | core | Routing and relay | gateway | +| CostReserved, CallSettled, ReservationRecovered, WalletAdjusted | Usage accounting | core | Accounting | metering | +| UsageQueried | Usage reporting | supporting | Usage analytics | metering | +| TenantCreated, PlanAssigned, KeyIssued, KeyRevoked, ConsoleUserSignedIn | Tenant and access | supporting | Tenant and access | access | +| ChannelRegistered, PriceVersionPublished | Channel catalogue | supporting | Channel catalogue | access | +| (no business events) | Metrics, logs, password hashing, sessions | generic | none; shared libraries in `pkg/` and off-the-shelf tools | all | + +Five contexts become three deployed services: + +- **Tenant and access** and **Channel catalogue** share the access service. They are + both configuration, change at the same slow rate, and are read together on every + `keys/verify`. +- **Usage analytics** stays with **Accounting** in metering, because it is a read + model of the accounting ledger. +- **Routing and relay** has no tables and the most traffic, so it is its own service + and the only one that autoscales. + +The proposal (section 5.3) planned this mapping; the code kept it. + +### 2.2 Context map + Each arrow points from the upstream context (U), which publishes a contract, to the downstream context (D), which uses it. The internal contracts are an *Open Host Service* with a *Published Language*: [api/access.yaml](../../api/access.yaml), @@ -168,54 +213,142 @@ Service* with a *Published Language*: [api/access.yaml](../../api/access.yaml), ```mermaid flowchart LR subgraph prod["Producers"] - OP["Platform operator
registers channels and prices
(S1: seed command)"] + OP["Platform operator
registers tenants, channels,
prices, plans (seed commands)"] + EXT["Model channels
Anthropic Messages,
OpenAI Responses"] end subgraph accessSvc["access service"] - TA["Tenant and access
Tenant, ApiKey"] - CC["Channel catalogue
Channel by protocol,
ChannelModel, PriceVersion"] + TA["Tenant and access
Tenant, Plan, ApiKey,
ConsoleUser"] + CC["Channel catalogue
Channel, ChannelModel,
PriceVersion"] end subgraph gatewaySvc["gateway service"] - RR["Routing and relay
live call, outcome"] + RR["Routing and relay
live call, plan limits,
outcome"] end subgraph meteringSvc["metering service"] - AC["Accounting
Wallet, Reservation,
LedgerEntry"] - UA["Usage analytics
(planned, UC7)"] + AC["Accounting
Wallet, Reservation,
LedgerEntry, WalletAdjustment"] + UA["Usage analytics
read model of the ledger"] end subgraph cons["Consumers"] - DEV["Developers' coding tools
OpenCode, Codex"] + DEV["Coding tools
OpenCode, Codex CLI"] + CON["Console
portal, admin, playground"] end - EXT["Model providers
Anthropic Messages,
OpenAI Responses"] - OP --> CC - TA -->|"U: OHS keys/verify
D: gateway"| RR + OP --> TA + TA -->|"U: OHS keys/verify, with plan limits
D: gateway"| RR + CC -->|"U: OHS keys/verify, channel
D: gateway"| RR CC -->|"U: OHS price-versions
D: accounting"| AC AC -->|"U: OHS reservations, settle
D: gateway"| RR EXT -->|"U: provider protocol
D: conformist"| RR - RR -->|"U: native model APIs
D: tools"| DEV - AC -.-> UA - - classDef planned stroke-dasharray:5 5 - class UA planned + RR -->|"U: native model APIs
D: tools, playground"| DEV + RR --> CON + AC --> UA + TA -->|"U: OHS tenants, keys, login
D: console"| CON + UA -->|"U: OHS usage, wallets
D: console"| CON ``` | Context | Service | Owns | Relationship | |---|---|---|---| -| Tenant and access | access | `tenants`, `api_keys`, `api_key_channels` | Upstream of the gateway through `keys/verify` | -| Channel catalogue | access | `channels`, `channel_models`, `price_versions` (immutable) | Upstream of accounting through `price-versions` | -| Routing and relay | gateway | no tables; the live call only | Downstream of access and accounting. Conformist to the provider protocols: the body goes upstream unchanged, and the wire types in `pkg/protocol/` are a shared kernel of technical types only, with no business rules | -| Accounting | metering | `wallets`, `reservations`, `ledger_entries` | Upstream of the gateway through reserve and settle | -| Usage analytics | metering (planned) | reads the ledger | Planned with UC7; no API yet | +| Tenant and access | access | `tenants`, `plans`, `api_keys`, `api_key_channels`, `console_users` | Upstream of the gateway through `keys/verify`, which returns the tenant's plan limits, and of the console | +| Channel catalogue | access | `channels`, `channel_models`, `price_versions` (immutable) | Upstream of the gateway (the channel in `keys/verify`) and of accounting (`price-versions`) | +| Routing and relay | gateway | no tables; the live call, and limit counters in Redis | Downstream of access and accounting. Conformist to the provider protocols: the body goes upstream unchanged, and the wire types in `pkg/protocol/` are a shared kernel of technical types only, with no business rules | +| Accounting | metering | `wallets`, `reservations`, `ledger_entries`, `wallet_adjustments` | Upstream of the gateway through reserve and settle | +| Usage analytics | metering | no tables of its own; reads `reservations` | Upstream of the console through `usage` and `wallets` | The `test/archtest` package fails the build when one service imports another -service's packages, so the boundaries above hold in the code as well as at runtime. +service's packages, so these boundaries hold in the code as well as at runtime. -Checked against: `services/*/internal/`, `pkg/protocol/`, `test/archtest/`, -`api/*.yaml`, `services/access/internal/seed/seed.go`. +### 2.3 Aggregates + +Each box is an aggregate root with the entities and value objects it owns. The +invariants are enforced by the database (check constraints, unique keys, triggers), +so they hold for every replica and for the recovery job. + +```mermaid +classDiagram + direction LR + class Tenant { + <> + id + name + plan_id + } + class Plan { + <> + id + rpm_limit + max_concurrent + 0 means no limit + } + class ApiKey { + <> + id + key_hmac unique + price_version_id + revoked_at + granted channels + } + class ConsoleUser { + <> + username unique + password_hash PBKDF2 + role user or admin + a user has a tenant, an admin none + } + class Channel { + <> + id + protocol + base_url + credential_env https only + model patterns + } + class PriceVersion { + <> + input_micro_per_token + output_micro_per_token + } + class Wallet { + <> + tenant_id + balance + held + balance - held >= 0 + } + class Reservation { + <> + call_id, attempt_id unique + max_cost + status reserved or settled + outcome + charged <= max_cost + one LedgerEntry when settled + } + class WalletAdjustment { + <> + adjustment_id idempotency key + amount not 0 + balance_after + } + Tenant --> Plan : on + ApiKey --> Tenant : belongs to + ApiKey --> PriceVersion : priced by + ApiKey --> Channel : may use + ConsoleUser --> Tenant : member of + Reservation ..> Wallet : holds max_cost in + WalletAdjustment ..> Wallet : changes balance of + Reservation ..> PriceVersion : copies prices of, by id only +``` + +Solid arrows are references inside one schema; dotted arrows cross the service +boundary or link by value only, with no foreign key. + +Checked against: `services/access/migrations/`, `services/metering/migrations/`, +`services/access/internal/store/queries.go`, `test/archtest/`, `api/*.yaml`, +`services/access/internal/seed/seed.go`, proposal section 5.3. ## 3. Physical view: local stack @@ -227,13 +360,15 @@ healthy, or, for the seed steps, has finished. flowchart TB subgraph host["Developer machine: Docker, ports on 127.0.0.1"] PG[("postgres:17 :5432
roles access, metering
from postgres-init.sql")] - RD[("redis:8 :6379
not used by any service yet")] + RD[("redis:8 :6379
plan limits, on by default here")] AC["access :8081"] ME["metering :8082"] - SA["seed-access
one-off: tenants, keys,
channels, pv-seed-1"] + SA["seed-access
one-off: tenants, plans, keys,
channels, pv-seed-1"] SM["seed-metering
one-off: wallets"] MU["mock-upstream :8090"] GW["gateway :8080"] + CO["console :8070"] + PM["prometheus :9090"] OLC["ollama :11434
optional, profile ollama, CPU"] end OLH["Ollama on the host :11434
host.docker.internal"] @@ -249,16 +384,20 @@ flowchart TB ME --> GW MU --> GW SM --> GW + GW --> CO GW -.->|"ollama channel
OLLAMA_BASE_URL"| OLH GW -.-> OLC classDef optional stroke-dasharray:5 5 - class OLC,OLH,RD optional + class OLC,OLH optional ``` - Each service image is built from one `deploy/docker/Dockerfile` and runs as a distroless, non-root binary. The binary checks its own `/healthz` for the health checks, because the image has no shell or curl. +- In compose the gateway enforces plan limits (`RATELIMIT_ENABLED` defaults to + `true`); `SEED_PLANS` assigns plans to tenants, and every other tenant stays on + the `unlimited` plan. - Containers stop with a 15 s grace period, longer than the services' 10 s drain. - The seed steps run on every `up` and change nothing when the data exists. Keys are printed only in the `seed-access` log. @@ -268,176 +407,154 @@ Checked against: `deploy/compose/compose.yaml`, `deploy/compose/postgres-init.sq ## 4. Physical view: cloud -*Run on 17 October 2026, torn down daily.* The Terraform (#90) was applied to -ap-southeast-1 on 17 October and destroyed the same day. A second apply that day -used m6i.2xlarge for the load-generator node. Both times the Helm chart (#106), -Prometheus and KEDA (#112, #118) and the AWS Load Balancer Controller were -installed, and k6 ran inside the cluster (operator record, 17 Oct). The measured -results and the report are not published yet, so this view shows no measured -value. Grafana (#115) is verified on kind and optional in the cloud. The deployment -script (#111) and the `deploy/aws` scripts (#120) have no recorded run on EKS. The -image pipeline (#83) and GHCR run on every push to `main`. A tag on a box or a -group gives its state (see the legend). +Terraform creates the environment for each cloud day and destroys it the same day. +The view shows the configuration of the runs on 18 October and the recording on +19 October ("cfg-D", set at install time over `values-aws.yaml`), and marks what +the repository defaults differ in. Plan limits and recovery are built and accepted +on kind, but never ran on EKS. ```mermaid flowchart LR OPR["Operator, bibi45c
coding tools, smoke checks"] subgraph ops["Operator and test tools, run from the operator's machine (not product components)"] - TFC["terraform plan, apply, destroy
deploy/terraform [ran 17 Oct]"] - HLM["helm install: chart, Prometheus, KEDA,
AWS Load Balancer Controller [ran 17 Oct]"] - DEP["deploy/helm/deploy.py
one service by digest, rollback record [kind]"] - DSH["deploy/helm/dashboard.py
installs the Grafana release [optional]"] - AWSS["deploy/aws: reset_db.py [kind],
secrets.py, leftovers.py [AWS only]"] - LT["loadtest/kube_run.py [kind],
prom_export.py, derive.py"] + TFC["terraform apply, destroy
deploy/terraform [EKS]"] + HLM["helm install: chart, Prometheus, KEDA,
AWS Load Balancer Controller [EKS]"] + DEP["deploy/helm/deploy.py
one service by digest, rollback record [EKS]"] + LT["loadtest/kube_run.py, prom_export.py,
derive.py [EKS]"] end - subgraph aws["AWS ap-southeast-1, one VPC, two AZs [ran 17 Oct]"] - subgraph pub["Public subnets"] - ALB["ALB from the chart's Ingress
plain HTTP, no TLS
inbound-cidrs: the operator's /32
/v1/messages, /v1/responses"] - NAT["one NAT gateway"] + subgraph aws["AWS ap-southeast-1, VPC 10.40.0.0/16, AZs 1a and 1b"] + subgraph pub["Public subnets 10.40.0.0/24, 10.40.1.0/24"] + ALB["ALB from the chart's Ingress
plain HTTP :80, operator CIDR only
/v1/messages, /v1/responses
IP targets, 300 s idle timeout"] + NAT["one NAT gateway, AZ 1a"] end - subgraph priv["Private subnets"] - subgraph eks["EKS 1.35, API open to operator CIDRs"] - subgraph svc["node group services: 2 × m6i.xlarge"] - GW["gateway Pods, 2 to 4
values-aws.yaml: 1 vCPU, 2 GiB, request = limit
second apply: 0.25 vCPU limit, set at deploy"] - ACM["access, metering
one Pod each"] + subgraph priv["Private subnets 10.40.2.0/24, 10.40.3.0/24"] + subgraph eks["EKS 1.35, cluster gateflow-s3; API public to the operator CIDR, private on"] + subgraph svc["node group services: 2 × m6i.xlarge, one per AZ"] + GW["gateway Pods, 2 to 4 by KEDA
cfg-D: 250m, 1 GiB"] + AM["access ×3, metering ×5
50m, 64 Mi request"] + CON["console ×1 [EKS 19 Oct]
port-forward only"] + RDS_K["gf-redis ×1 [optional, kind]
emptyDir"] + RJ["gf-metering-recover CronJob
[optional, kind]"] JOB["bootstrap and seed Jobs"] - PROM["Prometheus server, emptyDir, 6 h
kube-state-metrics; cAdvisor from
each kubelet via the API server"] + PROM["Prometheus, emptyDir, 6 h;
kube-state-metrics"] KEDA["KEDA, gateway ScaledObject"] - GRAF["Grafana, separate release [optional]
256 MiB request, 1 GiB limit, no persistent volume
dashboard ConfigMap from the chart
localhost port-forward only"] + GRAF["Grafana [optional]
port-forward only"] end - subgraph gen["node group load-generator: 1 × c6i.xlarge (m6i.2xlarge in the second apply), tainted"] - MU["mock-upstream Pod"] + subgraph gen["node group load-generator: 1 × m6i.2xlarge, tainted"] + MU["mock-upstream ×1, 1 vCPU"] K6["k6 Pod, one per run"] end end - RDS[("RDS PostgreSQL 17, db.t4g.small
Single-AZ, 5432 from service nodes
schemas access, accounting")] + RDS[("RDS PostgreSQL 17, db.t4g.small
single-AZ, 5432 from service nodes only
schemas access, accounting")] end + SM[("Secrets Manager
RDS master password")] end - GA["GitHub Actions
images.yml on push to main"] - GHCR[("GHCR
ghcr.io/bibi45c/gateflow-*
tagged by commit, public")] + GHCR[("GHCR
ghcr.io/bibi45c/gateflow-*
pinned by digest")] - OPR -.->|"HTTP :80, from the /32 only"| ALB - ALB -.->|"IP targets, /readyz"| GW - GW -->|"verify, reserve, settle"| ACM + OPR -->|"HTTP :80, from the operator CIDR"| ALB + ALB -->|"per-request balancing to Pod IPs"| GW + GW -->|"verify, reserve, settle"| AM GW -->|"mock channels"| MU + GW -.->|"plan limits, when on"| RDS_K K6 -->|"gf-gateway:8080,
inside the cluster"| GW - ACM -->|"sslmode=require"| RDS + AM -->|"sslmode=require"| RDS + RJ -.->|"settle held reservations"| RDS JOB -->|"roles, schemas, seed"| RDS + CON --> AM PROM -->|"GET /metrics"| GW - PROM -->|"GET /metrics"| ACM KEDA -->|"two triggers"| PROM KEDA -->|"scales 2 to 4"| GW - GRAF -.->|"PromQL, private datasource"| PROM + GRAF -.->|"PromQL"| PROM TFC -->|"creates, destroys"| aws ops -->|"EKS API: helm, kubectl"| eks - GA --> GHCR svc -->|"image pulls"| NAT - NAT -->|"anonymous pull"| GHCR + NAT --> GHCR + RDS -.- SM - classDef notrun stroke-dasharray:2 2 classDef optional stroke-dasharray:5 5 - class DEP,AWSS,LT notrun - class DSH,GRAF optional + class RDS_K,RJ,GRAF optional ``` -- **Runs on every push:** the image pipeline. Every push to `main` publishes four - images tagged with the commit, with OCI source and revision labels, and records - their digests in the job summary; pull requests build without pushing. The images - pull without a login (being checked on team machines in #21). -- **[ran 17 Oct], Terraform:** the modules `network`, `cluster` and `database`, - checked in CI with `terraform validate` and `terraform test` against mocked - providers, were applied twice on 17 October (operator record, 17 Oct). The view - shows the defaults in `variables.tf`; the record names one Terraform override, the - generator instance type of the second apply, and `generator_instance_type` still - defaults to c6i.xlarge. Redis is left out, so no ElastiCache is created. The - load-generator node group carries the `gateflow.io/workload=load-generator:NoSchedule` - taint. The teardown steps are in - [deploy/terraform/README.md](../../deploy/terraform/README.md#daily-teardown-bibi45c-only). -- **[ran 17 Oct], in the cluster:** from the chart, gateway, access, metering, the - bootstrap and seed Jobs, mock-upstream and the Ingress; Prometheus and KEDA as - their own releases; the k6 Pods. The ALB comes from the AWS Load Balancer - Controller, installed separately with `alb-controller-values.yaml`; Terraform - creates only its IAM role. The chart's `environment=aws` path, with the ALB - annotations and the database TLS init check, is covered by the chart tests. The - record says k6 ran inside the cluster but does not name the tool, so - `kube_run.py` keeps its [kind] tag. The record does not cover traffic through the - ALB either, so the ALB arrows stay dotted. -- **Gateway CPU:** `values-aws.yaml` gives each gateway Pod 1 vCPU and 2 GiB, as - both request and limit. In the second apply, each gateway Pod was limited to - 0.25 vCPU, passed with `--set` and an overlay that is not in the repository - (operator record, 17 Oct). See [Deviations](#7-deviations-from-the-proposal). -- **[kind], no EKS run recorded:** CI installs the chart on kind, upgrades it and - rehearses a one-service deploy and rollback with `deploy.py` - (`.github/workflows/helm.yml`). The KEDA scale-up and scale-down check and the k6 - load check are kind rehearsals run by hand, not in CI - (`rehearse.py --autoscale-check`, `--load-check`). -- **[optional], Grafana (#115):** `dashboard.enabled`, off by default, renders the - dashboard JSON as a ConfigMap. `deploy/helm/dashboard.py` checks that ConfigMap - and the Prometheus release, then installs Grafana 13.2.4 (chart 14.0.0) as a - separate release in the application namespace. Grafana has no Ingress, no - Kubernetes RBAC objects (`rbac.create: false`) and does not mount an API token - (`automountServiceAccountToken: false`). It has no persistent volume and keeps - its data in an `emptyDir`. It requests 256 MiB of memory with a 1 GiB limit, - reads Prometheus through a private in-cluster datasource, and is opened through - a localhost port-forward. The dashboard is described in - [deploy/helm/README.md](../../deploy/helm/README.md#load-and-scaling-dashboard-74). - It was verified on kind on 16 October - ([screenshot](../../deploy/helm/evidence/dashboard-kind-20261016.jpg)); the - 17 October record does not list it. -- **Pod placement:** gateway, access, metering, the bootstrap and seed Jobs, - Prometheus, kube-state-metrics, KEDA and Grafana select - `gateflow.io/workload=services`. mock-upstream and k6 select - `gateflow.io/workload=load-generator` and tolerate its taint, so they share the one - generator node. -- **Scaling:** with `values-aws.yaml`, KEDA's HPA owns the gateway's replica count, - between 2 and 4. T is 70 and D is 20 s, still marked provisional until the - single-Pod calibration (#73) is published. access and metering stay at one replica. -- **Ingress limits:** the chart refuses to render for AWS without +| Namespace (release) | Workload | Replicas | CPU / memory | Node group | +|---|---|---|---|---| +| `gateflow` (`gf`) | gateway | 2–4, KEDA; T = 575 per Pod, D = 21.0072 s, scale-down window 60 s | 250m / 1 GiB, request = limit | services | +| | access | 3 | 50m / 64 Mi request, 256 Mi limit | services | +| | metering | 5 | 50m / 64 Mi request, 256 Mi limit | services | +| | console | 1 (19 October) | 20m / 32 Mi request, 128 Mi limit | services | +| | gf-redis [optional] | 1 | 50m / 64 Mi request, 128 Mi limit | services | +| | gf-metering-recover [optional] | CronJob, default every 5 min, threshold 30 min | 10m / 32 Mi request, 64 Mi limit | services | +| | mock-upstream | 1 | 1 vCPU / 1 GiB request, 2 GiB limit | load-generator | +| | k6 Job | one per run | 4 CPU / 16 GiB | load-generator | +| `monitoring` (`prometheus`) | Prometheus server, kube-state-metrics | 1 each | 100m / 256 Mi (server) | services | +| `keda` (`keda`) | KEDA operator, metrics API server | 1 each | chart default | services | +| `kube-system` | AWS Load Balancer Controller, CoreDNS, VPC CNI, kube-proxy | chart or EKS default | — | services; CNI and kube-proxy on every node | + +- **Repository defaults against cfg-D.** `values-aws.yaml` gives each gateway Pod + 1 vCPU and 2 GiB, with the provisional T = 70 and D = 20 s. The chart default for + access and metering is one replica each. cfg-D (250m gateway Pods, access 3, + metering 5, T 575, D 21.0072 s from the single-Pod calibration) was set at + install time; its values are recorded in the evidence bundles. +- **Optional parts.** `ratelimit.enabled` adds `gf-redis` and turns limits on in + the gateway. `recovery.enabled` adds the CronJob. Both are off by default, so an + install without them behaves as on the cloud days. +- **Probes.** Readiness, liveness and startup probes are now set from + `probes.*`, with a 1 s timeout by default, the same as in the cloud runs. +- **Graceful shutdown.** A gateway Pod has an 85 s grace period: a 5 s preStop, a + 10 s drain, up to 60 s for open streams and a 10 s margin. +- **Pod placement.** Every product Pod selects `gateflow.io/workload=services`; + mock-upstream and k6 select `gateflow.io/workload=load-generator` and tolerate + its taint. +- **Ingress limits.** The chart refuses to render for AWS without `ingress.inboundCidrs`, and refuses any `/0` unless `ingress.allowAnySource=true`. -- **[AWS only], `deploy/aws`:** `secrets.py` and `leftovers.py inventory` have unit - tests, and no run against AWS is recorded. `leftovers.py check` has run once, - read-only, against an empty AWS account. `reset_db.py` is [kind]: it has run on - kind, and no run against RDS is recorded. -- **Operator only:** bibi45c runs every operator tool from an allowed IP. GitHub - holds no AWS credentials, and no workflow deploys to AWS. On EKS the smoke suite - reads the ledger through a short-lived query Pod on a services node - (`--smoke-db-secret`), because there is no database Pod. - -Checked against: `deploy/terraform/` (`main.tf`, `variables.tf`, `modules/*`, `README.md`), -`deploy/helm/gateflow/` (`values.yaml`, `templates/deployments.yaml`, -`templates/ingress.yaml`, `templates/scaledobject.yaml`, `templates/dashboard.yaml`), -`deploy/helm/values-aws.yaml`, `deploy/helm/alb-controller-values.yaml`, -`deploy/helm/monitoring/` (including `grafana-values.yaml`, `grafana-version.json`), -`deploy/helm/README.md`, `deploy/helm/deploy.py`, `deploy/helm/dashboard.py`, -`deploy/aws/`, `loadtest/kube_run.py`, `loadtest/README.md`, -`.github/workflows/images.yml`, `.github/workflows/helm.yml`, -`deploy/docker/README.md`, #21, and the operator record of 17 October, which is not -in the repository. + `/metrics`, `/healthz` and `/readyz` return 404 through the ALB. +- **Operator only.** bibi45c runs every operator tool from an allowed IP. GitHub + holds no AWS credentials, and no workflow deploys to AWS. + +Checked against: `deploy/terraform/` (`main.tf`, `variables.tf`, `terraform.tfvars`, +`modules/*`), `deploy/helm/gateflow/` (`values.yaml`, `templates/*.yaml`), +`deploy/helm/values-aws.yaml`, `deploy/helm/monitoring/`, `deploy/helm/README.md`, +`deploy/helm/deploy.py`, `loadtest/kube_run.py`, and the evidence bundles of 17 and +18 October (`day*-config/`: `helm-list.txt`, `nodes.json`, `helm-values-gf.json`). ## 5. Persistence view -One PostgreSQL database, `gateflow`, holds two schemas. Each schema is owned by its -service's own login role, and every schema, `public` included, is closed to every -other role, so the access role cannot read accounting tables and the reverse. Each -service applies its own migrations at startup through `pkg/pgmigrate`. There are no -cross-schema foreign keys or joins: `tenant_id` matches between the schemas by value -only. +One PostgreSQL database, `gateflow`, holds two schemas: -### Schema access (role access) +- **Ownership.** Each schema is owned by its service's own login role. +- **Isolation.** Every schema, `public` included, is closed to every other role. The + access role cannot read accounting tables, and the reverse. +- **Migrations.** Each service applies its own migrations at startup through + `pkg/pgmigrate`. Every migration only adds tables or defaulted columns, so the + previous image still starts on the new schema. +- **No cross-schema links.** There are no cross-schema foreign keys or joins; + `tenant_id` matches between the schemas by value only. + +Redis holds only the plan-limit counters, which may be lost (section 5.3). + +### 5.1 Schema access (role access): 8 tables ```mermaid erDiagram + plans ||--o{ tenants : "limits" tenants ||--o{ api_keys : "owns" + tenants ||--o{ console_users : "has users" price_versions ||--o{ api_keys : "prices" api_keys ||--o{ api_key_channels : "may use" channels ||--o{ api_key_channels : "granted to" channels ||--o{ channel_models : "serves" + plans { + text id PK "unlimited, free, standard, premium" + integer rpm_limit "per sliding minute, 0 = none" + integer max_concurrent "calls in flight, 0 = none" + timestamptz created_at + } tenants { - text id PK "tenant-00 to tenant-10" + text id PK text name + text plan_id FK "default unlimited" timestamptz created_at } api_keys { @@ -445,6 +562,8 @@ erDiagram text tenant_id FK bytea key_hmac UK "HMAC-SHA256 with the pepper, 32 bytes" text price_version_id FK + text name "console label" + text key_prefix "shown to tell keys apart" timestamptz revoked_at "null unless revoked" timestamptz created_at } @@ -455,8 +574,8 @@ erDiagram timestamptz created_at "a trigger refuses update and delete" } api_key_channels { - text key_id PK - text channel_id PK + text key_id PK, FK + text channel_id PK, FK } channels { text id PK "mock, mock-responses, ollama, ollama-responses" @@ -466,22 +585,28 @@ erDiagram timestamptz created_at } channel_models { - text channel_id PK + text channel_id PK, FK text model_pattern PK "exact name or prefix*" } + console_users { + text id PK + text username UK + text password_hash "pbkdf2-sha256$iterations$salt$hash" + text role "user or admin" + text tenant_id FK "set for a user, null for an admin" + timestamptz created_at + } ``` -- The raw key is never stored; a lookup hashes the presented key and uses the unique - index on `key_hmac`. -- A credential is only allowed over `https`: a check constraint refuses a +- The raw key is never stored. A lookup hashes the presented key and uses the unique + index on `key_hmac`. Revocation takes effect on the next call, because the gateway + keeps no key cache. +- A credential is allowed only over `https`: a check constraint refuses a `credential_env` on an `http` channel. -- Migration `0002` widened the protocol check to `anthropic-messages` and - `openai-responses`, without changing a row. The seed gives each upstream a Messages - channel and a Responses twin (`mock-responses`, `ollama-responses`) that serve the - same models, and verify picks among the key's channels that speak the requested - protocol. +- `keys/verify` joins the key, its tenant and the tenant's plan, so the gateway + learns the limits with the key check and needs no extra call. -### Schema accounting (role metering) +### 5.2 Schema accounting (role metering): 4 tables ```mermaid erDiagram @@ -489,16 +614,17 @@ erDiagram reservations ||--o| ledger_entries : "settles into" wallets { text tenant_id PK "same value as access tenants id" - bigint balance "100000000 seeded, tenant-00 has 0" + bigint balance bigint held "open holds, balance minus held never below 0" timestamptz updated_at } reservations { text reservation_id PK text call_id UK "unique with attempt_id" - integer attempt_id "1 in S1" + integer attempt_id "always 1: no replay" text tenant_id FK text price_version_id + text model "named at reserve, for usage" bigint input_price "copied at reserve" bigint output_price "copied at reserve" bigint max_input_tokens "request body bytes" @@ -510,7 +636,7 @@ erDiagram bigint output_tokens "null until settled, null if unknown" text outcome "null until settled" bigint charged "never above max_cost" - bytea settle_digest "replay check" + bytea settle_digest "replay check; a fixed value marks a recovery" timestamptz settled_at timestamptz created_at } @@ -526,26 +652,172 @@ erDiagram text outcome timestamptz created_at "a trigger refuses update and delete" } + wallet_adjustments { + text adjustment_id PK "the caller's idempotency key" + text tenant_id "no foreign key, survives a wallet reset" + bigint amount "top-up above 0, correction below 0" + text note + bigint balance_after + timestamptz created_at "a trigger refuses update and delete" + } ``` - Check constraints keep a `reserved` row free of settlement values, and require a `settled` row to have an outcome, a charge and a time. -- `ledger_entries` is append-only: a correction is a new entry. +- `ledger_entries` and `wallet_adjustments` are append-only: a correction is a new + row. +- Index `reservations (tenant_id, created_at DESC)` serves the console's usage + pages. +- **Concurrency.** Reserve updates the wallet row in the same transaction that + inserts the reservation, so two reserves on one wallet serialise on that row. + Settle and recovery lock the reservation row (`SELECT … FOR UPDATE`) before they + write the ledger entry and release the hold. + +### 5.3 Redis: plan limit state + +```mermaid +flowchart LR + subgraph tenant["Two keys per limited tenant"] + K1[("gateflow:rl:{tenant}:rpm
sorted set
member: request id
score: start time, ms
expires 60 s after the last call")] + K2[("gateflow:rl:{tenant}:concurrent
sorted set
member: request id
score: lease expiry, ms
expires one lease TTL after the last call")] + end + ACQ["acquire (one Lua script, Redis server clock)
1. drop rpm members older than 60 s; refuse if count ≥ rpm_limit
2. drop expired leases; refuse if count ≥ max_concurrent
3. only if both admit: add the lease, then the start"] + REL["release, when the call reports
ZREM the lease"] + ACQ --> K1 + ACQ --> K2 + REL --> K2 +``` -Checked against: `services/access/migrations/0001_init.sql`, `.../0002_channel_protocols.sql`, -`services/access/internal/seed/seed.go`, `services/access/internal/store/queries.go`, -`services/metering/migrations/0001_init.sql`, `deploy/compose/postgres-init.sql`, -`pkg/pgmigrate/pgmigrate.go`, `services/metering/main.go`. +- The hash tag `{tenant}` keeps both keys of a tenant in one slot of a Redis cluster. +- The script runs atomically on the Redis server's clock, so replicas never disagree + on time and never admit past a limit together. +- A lease whose gateway died is dropped when it expires: `RATELIMIT_LEASE_TTL`, + 10 minutes by default. +- The state is not durable. The chart's `gf-redis` keeps it in an `emptyDir`, so a + restart starts every window and lease from zero. -## 6. Use case design: reserve and settle one call +Checked against: `services/access/migrations/0001_init.sql` to `0004_plans.sql`, +`services/metering/migrations/0001_init.sql`, `0002_console.sql`, +`services/access/internal/store/queries.go`, +`services/metering/internal/store/reservations.go`, `.../store/recover.go`, +`services/gateway/internal/ratelimit/ratelimit.go`, `deploy/compose/postgres-init.sql`, +`deploy/helm/gateflow/files/bootstrap.sql`, `deploy/helm/gateflow/templates/redis.yaml`, +`pkg/pgmigrate/pgmigrate.go`. -The representative use case is **UC6, reserve and settle simulated usage**, as it -runs inside **UC4**, one `POST /v1/messages` call. The `/v1/responses` call follows -the same steps; its differences are listed below the sequence. It crosses all three services and -both schemas, and it carries most of the S1 rules: the order of checks, the -idempotent internal calls and the charge rule. +## 6. Use case design: bill one streamed call + +The first representative use case is **UC6, reserve and settle usage**, as it runs +inside **UC4**, one streamed `POST /v1/messages` call. The `/v1/responses` call +follows the same steps; its differences are listed below the sequence. It crosses all +three services and both schemas. It carries most of the billing rules: the +idempotent internal calls, the outcome rule, the charge rule and recovery. + +### 6.1 Classes + +```mermaid +classDiagram + direction LR + class handler { + <> + ServeHTTP() + } + class protocol { + <> + checkBody() + forwardedHeaders() + setCredential() + errorBody() + newObserver() + } + class messages { + <> + } + class responses { + <> + } + class observer { + <> + event() + body() + counts() + } + class Admission_port { + <> + Admit(Call) Admitted + } + class Admitted { + <> + Channel + Report(Result) + } + class Admission { + <> + Admit() + verify() + limit() + reserve() + settle() + } + class meteringclient_Client { + <> + Reserve() + Settle() + } + class api_Handler { + <> + reserve() + settle() + } + class Store { + <> + FindReservation() + Reserve() + Settle() + } + class Reservations { + <> + Reserve() + Settle() + StaleReservations() + Recover() + } + class billing { + <> + MaxCost(Prices, Bounds) + Charge(Prices, Bounds, maxCost, Usage, Outcome) + } + handler --> protocol + protocol <|.. messages + protocol <|.. responses + protocol ..> observer : creates + handler --> Admission_port + Admission_port <|.. Admission + Admission_port ..> Admitted : returns + Admission --> meteringclient_Client + meteringclient_Client ..> api_Handler : HTTP + api_Handler --> Store + Store <|.. Reservations + api_Handler --> billing +``` -### Sequence +Design patterns, as they appear in the code: + +- **Strategy:** `relay.protocol`, one implementation per wire protocol (`messages.go`, + `responses.go`). The size check, admission, event cap, failure handling and outcome + rule are shared. +- **Ports and adapters:** the relay depends only on the `relay.Admission` interface, + which `admission.Admission` implements. Likewise, the metering API depends on the + `api.Store` and `api.PriceSource` interfaces, implemented by the PostgreSQL store + and the access client. Tests swap in stand-ins. +- **Anti-corruption client:** each caller writes its own client of another service's + contract (`gateway/internal/accessclient`, `.../meteringclient`, + `metering/internal/accessclient`), so no service imports another's types. +- **Idempotent receiver:** reserve and settle return the recorded result for a replay + with the same content, and refuse different content with 422. +- **Callback:** `Admitted.Report` is called exactly once when the relay ends; it + frees the concurrency lease and settles. + +### 6.2 Sequence ```mermaid sequenceDiagram @@ -553,17 +825,18 @@ sequenceDiagram participant T as Coding tool participant G as gateway participant A as access + participant R as Redis participant M as metering participant DB as accounting schema participant U as Upstream - T->>G: POST /v1/messages, gateway key - Note over G: Checks in order: body at most 2 MiB (413),
key present (401), JSON, model,
max_tokens 1 to 32768 (400) + T->>G: POST /v1/messages, gateway key, stream true + Note over G: Request checks (section 7) G->>A: POST /internal/v1/keys/verify {key, model} - A-->>G: 200 tenant_id, key_id, price_version_id, channel - Note over G: Channel check: base_url form,
credential variable set (else 500, no hold) - G->>M: POST /internal/v1/reservations
call_id = request id, attempt_id 1,
max_input_tokens = body bytes, max_output_tokens - M->>DB: find (call_id, attempt_id): a replay returns the record + A-->>G: 200 tenant, key, price version, channel, plan limits + G->>R: acquire (only for a limited plan) + R-->>G: admitted + G->>M: POST /internal/v1/reservations
call_id = request id, attempt_id 1, model,
max_input_tokens = body bytes, max_output_tokens M->>A: GET /internal/v1/price-versions/{id} (cached) A-->>M: input and output price M->>DB: one transaction: insert reservation,
held = held + max_cost if balance - held covers it @@ -572,25 +845,16 @@ sequenceDiagram U-->>G: SSE events G-->>T: each event relayed as it arrives Note over G: Usage from message_start and message_delta.
Outcome by the first rule that matches. + G->>R: release the lease G->>M: POST /internal/v1/reservations/{id}/settle
{input_tokens, output_tokens, outcome}, fresh context M->>DB: one transaction: lock the reservation,
insert one ledger entry, mark it settled,
held = held - max_cost, balance = balance - charged M-->>G: 200 charged, ledger_entry_id ``` -Refusals end the call before the upstream is reached, and leave no ledger entry: - -| Where | Internal answer | What the tool gets | -|---|---|---| -| verify | 401 `invalid_key` | 401 `authentication_error` | -| verify | 403 `model_not_allowed` | 403 `permission_error` | -| reserve | 409 `insufficient_balance`, 404 `wallet_not_found` | 402 `billing_error` | -| verify or reserve | timeout, unreachable, `dependency_unavailable` | 503 `api_error` | -| anything else | any other code | 500 `api_error`, logged | - -### The same flow on `/v1/responses` +### 6.3 The same flow on `/v1/responses` -`POST /v1/responses` (S2, #65) runs the same admission, reserve, relay and settle -steps; only the wire format differs: +`POST /v1/responses` runs the same admission, relay and settle steps; only the wire +format differs: | Step | `/v1/messages` | `/v1/responses` | |---|---|---| @@ -600,22 +864,24 @@ steps; only the wire format differs: | Usage source | `message_start` and `message_delta`, cache tokens added to input | `response.completed` usage; cached input tokens are part of `input_tokens` | | Public errors | Anthropic error body | OpenAI error body | -### Reservation states +### 6.4 Reservation states ```mermaid stateDiagram-v2 [*] --> reserved: reserve, hold max_cost - reserved --> settled: settle, charge and release the rest reserved --> reserved: reserve replay, same content - settled --> settled: settle repeat, same content - reserved --> [*]: never settled, hold stays until UC6 recovery + reserved --> settled: settle by the gateway,
charge and release the rest + reserved --> settled: metering recover, after the threshold,
outcome interrupted, charge max_cost + settled --> settled: settle replay, same content note right of settled A replay with different content gets - 422 idempotency_conflict + 422 idempotency_conflict. A late settle + after a recovery gets a conflict too, + because the recovery records its own digest. end note ``` -### Charge rule +### 6.5 Charge rule `max_cost = max_input_tokens × input price + max_output_tokens × output price`. At settlement, an unknown count is replaced by its maximum, and @@ -630,37 +896,232 @@ micro-units per token), a call with 6,328 input and 449 output tokens is charged | `completed` | `message_stop` arrived, or a full 2xx body was read | counts as reported | | `client_cancelled` | the client disconnected first | unknown counts at their maximum | | `upstream_error` | the upstream answered non-2xx before any output | 0 | -| `interrupted` | anything else, such as a dropped stream | unknown counts at their maximum | +| `interrupted` | anything else, such as a dropped stream; also every recovery | unknown counts at their maximum | + +**Recovery** (#154, #155). A gateway that dies before it settles leaves the hold in +place. `metering recover` lists reservations still `reserved` after `--older-than` +(chart default 30 minutes, every 5 minutes), oldest first, at most `--limit`. It +settles each one through the same transaction as a settle request, as `interrupted` +with unknown counts, so it charges `max_cost`. It logs one JSON line per reservation +and fails the run if any recovery fails. + +Checked against: `services/gateway/internal/relay/relay.go`, `.../relay/protocol.go`, +`.../relay/messages.go`, `.../relay/responses.go`, `.../relay/result.go`, +`services/gateway/internal/admission/`, `services/gateway/internal/meteringclient/client.go`, +`services/metering/internal/api/api.go`, `services/metering/internal/billing/billing.go`, +`services/metering/internal/store/reservations.go`, `.../store/recover.go`, +`services/metering/recover.go`, [api/metering.yaml](../../api/metering.yaml). + +## 7. Use case design: admit one call + +The second representative use case is **admission**: every call is checked in the +order verify → channel check → plan limits → reserve, and only a call with a +confirmed reservation reaches the upstream. A refused call leaves no reservation and +no ledger entry. It carries the multi-tenant rules: key and model permission, fair +use by plan (#157), and the wallet as the spending limit. + +### 7.1 Classes + +```mermaid +classDiagram + direction LR + class Admission { + <> + -access accessclient.Client + -metering meteringclient.Client + -limiter Limiter + +Admit(ctx, Call) Admitted, PublicError + -verify() + -limit() + -reserve() + } + class Limiter { + <> + Acquire(ctx, tenant, requestID, Limits) release, Refusal, error + } + class ratelimit_Limiter { + <> + Acquire() + -release() + } + class Limits { + <> + RPM + MaxConcurrent + Unlimited() bool + } + class Refusal { + <> + Reason rpm or concurrency + RetryAfter + } + class accessclient_Client { + <> + Verify(ctx, requestID, key, model, protocol) Verified + } + class Verified { + <> + TenantID + KeyID + PriceVersionID + Channel + Limits plan_id, rpm, max_concurrent + } + class PublicError { + <> + Status + Type + Message + RetryAfter + } + Admission --> Limiter + Limiter <|.. ratelimit_Limiter + ratelimit_Limiter ..> Limits + ratelimit_Limiter ..> Refusal : returns + Admission --> accessclient_Client + accessclient_Client ..> Verified : returns + Admission ..> PublicError : returns on refusal +``` + +The limiter is optional: with `RATELIMIT_ENABLED=false` the gateway has no +`Limiter`, and `limit()` admits without asking, which is the behaviour before #157. + +### 7.2 Sequence, with the refusal paths + +```mermaid +sequenceDiagram + autonumber + participant T as Coding tool + participant G as gateway + participant A as access + participant R as Redis + participant M as metering + + T->>G: POST /v1/messages or /v1/responses + Note over G: Request checks: body at most 2 MiB (413),
key present (401), JSON, model,
output bound 1 to 32768 (400) + G->>A: keys/verify {key, model, protocol} + alt key invalid or revoked, or model not allowed + A-->>G: 401 invalid_key or 403 model_not_allowed + G-->>T: 401 authentication_error or 403 permission_error + else verified + A-->>G: 200 tenant, channel, plan limits + Note over G: Channel check: base_url form, credential
variable set (else 500, no hold) + alt plan is unlimited + Note over G: no Redis call + else plan has limits + G->>R: acquire(tenant, request id, rpm, max_concurrent) + alt Redis error or timeout + G-->>T: 503 api_error (fail closed) + else over a limit + R-->>G: refused, retry after N ms + G-->>T: 429 rate_limit_error, retry-after + else admitted + R-->>G: admitted, lease recorded + end + end + G->>M: reserve + alt insufficient balance or no wallet + M-->>G: 409 insufficient_balance, 404 wallet_not_found + G->>R: release the lease + G-->>T: 402 billing_error + else held + M-->>G: 201 reservation + Note over G: relay as in section 6.2 + end + end +``` + +| Where | Internal answer | What the tool gets | +|---|---|---| +| verify | 401 `invalid_key` | 401 `authentication_error` | +| verify | 403 `model_not_allowed` | 403 `permission_error` | +| plan limits | over `rpm_limit` or `max_concurrent` | 429 `rate_limit_error` with `retry-after`; the message names the gateway's own limit, so a client can tell it from an upstream 429 | +| plan limits | Redis unreachable or slow | 503 `api_error`: admitting would let a tenant exceed its limits | +| reserve | 409 `insufficient_balance`, 404 `wallet_not_found` | 402 `billing_error` | +| verify or reserve | timeout, unreachable, `dependency_unavailable` | 503 `api_error` | +| anything else | any other code | 500 `api_error`, logged | + +Checked against: `services/gateway/internal/admission/admission.go`, `.../admission/limit.go`, +`.../admission/channel.go`, `services/gateway/internal/ratelimit/ratelimit.go`, +`services/gateway/internal/accessclient/client.go`, `services/gateway/internal/relay/relay.go`, +[api/access.yaml](../../api/access.yaml), [api/errors.md](../../api/errors.md). + +## 8. Delivery pipeline + +Six GitHub Actions workflows, plus Dependabot. Every pull request runs about 24 +checks; a push to `main` runs them again and publishes the images. + +```mermaid +flowchart LR + DEV["Branch and PR
one issue per PR"] + subgraph ci["CI on every pull request and push to main"] + TS["Tests
repository tests, 3 OSes"] + GO["Go
gofmt, vet, test, build, 3 OSes"] + IT["Go integration, ubuntu
PostgreSQL + Redis, no skips,
statement coverage"] + CS["Compose smoke
health endpoints, end-to-end"] + HK["Helm kind smoke (chart paths)
lint, render tests, install, ingress,
upgrade, deploy one service, roll back"] + TF["Terraform validation
mocked tests, no AWS calls"] + IB["Service images
build 5 images, no push"] + SEC["Security
govulncheck; Trivy on 5 images;
Trivy config on Terraform, chart, Dockerfile;
SARIF to code scanning"] + end + IF["Issue format check
on issue events"] + DB["Dependabot
weekly: Go modules, Actions, Docker"] + MG["Squash merge to main
by the integrator"] + PUB["Publish images
on main and v* tags
tag = commit SHA, digest recorded"] + GHCR[("GHCR")] + CHART["Chart default imageTag
set in a reviewed PR"] + subgraph cd["Delivery, run by the operator"] + TFA["terraform apply
per cloud day"] + INS["helm install
values-aws + overrides"] + DPL["deploy.py: one service by digest,
smoke, record"] + RB["deploy.py rollback
from the record"] + TFD["terraform destroy,
leftover check"] + end + + DEV --> ci + DB --> DEV + ci --> MG + MG --> PUB --> GHCR + PUB --> CHART + GHCR --> INS + TFA --> INS --> DPL --> RB + RB --> TFD +``` + +- Required checks and the merge rules are in + [CONTRIBUTING.md](../../.github/CONTRIBUTING.md); test levels and evidence are in + [docs/testing.md](../testing.md). +- Scanners are installed at a pinned version with a checksum check, not through + third-party actions. Accepted findings are listed with a reason and an expiry date + in [.trivyignore](../../.trivyignore). +- There is one cloud environment, created and destroyed per experiment day. The + owner approves every cloud action; no workflow holds AWS credentials. -Checked against: `services/gateway/internal/relay/relay.go`, `.../relay/request.go`, -`.../relay/result.go`, `services/gateway/internal/admission/`, -`services/metering/internal/api/`, `services/metering/internal/billing/billing.go`, -`services/metering/internal/store/reservations.go`, `services/gateway/internal/relay/responses.go`, -[api/metering.yaml](../../api/metering.yaml). +Checked against: `.github/workflows/tests.yml`, `helm.yml`, `images.yml`, +`security.yml`, `terraform.yml`, `issue-check.yml`, `.github/dependabot.yml`, +`.trivyignore`, `deploy/helm/deploy.py`, `deploy/terraform/README.md`. -## 7. Deviations from the proposal +## 9. Deviations from the proposal -Recorded in #18, the gate record (#20) and the [feasibility plan](../feasibility-plan.md). -The S3 rows come from [deploy/helm/README.md](../../deploy/helm/README.md), -[loadtest/README.md](../../loadtest/README.md), -[docs/observability.md](../observability.md) and the Terraform -[variables.tf](../../deploy/terraform/variables.tf). Facts about the 17 October -runs cite "operator record, 17 Oct"; that record is not in the repository. +Recorded in #18, the gate record (#20), the [feasibility plan](../feasibility-plan.md) +and the issues named in each row. | Proposal | As built | Why | |---|---|---| -| Transactional outbox, SQS and idempotent consumers for usage views (sections 5.1, 5.4) | Reservation and settlement in PostgreSQL transactions only; no outbox, no queue | Keeps S1 small; planned with UC6 after the gate | -| A pending reservation waits for recovery when usage is unknown | An unknown count is charged at its maximum; a hold whose settle fails stays held until UC6 | No recovery process in S1 | -| Redis for limits and the key cache | Redis runs in compose, but no service uses it; the chart and Terraform leave it out; no key cache | Not needed until rate limits or caching land | -| Keys and channels created through APIs (UC1, UC2) | Seed commands write each service's own schema | The key and channel APIs come later | -| Separate prices for cache and reasoning tokens | Cache tokens are charged at the full input price, on Messages and on Responses (where cached tokens are inside `input_tokens`); thinking tokens arrive in `output_tokens` | One price pair per version in S1 | +| Transactional outbox, SQS and idempotent consumers for usage views (sections 5.1, 5.4) | Reservation and settlement in PostgreSQL transactions only; the console reads usage straight from `reservations`; no outbox, no queue | Kept S1 small; not added later | +| A pending reservation waits for recovery when usage is unknown | An unknown count is charged at its maximum at settle. A hold that is never settled is recovered by `metering recover` and charged `max_cost` (#155) | One charge rule for every lost count | +| Redis for limits and the key cache | Redis holds the per-tenant plan limits (#157), optional in the chart, on in compose. No key cache, so revocation is immediate | Limits must hold across replicas; a key cache would delay revocation | +| Keys and channels created through APIs (UC1, UC2) | The console creates and revokes keys and adjusts wallets through the internal APIs (#137). Tenants, channels, price versions and plans are written by seed commands; the console only lists them | The write APIs for the catalogue were not built | +| Separate prices for cache and reasoning tokens | Cache tokens are charged at the full input price, on Messages and on Responses (where cached tokens are inside `input_tokens`); thinking tokens arrive in `output_tokens` | One price pair per version | | Conversation state through the gateway | `/v1/responses` is stateless: `previous_response_id` is refused, and clients resend the conversation | The gateway keeps no response state | -| Per-caller internal identities | One shared token for access; metering trusts the gateway's `tenant_id` and `price_version_id` | mTLS and workload identity are out of S1 | +| Per-caller internal identities | One shared token per callee; metering trusts the gateway's `tenant_id` and `price_version_id` | mTLS and workload identity were out of scope | | Claude Code and Codex CLI as the acceptance clients (UC3) | OpenCode on Messages; Codex CLI on Responses (guide #103) | Claude Code needs a Claude service | | Aurora, Argo CD, Cognito and CloudFront (section 5.5) | Plain RDS PostgreSQL in Terraform, images in GHCR instead of ECR, and the operator-run `deploy/helm/deploy.py` instead of Argo CD: one service by digest, rolled back from its record (#111) | Cost and time | | Distributed tracing (UC15) | Prometheus metrics and JSON logs only (`docs/observability.md`) | Tracing is optional and its overhead unmeasured | -| Public HTTPS ingress (section 5.5) | The ALB listens on plain HTTP, with no TLS. It admits only `ingress.inboundCidrs`, the operator's /32; the chart refuses an empty list, and any `/0` unless `ingress.allowAnySource=true` (#116) | HTTPS has not landed; the source allow-list is the stopgap, and keys still cross the internet in clear text | -| Prometheus and one Grafana dashboard (section 5.5) | The Prometheus server runs on an `emptyDir` with 6 h retention, so its data is lost with the Pod; `loadtest/prom_export.py` exports each run's series before teardown. The one dashboard is on `main` (#115) but opt-in: `dashboard.enabled` is off by default, and `deploy/helm/dashboard.py` installs Grafana as a separate release, with its data on an `emptyDir` too. It is verified on kind and optional in the cloud | Terraform installs no EBS CSI driver, so a volume claim would stay Pending; the scaling queries need only the last minutes. The dashboard is a view: k6 supplies load, client latency and formal completion, and ledger queries the money evidence | -| 1 vCPU and 2 GiB per gateway Pod (section 5.6) | `values-aws.yaml` keeps 1 vCPU and 2 GiB, as both request and limit. The second 17 October apply limited each gateway Pod to 0.25 vCPU, passed with `--set` and an overlay that is not in the repository (operator record, 17 Oct) | The repository does not record why. The results are not published, so this page draws no conclusion from either setting | -| Generator and simulator compute isolated from the relay nodes (section 5.6) | Isolated from the services nodes, but k6 and mock-upstream share one load-generator node: c6i.xlarge by default (`generator_instance_type`), m6i.2xlarge in the second 17 October apply (operator record, 17 Oct) | Cost; a saturated node slows both, so rising dropped iterations read as a generator limit, not gateway capacity | -| A calibrated per-Pod admission cap C, with capacity rejections (section 5.6) | No admission cap: every call whose cost is held is admitted, so eligible arrivals equal admitted ones. C is measured, as the mean active requests of the single-Pod calibration (`loadtest/derive.py`), and sets T = floor(0.7 × C); nothing enforces it | The capacity gate is not built yet | +| Public HTTPS ingress (section 5.5) | The ALB listens on plain HTTP, with no TLS. It admits only `ingress.inboundCidrs`; the chart refuses an empty list, and any `/0` unless `ingress.allowAnySource=true` (#116) | HTTPS has not landed; the source allow-list is the stopgap, and keys still cross the internet in clear | +| Prometheus and one Grafana dashboard (section 5.5) | Prometheus keeps 6 h on an `emptyDir`; `loadtest/prom_export.py` exports each run's series before teardown. The dashboard (#115) is opt-in | Disposable environments | +| 1 vCPU and 2 GiB per gateway Pod (section 5.6) | `values-aws.yaml` keeps 1 vCPU and 2 GiB. The cloud runs used 250m and 1 GiB per Pod (cfg-D), set at install time | Owner's decision on 17 October; the results describe 250m Pods and do not transfer to 1 vCPU | +| Generator and simulator compute isolated from the relay nodes (section 5.6) | Isolated from the services nodes, but k6 and mock-upstream share one load-generator node: c6i.xlarge by default (`generator_instance_type`), m6i.2xlarge in the cloud runs | Cost; the runs checked that the generator was not the limit | +| A calibrated per-Pod admission cap C, with capacity rejections (section 5.6) | No admission cap. C is measured in the single-Pod calibration (`loadtest/derive.py`) and sets the KEDA target T = floor(0.7 × C); nothing enforces C | The capacity gate is not built | +| Fallback, 8 replicas, weighted routing, CSV export (UC11–UC14) | Not built | Should and Could items | +| Real-model capacity tests | A kit for vLLM on a GPU node (#147–#149) is merged; the cloud run is deferred because the GPU instance quota was 0 | AWS quota |