Skip to content

Assert what the served endpoints return, not just a 200 #420

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nicknikolakakis wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Assert what the served endpoints return, not just a 200 #420

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7915b0f
Answer real requests in the local e2e mock engine
nicknikolakakis Aug 16, 2026
4da464a
Assert what the served endpoints return, not just a 200
nicknikolakakis Aug 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions e2e/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ against. The control-plane cluster needs no DRA.

```bash
nix run .#e2e # bring up both clusters + deploy the mock model
nix run .#e2e -- --verify # same, then wait for readiness and assert a live 200
nix run .#e2e -- --verify # same, then wait for readiness and run the behavioral suite
nix run .#e2e -- --clean # tear both clusters down
```

Expand All @@ -119,10 +119,16 @@ kubectl run curl -n ml-team --rm -it --image=curlimages/curl@sha256:7c12af72ceb3
-d '{"model":"mock","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}'
```

`--verify` runs both of those (OpenAI then Anthropic) and exits non-zero on
failure. It's the exact command the `E2E` CI workflow runs, so a green `--verify`
locally and a green CI run mean the same thing; use the manual curls above to
poke the endpoints interactively.
`--verify` goes further than those curls: it runs [`verify/test_serving.py`](verify/test_serving.py)
from a pod on the control plane and asserts what came back — the response shapes
both surfaces promise, that the last user turn reaches the engine, that a
streamed response arrives as `text/event-stream` frames rather than one buffered
blob, and that malformed input and unrouted paths fail. The suite goes in as a
ConfigMap and runs under a stock `python:3.12-alpine` pod; it imports only the
stdlib, so nothing is built or installed. It exits non-zero on failure and is the
exact command the `E2E` CI workflow runs, so a green `--verify` locally and a
green CI run mean the same thing; use the manual curls above to poke the
endpoints interactively.

## How it's structured

Expand Down
66 changes: 57 additions & 9 deletions e2e/manifests/40-model-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,13 +37,42 @@ spec:
- |
import json, http.server
class H(http.server.BaseHTTPRequestHandler):
# HTTP/1.1 so the streaming path can chunk, the way a real engine does.
protocol_version = "HTTP/1.1"
def _s(self, o, c=200):
b = json.dumps(o).encode()
self.send_response(c)
self.send_header("content-type", "application/json")
self.send_header("content-length", str(len(b)))
self.end_headers()
self.wfile.write(b)
def _sse(self, frames):
self.send_response(200)
self.send_header("content-type", "text/event-stream")
self.send_header("cache-control", "no-cache")
self.send_header("transfer-encoding", "chunked")
self.end_headers()
for f in [*frames, "[DONE]"]:
b = ("data: " + (f if isinstance(f, str) else json.dumps(f)) + "\n\n").encode()
self.wfile.write(b"%X\r\n" % len(b) + b + b"\r\n")
self.wfile.flush()
self.wfile.write(b"0\r\n\r\n")
def _body(self):
# The endpoint picker's ext_proc drops content-length, so bodies arrive chunked.
if "chunked" not in (self.headers.get("transfer-encoding") or "").lower():
return self.rfile.read(int(self.headers.get("content-length") or 0))
b = b""
while n := int(self.rfile.readline().split(b";")[0] or b"0", 16):
b += self.rfile.read(n)
self.rfile.readline()
self.rfile.readline()
return b
def _read(self):
try:
r = json.loads(self._body() or b"{}")
except ValueError:
return None
return r if isinstance(r, dict) and r.get("messages") else None
def do_GET(self):
if self.path == "/health":
self._s({"status": "ok"})
Expand All @@ -52,27 +81,46 @@ spec:
else:
self._s({"error": "not found"}, 404)
def do_POST(self):
n = int(self.headers.get("content-length") or 0)
self.rfile.read(n)
text = "Hello from the Modelplane local mock engine."
if self.path.startswith("/v1/messages"):
req = self._read()
anthropic = self.path.startswith("/v1/messages")
if not anthropic and not self.path.startswith("/v1/chat/completions"):
self._s({"error": "not found"}, 404)
elif req is None:
self._s({"error": "messages is required"}, 400)
else:
self._answer(req, anthropic)
def _answer(self, req, anthropic):
model = req.get("model") or "mock"
# Echo the last user turn back, so a caller can tell its request reached the engine.
turns = [m.get("content") or "" for m in req["messages"] if m.get("role") == "user"]
text = "Echo: " + (turns[-1] if turns else "")
if anthropic:
# Anthropic Messages API; vLLM serves it alongside the OpenAI routes.
self._s({
"id": "msg-mock", "type": "message", "role": "assistant",
"model": "mock", "stop_reason": "end_turn",
"model": model, "stop_reason": "end_turn",
"content": [{"type": "text", "text": text}],
"usage": {"input_tokens": 1, "output_tokens": 1},
"usage": {"input_tokens": len(turns), "output_tokens": 1},
})
elif req.get("stream"):
head = {"id": "chatcmpl-mock", "object": "chat.completion.chunk", "model": model}
frames = [dict(head, choices=[{"index": 0, "delta": {"role": "assistant"}}])]
frames += [dict(head, choices=[{"index": 0, "delta": {"content": w + " "}}]) for w in text.split()]
frames.append(dict(head, choices=[{"index": 0, "delta": {}, "finish_reason": "stop"}]))
self._sse(frames)
else:
self._s({
"id": "chatcmpl-mock", "object": "chat.completion", "model": "mock",
"id": "chatcmpl-mock", "object": "chat.completion", "model": model,
"choices": [{"index": 0, "finish_reason": "stop",
"message": {"role": "assistant", "content": text}}],
"usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
"usage": {"prompt_tokens": len(turns), "completion_tokens": 1,
"total_tokens": len(turns) + 1},
})
def log_message(self, *a):
pass
http.server.HTTPServer(("0.0.0.0", 8000), H).serve_forever()
# Threaded: HTTP/1.1 keeps connections alive, so a single-threaded server
# would block the probes and the endpoint-picker behind one client.
http.server.ThreadingHTTPServer(("0.0.0.0", 8000), H).serve_forever()
# The ModelDeployment container template is a curated subset
# (name/image/command/args/env); the composition wires the port,
# readiness, and any GPU resources itself (it assumes port 8000). So
Expand Down
108 changes: 59 additions & 49 deletions e2e/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ WL=modelplane-e2e-workload
# v1.34.2 or newer: older kubelets deadlock on an idle DRA connection (k/k#133934).
WL_NODE_IMAGE=kindest/node:v1.34.8@sha256:02722c2dedddcfc00febf5d27fbeb9b7b2c14294c82109ff4a85d89ac9ba3256
METALLB_URL=https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml
# Pinned by digest (a multi-arch manifest list) so a moving :latest can't flake
# the verify curl pod.
CURL_IMAGE=curlimages/curl@sha256:7c12af72ceb38b7432ab85e1a265cff6ae58e06f95539d539b654f2cfa64bb13
# The same python:3.12-alpine digest the mock engine runs, so the e2e pins one
# image. The suite is stdlib-only, so the stock image needs no pip install.
PYTHON_IMAGE=python@sha256:6d43704baacd1bfbe7c295d7f13079d5d8104ed33568873133f8fc69980419df
ROOT="$(git rev-parse --show-toplevel)"

log() { printf '\n\033[1;34m==> %s\033[0m\n' "$*"; }
Expand Down Expand Up @@ -186,9 +186,9 @@ fi

# --verify: project run returns once the config is healthy and the resources are
# applied, so the serving-stack install and model rollout are still reconciling.
# Wait for the ModelService to publish an address, then route a real request to
# the engine and assert a 200. Any failure exits non-zero — that is what makes
# this usable as a CI gate.
# Wait for the ModelService to publish an address, then run the behavioral suite
# in e2e/verify against it. Any failure exits non-zero — that is what makes this
# usable as a CI gate.
log "Verifying the model serves end to end"
ns=ml-team
svc=mock
Expand All @@ -206,52 +206,62 @@ done
log "ModelService address: $addr"

# The address is on the kind Docker subnet the host can't route to on macOS, so
# curl from a pod on the control plane, reading the status from the pod's logs
# (not `run -i`, whose attach drops output on a headless runner). curl_status
# runs one throwaway pod per call and echoes the HTTP code; it polls the logs
# (curl writes the code once, then exits) so a failed attempt costs seconds, and
# a unique pod name per call keeps retries from reading a prior pod's output.
curl_status() {
local pod="$1" url="$2"
shift 2
kubectl --context "$cpctx" -n "$ns" run "$pod" --restart=Never \
--labels=app.kubernetes.io/name=e2e-verify --image="$CURL_IMAGE" \
--command -- curl -sS --max-time 15 -o /dev/null -w '%{http_code}' "$url" "$@" \
>/dev/null 2>&1 || true
local c=""
for _ in $(seq 1 30); do
c="$(kubectl --context "$cpctx" -n "$ns" logs "$pod" 2>/dev/null | tr -dc '0-9' || true)"
[ -n "$c" ] && break
sleep 2
done
printf '%s' "$c"
}
# the suite runs from a pod on the control plane. It ships as a ConfigMap rather
# than an image we'd have to build and push, and the stock python image runs it
# as-is because it imports nothing outside the stdlib.
log "Running the behavioral suite (e2e/verify) against $addr"
kubectl --context "$cpctx" -n "$ns" create configmap e2e-verify-suite \
--from-file=test_serving.py="$ROOT/e2e/verify/test_serving.py" \
--dry-run=client -o yaml | kubectl --context "$cpctx" apply -f -

# A rerun against a live cluster would otherwise hit an immutable completed pod.
kubectl --context "$cpctx" -n "$ns" delete pod e2e-verify --now >/dev/null 2>&1 || true
kubectl --context "$cpctx" apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: e2e-verify
namespace: $ns
labels:
app.kubernetes.io/name: e2e-verify
spec:
restartPolicy: Never
containers:
- name: verify
image: $PYTHON_IMAGE
command: [python, -u, /suite/test_serving.py, -v]
env:
- name: MODELPLANE_ADDRESS
value: "$addr"
- name: MODELPLANE_MODEL
value: "$svc"
volumeMounts:
- name: suite
mountPath: /suite
volumes:
- name: suite
configMap:
name: e2e-verify-suite
EOF

# OpenAI /v1/chat/completions, retried: the address can publish a moment before
# the cross-cluster route is serving, and a slower CI runner widens that gap.
oai='{"model":"'"$svc"'","messages":[{"role":"user","content":"ping"}]}'
code=""
for attempt in $(seq 1 10); do
code="$(curl_status "e2e-verify-oai-$attempt" "$addr/v1/chat/completions" -H 'content-type: application/json' -d "$oai")"
log "verify attempt $attempt (OpenAI): HTTP ${code:-none}"
[ "$code" = "200" ] && break
# The suite waits for the route itself, so the only wait here is for the pod to
# finish. Poll the phase: kubectl wait has no single condition for "ran, either
# way", and a Failed pod is a result to report, not an error to time out on.
phase=""
for _ in $(seq 1 90); do
phase="$(kubectl --context "$cpctx" -n "$ns" get pod e2e-verify -o jsonpath='{.status.phase}' 2>/dev/null || true)"
case "$phase" in
Succeeded | Failed) break ;;
esac
sleep 10
done
[ "$code" = "200" ] || {
echo "verify: $addr/v1/chat/completions did not return 200 within retries (last: ${code:-none})" >&2
kubectl --context "$cpctx" -n "$ns" delete pod -l app.kubernetes.io/name=e2e-verify --now >/dev/null 2>&1 || true
exit 1
}

# Anthropic Messages API on the same address: vLLM serves /v1/messages alongside
# the OpenAI routes (PR #360) and the route preserves the path. Serving is up by
# now, so one attempt suffices.
ant='{"model":"'"$svc"'","max_tokens":16,"messages":[{"role":"user","content":"ping"}]}'
mcode="$(curl_status e2e-verify-anthropic "$addr/v1/messages" -H 'content-type: application/json' -H 'anthropic-version: 2023-06-01' -d "$ant")"
log "verify (Anthropic /v1/messages): HTTP ${mcode:-none}"
kubectl --context "$cpctx" -n "$ns" delete pod -l app.kubernetes.io/name=e2e-verify --now >/dev/null 2>&1 || true
[ "$mcode" = "200" ] || {
echo "verify: $addr/v1/messages did not return 200 (last: ${mcode:-none})" >&2
# unittest writes to stderr, and the logs are the failure report on a red run.
kubectl --context "$cpctx" -n "$ns" logs e2e-verify 2>&1 || true
kubectl --context "$cpctx" -n "$ns" delete pod e2e-verify --now >/dev/null 2>&1 || true
kubectl --context "$cpctx" -n "$ns" delete configmap e2e-verify-suite >/dev/null 2>&1 || true
[ "$phase" = "Succeeded" ] || {
echo "verify: the behavioral suite did not pass (pod phase: ${phase:-none})" >&2
exit 1
}
log "End to end OK: $addr serves OpenAI (/v1/chat/completions) and Anthropic (/v1/messages)"
log "End to end OK: $addr passes the behavioral suite in e2e/verify"
Loading
Loading