test(otel): add vLLM, KServe and Knative solution integration tests

[Aakash-Dantre]

Summary

End-to-end integration tests for the OTEL Container Insights LLM model-serving solutions in the helm chart: vLLM (aws-observability/helm-charts#398), KServe control plane (aws-observability/helm-charts#400, merged) and Knative Serving (aws-observability/helm-charts#401).

They follow test/otel/solutions/keda and karpenter: a terraform root builds an EKS cluster, installs the workloads and the chart, then Go suites query CloudWatch's PromQL endpoint and check what the agent published.

Test cluster: terraform/eks/daemon/otel-llm-serving

• Cluster: EKS with 2 × m5.2xlarge CPU nodes, Pod Identity for the agent.
• Serving stack: cert-manager, Istio, Knative Serving and KServe in Serverless mode (versions are variables: 1.17 / 1.27 / 1.21 / v0.20 by default).
  • Knative is configured to serve Prometheus metrics (metrics-protocol, request-metrics-protocol) and restarted, since it reads them at startup.
  • Tag resolution is skipped for public.ecr.aws, because resolving the multi-GB vLLM image's tag times out.
• Workloads in namespace llm. Each exercises one chart rule:

Workload	Exercises
vllm-isvc: InferenceService, vLLM CPU image, enable-prometheus-scraping	vLLM found as KServe's kserve-container; the Knative data plane; KServe's own prometheus.io/* must not make the queue-proxy a target
vllm-isvc transformer (hello-world container, component: transformer)	same InferenceService label as the predictor: must not be scraped as vLLM and must get no aws.service.type
vllm-standalone: Deployment, vLLM CPU image	vLLM found by image name
hello-ksvc: plain Knative Service (hello-world)	Knative data plane scraped, but no aws.service.type
vllm-annotated-mock: busybox serving static vllm:* and http_*, with prometheus.io/* annotations	negative control: a vLLM server from a custom-named image is not scraped; the chart reads no scrape annotations
generic-exporter: busybox, prometheus.io/*	negative control: a non-vLLM exporter adds nothing
loadgen	requests to the predictor (through its queue-proxy), the standalone server, the transformer and the plain Knative Service

Both vLLM servers run Qwen2.5-0.5B-Instruct with a memory-backed /dev/shm. The chart is cloned from helm_chart_repo at helm_chart_branch, so a fork branch can be tested before merge, and installed with otelContainerInsights.enabled=true (the solutions default on). The agent image is then replaced with cwagent_image_repo:cwagent_image_tag, as in the other OTEL roots.

The cluster-scraper image patch and restart fail the run if the scraper is missing, since the KServe and Knative suites need it. The root also fails fast if the cloned chart has no vllm, kserve or knative solution. The validator waits 4 minutes for data, then runs test_dir (default ./test/otel/solutions/vllm) followed by solutions/kserve and solutions/knative on the same cluster; every suite runs, and the run fails at the end if any did.

Suites: test/otel/solutions/{vllm,kserve,knative}

Same layout as the KEDA suite (setup_test.go, metrics_test.go, *_test.go), using util/otelmetrics with new SourceVLLM, SourceKServe and SourceKnative constants.

vllm (11 tests): 12 metrics from both real servers. Also checks:

• scope github.com/vllm-project/vllm, pipeline vllm, and datapoint labels
• InferenceService attribution: service.name, KServe's pod label k8s.pod.label.serving.kserve.io/inferenceservice (and no custom inferenceservice key), kserve-container, the revision Deployment, aws.service.type = ai_inference, cloud identity
• one series per pod and never a queue-proxy container
• image-detected naming
• nothing from either annotated pod, or from the transformer, checked by listing what the vllm pipeline published for those pods (positive controls: the pods' container metrics, and the transformer's queue-proxy series, exist)
• request counters (vllm:request_success_total, http_requests_total) increase on both servers
• the vllm pipeline publishes no python_* or process_* names

kserve (5 tests): 8 controller-runtime and leader-election metrics. Also checks:

• scope github.com/kserve/kserve, pipeline kserve-controlplane, labels
• controller pod and workload (k8s.workload.*) attribution and cloud identity
• no AZ or host from the cluster-scraper's node
• the pipeline publishes no workqueue_*, rest_client_*, go_* or process_* names

knative (6 tests):

• control plane: 8 kn_* metrics attributed to their component pod and workload (component also as k8s.pod.label.app, no custom knative_component key), cloud identity, no AZ or host, and no go_*, process_* or *_go_* names published
• data plane, predictor revision: kn_serving_invocation_duration_seconds attributed to the revision Deployment and InferenceService (KServe's pod labels, no custom key), with aws.service.type = ai_inference
• data plane, transformer revision and plain Knative Service: series exist, with no aws.service.type
• Knative <= 1.18 metric names are not covered: the suite uses the >= 1.19 kn_* names

Shared helpers (Increase, PublishedMetricNames, RequireNoPrefix, RequireCloudIdentity, RequireNoAttribute) live in util/otelmetrics/solution_helpers.go. QueryCache.GetWithFilter now quotes @instrumentation.* keys and shares the cache's concurrency limit, and the client retries HTTP 429.

Every suite checks cloud identity as KEDA and Karpenter do: cloud.provider, cloud.platform, region, account, and the cluster ARN in cloud.resource_id.

CI

Added to the eks_daemon matrix as eks_daemon:otel_solutions_vllm_test (m5.2xlarge, AL2023, Kubernetes 1.35) with wip: true, so failures don't block CI until it has a run history. One cluster per run covers all three suites.

CI installs the chart from the newest helm-charts release-* branch. Until a release branch includes aws-observability/helm-charts#398, aws-observability/helm-charts#400 and aws-observability/helm-charts#401, the run stops at the chart guard (a warning, with wip).

Testing

• Full terraform apply of the root: all three suites pass. As a negative check, the same cluster with attribution and filtering removed from the chart makes the attribution and filter tests fail.
• All three suites also pass against an existing GPU EKS cluster with KServe Serverless and GPU vLLM (vLLM 11/11, KServe 5/5, Knative 6/6), with workload names pointed at that cluster, a transformer-labelled revision and a plain Knative Service added, and load on both vLLM servers. After the review changes, every test whose workloads exist on that cluster passes again (vLLM 9/9, KServe 5/5, Knative 5/5).
• go vet -tags integration, gofmt and go test ./util/otelmetrics/ (new TestFilterMatchers) are clean. terraform fmt and terraform validate pass. go run ./generator emits the new matrix entry.

PR Checklist

• Commits are squashed into a logical, reviewable set
• Commits and PR description comply with Amazon internal guidelines
• go vet / gofmt / terraform validate pass locally
• All GitHub Actions checks on the PR are passing
• Integration test evidence: full terraform apply run above
• New or updated integration test coverage: this PR
• New functionality has tests
• No breaking changes: new root, suites and matrix entry (wip); three new otelmetrics source constants

[miconeilaws]

Coverage gaps

1. Transformer / explainer revisions. Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.11.49 to 1.11.88 #353 scrapes only the predictor (component: predictor), and adding test #390 should match it. vllm-isvc has no transformer, so a regression here wouldn't be caught. Could we add a small transformer to the InferenceService and assert:
   • no vllm pipeline series come from the transformer pod
   • the transformer's queue-proxy series do not carry aws.service.type
2. Plain Knative Service. adding test #390 says plain Knative Services are scraped but get no aws.service.type, and there isn't one in the cluster. A tiny ksvc (any hello-world image) with a test asserting its data-plane series exist without aws.service.type would cover both claims.
3. Non-vLLM KServe predictor. Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.11.49 to 1.11.88 #353 matches any predictor kserve-container regardless of image and relies on the keep filter for the rest. An sklearn (or similar) InferenceService, with an assertion that nothing reaches cloudwatch.pipeline=vllm, would pin that behaviour.
4. Knative <= 1.18. The chart keeps both name sets (revision_* / autoscaler_*) and the .*_go_.* drop, but the suite hard-codes the kn_* names. knative_version is a variable, but changing it would make the suite fail rather than test the old names. Either add a matrix entry with version-keyed metric lists, or note somewhere that <= 1.18 is not covered by e2e.
5. Follow-ups from adding test #390 review. Once those changes land, please assert there is no knative_component attribute, and that aws.service.type is set on predictor revisions only.

Nits

• Negative controls: vllm-annotated-mock and generic-exporter were built for the prometheus.io/* opt-in, which has since been removed. Both use busybox, so they pass trivially now. It would be more useful to keep one of them and add an image whose repo path contains vllm but isn't vLLM (e.g. .../vllm-team/sidecar). That tests the boundary of the image match.
• Values, not just presence: load is generated, but nothing checks that counters move. One rate/increase check (e.g. vllm:request_success_total > 0, or a non-zero kn_serving_invocation_duration_seconds count) would catch start-time / reset regressions.
• || true on the cluster-scraper image patch and restart: if the patch fails, the KServe and Knative suites silently run against the chart's default agent image. Since both depend on the cluster-scraper, could that step fail instead?
• Shared helpers: series, pipeline, metricNames, requireCloudIdentity and requireNoCustomISVCKey are duplicated across all three packages. Could they move to util/otelmetrics?
• sleep 240: fine as it is (same pattern as the other roots), but polling for the first datapoint would cut both flakiness and runtime.

[Aakash-Dantre]

Thanks, updated:

1. Added a transformer to the InferenceService. Tests check nothing from it reaches the vllm pipeline (with a positive control that its pod is present) and that its queue-proxy series have no aws.service.type.
2. Added a plain Knative Service. Its data-plane series exist without aws.service.type.
3. The non-vLLM predictor case raises a question about what the vLLM solution (feat(otel-ci): add vLLM model-server metrics solution aws-observability/helm-charts#398) should do with http_* from other runtimes. Let's discuss that separately before pinning it in a test.
4. Knative <= 1.18 is now noted as not covered by e2e.
5. Added assertions for no knative_component and aws.service.type on predictor revisions only.

Nits: counters are now checked to increase; the cluster-scraper patch and restart fail the run instead of || true; the shared helpers moved to util/otelmetrics. I kept one busybox control, because a non-vLLM image with vllm in its path would need publishing one; happy to revisit. sleep 240 stays, matching the other setups.

Chart PRs now: vLLM aws-observability/helm-charts#398, KServe aws-observability/helm-charts#400 (merged), Knative aws-observability/helm-charts#401 (after #398).

[olowosulu]

Thanks for the thorough suite, the kserve/knative contract checks line up with the chart. One thing I think blocks merge and two I'd want sorted before we rely on the run history.

Blocker: vllm_test.go:114 asserts k8s.container.name == "kserve-container", but the chart never sets that attribute. The vllm scrape job only uses __meta_kubernetes_pod_container_name for target selection and vllm_promote promotes pod/namespace/aws.service.type only (DCGM does label the container, vllm doesn't). res["k8s.container.name"] is "" so this fails every run. Easiest fix is to drop the line (the per-pod count in TestVLLMNoDuplicateSeries already covers queue-proxy); otherwise it needs a chart change adding target_label: container plus a promote. Line 143 (NotEqual "queue-proxy") is vacuous for the same reason.

Suggestion: chart source in CI. test-artifacts.yml installs the chart from the newest release-* branch (currently release-6.7.0), which has no vllm/kserve/knative keys. So the default run fails all three suites and wip: true turns that into a warning until a release branch includes helm-charts #398/#400/#401. Could you note that in the PR / generator comment, or add a guard that fails explicitly when the installed chart has no vllm: solution?

Suggestion: negative tests are vacuous. The annotated/generic mock tests (vllm_test.go:325-344) query only pipeline="vllm" with no positive control that the pods exist, and the dropped-family checks (knative_test.go:113, kserve_test.go:134, vllm_test.go:350) assert fixed names are absent; http_request_size_bytes can't appear anyway since CloudWatch rejects Summaries. Suggest using client.MetricNames to list what's actually there, require.NotEmpty, then assert no forbidden prefixes.

Nits:

• solution_helpers.go:45-47: Increase returns 0 for both "no data" and "no increase", so the failure message is misleading.
• solution_helpers.go:23-35: Series duplicates QueryCache.GetWithFilter; root cause is query_cache.go:237-238 not quoting @instrumentation.* keys. Adding that case lets Series/PipelineMatcher go.
• client.go:171-172 treats 429 as fatal and vllm fires ~66 uncached parallel queries; flake risk.
• main.tf:583-599: with set -e a vllm failure skips kserve/knative. Collect rc per suite instead.
• Description drift: reply says one busybox control kept but main.tf:399 still deploys both; body says vllm has 9 tests, file has 11; main.tf:4-14 header omits transformer/hello-ksvc.
• vllm/setup_test.go:99-110 registry wiring is unused by the suite; attribution keys differ across suites (k8s.workload.* vs k8s.deployment.name).

[Aakash-Dantre]

Thanks! Updated in c5cfa07:

• k8s.container.name: the chart doesn't set it, but the Prometheus receiver does from the target's __meta_kubernetes_pod_container_name (addKubernetesResource in prom_to_otlp.go), so the KServe series carry kserve-container. Kept the assertion with a comment; the queue-proxy check also requires a non-empty name now.
• Chart source: the root fails fast if the cloned chart has no vllm/kserve/knative solution, and the generator entry notes the release-branch dependency.
• Negative tests: they list what each pipeline published (MetricNames) and assert no forbidden prefixes. The mocks and transformer have positive controls. Dropped the Summary check.
• Nits: all done (Increase fails on no data, @instrumentation.* keys in GetWithFilter so Series/PipelineMatcher are gone, 429 retried, every suite runs, keys aligned on k8s.workload.*, unused registry removed). On the busybox controls: I meant both are kept. The description is fixed.

[olowosulu]

LGTM.