Self-hosted LLM inference on Kubernetes: agentgateway + Gateway API Inference Extension + llm-d EPP, vLLM (T4) and llama.cpp, KEDA queue-depth autoscaling, Kueue/DRA GPU scheduling, Prometheus/Grafana, and an emulated BGP/EVPN leaf-spine AI fabric with RoCE v2 design.
kubernetes bgp gpu rdma roce gateway-api nccl mlops gpu-benchmark keda gpu-scheduling mlops-project llama-cpp vllm llm-inference ai-infrastructure kueue vllm-serve llm-d container-lab
-
Updated
Nov 4, 2026 - Python