An end-to-end cloud-native platform for containerized LLM inference serving, automated CI/CD pipelines, Infrastructure as Code (Terraform), and full-stack Kubernetes observability on AWS EKS.
- Executive Project Summary
- Production Architecture
- Core Production Highlights
- Technology Stack
- Key Engineering Decisions and Tradeoffs
- 10-Day Implementation Milestone Summary
- Detailed Day-by-Day Engineering and Debugging Logs
- Day 1: Infrastructure and Local Environment Baseline
- Day 2: Containerized Ollama Deployment into kind
- Day 3: Kubernetes Service Discovery and FastAPI Integration
- Day 4: Automated CI/CD Pipeline and Container Registry Integration
- Day 5: Observability Stack — Prometheus, Grafana and Golden Signals Dashboard
- Day 6: Alerting — Alertmanager, PrometheusRule and Incident Lifecycle Validation
- Day 7: Cloud Migration — AWS EC2 via Terraform
- Day 8: Cloud-Native Migration — EKS, IRSA and Production Kubernetes Friction
- Day 9: Observability on EKS, Public Exposure and Cross-Environment Validation
- Day 10: CD Loop Completion, GitOps Automation and Portfolio Consolidation
- Repository Structure
- Quickstart and Reproduction Guide
This project demonstrates the complete engineering lifecycle of deploying and operating an open-source Large Language Model (LLM) serving platform in production. The system transitions from a local multi-node prototyping environment (kind) to a fully automated, observable, and secured cloud deployment on Amazon Web Services (AWS EKS) using Infrastructure as Code (Terraform).
- Resilient Model Serving: Serves quantized open-source LLMs (
qwen2.5:0.5b) on CPU-optimized nodes using Ollama with an isolated FastAPI gateway, decoupling model lifecycle and storage from inference execution. - Infrastructure as Code (IaC): Modular Terraform configurations provisioning an Amazon EKS cluster (Kubernetes v1.35), custom VPC networking, public subnets across multiple AZs, managed node groups, and AWS EBS CSI Driver integrations.
- Enterprise IAM & Security: Complete least-privilege security model using AWS IAM Roles for Service Accounts (IRSA) via OIDC, eliminating static AWS credentials and restricting cloud API access per Kubernetes ServiceAccount.
- Automated CI/CD & GitOps: GitHub Actions pipeline executing static linting (
ruff), unit test suites (pytest), container image compilation, and immutable SHA-based image publishing to GitHub Container Registry (GHCR). - Production Observability and SRE Alerting: Deployed
kube-prometheus-stackto monitor SRE Golden Signals (Traffic, Latency, Error Rate). Implemented declarativePrometheusRulealerting, debounced incident thresholds, and Alertmanager routing, validated through live fault-injection testing over public AWS Load Balancer endpoints.
PUBLIC INTERNET
│
│ HTTPS / HTTP (:8000)
▼
┌────────────────────────────────────┐
│ AWS Classic Load Balancer │
│ (*.ap-south-1.elb.amazonaws.com) │
└──────────────────┬─────────────────┘
│
▼
AWS EKS CLUSTER (ap-south-1)
┌─────────────────────────────────────────────────────────────────────────────┐
│ VPC: 10.0.0.0/16 │
│ │
│ ┌─────────────────────────────────┐ ┌─────────────────────────────────┐ │
│ │ Public Subnet 1 (ap-south-1a) │ │ Public Subnet 2 (ap-south-1b) │ │
│ │ Managed Node: t3.small │ │ Managed Node: t3.small │ │
│ │ │ │ │ │
│ │ ┌───────────────────────────┐ │ │ ┌───────────────────────────┐ │ │
│ │ │ FastAPI Serving Pod │ │ │ │ Ollama Backend Pod │ │ │
│ │ │ Namespace: llm-serving │ │ │ │ Namespace: llm-serving │ │ │
│ │ │ Port: 8000 │──┼───┼─▶│ Port: 11434 │ │ │
│ │ └─────────────┬─────────────┘ │ │ └─────────────┬─────────────┘ │ │
│ │ │ │ │ │ │ │
│ │ │ /metrics │ │ │ Dynamic Mount │ │
│ │ ▼ │ │ ▼ │ │
│ │ ┌───────────────────────────┐ │ │ ┌───────────────────────────┐ │ │
│ │ │ Prometheus Server │ │ │ │ AWS EBS gp3 Volume │ │ │
│ │ │ Namespace: monitoring │ │ │ │ (Provisioned by EBS CSI) │ │ │
│ │ └─────────────┬─────────────┘ │ │ └───────────────────────────┘ │ │
│ │ │ │ │ │ │
│ │ ┌────────┴────────┐ │ │ ┌───────────────────────────┐ │ │
│ │ ▼ ▼ │ │ │ EBS CSI Controller (IRSA) │ │ │
│ │ ┌───────────┐ ┌───────────┐ │ │ │ Namespace: kube-system │ │ │
│ │ │Alertmanager│ │ Grafana │ │ │ └─────────────┬─────────────┘ │ │
│ │ └───────────┘ └───────────┘ │ │ │ │ │
│ └─────────────────────────────────┘ └────────────────┼────────────────┘ │
└─────────────────────────────────────────────────────────┼───────────────────┘
│ OIDC AssumeRole
▼
┌───────────────────────────┐
│ AWS IAM Role: ebs-csi-irsa│
│ Policy: AmazonEBSCSIDriver│
└───────────────────────────┘
Key verification milestones across the project's cloud and observability stack:
Live prompt inference and health probe verification over the public AWS Classic Load Balancer endpoint (ap-south-1.elb.amazonaws.com:8000):

Production Grafana dashboard tracking real-time Request Rate, P95 Latency, and Error Rate under live cluster traffic:

Prometheus HighErrorRate rule transitioning to FIRING during simulated backend failure on AWS EKS:

| Domain | Technology / Tool | Version | Purpose |
|---|---|---|---|
| Cloud and IaC | AWS EKS, EC2, EBS gp3, ELB, IAM, VPC | AWS CLI v2 | Production cloud hosting and network isolation |
| HashiCorp Terraform | v1.15 | Declarative Infrastructure as Code and state management | |
| Orchestration and Storage | Kubernetes | v1.35 / v1.36 | Workload orchestration, scheduling, and service discovery |
kind (Kubernetes in Docker) |
v0.22 | Local multi-node development and prototyping | |
| AWS EBS CSI Driver | v1.40+ | Dynamic persistent storage provisioning via IRSA | |
| Docker | v29 | Containerization and multi-stage image packaging | |
| Application and Serving | FastAPI & Uvicorn | Latest | High-throughput asynchronous REST inference API |
| Ollama & Qwen 2.5 (0.5B GGUF) | Latest | Quantized local and cloud CPU-based LLM inference engine | |
| Pydantic & HTTPX | v2 / Latest | Data validation, schema enforcement, and async backend client | |
| Observability and SRE | Prometheus Operator (kube-prometheus-stack) |
Helm Chart | Declarative Kubernetes metrics collection and ServiceMonitors |
| Alertmanager & PrometheusRule | Latest | Automated alert evaluation, debouncing, and notification routing | |
| Grafana | Latest | Version-controlled Golden Signals dashboards (Dashboard-as-Code) | |
prometheus-fastapi-instrumentator |
Latest | Custom latency and HTTP request status metrics instrumentation | |
| CI/CD and Code Quality | GitHub Actions | Latest | Automated multi-stage build, test, and release pipelines |
| GitHub Container Registry (GHCR) | Latest | Immutable container artifact storage tagged by Git commit SHA | |
| Ruff & Pytest | Latest | High-speed Python linting, style formatting, and unit testing |
- Context: AWS managed control planes and multi-node compute clusters quickly accumulate costs if over-provisioned.
- Decision: Architected the cluster around
t3.smallnodes (2 vCPU, 2GB RAM) under AWS Free Tier eligibility instead of unconstrained default instance sizes liket3.mediumorm5.large. - Engineering Judgment:
- Uncovered and resolved an
InvalidParameterCombinationASG failure wheret3.mediumwas rejected by AWS Free Tier policies. - Sized Ollama's memory requests down from
1Gi(carried over from unconstrained local Docker environments) to512Mirequest and1.5Gilimit. This ensured system pods (AWS VPC CNI, EBS CSI Driver, CoreDNS, kube-proxy) and application workloads coexisted on 2GB nodes without OOM-kills or scheduling gridlocks.
- Uncovered and resolved an
- Context: The AWS EBS CSI Controller requires IAM permissions (
ec2:CreateVolume,ec2:AttachVolume, etc.) to dynamically provision EBS volumes for the Ollama model store. - Decision: Implemented an IAM OIDC Identity Provider with a dedicated IAM role bound directly to the Kubernetes ServiceAccount (
system:serviceaccount:kube-system:ebs-csi-controller-sa) via IRSA, instead of granting the EC2 Node Instance Role broad AWS administrator permissions. - Engineering Judgment:
- EC2 node-role permissions grant broad access to every pod running on that node, violating the principle of least privilege.
- IRSA injects temporary STS credentials specifically into the CSI controller pod via Kubernetes service account token projection, isolating AWS API access exclusively to the storage control plane.
- Context: The Ollama LLM backend utilizes persistent EBS
gp3storage (ollama-pvc) to persist model weights across restarts. - Decision: Configured the
StorageClasswithvolumeBindingMode: WaitForFirstConsumerand analyzed cross-AZ volume affinity constraints. - Engineering Judgment:
- AWS EBS volumes are physical block devices bound to a single Availability Zone (e.g.
ap-south-1a). When a multi-AZ node group attempts to reschedule a stateful pod to a node inap-south-1b, scheduling fails due toPersistentVolume node affinitymismatch. - In production systems, this tradeoff is addressed by either:
- Pinning stateful worker node groups to a single AZ for lightweight workloads.
- Utilizing multi-AZ shared network file storage (AWS EFS CSI Driver) for read-heavy model artifacts.
- Deploying StatefulSets with topology-aware pod scheduling constraints.
- AWS EBS volumes are physical block devices bound to a single Availability Zone (e.g.
- Context: Cloud infrastructure left running outside active development sessions causes budget exhaustion.
- Decision: Enforced an automated, dependency-aware teardown workflow executed immediately following verification runs.
- Engineering Judgment:
- Deletion Ordering: Kubernetes
Serviceobjects oftype: LoadBalancermust be deleted viakubectlprior to runningterraform destroy. If Terraform deletes the VPC/subnets before Kubernetes deprovisions the ELB, the AWS Load Balancer becomes orphaned in AWS and continues billing silently. - Orphan Sweeps: Integrated automated AWS CLI queries into teardown procedures to confirm zero unattached EBS volumes (
status=available) or dangling ELBs remained.
- Deletion Ordering: Kubernetes
| Day | Focus Domain | Key Accomplishments | Core Architectural Pattern | Detailed Log Link |
|---|---|---|---|---|
| Day 1 | Local Baseline and Guardrails | AWS billing alerts configured, non-root IAM admin role, toolchain verification, 2-node kind cluster setup. |
Cost-constrained local sandbox environment | Day 1 Log |
| Day 2 | Containerized LLM Serving | Ollama containerized in kind, 5Gi PVC for model weights, initContainer bootstrap pattern for qwen2.5:0.5b. |
Decoupled model storage and lifecycle isolation | Day 2 Log |
| Day 3 | Microservice Discovery | FastAPI integration, Kubernetes ClusterIP Service, CoreDNS FQDN internal routing. | Dynamic DNS-based microservice discovery | Day 3 Log |
| Day 4 | CI/CD and Image Registry | GitHub Actions workflow, Ruff static linting, Pytest test suite, immutable SHA image tagging to GHCR. | Immutable artifact pipeline and GitOps boundary | Day 4 Log |
| Day 5 | Observability Baseline | kube-prometheus-stack via Helm, /metrics instrumentation, ServiceMonitor CRD, Grafana Golden Signals. |
Kubernetes-native declarative metrics scraping | Day 5 Log |
| Day 6 | Alerting and Incident SRE | Alertmanager enabled, PrometheusRule CRDs (HighErrorRate, PodNotReady), live fault injection validation. |
SRE alert debounce windows and dashboard-as-code | Day 6 Log |
| Day 7 | Cloud Migration (EC2) | Terraform IaC for EC2 t3.micro, Docker bridge network llm-net, IP-scoped security groups, GHCR pull. |
Production IaC scaffolding and zero-leakage teardown | Day 7 Log |
| Day 8 | Managed EKS and IRSA | EKS v1.35, managed node groups, OIDC / IRSA for EBS CSI driver, dynamic gp3 StorageClass, resource tuning. | Least-privilege IRSA and portable K8s manifests | Day 8 Log |
| Day 9 | Cloud Observability and ELB | EKS re-provisioning via IaC, public AWS Classic ELB, live alert validation, multi-AZ EBS analysis. | End-to-end cloud reproducibility and SRE validation | Day 9 Log |
| Day 10 | CD Loop and Consolidation | Closed CD loop with automated GitOps manifest patch workflow, curated screenshots, executive documentation. | Closed-loop GitOps deployment and portfolio consolidation | Day 10 Log |
- AWS Billing Guardrails configured with alert caps
- AWS CLI v2 configured with non-root IAM user (
yash-IAM-Admin) - Local toolchain verified: Docker v29, kubectl v1.36, kind v0.22, Terraform v1.15
- Local 2-node Kubernetes cluster created using
kind(devops-ai-cluster) - Ollama local model serving verified
- Non-root AWS IAM Access: Used a dedicated IAM administrator identity instead of the AWS root account to reduce operational and security risk.
- Local Kubernetes via kind: Chosen to provide a reproducible multi-node Kubernetes environment without incurring cloud infrastructure costs during development.
- Cost Guardrails: AWS billing alerts were configured before beginning cloud infrastructure work to prevent unexpected resource consumption.
- Ollama containerized and deployed into the local
kindcluster - Dedicated Kubernetes namespace configured for LLM serving (
llm-serving) - Persistent Volume Claim (
ollama-pvc) created (5Gi) - Ollama model storage mounted at
/root/.ollama - Init container implemented for automatic model bootstrap
qwen2.5:0.5bmodel configured for automatic download- Ollama serving container verified
- CPU-based Serving: Retained Ollama GGUF quantized models on CPU to avoid GPU overhead and unnecessary infrastructure costs in local
kindclusters. - Persistent Storage (
ollama-pvc): Mounted a 5Gi PVC to/root/.ollamato decouple model-weight storage from pod lifecycles, eliminating unnecessary model re-downloads after pod restarts. - Init Container Bootstrap Pattern: Introduced an
initContainernamedmodel-pullerto verify Ollama readiness usingollama listand fetchqwen2.5:0.5bmodel weights before the primary serving container starts. - Container Lifecycle Separation: Model initialization is isolated from the serving process, allowing the main Ollama container to start only after the required model artifacts are available.
- FastAPI application integrated with the Ollama backend
- Kubernetes Service created for Ollama
- FastAPI configured to communicate with Ollama through Kubernetes DNS
- Hardcoded Pod IP addressing eliminated
- Internal service communication established using the Kubernetes FQDN
- Kubernetes DNS-Based Service Discovery: The FastAPI layer communicates with the Ollama backend using the Kubernetes CoreDNS domain:
http://ollama-service.llm-serving.svc.cluster.local:11434 - Dynamic IP Assignment: Pod IPs and Service ClusterIPs are dynamic. Hardcoding IP addresses makes the application fragile whenever pods are rescheduled or services are recreated.
- Namespace-Scoped Discovery: Kubernetes Fully Qualified Domain Names follow the standard format:
<service-name>.<namespace>.svc.cluster.localThis allows services to communicate reliably across namespaces without depending on the underlying cluster topology. - Decoupled Architecture: FastAPI depends on the stable Kubernetes Service abstraction rather than individual Ollama pods. This allows the Ollama backend to scale horizontally without requiring application-level configuration changes.
Kubernetes Cluster
┌─────────────────────────────────────────────────────────────┐
│ │
│ ┌──────────────────┐ │
│ │ FastAPI Pod │ │
│ │ │ │
│ │ API / Inference │ │
│ └────────┬─────────┘ │
│ │ │
│ │ Kubernetes DNS │
│ │ ollama-service.llm-serving.svc.cluster.local │
│ ▼ │
│ ┌──────────────────┐ │
│ │ Ollama Service │ │
│ │ │ │
│ │ ClusterIP:11434 │ │
│ └────────┬─────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ Ollama Pod │ │
│ │ │ │
│ │ qwen2.5:0.5b │ │
│ └──────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
- Automated CI/CD workflow created using GitHub Actions (
.github/workflows/ci.yaml) - Code quality and linting automated using
ruff - Automated unit test suite implemented using
pytest - Pydantic schema validation verified
/healthprobe logic verified- Container image builds automated
- Container images published to GitHub Container Registry (GHCR)
- Immutable image tagging strategy implemented using Git commit SHAs (
${{ github.sha }})
- Static Code Analysis (
ruff): Integrated high-performance Python linting into the CI pipeline to enforce code-quality standards and maintain consistent import ordering before container image assembly. - Automated Verification (
pytest): Unit tests execute automatically on every push and pull request targetingmain. This provides an early validation layer for API behavior, Pydantic schemas, and health-check endpoints before artifacts are built. - Immutable Artifact Strategy: Container images are tagged using the Git commit SHA (
${{ github.sha }}), creating a deterministic relationship between source code and the resulting artifact. This avoids the ambiguity associated with mutable tags such as:latest. - GitHub Container Registry (GHCR): GHCR is used as the centralized container artifact registry, providing a persistent location for versioned images that can later be consumed by Kubernetes or a GitOps deployment controller.
- Deliberate Deployment Boundary: The CI pipeline intentionally terminates after publishing the container image and updating the deployment manifest.
Git Repository
│
│ Push / Pull Request
▼
┌─────────────────────┐
│ GitHub Actions │
│ CI Runner │
└──────────┬──────────┘
│
┌────────────┼────────────┐
│ │ │
▼ ▼ ▼
┌────────┐ ┌────────┐ ┌─────────────┐
│ Ruff │ │ Pytest │ │ Pydantic / │
│ Linting│ │ Tests │ │ Health Check│
└────┬───┘ └────┬───┘ └──────┬──────┘
│ │ │
└────────────┼──────────────┘
│
Validation Pass
│
▼
┌─────────────────────┐
│ Docker Image Build │
└──────────┬──────────┘
│
│ Tag: <github-sha>
▼
┌─────────────────────┐
│ GHCR │
│ GitHub Container │
│ Registry │
└──────────┬──────────┘
│
│ GitOps Sync
▼
┌─────────────────────┐
│ k8s Manifest Update │
└─────────────────────┘
- FastAPI application instrumented using
prometheus-fastapi-instrumentator /metricsendpoint exposed on the API servicekube-prometheus-stackdeployed via Helm into a dedicatedmonitoringnamespaceServiceMonitor(llm-api-servicemonitor) configured to scrape the API service- Prometheus target verified as
UPwith successful scrape health - Grafana dashboard built with three golden-signal panels: Request rate, P95 latency, Error rate
- Dashboard exported as JSON and version-controlled (
monitoring/dashboards/golden-signals.json) - Load-tested end-to-end pipeline to validate live metric flow
kube-prometheus-stackover hand-rolled manifests: Used the community Helm chart (Prometheus + Grafana + Alertmanager bundled) rather than deploying each component manually, mirroring standard enterprise operations.- ServiceMonitor as the scrape-config abstraction: Declaratively defines metric endpoints reconciled by the Prometheus Operator.
- Golden Signals over exhaustive metrics: Dashboard limits scope to request rate, latency (p95), and error rate—the three signals directly tied to service health.
-
Prometheus silently ignoring the ServiceMonitor (label selector mismatch)
- Symptom:
ServiceMonitorexisted, but Prometheus's Service Discovery page showed nothing. - Root cause:
kube-prometheus-stack's Prometheus CR only watchesServiceMonitorscarrying arelease: prometheus-stacklabel by default (spec.serviceMonitorSelector.matchLabels). The customServiceMonitorlacked this label. - Fix: Added
labels: { release: prometheus-stack }to theServiceMonitormetadata and re-applied.
- Symptom:
-
ServiceMonitor discovered, but "0/0 No targets"
- Symptom: Label fix resolved discovery, but the scrape pool showed zero targets.
- Root cause: The underlying
Service(llm-api-service) had zero running pods (0/0replicas). - Fix: Scaled deployments to 1 replica (
kubectl scale deployment <name> -n llm-serving --replicas=1).
-
Target
UPin discovery, but scraping failed with404 Not Found- Symptom: Prometheus reached the pod over the network, but
/metricsreturned 404. - Root cause: The running pod was serving an older image built before Prometheus instrumentation was added to
main.py. - Fix: Rebuilt image, reloaded into
kind, bumped manifest tag, and re-applied.
- Symptom: Prometheus reached the pod over the network, but
-
Port drift between local YAML, live cluster state, and the container
- Symptom:
kubectl port-forwardfailed with "Service does not have a service port 8001." - Root cause:
api-service.yamlhad been locally edited toport: 8001but never re-applied, while Dockerfile and Service were on8000. - Fix: Reverted Service manifest to
8000, re-applied, and verified parity.
- Symptom:
# Request Rate
sum(rate(http_requests_total{job="llm-api-service"}[1m]))
# P95 Latency
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="llm-api-service"}[1m])) by (le))
# Error Rate
sum(rate(http_requests_total{job="llm-api-service", status=~"4..|5.."}[1m])) or vector(0)
- Alertmanager enabled via Helm upgrade
- Verified Prometheus to Alertmanager delivery path via
alerting.alertmanagersconfig - Three
PrometheusRulealerts authored and deployed (k8s/api-alerts.yaml):HighErrorRate— fires when 5xx ratio exceeds 5% for 2+ minutesPodNotReady— fires when any pod inllm-servingis not-ready for 1+ minuteHighLatencyP95— fires when p95 latency exceeds 2s for 5+ minutes
- Forced a real outage (Ollama scaled to zero) to validate the full alert lifecycle
- Confirmed alert transition:
Inactive->Pending->Firing-> resolved - Confirmed alert delivery into Alertmanager's UI
- Diagnosed and fixed Grafana dashboard persistence bug via PVC configuration
Threshold breached; alert enters the 2-minute debounce window (for: 2m) to prevent transient false alarms:

Failure condition sustained for over 2 minutes; alert transitions to FIRING and routes to Alertmanager:

PrometheusRuleCRD: Alert rules live as version-controlled Kubernetes objects reconciled automatically by the Prometheus Operator.for:duration debouncing: Every alert requires its condition to hold for a sustained window (1-5 minutes) before firing, preventing alert fatigue from transient spikes.- Real outage validation: Validated alerts by taking Ollama offline and observing real HTTP 503 error rates from the FastAPI gateway.
- Dashboard-as-Code: Grafana dashboards versioned in JSON to prevent data loss across pod lifecycles.
-
Alertmanager pods not running
- Root cause:
alertmanager.enabled: falsewas set in base values. - Fix: Upgraded Helm release with
--set alertmanager.enabled=trueand custom resource bounds.
- Root cause:
-
Control-plane noise alerts in kind
- Symptom: 10 alerts in
kube-systemfired continuously (etcdInsufficientMembers, etc.). - Root cause:
kinddoes not expose cloud control-plane metrics. Identified as expected local noise.
- Symptom: 10 alerts in
-
Grafana dashboard lost after Helm upgrade
- Root cause: Grafana default uses ephemeral SQLite DB.
- Fix: Re-imported from
golden-signals.jsonand enabled persistent storage via--set grafana.persistence.enabled=true --set grafana.persistence.size=1Gi.
- Provisioned real AWS infrastructure using Terraform (
terraform/):- Single
t3.microEC2 instance (Ubuntu, 25Gi gp3 root volume) - Dedicated security group scoped to the operator's IP (SSH + port 8000)
- SSH key pair generated and managed via Terraform
- Single
- Installed Docker and deployed containers on EC2
- Configured custom Docker bridge network (
llm-net) for container-name DNS resolution - Pulled API image from GHCR
- Verified
/healthand/generateendpoints over the public EC2 IPv4 address - Executed full teardown via
terraform destroywith zero residual billing
- EC2 + Docker staging step: Scoped initial cloud migration to plain Docker on EC2 to validate cloud networking, security groups, and registry authentication before incurring managed EKS control plane costs.
- Strict security group scoping: SSH (22) and API (8000) locked to the operator's IP using
data.httplookup at apply time. - Docker bridge network: Enabled internal hostname discovery (
http://ollama:11434), mirroring Kubernetes CoreDNS patterns. - Aggressive teardown discipline: Infrastructure destroyed immediately after verification to stay within cloud budget.
-
GHCR pull denied
- Root cause: Container image repository name was
llm-devops, notllm-api. - Fix: Corrected image path to
ghcr.io/yash55-max/llm-devops:latest.
- Root cause: Container image repository name was
-
SSH session terminated mid-verification
- Root cause:
terraform destroytriggered from local terminal destroyed instance immediately upon verification completion. Expected lifecycle behavior.
- Root cause:
- Extended Terraform configuration to provision a production EKS cluster:
- EKS control plane (Kubernetes 1.35)
- Managed node group (2x
t3.small, Free Tier eligible) - IAM OIDC Provider for IAM Roles for Service Accounts (IRSA)
- AWS EBS CSI Driver addon with dedicated IRSA role
- Reapplied existing Kubernetes manifests (
k8s/) with minimal changes - Created dynamic
StorageClass(ebs-gp3) set as default - Right-sized Ollama resource requests/limits for 2GB node capacity
- Created GHCR image pull secrets for private container registry access
- Verified full request flow:
curl-> API pod -> Kubernetes DNS -> Ollama pod -> response - Teardown verified via
terraform destroyand manual EBS orphan checks
- IRSA over node-role IAM: The EBS CSI controller calls AWS APIs using a dedicated IAM role bound to its service account via OIDC, enforcing least privilege.
- Manifest Portability: Kubernetes manifests written for
kinddeployed onto AWS EKS with zero structural rewrites, proving cloud-native portability. - Resource Right-Sizing: Tuned container requests to operate reliably on resource-constrained cloud nodes.
-
EKS cluster created with end-of-support Kubernetes version
- Root cause: Hardcoded
version = "1.30"had aged out of AWS standard support. - Fix: Re-provisioned with
version = "1.35"(standard support through March 2027).
- Root cause: Hardcoded
-
Node group launch failure (
InvalidParameterCombination)- Symptom: Node group stuck in
CREATINGfor 30+ minutes without launching instances. - Root cause: Configured
t3.mediuminstance type was not Free Tier eligible on this account. - Fix: Switched to
t3.small(verified Free Tier eligible); node group came up in under 2 minutes.
- Symptom: Node group stuck in
-
EBS CSI controller pods CrashLoopBackOff with HTTP 500
- Root cause: CSI controller pod lacked dedicated IAM role via IRSA.
- Fix: Added
aws_iam_openid_connect_provider, createdaws_iam_role.ebs_csi_irsatrusted bysystem:serviceaccount:kube-system:ebs-csi-controller-sa, and attached to the addon.
-
PVC stuck Pending
- Root cause: EKS provides no default StorageClass out of the box.
- Fix: Deployed
k8s/storageclass.yaml(ebs-gp3provisioner withvolumeBindingMode: WaitForFirstConsumer).
-
Ollama pod unschedulable (Insufficient Memory)
- Root cause: Inherited
1Gimemory request from unconstrained localkind. - Fix: Sized down request to
512Mi(limit1.5Gi), pod scheduled immediately.
- Root cause: Inherited
-
API pod ImagePullBackOff
- Root cause: Missing registry prefix and missing private registry pull secret.
- Fix: Set image path to
ghcr.io/yash55-max/llm-devops:<sha>and addedimagePullSecrets: [name: ghcr-secret].
- Recreated full EKS cluster from scratch via Terraform, validating complete IaC reproducibility
- Deployed
kube-prometheus-stackvia Helm with pre-sized resource limits - Reapplied
ServiceMonitorandPrometheusRulemanifests on live cloud cluster - Re-imported Golden Signals Grafana dashboard from version-controlled JSON
- Exposed FastAPI serving layer via AWS Classic Load Balancer (
type: LoadBalancer) - Verified end-to-end inference over public ELB DNS endpoint
- Triggered real
HighErrorRatealert over public endpoint and verified Prometheus/Grafana state transition - Diagnosed multi-AZ EBS scheduling constraint during pod recreation
- Clean teardown: Deprovisioned LoadBalancer Service first, followed by
terraform destroyand orphan verification
- IaC Reproducibility: Tested full cluster rebuild from empty state to prove zero drift.
- Public Ingress via LoadBalancer Service: Integrated AWS Classic Load Balancer for public inference traffic.
- Teardown Sequencing: Deleting the Kubernetes
LoadBalancerService prior toterraform destroyensures the cloud-provider controller deprovisions the underlying ELB cleanly, avoiding orphaned cloud resources.
-
Grafana pod stuck 2/3 Running (SQLITE_BUSY / timeouts)
- Root cause: Plugin initializations during startup throttled by default
200mCPU limit. - Fix: Upgraded Helm values to
grafana.resources.limits.cpu: 750m; pod reached3/3 Running.
- Root cause: Plugin initializations during startup throttled by default
-
ELB DNS unresolvable immediately after provisioning
- Symptom:
curlreturnedNXDOMAINfor several minutes afterkubectl get svcshowed external hostname. - Root cause: Standard DNS propagation delay for newly registered AWS ELB records.
- Resolution: Verified AWS ELB state was active via AWS CLI and waited for DNS propagation (~10 minutes).
- Symptom:
-
Multi-AZ EBS Volume Affinity Scheduling Failure
- Symptom: Ollama pod stayed
PendingwithPersistentVolume's node affinity mismatch. - Root cause: EBS volume was provisioned in
ap-south-1a, while the worker node with free memory was inap-south-1b. EBS volumes cannot attach across Availability Zones. - Takeaway: Single-replica stateful workloads on multi-AZ node groups require AZ pinning, StatefulSet topology spread constraints, or shared multi-AZ storage (EFS).
- Symptom: Ollama pod stayed
curl http://a8931d61cc54f4205beb65058c6ebfc5-1730394859.ap-south-1.elb.amazonaws.com:8000/health
# {"status":"ok","ollama_host":"http://ollama-service.llm-serving.svc.cluster.local:11434"}
curl -X POST http://a8931d61cc54f4205beb65058c6ebfc5-1730394859.ap-south-1.elb.amazonaws.com:8000/generate -H "Content-Type: application/json" -d '{"prompt": "Explain load balancers in one sentence."}'
# {"model":"qwen2.5:0.5b","response":"A load balancer is a type of network service
# that distributes traffic evenly across multiple servers or instances, enabling the
# server or instance to handle an increasing number of concurrent connections while
# minimizing the impact of a single server or instance failure.","done":true}- Automated GitOps Continuous Delivery (CD) loop engineered within GitHub Actions (
.github/workflows/ci.yaml):- Added dedicated
deployworkflow job executing on pushes tomain - Dynamically extracts the published container image tag (Git commit SHA) from the build step
- Automatically updates the deployment manifest (
k8s/api-deployment.yaml) with the exact immutable image reference - Automatically commits and pushes the updated manifest back to the repository using
[skip ci]to prevent workflow loops
- Added dedicated
- Curated and centralized top production verification artifacts into a dedicated
docs/screenshots/repository directory:01_eks_elb_public_endpoint_verification.png— Public Load Balancer live endpoint verification02_eks_golden_signals_dashboard.png— Full Grafana Golden Signals observability on AWS EKS03_eks_prometheus_alert_firing.png— Live Prometheus incident alerting lifecycle
- Authored professional top-level platform documentation (
README.md):- Tech stack badges and executive project summary
- Production architecture and service topology diagrams
- Dedicated "Key Engineering Decisions" section covering Free Tier capacity planning, IRSA least privilege, EBS multi-AZ tradeoffs, and teardown discipline
- 10-day milestone summary table with direct links to all daily technical and debugging logs
- Zero-emoji formatting strictly maintained across the entire portfolio
- GitOps Manifest Synchronization over Imperative Cluster Mutation: Rather than executing imperative
kubectl set imagecommands against a remote cluster from CI runners (which requires storing long-lived cloud credentials in GitHub Secrets and creates configuration drift between Git and the cluster), the CD loop declaratively updatesk8s/api-deployment.yamlin Git. This makes the Git repository the absolute single source of truth for the desired cluster state. - Workflow Loop Prevention (
[skip ci]): Automated commits generated by GitHub Actions bots include the[skip ci]commit directive to prevent recursive CI triggering while preserving full commit history and audit trails. - Documentation as an SRE Artifact: Designed the README to serve as an operational runbook and architectural defense document, explicitly articulating failure modes, root cause analyses, and trade-off rationales encountered throughout the 10-day engineering sprint.
Developer / Feature Branch
│
│ Git Push / Pull Request
▼
┌──────────────────────┐
│ GitHub Actions CI/CD │
└──────────┬───────────┘
│
├─► Job 1: Lint & Unit Tests (Ruff, Pytest)
│
├─► Job 2: Build & Push Container (GHCR with Git SHA)
│
└─► Job 3: GitOps CD Sync
│
├─► Extract Image Tag (SHA)
├─► Update k8s/api-deployment.yaml
└─► Git Commit & Push to main [skip ci]
│
▼
┌──────────────────────┐
│ Git Repository │
│ (Single Source Truth)│
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ AWS EKS Cluster │
│ (llm-serving / apps) │
└──────────────────────┘
llm-devops/
├── .github/
│ └── workflows/
│ └── ci.yaml # Automated lint, test, build, push, & manifest
├── app/
│ ├── Dockerfile # Optimized Python 3.11 container image
│ ├── main.py # FastAPI inference gateway with Prometheus metrics
│ ├── requirements.txt # App dependencies (FastAPI, httpx, instrumentator)
│ └── tests/ # Tests folder
│ ├── __init__.py
│ └── test_main.py # Automated pytest suite for API & health logic
├── docs/
│ └── screenshots/ # Curated production verification highlights
│ ├── 01_eks_elb_public_endpoint_verification.png
│ ├── 02_eks_golden_signals_dashboard.png
│ └── 03_eks_prometheus_alert_firing.png
├── k8s/
│ ├── api-alerts.yaml # PrometheusRule alerts (HighErrorRate, PodNotReady)
│ ├── api-deployment.yaml # FastAPI Deployment with GHCR image & pull secrets
│ ├── api-service.yaml # LoadBalancer / ClusterIP Service for API ingress
│ ├── api-servicemonitor.yaml # ServiceMonitor CRD for Prometheus Operator
│ ├── ollama-deployment.yaml # Ollama LLM pod with initContainer model bootstrap
│ ├── ollama-pvc.yaml # PersistentVolumeClaim for model storage
│ ├── ollama-service.yaml # Internal ClusterIP Service for Ollama
│ └── storageclass.yaml # AWS EBS gp3 StorageClass (WaitForFirstConsumer)
├── monitoring/
│ └── dashboards/
│ ├── golden-signals.json # Exported Grafana Golden Signals dashboard (IaC)
│ └── *.png # Raw dashboard & alert verification captures
├── terraform/
│ ├── main.tf # VPC, EKS Cluster, Node Group, IRSA, & EBS CSI Addon
│ ├── outputs.tf # EKS cluster endpoint, CA data, & security group IDs
│ ├── variables.tf # Region, cluster name, and instance type variables
│ └── screenshots/ # Terraform & EC2/EKS verification captures
├── kind-config.yaml # 2-node local Kubernetes cluster configuration
└── README.md # Top-level platform documentation
- AWS CLI v2 configured with appropriate IAM permissions
- Terraform v1.5+
- kubectl v1.30+
- Helm v3+
- Docker v25+
cd terraform
terraform init
terraform apply -auto-approveaws eks update-kubeconfig --region ap-south-1 --name devops-ai-eks
kubectl get nodeskubectl apply -f k8s/storageclass.yaml
kubectl create namespace llm-serving
kubectl apply -f k8s/ollama-pvc.yaml# Create GHCR registry secret for private image pulls
kubectl create secret docker-registry ghcr-secret --docker-server=ghcr.io --docker-username=<GITHUB_USERNAME> --docker-password=<GITHUB_PAT> -n llm-serving
# Deploy Ollama and FastAPI
kubectl apply -f k8s/ollama-deployment.yaml
kubectl apply -f k8s/ollama-service.yaml
kubectl apply -f k8s/api-deployment.yaml
kubectl apply -f k8s/api-service.yamlhelm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus-stack prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace --set alertmanager.enabled=true --set grafana.persistence.enabled=true --set grafana.persistence.size=1Gi --set grafana.resources.limits.cpu=750m
# Apply ServiceMonitor and Alerting Rules
kubectl apply -f k8s/api-servicemonitor.yaml
kubectl apply -f k8s/api-alerts.yaml# Retrieve Load Balancer public hostname
kubectl get svc llm-api-service -n llm-serving -w
# Test inference
curl -X POST http://<ELB_HOSTNAME>:8000/generate -H "Content-Type: application/json" -d '{"prompt": "What is Site Reliability Engineering?"}'# 1. Delete LoadBalancer Service first to deprovision AWS ELB cleanly
kubectl delete svc llm-api-service -n llm-serving
# 2. Destroy Terraform infrastructure
cd terraform
terraform destroy -auto-approve
# 3. Verify zero orphaned AWS resources
aws ec2 describe-volumes --region ap-south-1 --filters "Name=status,Values=available" --query 'Volumes[].VolumeId'
aws elb describe-load-balancers --region ap-south-1 --query 'LoadBalancerDescriptions[].LoadBalancerName'











