AI Observability Platform

GitOps mono-repo for a production-ready AI observability platform on Kubernetes.

Overview

This repo drives all Kubernetes manifests via ArgoCD GitOps. The core design principle is zero-touch observability — any AI app deployed in the apps namespace is automatically instrumented and traced with no code changes required.

Two enforcement layers

1. OTel Operator Mutating Webhook — namespace-level annotations trigger automatic SDK injection (Python/Node/Java/.NET) into every pod at admission time. Apps emit traces, metrics, and logs to the Collector without any manual exporter setup.

2. AI Gateway as mandatory LLM egress — Cilium FQDN NetworkPolicy blocks direct egress to LLM provider APIs. All LLM calls must traverse the AI Gateway (LiteLLM Proxy), which auto-captures model, tokens, latency, cost, and emits OTel spans — invisible to the app.

Architecture

                 ┌─────────────┐
                 │  Grafana UI │
                 └──────┬──────┘
        ┌───────────────┼──────────────┐
        ▼               ▼              ▼
  Prometheus          Loki           Tempo
        ▲               ▲              ▲
        └───────────────┼──────────────┘
                        │ (OTel Collector)
              ┌─────────┴───────────┐
              │  OTel Collector     │
              │  (DaemonSet/Deploy) │
              └─────────┬───────────┘
                        │ OTLP
        ┌───────────────┼───────────────┐
        ▼               ▼               ▼
  App Pods         AI Gateway        Infra
  (auto-instr)   (LiteLLM Proxy)   (node/kube metrics)
                        │
              LLM Providers (OpenAI, Anthropic, ...)
              [Cilium FQDN policy — apps namespace blocked]

Tech Stack

Component	Tool
Kubernetes (local)	kind
Kubernetes (cloud)	TBD (EKS/GKE)
CNI	Cilium
GitOps	ArgoCD
OTel Operator	opentelemetry-operator
OTel Collector	OpenTelemetryCollector CR
AI Gateway	LiteLLM Proxy
Metrics	kube-prometheus-stack (Prometheus + Grafana)
Logs	Loki
Traces	Tempo
Secrets	Sealed Secrets
CI	GitHub Actions

Repo Structure

.
├── clusters/
│   ├── local/          # kind cluster — dev environment
│   └── cloud/          # cloud cluster — production
├── infra/
│   ├── argocd/         # ArgoCD installation + App-of-Apps
│   ├── cilium/         # Cilium CNI ArgoCD Application
│   ├── otel-operator/  # OTel Operator + Collector + Instrumentation CRs
│   ├── observability/  # kube-prometheus-stack, Loki, Tempo, Grafana
│   ├── ai-gateway/     # LiteLLM Proxy ArgoCD Application + config
│   ├── network-policies/ # Cilium FQDN policies (LLM egress enforcement)
│   └── sealed-secrets/ # Sealed Secrets controller
├── apps/
│   └── demo-ai-app/    # Sample instrumented Python FastAPI app
└── .github/
    └── workflows/      # GitHub Actions CI

Getting Started

Prerequisites

• kubectl >= 1.27 : https://kubernetes.io/docs/tasks/tools/install-kubectl-linux
• helm >= 3.12 : https://helm.sh/docs/intro/install
• kind (for local dev) : https://kind.sigs.k8s.io/docs/user/quick-start/#installation
• cilium CLI (for CNI debugging) : https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-the-cilium-cli
• argocd CLI
• kubeseal (for sealed secrets)

Local Development

Manual setup steps for the local kind cluster. In production, all of this will be automated via GitOps.

# 1. Create kind cluster
kind create cluster --config clusters/local/kind-config.yaml

# 2. Install Cilium CNI
kubectl create namespace cilium
cilium install --version 1.19.4 --namespace cilium --wait
cilium status

# 3. Bootstrap ArgoCD -  https://argo-cd.readthedocs.io/en/latest/try_argo_cd_locally
kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# 4. Apply the local bootstrap manifests (root Application + ApplicationSet)
kubectl apply -k clusters/local

# 5. Watch everything reconcile
argocd app list
kubectl get pods -A

Namespaces

Namespace	Purpose
argocd	ArgoCD GitOps engine
otel-system	OTel Operator + Collectors
ai-gateway	LiteLLM Proxy
observability	Prometheus, Grafana, Loki, Tempo
apps	Instrumented AI workloads

Contributing

See CONTRIBUTING.md.