feat(litellm): deploy LiteLLM Proxy for OpenAI + Anthropic routing - #2150
Open
jfroy wants to merge 23 commits into
Open
feat(litellm): deploy LiteLLM Proxy for OpenAI + Anthropic routing#2150jfroy wants to merge 23 commits into
jfroy wants to merge 23 commits into
Conversation
Used by the subagent-driven-development skill for per-plan progress ledgers and review packages; git-ignored scratch, not repo content. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Deployment's init-db copy inherits runAsNonRoot from pod-level podSecurityContext, but the chart doesn't apply that to the migration Job's pod, so the shared &initDb anchor needs it set explicitly.
📝 WalkthroughWalkthroughChangesLiteLLM deployment
Pinchflat cleanup
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant HTTPRoute
participant LiteLLM
participant PostgreSQL
participant Prometheus
participant Grafana
Client->>HTTPRoute: send request to litellm.kantai.xyz
HTTPRoute->>LiteLLM: route request to port 4000
LiteLLM->>PostgreSQL: use configured database
Prometheus->>LiteLLM: scrape /metrics/
Grafana->>Prometheus: load dashboard data
Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
konflate — summaryNote +42 added · 0 changed · −0 removed — 42 resources · 4 apps Image changes
konflate · rendered |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kubernetes/apps/default/litellm/app/helmrelease.yaml`:
- Around line 90-102: Add an OpenAI model entry to the model_list alongside the
existing Anthropic routes, configuring its OpenAI model identifier and
referencing OPENAI_API_KEY from the application secret via the established
environment-variable pattern.
In `@kubernetes/apps/default/litellm/ks.yaml`:
- Around line 7-16: Add a dependsOn entry to the Kustomization spec for the
cnpg-pg18vc Kustomization in the cnpg-system namespace, while preserving the
existing path, sourceRef, pruning, and timing settings.
- Around line 7-19: Add the VolSync component reference to the Flux
Kustomization spec alongside the existing source configuration, and include the
required PVC backup configuration so the application receives the daily Kopia
backup to Cloudflare R2. Preserve the existing substitution and reconciliation
settings.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 14ed881a-4ca7-4409-8235-df0b42a79d4c
📒 Files selected for processing (12)
.gitignorekubernetes/apps/default/kustomization.yamlkubernetes/apps/default/litellm/app/externalsecret.yamlkubernetes/apps/default/litellm/app/grafanadashboard.yamlkubernetes/apps/default/litellm/app/helmrelease.yamlkubernetes/apps/default/litellm/app/httproute.yamlkubernetes/apps/default/litellm/app/kustomization.yamlkubernetes/apps/default/litellm/app/objectbucketclaim.yamlkubernetes/apps/default/litellm/app/ocirepository.yamlkubernetes/apps/default/litellm/app/prometheusrule.yamlkubernetes/apps/default/litellm/app/servicemonitor.yamlkubernetes/apps/default/litellm/ks.yaml
Comment on lines
+7
to
+16
| spec: | ||
| path: ./kubernetes/apps/default/litellm/app | ||
| prune: true | ||
| sourceRef: | ||
| kind: GitRepository | ||
| name: flux-system | ||
| namespace: flux-system | ||
| interval: 1h | ||
| retryInterval: 2m | ||
| timeout: 10m |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Declare the CNPG dependency.
The HelmRelease initializes and connects to pg18vc-rw.database.svc.cluster.local, but this Kustomization does not wait for the CNPG Kustomization. Add dependsOn for cnpg-pg18vc in namespace cnpg-system.
As per coding guidelines, “For Postgres apps in ks.yaml files, add dependsOn: cnpg-pg18vc in cnpg-system namespace.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kubernetes/apps/default/litellm/ks.yaml` around lines 7 - 16, Add a dependsOn
entry to the Kustomization spec for the cnpg-pg18vc Kustomization in the
cnpg-system namespace, while preserving the existing path, sourceRef, pruning,
and timing settings.
Source: Coding guidelines
Comment on lines
+7
to
+19
| spec: | ||
| path: ./kubernetes/apps/default/litellm/app | ||
| prune: true | ||
| sourceRef: | ||
| kind: GitRepository | ||
| name: flux-system | ||
| namespace: flux-system | ||
| interval: 1h | ||
| retryInterval: 2m | ||
| timeout: 10m | ||
| postBuild: | ||
| substitute: | ||
| APP: *app |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Add the VolSync component.
This Kustomization has no components/volsync reference. Flux will not configure the required daily Kopia backup for this application. Add the VolSync component and its required PVC backup configuration.
As per coding guidelines, “In ks.yaml Flux Kustomization files, reference components/volsync to wire up daily Kopia backups to Cloudflare R2.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kubernetes/apps/default/litellm/ks.yaml` around lines 7 - 19, Add the VolSync
component reference to the Flux Kustomization spec alongside the existing source
configuration, and include the required PVC backup configuration so the
application receives the daily Kopia backup to Cloudflare R2. Preserve the
existing substitution and reconciliation settings.
Source: Coding guidelines
Matches every other namespace.yaml in the repo — the NamespaceTransformer overwrites metadata.name regardless, so the literal value here was a second, non-authoritative source of truth for the namespace name.
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
kubernetes/apps/litellm/litellm/app/helmrelease.yaml (1)
13-14: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winPin the LiteLLM Deployment image digest.
This HelmRelease does not override
image.repositoryorimage.tag, so the chart uses its default repository and tag from.Chart.AppVersion.postgres-initis digest-pinned, but the primary LiteLLM container is not. Set both values using Renovate-managed tag-plus-digest image references.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kubernetes/apps/litellm/litellm/app/helmrelease.yaml` around lines 13 - 14, Update the HelmRelease values for the primary LiteLLM container by explicitly setting image.repository and image.tag to the Renovate-managed tag-plus-digest reference format, matching the existing postgres-init pinning convention. Keep replicaCount unchanged and ensure the chart no longer relies on .Chart.AppVersion.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@kubernetes/apps/litellm/litellm/app/helmrelease.yaml`:
- Around line 13-14: Update the HelmRelease values for the primary LiteLLM
container by explicitly setting image.repository and image.tag to the
Renovate-managed tag-plus-digest reference format, matching the existing
postgres-init pinning convention. Keep replicaCount unchanged and ensure the
chart no longer relies on .Chart.AppVersion.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 4b21c4aa-71c4-4884-86a7-f39b22f5b674
📒 Files selected for processing (13)
kubernetes/apps/litellm/kustomization.yamlkubernetes/apps/litellm/litellm/app/externalsecret.yamlkubernetes/apps/litellm/litellm/app/grafanadashboard.yamlkubernetes/apps/litellm/litellm/app/helmrelease.yamlkubernetes/apps/litellm/litellm/app/httproute.yamlkubernetes/apps/litellm/litellm/app/kustomization.yamlkubernetes/apps/litellm/litellm/app/objectbucketclaim.yamlkubernetes/apps/litellm/litellm/app/ocirepository.yamlkubernetes/apps/litellm/litellm/app/prometheusrule.yamlkubernetes/apps/litellm/litellm/app/servicemonitor.yamlkubernetes/apps/litellm/litellm/ks.yamlkubernetes/apps/litellm/namespace.yamlkubernetes/apps/litellm/transformers/kustomization.yaml
…s; enable rust gateway
pg18vc-r load-balances across all CNPG instances including the primary (selector: podRole=instance); pg18vc-ro is the replica-only service (selector: instanceRole=replica). The prior value defeated the point of read-replica routing by still sending roughly half of reads to the primary. Corrects the inverted claim in the design spec and plan too.
…, split salt key from master key Three fixes from final review: - litellm-oidc envFrom is now optional via postRenderers — a missing 1Password item degrades SSO gracefully instead of blocking the entire proxy from starting. - Image digest pin now carries a renovate: marker directly above the tag field, matching this repo's established custom-manager convention (see rook-ceph cephImage) — the prior comment claimed Renovate tracked it but nothing in the file made that true, so the chart would have kept advancing while the image silently froze. - LITELLM_SALT_KEY moved to its own litellm-salt Secret, separate from litellm-keys, so rotating the master key (by deleting litellm-keys) can never also regenerate the salt key, which must never change once set. Rotation comment rewritten to describe an actually-safe, executable procedure. Also sets GENERIC_USER_ID_ATTRIBUTE=sub so the admin identity binds to the immutable OIDC subject rather than a renameable Pocket ID username.
BerriAI signs the litellm-helm OCI chart with the same cosign key used for the litellm container images (confirmed: cosign verify succeeds against ghcr.io/berriai/litellm-helm, not just the -database image). Wires Flux's native OCIRepository.spec.verify so every pull is checked against the published key, not just a one-time manual check.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/superpowers/specs/2026-08-06-litellm-production-tuning-design.md`:
- Around line 14-16: Update the implementation example’s read-replica hostname
to use the replica-only service `pg18vc-ro` consistently with the documented
`DATABASE_URL_READ_REPLICA` configuration, replacing `pg18vc-r` while leaving
other routing details unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e190f3fe-62e0-4827-ba11-b1efae2ec8a5
📒 Files selected for processing (4)
docs/superpowers/plans/2026-08-06-litellm-production-tuning-implementation.mddocs/superpowers/specs/2026-08-06-litellm-production-tuning-design.mdkubernetes/apps/litellm/litellm/app/externalsecret.yamlkubernetes/apps/litellm/litellm/app/helmrelease.yaml
Comment on lines
+14
to
+16
| - Read-replica routing: `DATABASE_URL_READ_REPLICA` → `pg18vc-ro.database.svc.cluster.local` | ||
| (confirmed live: a replica-only CNPG service, distinct from `pg18vc-r` which load-balances | ||
| across all instances, including the primary). |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Use the replica-only service name consistently.
The scope defines pg18vc-ro as the replica-only service and distinguishes it from pg18vc-r. The implementation example at Lines 96-98 uses pg18vc-r. Update that example to pg18vc-ro.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/superpowers/specs/2026-08-06-litellm-production-tuning-design.md` around
lines 14 - 16, Update the implementation example’s read-replica hostname to use
the replica-only service `pg18vc-ro` consistently with the documented
`DATABASE_URL_READ_REPLICA` configuration, replacing `pg18vc-r` while leaving
other routing details unchanged.
OPENAI_API_KEY was already provisioned in the litellm 1Password item from the original deployment, just unused until now.
These are working-session scratch docs, not meant to ship in the PR — got committed by accident alongside a code fix. Working-tree copies are kept locally (untracked), matching how the rest of docs/superpowers/ is handled in this branch.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
kubernetes/apps/litellm/litellm/app/helmrelease.yaml (2)
196-204: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winDo not rely on
envFromindex 3.This patch changes the
optionalflag for the fourthenvFromentry. In thisenvironmentSecretslist, that islitellm-oidcnow, but any later chart change can insert or reorder entries. Add a JSON PatchtestforsecretRef.name == litellm-oidcbefore settingoptional, or use chart-supported named configuration.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kubernetes/apps/litellm/litellm/app/helmrelease.yaml` around lines 196 - 204, Make the envFrom patch targeting Deployment litellm validate that index 3 references secretRef.name litellm-oidc before updating optional, using a JSON Patch test operation; preserve the existing optional=true change and avoid relying on the positional index alone.
175-195: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winPreserve chart-provided migration Job mount entries.
These JSON patches write complete arrays at
/volumesand/volumeMounts. If the chart renders either array by default, the existing mount entries are replaced by these write paths, which can block migration writes. Render the chart first; use guarded appends if either array is non-empty.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kubernetes/apps/litellm/litellm/app/helmrelease.yaml` around lines 175 - 195, Update the volume and volumeMount patches in the Litellm HelmRelease to preserve chart-rendered entries: render the chart first, then append the tmp, cache, and npm entries when the corresponding arrays already exist, while retaining guarded creation only when they are absent. Ensure existing migration Job mounts are never overwritten.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kubernetes/apps/litellm/litellm/app/ocirepository.yaml`:
- Around line 12-18: Add namespace metadata to the OCIRepository definition and
its referenced Secret configuration, ensuring both use the intended litellm
application namespace and remain aligned with the Helm chart deployment. Update
the relevant namespace fields without changing the existing OCI verification
settings.
---
Nitpick comments:
In `@kubernetes/apps/litellm/litellm/app/helmrelease.yaml`:
- Around line 196-204: Make the envFrom patch targeting Deployment litellm
validate that index 3 references secretRef.name litellm-oidc before updating
optional, using a JSON Patch test operation; preserve the existing optional=true
change and avoid relying on the positional index alone.
- Around line 175-195: Update the volume and volumeMount patches in the Litellm
HelmRelease to preserve chart-rendered entries: render the chart first, then
append the tmp, cache, and npm entries when the corresponding arrays already
exist, while retaining guarded creation only when they are absent. Ensure
existing migration Job mounts are never overwritten.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 117461cb-0ac9-43e6-a2d9-ee471cab23bd
📒 Files selected for processing (4)
kubernetes/apps/default/kustomization.yamlkubernetes/apps/default/pinchflat/app/helmrelease.yamlkubernetes/apps/litellm/litellm/app/helmrelease.yamlkubernetes/apps/litellm/litellm/app/ocirepository.yaml
💤 Files with no reviewable changes (2)
- kubernetes/apps/default/kustomization.yaml
- kubernetes/apps/default/pinchflat/app/helmrelease.yaml
Comment on lines
+12
to
+18
| ref: | ||
| tag: 1.95.0 | ||
| url: oci://ghcr.io/berriai/litellm-helm | ||
| verify: | ||
| provider: cosign | ||
| secretRef: | ||
| name: litellm-cosign-pub |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
repo_root="$(pwd)"
source_file="kubernetes/apps/litellm/litellm/app/ocirepository.yaml"
chart="ghcr.io/berriai/litellm-helm"
version="1.95.0"
key_file="$(mktemp)"
trap 'rm -f "$key_file"' EXIT
yq -r 'select(.kind == "Secret") | .stringData."cosign.pub"' \
"$repo_root/$source_file" > "$key_file"
test -s "$key_file"
helm show chart "oci://$chart" --version "$version" >/dev/null
cosign verify --key "$key_file" "$chart:$version"
yq -r '
select(.kind == "OCIRepository" or .kind == "Secret") |
[.kind, .metadata.name, (.metadata.namespace // "<unset>")] | `@tsv`
' "$repo_root/$source_file"
rg -n -C 4 'namespace:|kind: OCIRepository|kind: Secret|litellm-cosign-pub' \
"$repo_root/kubernetes/apps/litellm"Repository: jfroy/flatops
Length of output: 192
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== locating relevant file =="
git ls-files | rg '^kubernetes/apps/litellm/.*/ocirepository\.yaml$|ocirepository\.yaml$|kubens|ns' || true
echo
echo "== relevant manifest =="
if [ -f kubernetes/apps/litellm/litellm/app/ocirepository.yaml ]; then
nl -ba kubernetes/apps/litellm/litellm/app/ocirepository.yaml | sed -n '1,220p'
fi
echo
echo "== namespace/source references under litellm apps =="
rg -n -C 3 'kind:(| ) OCIRepository|kind:(| ) Secret|namespace:|metadata:|verify:|litellm-cosign-pub|cosign\.pub|ref:|url:' kubernetes/apps/litellm || true
echo
echo "== oci tag/remote availability using OCI CLI or registry inspection =="
[ -x "$(command -v oras)" ] && oras discover --artifact-type 'application/vnd.cncf.helm.config.v1+json' oci://ghcr.io/berriai/litellm-helm:1.95.0 || true
[ -x "$(command -v cosign)" ] && echo "cosign present" || echo "cosign missing"
[ -x "$(command -v helm)" ] && helm show chart "oci://ghcr.io/berriai/litellm-helm" --version "1.95.0" || true
curl -fsSL --max-time 20 --fail-with-body "https://ghcr.io/v2/berriai/litellm-helm/tags/list" | jq -r '.tags // [] | .[]' | rg '^1\.95\.0$' || trueRepository: jfroy/flatops
Length of output: 9010
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== relevant manifest =="
awk '{printf "%6d %s\n", NR, $0}' kubernetes/apps/litellm/litellm/app/ocirepository.yaml
echo
echo "== namespace/source references under litellm apps =="
rg -n -C 3 'kind:(| ) OCIRepository|kind:(| ) Secret|namespace:|metadata:|verify:|litellm-cosign-pub|cosign\.pub|ref:|url:' kubernetes/apps/litellm || true
echo
echo "== registry tag availability =="
python3 - <<'PY'
import json, urllib.request, urllib.error
url="https://ghcr.io/v2/berriai/litellm-helm/tags/list"
try:
with urllib.request.urlopen(url, timeout=20) as r:
data=json.load(r)
except Exception as e:
print(f"fetch_error={type(e).__name__}: {e}")
raise SystemExit
tags=data.get("tags", [])
print("available_count=", len(tags))
print("1.95.0=", "1.95.0" in tags)
PY
echo
echo "== OCI tag digest existence =="
{
[ -x "$(command -v oras)" ] && oras discover --artifact-type 'application/vnd.cncf.helm.config.v1+json' oci://ghcr.io/berriai/litellm-helm:1.95.0 || true
[ -x "$(command -v cosign)" ] && cosign verify-attestation --type="dsse" --key "<placeholder>" oci://ghcr.io/berriai/litellm-helm:1.95.0 2>&1 | head -80 || true
} || trueRepository: jfroy/flatops
Length of output: 17079
Add namespace metadata to the OCI source objects.
OCIRepository and the referenced Secret omit spec.targetNamespace, and no Kustomize transformer sets a namespace for kubernetes/apps/litellm/litellm/app. Flux can render these resources in different namespaces unless a chart default injects the namespace elsewhere, so add namespace metadata to keep the source objects and Helm chart deployment aligned.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kubernetes/apps/litellm/litellm/app/ocirepository.yaml` around lines 12 - 18,
Add namespace metadata to the OCIRepository definition and its referenced Secret
configuration, ensuring both use the intended litellm application namespace and
remain aligned with the Helm chart deployment. Update the relevant namespace
fields without changing the existing OCI verification settings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
New Features
Removed
Chores