Skip to content

Feat/add litellm provider#6856

Open
RheagalFire wants to merge 2 commits into
ChatGPTNextWeb:mainfrom
RheagalFire:feat/add-litellm-provider
Open

Feat/add litellm provider#6856
RheagalFire wants to merge 2 commits into
ChatGPTNextWeb:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire

@RheagalFire RheagalFire commented Jul 16, 2026

Copy link
Copy Markdown

💻 Change Type

  • feat
  • fix
  • refactor
  • perf
  • style
  • test
  • docs
  • ci
  • chore
  • build

🔀 Description of Change

Adds LiteLLM as a first-class AI gateway provider, giving NextChat users access to 100+ LLM providers (AWS Bedrock, Vertex AI, Groq, Mistral, Cohere, etc.) through a single proxy endpoint.

Changes (8 files, 451 insertions):

New files:

  • app/client/platforms/litellm.ts - Client-side provider implementing LLMApi (chat with streaming, model auto-discovery via /v1/models, tool calling, reasoning content support)
  • app/api/litellm.ts - Server-side proxy handler with auth and model filtering
  • test/litellm.test.ts - 13 unit tests (path resolution, message extraction, model discovery, error handling, chat payload)

Modified files:

  • app/constant.ts - Added LITELLM_BASE_URL, LiteLLM to ServiceProvider/ModelProvider enums, ApiPath.LiteLLM, LiteLLMConst config
  • app/client/api.ts - Imported LiteLLMApi, added dispatch cases in ClientApi and getClientApi()
  • app/store/access.ts - Added litellmUrl, litellmApiKey state, isValidLiteLLM() validation
  • app/config/server.ts - Added LITELLM_URL, LITELLM_API_KEY env vars
  • app/api/auth.ts - Added LiteLLM auth case
  • app/api/[provider]/[...path]/route.ts - Registered LiteLLM route handler

Tests*

Unit tests (13/13 pass):

PASS test/litellm.test.ts
  LiteLLMApi
    path()
      ✓ builds correct path with custom config URL
      ✓ strips trailing slash from base URL
    extractMessage()
      ✓ extracts content from standard response
      ✓ returns empty string for missing choices
      ✓ returns empty string for null content
    models()
      ✓ fetches and parses model list from /v1/models
      ✓ returns empty array on HTTP error
      ✓ returns empty array when data is null
      ✓ preserves provider/model format in model IDs
    usage()
      ✓ returns zero usage
    speech()
      ✓ throws not implemented error
    chat()
      ✓ sends correct payload for non-streaming
      ✓ calls onError when fetch fails
Tests: 13 passed, 13 total

Live E2E (Azure Foundry Claude Sonnet 4.6 via LiteLLM proxy):

--- Chat completion ---
Status: 200, Content: "10", Usage: prompt=20, completion=5, total=25
--- Model auto-discovery ---
Models found: 1 (anthropic/claude-sonnet-4-6)
--- Response structure ---
id: chatcmpl-..., object: chat.completion, role: assistant, finish_reason: stop

📝 Additional Information

  • Additive only - all 14 existing providers untouched
  • No new npm dependencies
  • Follows the SiliconFlow provider pattern exactly
  • Server-side deployment supports LITELLM_URL and LITELLM_API_KEY env vars
  • Default proxy URL http://localhost:4000 matches LiteLLM's standard port
  • Models auto-populate from the proxy's /v1/models endpoint

Client-side setup:

  1. Select "LiteLLM" as the provider in Settings
  2. Enter your LiteLLM proxy URL (default: http://localhost:4000)
  3. Enter your LiteLLM API key

Server-side (env vars):

LITELLM_URL=http://your-litellm-proxy:4000
LITELLM_API_KEY=sk-your-litellm-key

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant