cfd0e2cf99
* Update AGENTS.md with additional Cursor Cloud setup notes - Add note about openapi-core dependency needed for OpenAPI compliance tests - Add note about poetry lock fallback when lock file is out of sync Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * Sync lock files with current dependency specs - poetry.lock: regenerated to match pyproject.toml (litellm-proxy-extras 0.4.50 -> 0.4.51) - package-lock.json: updated from npm install Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * Polish MCP Servers UI for enterprise-grade look and feel 10 improvements to the MCP Servers table and related components: 1. Remove debug console.logs from mcp_servers.tsx 2. Fix health status icons: distinct ✓/✗/? per state instead of identical dots 3. Health status badges: proper pill styling with rounded-full and borders 4. Health loading state: subtle pulsing dot instead of raw SVG spinner 5. Transport column: color-coded badges (HTTP=blue, SSE=purple, STDIO=amber, OPENAPI=teal) 6. Auth type column: color-coded badges (oauth2=indigo, bearer_token=sky, api_key=emerald) 7. Server ID chip: rounded corners, border, and transition effect 8. Filter bar: lighter border, cleaner labels, vertical divider between filters 9. Network Access: pill badges with colored dots (Public/Internal) 10. Date columns: shorter headers, dash for missing values, tooltip with full datetime Also: - Improved delete modal: cleaner layout, neutral background instead of red - Access Groups column: shows first group with +N count instead of truncated text - Empty state message includes CTA guidance - Updated test to match renamed filter label Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * Polish MCP server detail views and table refinements (round 2) 10 more enterprise polish improvements: 1. Overview cards: use color-coded badges for Transport and Auth Type values 2. Overview cards: fix 'Host Url' typo -> 'Host URL', uppercase card labels 3. Settings tab: show em-dash placeholder for empty/missing values 4. Settings tab: use consistent Transport/Auth/Network badge styling matching table 5. Settings tab: definition-list layout with label/value grid columns 6. Server detail header: show server name prominently with alias as badge 7. Server detail header: show description below name, smaller server ID 8. Actions column: improved hover states with background color transitions 9. Credential column: pill badge for Connected state, shadow on Connect button 10. Table header: server count badge next to title, CTA button moved right Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * Revert colorful transport/auth badges to neutral gray Color should only carry semantic meaning. Transport type (HTTP/SSE) and auth type (oauth2/bearer_token) are informational labels, not status indicators, so they use a uniform gray badge. Color remains on: - Health status: green (healthy), red (unhealthy) - Network access: green (public), orange (internal) - Credential: green (connected) Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> |
||
|---|---|---|
| .circleci | ||
| .claude | ||
| .devcontainer | ||
| .github | ||
| .semgrep/rules | ||
| ci_cd | ||
| cookbook | ||
| db_scripts | ||
| deploy | ||
| dist | ||
| docker | ||
| docs/my-website | ||
| enterprise | ||
| litellm | ||
| litellm-js | ||
| litellm-proxy-extras | ||
| scripts | ||
| tests | ||
| ui/litellm-dashboard | ||
| .dockerignore | ||
| .env.example | ||
| .flake8 | ||
| .git-blame-ignore-revs | ||
| .gitattributes | ||
| .gitguardian.yaml | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| .trivyignore | ||
| AGENTS.md | ||
| ARCHITECTURE.md | ||
| CLAUDE.md | ||
| codecov.yaml | ||
| CONTRIBUTING.md | ||
| dev_config.yaml | ||
| docker-compose.hardened.yml | ||
| docker-compose.yml | ||
| Dockerfile | ||
| GEMINI.md | ||
| index.yaml | ||
| LICENSE | ||
| license_cache.json | ||
| Makefile | ||
| mcp_servers.json | ||
| model_prices_and_context_window.json | ||
| package-lock.json | ||
| package.json | ||
| poetry.lock | ||
| policy_templates.json | ||
| prometheus.yml | ||
| provider_endpoints_support.json | ||
| proxy_server_config.yaml | ||
| pyproject.toml | ||
| pyrightconfig.json | ||
| README.md | ||
| render.yaml | ||
| requirements.txt | ||
| ruff.toml | ||
| schema.prisma | ||
| security.md | ||
| taplo.toml | ||
| uv.lock | ||
🚅 LiteLLM
Call 100+ LLMs in OpenAI format. [Bedrock, Azure, OpenAI, VertexAI, Anthropic, Groq, etc.]
LiteLLM Proxy Server (AI Gateway) | Hosted Proxy | Enterprise Tier
Use LiteLLM for
LLMs - Call 100+ LLMs (Python SDK + AI Gateway)
All Supported Endpoints - /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, /messages and more.
Python SDK
pip install litellm
from litellm import completion
import os
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"
# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])
# Anthropic
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])
AI Gateway (Proxy Server)
Getting Started - E2E Tutorial - Setup virtual keys, make your first request
pip install 'litellm[proxy]'
litellm --model gpt-4o
import openai
client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello!"}]
)
Agents - Invoke A2A Agents (Python SDK + AI Gateway)
Supported Providers - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI
Python SDK - A2A Protocol
from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4
client = A2AClient(base_url="http://localhost:10001")
request = SendMessageRequest(
id=str(uuid4()),
params=MessageSendParams(
message={
"role": "user",
"parts": [{"kind": "text", "text": "Hello!"}],
"messageId": uuid4().hex,
}
)
)
response = await client.send_message(request)
AI Gateway (Proxy Server)
Step 1. Add your Agent to the AI Gateway
Step 2. Call Agent via A2A SDK
from a2a.client import A2ACardResolver, A2AClient
from a2a.types import MessageSendParams, SendMessageRequest
from uuid import uuid4
import httpx
base_url = "http://localhost:4000/a2a/my-agent" # LiteLLM proxy + agent name
headers = {"Authorization": "Bearer sk-1234"} # LiteLLM Virtual Key
async with httpx.AsyncClient(headers=headers) as httpx_client:
resolver = A2ACardResolver(httpx_client=httpx_client, base_url=base_url)
agent_card = await resolver.get_agent_card()
client = A2AClient(httpx_client=httpx_client, agent_card=agent_card)
request = SendMessageRequest(
id=str(uuid4()),
params=MessageSendParams(
message={
"role": "user",
"parts": [{"kind": "text", "text": "Hello!"}],
"messageId": uuid4().hex,
}
)
)
response = await client.send_message(request)
MCP Tools - Connect MCP servers to any LLM (Python SDK + AI Gateway)
Python SDK - MCP Bridge
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import experimental_mcp_client
import litellm
server_params = StdioServerParameters(command="python", args=["mcp_server.py"])
async with stdio_client(server_params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
# Load MCP tools in OpenAI format
tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")
# Use with any LiteLLM model
response = await litellm.acompletion(
model="gpt-4o",
messages=[{"role": "user", "content": "What's 3 + 5?"}],
tools=tools
)
AI Gateway - MCP Gateway
Step 1. Add your MCP Server to the AI Gateway
Step 2. Call MCP tools via /chat/completions
curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Summarize the latest open PR"}],
"tools": [{
"type": "mcp",
"server_url": "litellm_proxy/mcp/github",
"server_label": "github_mcp",
"require_approval": "never"
}]
}'
Use with Cursor IDE
{
"mcpServers": {
"LiteLLM": {
"url": "http://localhost:4000/mcp/",
"headers": {
"x-litellm-api-key": "Bearer sk-1234"
}
}
}
}
How to use LiteLLM
You can use LiteLLM through either the Proxy Server or Python SDK. Both gives you a unified interface to access multiple LLMs (100+ LLMs). Choose the option that best fits your needs:
| LiteLLM AI Gateway | LiteLLM Python SDK | |
|---|---|---|
| Use Case | Central service (LLM Gateway) to access multiple LLMs | Use LiteLLM directly in your Python code |
| Who Uses It? | Gen AI Enablement / ML Platform Teams | Developers building LLM projects |
| Key Features | Centralized API gateway with authentication and authorization, multi-tenant cost tracking and spend management per project/user, per-project customization (logging, guardrails, caching), virtual keys for secure access control, admin dashboard UI for monitoring and management | Direct Python library integration in your codebase, Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router, application-level load balancing and cost tracking, exception handling with OpenAI-compatible errors, observability callbacks (Lunary, MLflow, Langfuse, etc.) |
LiteLLM Performance: 8ms P95 latency at 1k RPS (See benchmarks here)
Jump to LiteLLM Proxy (LLM Gateway) Docs
Jump to Supported LLM Providers
Stable Release: Use docker images with the -stable tag. These have undergone 12 hour load tests, before being published. More information about the release cycle here
Support for more providers. Missing a provider or LLM Platform, raise a feature request.
OSS Adopters
Netflix |
Supported Providers (Website Supported Models | Docs)
Run in Developer mode
Services
- Setup .env file in root
- Run dependant services
docker-compose up db prometheus
Backend
- (In root) create virtual environment
python -m venv .venv - Activate virtual environment
source .venv/bin/activate - Install dependencies
pip install -e ".[all]" pip install prismaprisma generate- Start proxy backend
python litellm/proxy/proxy_cli.py
Frontend
- Navigate to
ui/litellm-dashboard - Install dependencies
npm install - Run
npm run devto start the dashboard
Enterprise
For companies that need better security, user management and professional support
This covers:
- ✅ Features under the LiteLLM Commercial License:
- ✅ Feature Prioritization
- ✅ Custom Integrations
- ✅ Professional Support - Dedicated discord + slack
- ✅ Custom SLAs
- ✅ Secure access with Single Sign-On
Contributing
We welcome contributions to LiteLLM! Whether you're fixing bugs, adding features, or improving documentation, we appreciate your help.
Quick Start for Contributors
This requires poetry to be installed.
git clone https://github.com/BerriAI/litellm.git
cd litellm
make install-dev # Install development dependencies
make format # Format your code
make lint # Run all linting checks
make test-unit # Run unit tests
make format-check # Check formatting only
For detailed contributing guidelines, see CONTRIBUTING.md.
Code Quality / Linting
LiteLLM follows the Google Python Style Guide.
Our automated checks include:
- Black for code formatting
- Ruff for linting and code quality
- MyPy for type checking
- Circular import detection
- Import safety checks
All these checks must pass before your PR can be merged.
Support / talk with founders
- Schedule Demo 👋
- Community Discord 💭
- Community Slack 💭
- Our numbers 📞 +1 (770) 8783-106 / +1 (412) 618-6238
- Our emails ✉️ ishaan@berri.ai / krrish@berri.ai
Why did we build this
- Need for simplicity: Our code started to get extremely complicated managing & translating calls between Azure, OpenAI and Cohere.