diff --git a/README.md b/README.md index 894b4bcb05..4a790ce430 100644 --- a/README.md +++ b/README.md @@ -39,8 +39,45 @@ Group 7154 (1) +--- -## Use LiteLLM for +## What is LiteLLM + +LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format. + +Use it as a **Python SDK** for direct library integration, or deploy the **AI Gateway (Proxy Server)** as a centralized service for your team or organization. + +[**Jump to LiteLLM Proxy (LLM Gateway) Docs**](https://docs.litellm.ai/docs/simple_proxy)
+[**Jump to Supported LLM Providers**](https://docs.litellm.ai/docs/providers) + +--- + +## Why LiteLLM + +Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction: + +- **Unified API** — one interface for 100+ LLMs, no provider-specific SDK juggling +- **Drop-in OpenAI compatibility** — swap providers without rewriting your code +- **Production-ready gateway** — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box +- **8ms P95 latency** at 1k RPS ([benchmarks](https://docs.litellm.ai/docs/benchmarks)) + +### OSS Adopters + + + + + + + + + + + +
StripeimageGoogle ADKGreptileOpenHands

Netflix

OpenAI Agents SDK
+ +--- + +## Features
LLMs - Call 100+ LLMs (Python SDK + AI Gateway) @@ -223,63 +260,7 @@ curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
---- - -## How to use LiteLLM - -You can use LiteLLM through either the Proxy Server or Python SDK. Both gives you a unified interface to access multiple LLMs (100+ LLMs). Choose the option that best fits your needs: - - - - - - - - - - - - - - - - - - - - - - - - - - -
LiteLLM AI GatewayLiteLLM Python SDK
Use CaseCentral service (LLM Gateway) to access multiple LLMsUse LiteLLM directly in your Python code
Who Uses It?Gen AI Enablement / ML Platform TeamsDevelopers building LLM projects
Key FeaturesCentralized API gateway with authentication and authorization, multi-tenant cost tracking and spend management per project/user, per-project customization (logging, guardrails, caching), virtual keys for secure access control, admin dashboard UI for monitoring and managementDirect Python library integration in your codebase, Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router, application-level load balancing and cost tracking, exception handling with OpenAI-compatible errors, observability callbacks (Lunary, MLflow, Langfuse, etc.)
- -LiteLLM Performance: **8ms P95 latency** at 1k RPS (See benchmarks [here](https://docs.litellm.ai/docs/benchmarks)) - -[**Jump to LiteLLM Proxy (LLM Gateway) Docs**](https://docs.litellm.ai/docs/simple_proxy)
-[**Jump to Supported LLM Providers**](https://docs.litellm.ai/docs/providers) - -**Stable Release:** Use docker images with the `-stable` tag. These have undergone 12 hour load tests, before being published. [More information about the release cycle here](https://docs.litellm.ai/docs/proxy/release_cycle) - -Support for more providers. Missing a provider or LLM Platform, raise a [feature request](https://github.com/BerriAI/litellm/issues/new?assignees=&labels=enhancement&projects=&template=feature_request.yml&title=%5BFeature%5D%3A+). - -## OSS Adopters - - - - - - - - - - - -
StripeimageGoogle ADKGreptileOpenHands

Netflix

OpenAI Agents SDK
- -## Supported Providers ([Website Supported Models](https://models.litellm.ai/) | [Docs](https://docs.litellm.ai/docs/providers)) +### Supported Providers ([Website Supported Models](https://models.litellm.ai/) | [Docs](https://docs.litellm.ai/docs/providers)) | Provider | `/chat/completions` | `/messages` | `/responses` | `/embeddings` | `/image/generations` | `/audio/transcriptions` | `/audio/speech` | `/moderations` | `/batches` | `/rerank` | |-------------------------------------------------------------------------------------|---------------------|-------------|--------------|---------------|----------------------|-------------------------|-----------------|----------------|-----------|-----------| @@ -386,12 +367,49 @@ Support for more providers. Missing a provider or LLM Platform, raise a [feature [**Read the Docs**](https://docs.litellm.ai/docs/) -## Run in Developer mode -### Services +--- + +## Get Started + +You can use LiteLLM through either the Proxy Server or Python SDK. Both give you a unified interface to access multiple LLMs (100+ LLMs). Choose the option that best fits your needs: + + + + + + + + + + + + + + + + + + + + + + + + + + +
LiteLLM AI GatewayLiteLLM Python SDK
Use CaseCentral service (LLM Gateway) to access multiple LLMsUse LiteLLM directly in your Python code
Who Uses It?Gen AI Enablement / ML Platform TeamsDevelopers building LLM projects
Key FeaturesCentralized API gateway with authentication and authorization, multi-tenant cost tracking and spend management per project/user, per-project customization (logging, guardrails, caching), virtual keys for secure access control, admin dashboard UI for monitoring and managementDirect Python library integration in your codebase, Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router, application-level load balancing and cost tracking, exception handling with OpenAI-compatible errors, observability callbacks (Lunary, MLflow, Langfuse, etc.)
+ +**Stable Release:** Use docker images with the `-stable` tag. These have undergone 12 hour load tests, before being published. [More information about the release cycle here](https://docs.litellm.ai/docs/proxy/release_cycle) + +Support for more providers. Missing a provider or LLM Platform, raise a [feature request](https://github.com/BerriAI/litellm/issues/new?assignees=&labels=enhancement&projects=&template=feature_request.yml&title=%5BFeature%5D%3A+). + +### Run in Developer Mode +#### Services 1. Setup .env file in root 2. Run dependant services `docker-compose up db prometheus` -### Backend +#### Backend 1. (In root) create virtual environment `python -m venv .venv` 2. Activate virtual environment `source .venv/bin/activate` 3. Install dependencies `pip install -e ".[all]"` @@ -399,12 +417,12 @@ Support for more providers. Missing a provider or LLM Platform, raise a [feature 5. `prisma generate` 6. Start proxy backend `python litellm/proxy/proxy_cli.py` -### Frontend +#### Frontend 1. Navigate to `ui/litellm-dashboard` 2. Install dependencies `npm install` 3. Run `npm run dev` to start the dashboard -# Verify Docker Image Signatures +### Verify Docker Image Signatures All LiteLLM Docker images published to GHCR are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](https://github.com/BerriAI/litellm/commit/0112e53046018d726492c814b3644b7d376029d0). @@ -430,6 +448,8 @@ cosign verify \ Replace `` with the version you are deploying (e.g. `v1.83.0-stable`). +--- + # Enterprise For companies that need better security, user management and professional support @@ -486,10 +506,6 @@ All these checks must pass before your PR can be merged. - [Community Slack 💭](https://www.litellm.ai/support) - Our emails ✉️ ishaan@berri.ai / krrish@berri.ai -# Why did we build this - -- **Need for simplicity**: Our code started to get extremely complicated managing & translating calls between Azure, OpenAI and Cohere. - # Contributors