9f68081f6d
* feat: prisma migrate deploy with lock
Author: Mini Jeong <mini.jeong@navercorp.com>
* fix: use redis cache from proxy server
Author: Mini Jeong <mini.jeong@navercorp.com>
* fix: add type checks and fix unit tests for migration lock
- Add DATABASE_URL validation in _create_baseline_migration() and _resolve_all_migrations()
- Fix MyPy type errors by adding None checks before using database_url in subprocess calls
- Add _resolve_all_migrations mock to failing unit tests to prevent filesystem errors
- Apply Black formatting to modified files
Fixes:
- MyPy type errors: database_url could be None when passed to subprocess
- Unit test failures: _resolve_all_migrations tried to create directories in read-only /test path
* fix: resolve MyPy type error in vertex_ai vertex_llm_base
Fix MyPy type checking error where vertex_api_version parameter type
was incompatible with function signature expectation.
* fix: Return 403 exception when calling GET responses api
* fix: added new step into rotate master key function for processing credentials table
* Add redisvl in requirements.txt
* fix: fixed the issue of handling root paths when processing Discovery protected resource metadata and authorization server metadata URLs.
* fix: added additional grant type into oauth_authorization_server response for fixing mcp auth register bad request issue
* fix: added RFC RECOMMENDED property(scopes_supported) to protected resource and authorization server metadata
* fix: removed initialize the tool name to MCP server name mapping(oauth2) on startup for avoiding 401 error
* fix: upgraded mcp sdk depency version for fixing ClosedResourceError
* Use already configured opentelemetry providers
Users that instrument using opentelemetry-instrument can now setup exporters as per their environment.
* Handle all protocols for all telemetry
* Add more tests
* feat(mcp): parallelize tool fetching from multiple MCP servers (#18627)
* feat(mcp): parallelize tool fetching from multiple MCP servers
Replace sequential tool fetching with asyncio.gather() to reduce
client timeouts when using multiple MCP servers.
Changes:
- mcp_server_manager.py: list_tools() now fetches tools in parallel
- server.py: _get_tools_from_mcp_servers() now fetches tools in parallel
Real-world impact (7 MCP servers example):
- Sequential: ~4.5+ seconds (exceeds typical 5-second client timeouts)
- Parallel: ~1.2 seconds (max of all servers)
Fixes #18626
* fix: copy oauth2_headers to avoid shared dict mutation in parallel tasks
* feat: add display_name, model_vendor, and model_version metadata
* added the option of adding langsmith tenant id in the env (#18623)
* fix(router): Validate routing_strategy at startup to fail fast with helpful error. (#18624)
Invalid routing_strategy values (e.g., "simple" instead of "simple-shuffle") previously failed silently, causing confusing "No deployments available" errors downstream. This change adds upfront validation in routing_strategy_init() to:
- Check if the provided strategy matches valid string values or RoutingStrategy enum
- Raise a clear ValueError listing valid options if invalid
- Fail fast at startup instead of at request time
Fixes behavior reported in #11330 where users had to debug cryptic errors.
Valid strategies: simple-shuffle, least-busy, usage-based-routing, latency-based-routing, cost-based-routing, usage-based-routing-v2
Co-authored-by: Flibbert E. Gibbitz <flibbertygibbitz@runelabs.ai>
* Add libsndfile to database Docker image for audio processing (#18612)
The litellm-database Docker image was missing the libsndfile system
library, which is required by the soundfile Python package for audio
file processing. This caused failures when using audio transcription
endpoints that attempt to calculate audio duration.
This adds libsndfile to the runtime dependencies in Dockerfile.database,
consistent with Dockerfile.alpine which already includes this library.
* Fix: Map Gemini cached_tokens to Langfuse cache_read_input_tokens (#18614)
* Fix: Map Gemini cached_tokens to Langfuse cache_read_input_tokens
Fixes #18520
## Problem
Langfuse integration was not capturing cached tokens from Gemini models.
Gemini returns cached tokens in `usage.prompt_tokens_details.cached_tokens`,
but Langfuse only read from top-level `usage.cache_read_input_tokens`
(which only Anthropic populates).
## Solution
Updated langfuse.py to check both locations:
1. First check top-level cache_read_input_tokens (for Anthropic)
2. Then check prompt_tokens_details.cached_tokens (for Gemini, OpenAI, others)
This ensures all providers' cached tokens are properly reported to Langfuse.
## Changes
- Modified litellm/integrations/langfuse/langfuse.py (lines 742-761)
- Added 3 unit tests in tests/test_litellm/integrations/langfuse/test_gemini_cached_tokens.py
- All existing Langfuse tests still pass (11/11)
## Testing
- test_cached_tokens_extraction: Verifies Gemini cached_tokens extraction
- test_cached_tokens_not_present: Backward compatibility (no cached_tokens)
- test_cached_tokens_is_zero: Edge case when cached_tokens = 0
* Refactor: Extract cache token logic into helper function
Address review feedback from @officer47p
- Created _extract_cache_read_input_tokens() helper function
- Reduces code bloat in _log_langfuse_v2 method
- Improves testability and reusability
- All tests still passing (11/11)
* Adding Role Mappings
* Fixing Edit SSO Settings Modal
* feat: add user_mcp_management_mode for view_all visibility
* Fixing tests
* fix: missing mcp_allow_all_ui.png
* docs: add user_mcp_management_mode
* Align responses API streaming hooks with chat pipeline
* Clarify responses API streaming context
* Address review comments
* feat: Add GigaChat provider support (#18564)
* feat: Add GigaChat provider support
Add native support for GigaChat API (Sber AI, Russia's leading LLM).
Supported features:
- Chat completions (sync/async)
- Streaming (sync/async)
- Function calling / Tools
- Structured output via JSON schema (emulated through function calls)
- Image input (base64 and URL)
- Embeddings
Closes #18515
* fix: resolve mypy type errors in GigaChat handler
- Fix _prepare_file_data return type (use 3-tuple for cleaner type flow)
- Add type annotations for lists in _process_content_parts methods
- Add type annotations in _collapse_user_messages
- Use ChatCompletionToolCallChunk for proper tool_use typing
- Add type: ignore[override] for astreaming async generator
* refactor(gigachat): migrate to BaseConfig pattern
* fix: remove unused imports
* fix: resolve mypy type errors
* fix: mypy type errors
* refactor: address review feedback for GigaChat provider
- Remove singleton pattern, reuse litellm HTTPHandler
- Move constants/errors to transformation files, delete common_utils.py
- Add models to model_prices_and_context_window.json
- Fix ssl_verify not passed to HTTP client for embeddings
* docs: update GigaChat documentation with ssl_verify requirement
* Revert "Add redisvl in requirements.txt"
* Put reasoning summary behind feat flag
* fix: model eol
* fix: anthropic claude-3-opus-20240229 EOL
* Revert "fix: model eol"
This reverts commit
|
||
|---|---|---|
| .. | ||
| build_from_pip | ||
| tests | ||
| .env.example | ||
| build_admin_ui.sh | ||
| Dockerfile.alpine | ||
| Dockerfile.custom_ui | ||
| Dockerfile.database | ||
| Dockerfile.dev | ||
| Dockerfile.non_root | ||
| entrypoint.sh | ||
| install_auto_router.sh | ||
| prod_entrypoint.sh | ||
| README.md | ||
| supervisord.conf | ||
Docker Development Guide
This guide provides instructions for building and running the LiteLLM application using Docker and Docker Compose.
Prerequisites
- Docker
- Docker Compose
Building and Running the Application
To build and run the application, you will use the docker-compose.yml file located in the root of the project. This file is configured to use the Dockerfile.non_root for a secure, non-root container environment.
1. Set the Master Key
The application requires a MASTER_KEY for signing and validating tokens. You must set this key as an environment variable before running the application.
Create a .env file in the root of the project and add the following line:
MASTER_KEY=your-secret-key
Replace your-secret-key with a strong, randomly generated secret.
2. Build and Run the Containers
Once you have set the MASTER_KEY, you can build and run the containers using the following command:
docker compose up -d --build
This command will:
- Build the Docker image using
Dockerfile.non_root. - Start the
litellm,litellm_db, andprometheusservices in detached mode (-d). - The
--buildflag ensures that the image is rebuilt if there are any changes to the Dockerfile or the application code.
3. Verifying the Application is Running
You can check the status of the running containers with the following command:
docker compose ps
To view the logs of the litellm container, run:
docker compose logs -f litellm
4. Stopping the Application
To stop the running containers, use the following command:
docker compose down
Hardened / Offline Testing
To ensure changes are safe for non-root, read-only root filesystems and restricted egress, always validate with the hardened compose file:
docker compose -f docker-compose.yml -f docker-compose.hardened.yml build --no-cache
docker compose -f docker-compose.yml -f docker-compose.hardened.yml up -d
This setup:
- Builds from
docker/Dockerfile.non_rootwith Prisma engines and Node toolchain baked into the image. - Runs the proxy as a non-root user with a read-only rootfs and only two writable tmpfs mounts:
/app/cache(Prisma/NPM cache; backingPRISMA_BINARY_CACHE_DIR,NPM_CONFIG_CACHE,XDG_CACHE_HOME)/app/migrations(Prisma migration workspace; backingLITELLM_MIGRATION_DIR)
- Routes all outbound traffic through a local Squid proxy that denies egress, so Prisma migrations must use the cached CLI and engines.
You should also verify offline Prisma behaviour with:
docker run --rm --network none --entrypoint prisma ghcr.io/berriai/litellm:main-stable --version
This command should succeed (showing engine versions) even with --network none, confirming that Prisma binaries are available without network access.
Troubleshooting
build_admin_ui.sh: not found: This error can occur if the Docker build context is not set correctly. Ensure that you are running thedocker-composecommand from the root of the project.Master key is not initialized: This error means theMASTER_keyenvironment variable is not set. Make sure you have created a.envfile in the project root with theMASTER_KEYdefined.