Monesh Ram
083c8998e6
fix: add missing return type annotations to iterator protocol methods in streaming_handler ( #21750 )
...
* fix: add return type annotations to iterator protocol methods in streaming_handler
Add missing return type annotations to __iter__, __aiter__, __next__, and __anext__ methods in CustomStreamWrapper and related classes.
- __iter__(self) -> Iterator["ModelResponseStream"]
- __aiter__(self) -> AsyncIterator["ModelResponseStream"]
- __next__(self) -> "ModelResponseStream"
- __anext__(self) -> "ModelResponseStream"
Also adds AsyncIterator and Iterator to typing imports.
Fixes issue with PLR0915 noqa comments and ensures proper type checking support.
Related to: BerriAI/litellm#8304
* fix: add ruff PLR0915 noqa for files with too many statements
2026-02-21 19:50:38 -08:00
LeeJuOh
50f36d9ca6
fix(budget): fix timezone config lookup and replace hardcoded timezone map with ZoneInfo ( #21754 )
...
* fix(budget): fix timezone config lookup and replace hardcoded timezone map with ZoneInfo
* fix(budget): update stale docstring on get_budget_reset_time
2026-02-21 19:35:06 -08:00
Ishaan Jaffer
ab032c292c
docs fix
2026-02-21 16:36:22 -08:00
ryan-crabbe
0e1db3f7e4
Merge pull request #21140 from BerriAI/litellm_perf_user_api_key_auth
...
perf: optimize user_api_key_auth
2026-02-21 16:23:54 -08:00
ryan-crabbe
ce22524ac5
Merge branch 'main' into litellm_perf_user_api_key_auth
2026-02-21 16:05:33 -08:00
Ishaan Jaffer
7e2d6f2355
ui fix
2026-02-21 16:05:03 -08:00
Ryan Crabbe
643c9b6c04
Merge remote-tracking branch 'origin/main' into litellm_perf_user_api_key_auth
2026-02-21 16:03:31 -08:00
Ishaan Jaffer
d31d5b8486
fix failing tests
2026-02-21 15:48:26 -08:00
Ishaan Jaffer
775fb79260
fix
2026-02-21 15:45:03 -08:00
Ishaan Jaffer
26ea29afd3
test_get_usage_as_dict
2026-02-21 15:39:06 -08:00
Ishaan Jaffer
a58232ac5b
fix prisma
2026-02-21 15:35:50 -08:00
Krish Dholakia
1f7eeb274c
Agent Builder - improve rejected response detection based on agent response ( #21850 )
...
* fix: feat: add litellm_system_prompt support
* feat: support new 'litellm_agent' model provider
* feat: ui/ - new agent builder ui
* fix(anthropic/chat/transformation.py): normalize max_tokens if decimal
* feat(agentbuilderview.tsx): run compliance datasets against litellm agent
* feat: new response rejection detector
* fix: multiple fixes
* feat: add mcp tools support to agent builder
create an agent with access to llm's + mcp servers
2026-02-21 15:34:42 -08:00
Krish Dholakia
9fc6fd647c
Agent Builder - support new experimental agent builder, to ensure agents pass compliance checks ( #21817 )
...
* fix: feat: add litellm_system_prompt support
* feat: support new 'litellm_agent' model provider
* feat: ui/ - new agent builder ui
* fix(anthropic/chat/transformation.py): normalize max_tokens if decimal
* feat(agentbuilderview.tsx): run compliance datasets against litellm agent
2026-02-21 15:32:47 -08:00
Ishaan Jaffer
2acc93e451
BUMP
2026-02-21 15:28:18 -08:00
Ishaan Jaff
bab4127cae
fix(tests): fix flaky test_use_prisma_db_push_flag_behavior ( #21849 )
...
Replace Click CliRunner with standalone_mode=False to avoid
"I/O operation on closed file" errors caused by Click's stream
isolation in CI environments.
2026-02-21 15:23:55 -08:00
Ishaan Jaff
f74a1c94df
test(router): add coverage tests for _is_complexity_router_deployment and init_complexity_router_deployment ( #21848 )
2026-02-21 15:21:10 -08:00
Ishaan Jaff
6dc9823926
docs(release-notes): update v1.81.14 - split guardrail sections, add eval results, fix key highlights and section placement ( #21847 )
2026-02-21 15:18:46 -08:00
Ryan Crabbe
c7ad8053b1
Merge origin/main into litellm_perf_user_api_key_auth
...
Resolve conflicts:
- pass_through_endpoints.py: take main's version, re-apply
MAPPED_PASS_THROUGH_PREFIXES startswith(tuple) optimization
- test_user_api_key_auth.py: keep both auth optimization regression
tests and JWT admin identity field tests
2026-02-21 15:14:20 -08:00
Ishaan Jaff
eac3ae8121
docs: update v1.81.14 release notes - guardrail model garden, complexity router placement ( #21843 )
...
* docs(release-notes): update v1.81.14 key highlights and section placement
* docs(release-notes): rewrite key highlights and add guardrail narrative section
* docs(release-notes): rewrite guardrail narrative to match release notes style
* docs(release-notes): add guardrail eval results section
2026-02-21 15:10:21 -08:00
Ishaan Jaff
9f459c5c57
fix(logging): preserve pass-through endpoint response_cost ( #21844 )
...
* fix(logging): preserve pass-through endpoint response_cost in async_success_handler
Two places in the logging pipeline were overwriting response_cost that
pass-through handlers (Gemini/Vertex) had already calculated:
1. _process_hidden_params_and_response_cost fell through to
_response_cost_calculator which returns None for pass-through calls
2. async_success_handler pass-through branch unconditionally set
response_cost = None (introduced in PR #19887 )
Now both places check if response_cost is already set before overwriting.
* test: add regression test for pass-through endpoint response_cost preservation
2026-02-21 15:09:45 -08:00
Ishaan Jaff
3278fee714
fix(lint): fix ruff/flake8 violations - unused imports, PLR0915, print statements ( #21846 )
2026-02-21 15:07:47 -08:00
ryan-crabbe
77c3b6ab92
Merge pull request #21133 from BerriAI/litellm_perf_convert_model_response_frozensets
...
perf: cache model_fields.keys() as frozensets
2026-02-21 15:05:26 -08:00
Ishaan Jaff
d427ff6705
fix(logging): zero out response_cost for cache hits in _process_hidden_params_and_response_cost ( #21841 )
...
When a cached response had `response_cost` pre-calculated in its `_hidden_params`
(from the original response), `_process_hidden_params_and_response_cost` was using
it directly without checking `cache_hit`, so cached responses were reported with
non-zero cost instead of 0.
Fixes test_cost_tracking_with_caching.
2026-02-21 14:57:03 -08:00
Ryan Crabbe
bb24ebebd9
Merge origin/main into litellm_perf_convert_model_response_frozensets
...
Resolve conflict: keep main's provider_specific_fields passthrough
preservation while using frozenset set-difference optimization.
2026-02-21 14:52:43 -08:00
Ishaan Jaff
8c7f667df2
docs: v1.81.14-stable release notes ( #21839 )
...
* docs(release-notes): add v1.81.14-stable release notes
* fix(docs): fix MDX compilation errors in auto_routing.md
* docs(release-notes): polish v1.81.14 - narrative paragraph, consolidated guardrail templates, merged competitor bullet
2026-02-21 14:50:50 -08:00
Shivam Rawat
698c46830b
remove prompt from getting logged ( #21831 )
2026-02-21 14:49:11 -08:00
Ryan Crabbe
c039e505ba
docs: add inline comment clarifying intentional falsy id behavior
2026-02-21 14:47:09 -08:00
yuneng-jiang
3476452774
Merge pull request #21822 from dkindlund/fix/admin-ui-logging-metadata
...
fix(ui): preserve logging_settings in key metadata on update
2026-02-21 14:46:41 -08:00
Ishaan Jaff
8afeaf8da4
fix(tests): fix flaky test_create_vertex_fine_tune_jobs_mocked - handle background Datadog flush ( #21838 )
2026-02-21 14:44:01 -08:00
Ishaan Jaff
d7b22d340b
fix(tests): move test_router_azure_acompletion to llm_translation testing ( #21837 )
2026-02-21 14:41:53 -08:00
Ishaan Jaff
59e5b7e8c6
fix(tests): use monkeypatch.setenv for Redis pool max_connections tests ( #21834 )
...
Replace patch('litellm._redis._get_redis_client_logic') with monkeypatch.setenv
in test_max_connections_url_config and test_max_connections_url_config_string_value.
The mock was unreliable in CI (REDIS_URL is set to the real Redis Cloud server),
causing the pool to silently use the real config instead of the test config.
Using monkeypatch.setenv tests the full env-var→pool chain more robustly and
matches the actual production code path.
2026-02-21 14:38:28 -08:00
Ishaan Jaff
235a47c576
fix(tests): mock test_claude_tool_use_with_gemini to fix flaky CI ( #21832 )
...
* ui fixes
* fix(tests): mock test_claude_tool_use_with_gemini to avoid MALFORMED_FUNCTION_CALL flakiness
2026-02-21 14:34:54 -08:00
ryan-crabbe
3da77f310b
Merge pull request #21828 from BerriAI/fix-duplicate-issues-workflow
...
fix: pass prompt as env var in duplicate detection workflows
2026-02-21 14:30:03 -08:00
Ryan Crabbe
c7d3198d9a
fix: pass prompt as env var in duplicate detection workflows
...
Fixes "Input must be provided either through stdin or as a prompt
argument" error by moving the prompt to a PROMPT env variable
instead of inline multiline shell string.
2026-02-21 14:29:08 -08:00
Ishaan Jaff
fb4249005e
fix(tests): add atexit.register mock to prevent Click isolation stream closure in test_use_prisma_db_push_flag_behavior ( #21829 )
2026-02-21 14:28:02 -08:00
yuneng-jiang
f61dc66a02
Merge pull request #21827 from BerriAI/revert-21707-add-watchdog-prisma
...
Revert "fix(proxy): recover from prisma-query-engine zombie process"
2026-02-21 14:21:19 -08:00
yuneng-jiang
8c5be4cb62
Revert "fix(proxy): recover from prisma-query-engine zombie process ( #21707 )"
...
This reverts commit 977ad015ca
2026-02-21 14:20:06 -08:00
Ishaan Jaff
56a095a079
fix(tests): update deprecated Anthropic model in test_user_model_access ( #21826 )
...
* fix(tests): clear _async_success_callback in vertex fine-tune mocked tests to prevent Datadog interference
* fix(tests): update deprecated claude-3-5-haiku-20241022 to claude-haiku-4-5-20251001
2026-02-21 14:18:24 -08:00
Ishaan Jaff
a1ead765ed
fix(tests): clear _async_success_callback in vertex fine-tune mocked tests to prevent Datadog interference ( #21825 )
2026-02-21 14:16:18 -08:00
Ishaan Jaff
494aad4a68
fix(tests): isolate auth in vertex passthrough and spend logs date range tests ( #21824 )
...
test_vertex_passthrough_with_default_credentials and
test_view_spend_logs_with_date_range_summarized fail intermittently when a
prior xdist worker sets master_key — auth then rejects the unauthenticated
test requests before the code under test is reached.
- mock user_api_key_auth in test_vertex_passthrough_with_default_credentials
(same pattern used for test_vertex_passthrough_with_no_default_credentials
in #21810 )
- wrap test_view_spend_logs_with_date_range_summarized in
app.dependency_overrides[ps.user_api_key_auth] with try/finally cleanup
(same pattern used for the other spend log tests in #21810 )
2026-02-21 14:14:50 -08:00
Ishaan Jaff
dd6a74da63
fix(tests): isolate litellm.cache and CLI env vars in flaky tests ( #21821 )
...
- TestSpendLogsPayload: save/restore litellm.cache in setup_method/teardown_method
so tests that run after a cache-setting test don't see a non-None cache and get
a hash instead of "Cache OFF" in the cache_key field
- test_use_prisma_db_push_flag_behavior: apply clean_env pattern (strip DATABASE_URL/DIRECT_URL,
then set DATABASE_URL to test value) inside the with block instead of using @patch.dict
decorator, matching the pattern from test_skip_server_startup to avoid Click 8.3.x
StreamMixer stream lifecycle issues in CI
2026-02-21 14:11:48 -08:00
Ishaan Jaff
6acfa1c71d
fix(tests): clear ANTHROPIC_BASE_URL/ANTHROPIC_API_BASE in spend log api_base tests ( #21820 )
...
Tests hardcode expected api_base as https://api.anthropic.com/v1/messages but
if ANTHROPIC_BASE_URL is set in the environment the recorded api_base changes,
causing a mismatch. Clear both env vars via monkeypatch at the start of each test.
2026-02-21 14:11:18 -08:00
Ishaan Jaff
6ec16d583d
fix(test): add timeout to flush() call to prevent 300s hang in CI ( #21819 )
...
GLOBAL_LOGGING_WORKER.flush() calls queue.join() which blocks until all
items are task_done(). In CI with pytest-asyncio, each test gets a fresh
event loop so the worker reinitializes its queue - items from a previous
test never get task_done(), causing an infinite hang.
Fix: wrap flush() with asyncio.wait_for(..., timeout=10.0).
2026-02-21 14:10:57 -08:00
Darien Kindlund
5f0bef3133
fix(ui): preserve logging_settings in key metadata on update
...
The logging_settings condition used a bare truthiness check which
failed when the form field was undefined or not properly synced
from the EditLoggingSettings component. Changed to explicit
Array.isArray() check consistent with the tags field pattern.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-02-21 16:51:56 -05:00
ryan-crabbe
b17d37eceb
Merge pull request #21815 from BerriAI/litellm_fix_openai_init_params_immutable
...
fix: make cached OpenAI init params immutable and fix import ordering
2026-02-21 13:33:03 -08:00
Ryan Crabbe
dcbac4a4af
style: add missing PEP 8 blank line before top-level function
2026-02-21 13:31:19 -08:00
Ishaan Jaff
c810f5cd63
fix(tests): replace fake France Azure endpoint in test_router_azure_acompletion ( #21818 )
2026-02-21 13:26:59 -08:00
Ryan Crabbe
9e1d83e3de
fix: make LITELLM_CLIENT_SPECIFIC_PARAMS a tuple to prevent TypeError
...
tuple + list raises TypeError in get_openai_client_cache_key. Also add
test coverage for get_openai_client_cache_key to catch type mismatches.
2026-02-21 13:25:23 -08:00
shin-bot-litellm
1be30f5129
feat(router): Add complexity-based auto routing strategy ( #21789 )
...
* feat(router): Add complexity-based auto routing strategy
Adds a rule-based routing strategy that classifies requests by complexity
and routes them to appropriate models - with zero API calls and sub-millisecond
latency.
## Features
- **Zero external API calls** - all scoring is local
- **Sub-millisecond latency** - typically <1ms per classification
- **Weighted multi-dimensional scoring** across 7 dimensions:
- Token count (short=simple, long=complex)
- Code presence (code keywords → complex)
- Reasoning markers ("step by step" → reasoning tier)
- Technical terms (domain complexity)
- Simple indicators ("what is" → simple, negative weight)
- Multi-step patterns (numbered steps)
- Question complexity (multiple questions)
- **Configurable tier boundaries** and model mappings
- **Reasoning override** - 2+ reasoning markers force REASONING tier
## Usage
```yaml
model_list:
- model_name: smart-router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: gpt-4o-mini
MEDIUM: gpt-4o
COMPLEX: claude-sonnet-4
REASONING: o1-preview
```
Inspired by ClawRouter: https://github.com/BlockRunAI/ClawRouter
## Files Added
- litellm/router_strategy/complexity_router/complexity_router.py - Main router class
- litellm/router_strategy/complexity_router/config.py - Configuration and defaults
- litellm/router_strategy/complexity_router/__init__.py - Package exports
- litellm/router_strategy/complexity_router/README.md - Documentation
- tests/test_litellm/router_strategy/test_complexity_router.py - Test suite (37 tests)
## Files Modified
- litellm/router.py - Integration with pre_routing_hook
- litellm/types/router.py - New config params
* feat(router): Add complexity-based auto routing strategy
Adds a new rule-based routing strategy that classifies requests by complexity
and routes them to appropriate models - without any external API calls.
## Features
- Weighted scoring across 7 dimensions: token count, code presence, reasoning
markers, technical terms, simple indicators, multi-step patterns, questions
- Maps to 4 tiers: SIMPLE, MEDIUM, COMPLEX, REASONING
- Each tier configurable to a different model
- Zero API calls, <1ms latency
- Inspired by ClawRouter
## Configuration
```yaml
model_list:
- model_name: smart_router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: gemini-2.0-flash
MEDIUM: gpt-4o-mini
COMPLEX: claude-sonnet-4
REASONING: claude-opus-4
```
## Use Cases
- Cost optimization: route simple queries to cheaper models
- Quality optimization: route complex queries to capable models
- Zero configuration: works out of the box with sensible defaults
* feat(router): Add complexity-based auto routing strategy
Adds a new rule-based routing strategy that classifies requests by complexity
and routes them to appropriate models - without any external API calls.
- Weighted scoring across 7 dimensions: token count, code presence, reasoning
markers, technical terms, simple indicators, multi-step patterns, questions
- Maps to 4 tiers: SIMPLE, MEDIUM, COMPLEX, REASONING
- Each tier configurable to a different model
- Zero API calls, <1ms latency
- Inspired by ClawRouter
```yaml
model_list:
- model_name: smart_router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: gemini-2.0-flash
MEDIUM: gpt-4o-mini
COMPLEX: claude-sonnet-4
REASONING: claude-opus-4
```
- Cost optimization: route simple queries to cheaper models
- Quality optimization: route complex queries to capable models
- Zero configuration: works out of the box with sensible defaults
* feat: add enterprise presets for complexity router
Adds preset configurations for different cloud providers:
- bedrock: AWS Bedrock (Claude models)
- vertex: Google Vertex AI (Gemini models)
- azure: Azure OpenAI (GPT + o1)
- standard: Direct API (OpenAI + Anthropic)
- cost_optimized: Maximum savings (Gemini Flash + cheaper models)
Usage:
```yaml
complexity_router_config:
preset: bedrock # or vertex, azure, standard, cost_optimized
```
* feat(ui): update auto router submit handler for complexity router
- Handle complexity_router model type in submit handler
- Generate correct litellm_params for complexity router:
- model: auto_router/complexity_router
- complexity_router_config: { tiers: { SIMPLE, MEDIUM, COMPLEX, REASONING } }
- Keep existing semantic router handling intact
- Add success notification with router type name
* docs: update PR description with UI changes
* chore: remove preset feature, keep simple tier config
* fix: exclude complexity_router from auto_router check
The _is_auto_router_deployment() was matching all auto_router/* models,
causing complexity_router to fail initialization. Now it explicitly
excludes auto_router/complexity_router which has its own handler.
* fix(complexity_router): Address Greptile review feedback
Fixes 5 issues flagged in code review:
1. **Mutable singleton mutation bug** - Now always creates a new
ComplexityRouterConfig instance instead of reusing DEFAULT_COMPLEXITY_CONFIG
singleton, preventing cross-instance config pollution.
2. **Substring matching false positives** - Added word boundaries (spaces)
to short keywords like 'ok', 'try', 'api', 'git', 'node', 'java', 'vue'
to prevent matching within longer words (e.g., 'capital' matching 'api').
3. **Redundant message extraction** - Simplified to single reverse loop that
extracts both last user message and last system prompt efficiently.
4. **Unused imports** - Removed unused DEFAULT_CREATIVE_KEYWORDS and
DEFAULT_MULTI_STEP_PATTERNS imports.
5. **Missing async_pre_routing_hook tests** - Added comprehensive tests for:
- Multi-turn conversations
- List-type content handling
- No user message case
- Empty string content
- Message preservation
- Singleton mutation prevention
* fix(complexity_router): Address Greptile review feedback
- Use word boundary matching for short keywords (<5 chars) to avoid
false positives (e.g., 'api' matching 'capital', 'git' matching 'digital')
- Remove 'ok' from simple keywords (too many false positives)
- Add tests for keyword false positive prevention
- Fix test expectations for edge cases (empty string content, list content)
Addresses: 2/5 Greptile score feedback on PR #21789
* docs(auto_routing): Add complexity router documentation
- Add Complexity Router section to auto_routing.md
- Include comparison table with semantic auto router
- Add Python SDK and Proxy Server configuration examples
- Document all configuration options (tier boundaries, token thresholds, dimension weights)
- Explain how complexity scoring works
* feat(complexity_router): Add eval suite + tune scoring parameters
Added comprehensive evaluation suite with 29 test cases covering:
- SIMPLE tier: greetings, definitions, factual questions
- MEDIUM tier: technical explanations, comparisons, debugging
- COMPLEX tier: architecture design, complex coding
- REASONING tier: explicit reasoning requests
- Regression tests: substring false positive prevention
Tuned scoring parameters based on eval results:
- Lowered tier boundaries (0.15/0.35/0.60) for better tier distribution
- Increased code/technical weights (0.30/0.25) for complex prompts
- Reduced simple indicator weight (0.05) to avoid over-penalizing
- Fixed 'hey'/'hi' keywords to require leading space
Eval results: 29/29 passed (100%)
* fix(complexity_router): Address Greptile review round 2
1. **Empty user message handling** - Changed from falsy check to None check
to properly distinguish 'no user message' from 'empty string message'
2. **ReDoS prevention** - Changed 'first.*then' to 'first.*?then' (non-greedy)
to prevent regex backtracking on pathological inputs
3. **Documentation sync** - Updated README.md to match actual config values:
- Tier boundaries: 0.15/0.35/0.60 (not 0.25/0.50/0.75)
- Dimension weights: tokenCount=0.10, codePresence=0.30, technicalTerms=0.25,
simpleIndicators=0.05, multiStepPatterns=0.03, questionComplexity=0.02
4. **Missing UI component** - Added ComplexityRouterConfig.tsx with:
- Tier-to-model dropdown selectors
- Descriptions and examples for each tier
- How classification works explanation
5. **Inline import comment** - Added explanation for why ComplexityRouter
import is inline (matches AutoRouter pattern, avoids circular imports)
* docs(auto_routing): fix dimension weights and tier boundaries to match config.py defaults
* fix(complexity_router): skip empty string content in async_pre_routing_hook
* fix(router): remove or {} masking None complexity_router_config
* fix(config): remove unused DEFAULT_MULTI_STEP_PATTERNS and DEFAULT_CREATIVE_KEYWORDS exports
* fix(complexity_router): use word boundary matching for all single-word keywords, avoid double-scanning reasoning keywords
* fix(router): clarify circular import comment for ComplexityRouter
* docs(README): fix token thresholds to match config.py defaults
* test(complexity_router): add false positive tests for error/class/merge keyword matching
* fix(complexity_router): align .get() fallbacks with config.py defaults, document system prompt scoring
* fix(config): deduplicate keywords across code and technical lists
---------
Co-authored-by: OpenClaw Assistant <assistant@openclaw.ai>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
2026-02-21 13:23:37 -08:00
Ryan Crabbe
bbbec23c8b
fix: update tests to match tuple return type for cached init params
2026-02-21 13:18:03 -08:00
