* fix(vertex_ai): improve passthrough endpoint url parsing and construction (#17402)
* test(proxy): add test for vertex passthrough load balancing
Add a test that verifies _base_vertex_proxy_route uses
get_available_deployment for proper load balancing instead of
get_model_list. This ensures the correct deployment is selected
from the router and vertex credentials are properly fetched.
Also refactor the implementation to:
- Use get_available_deployment instead of get_model_list
- Add error handling for deployment retrieval
- Improve code structure with try-except block
* feat(proxy): add pass-through deployment filtering methods
Add dedicated methods to filter and select deployments for pass-through endpoints:
- Implement get_available_deployment_for_pass_through() to ensure only deployments with use_in_pass_through=True are considered
- Implement async_get_available_deployment_for_pass_through() for async operations
- Add _filter_pass_through_deployments() helper method to filter by use_in_pass_through flag
- Update vertex pass-through route to use the new dedicated method
This ensures pass-through endpoints respect the use_in_pass_through configuration and apply proper load balancing strategy only to configured deployments.
Add comprehensive tests to verify filtering and load balancing behavior.
* fix(text_completion): support token IDs (list of integers) as prompt
Add support for passing token IDs (list of integers) to the text_completion
endpoint for OpenAI-compatible providers (openai, azure, vllm, etc.).
Fixes#17118
* test(text_completion): replace live test with mock test for token IDs
Move token IDs test from local_testing to test_litellm with mocks
per PR review feedback.
* feat(bedrock): add OpenAI-compatible service_tier parameter translation
Translates OpenAI's service_tier parameter (string) to Bedrock's
serviceTier format (object with type field).
* docs(bedrock): add OpenAI-compatible service_tier parameter documentation
Document the automatic translation from OpenAI-style service_tier
parameter to Bedrock's native serviceTier format.
* feat(bedrock): add service_tier to response when present
According to OpenAI's API documentation, when service_tier is sent in the
request, it should be returned in the response. This commit implements
this behavior for Bedrock Converse API to maintain compatibility with
OpenAI's API.
Changes:
- Added serviceTier field to ConverseResponseBlock type definition
- Moved ServiceTierBlock definition before ConverseResponseBlock to fix
type reference order
- Added response transformation to map Bedrock serviceTier (object) to
OpenAI service_tier (string format)
- Added 4 new tests for response transformation with service_tier
The service_tier is only added to the response when present in Bedrock's
response, maintaining backward compatibility.
Fixes#18137
Similar to the fix for web_search_tool_result (#17746, #17798), this PR
preserves web_fetch_tool_result blocks in multi-turn conversations.
Changes:
- Add handling for web_fetch_tool_result in transformation.py (non-streaming)
- Add capture of web_fetch_tool_result in handler.py (streaming)
- Fix streaming tool arguments bug where empty input {} was prepended to
actual arguments by using empty string instead of str({})
- Add unit tests for web_fetch_tool_result handling
The OCI adapter now accepts both string and object formats for image_url:
- String: "image_url": "https://example.com/image.png"
- Object: "image_url": {"url": "https://example.com/image.png"}
This fixes compatibility with OpenAI Vision API format.
* fix(gemini): prevent negative text_tokens with explicit caching (#18750)
## Problem
When using Gemini with explicit caching (especially with images),
text_tokens would become negative (e.g., -3327) due to incorrectly
subtracting total cached_tokens from modality-specific text_tokens.
## Root Cause
The old code did:
```python
text_tokens = text_tokens - cached_tokens # 737 - 4064 = -3327
```
This was wrong because:
- cached_tokens includes ALL modalities (text + image + audio + video)
- text_tokens only contains text
- Subtracting total from specific caused negative values
## Solution
Parse cacheTokensDetails to get per-modality cached token breakdown:
```python
if "cacheTokensDetails" in usage_metadata:
cached_text_tokens = parse from cacheTokensDetails["TEXT"]
text_tokens = text_tokens - cached_text_tokens # Correct!
```
Now we subtract cached tokens per modality, preventing negatives.
## Changes
- Parse cacheTokensDetails field from Gemini response
- Calculate non-cached tokens per modality (text, image, audio)
- Remove incorrect global cached_tokens subtraction
- Add tests for explicit caching and implicit/no caching scenarios
## Testing
- Added test_gemini_cache_tokens_details_no_negative_values
- Added test_gemini_without_cache_tokens_details
- All existing Gemini caching tests pass
Fixes#18750
* feat: add cache_read_input_tokens to Usage object
Addresses reviewer feedback to include cached tokens at the top level
of the Usage object. This aligns with how Anthropic provider handles
cached tokens and ensures they are visible in the final usage response.
* fix: add cacheTokensDetails field to UsageMetadata TypedDict
Fixes mypy error where cacheTokensDetails was being accessed but not defined
in the UsageMetadata TypedDict type definition.
* fix(anthropic): prevent dropping thinking when any message has thinking_blocks
When Claude returns multiple assistant messages in a conversation, some may
have thinking_blocks while others may not (Claude's behavior varies). The
previous logic only checked the LAST assistant message with tool_calls,
dropping the thinking param if it had no thinking_blocks.
This caused errors when earlier messages still contained thinking_blocks:
"When thinking is disabled, an assistant message cannot contain thinking"
The fix adds a new check: only drop thinking if NO assistant messages
have thinking_blocks. If any message has thinking_blocks, we keep
thinking enabled.
Fixes#18926
* chore: re-trigger CI
