This commit fixes two critical test failures and two test isolation issues
in the SSL configuration tests.
## Critical Test Failures Fixed
### 1. test_get_ssl_configuration
**Problem:** Test was failing with assertion error that ssl.create_default_context
was never called (expected 1 call, got 0).
**Root Cause:** The get_ssl_configuration() function uses a caching mechanism
(_ssl_context_cache) to avoid creating duplicate SSL contexts with the same
configuration. When tests run in sequence, a previous test may have created an
SSL context with the same configuration (same cafile, ssl_security_level,
ssl_ecdh_curve). When this test runs, it retrieves the cached context instead
of creating a new one, so ssl.create_default_context() is never called, causing
the mock assertion to fail.
**Fix:** Clear the SSL context cache at the start of the test to ensure a fresh
context is created, allowing the mock to be called and verified.
### 2. test_ssl_ecdh_curve
**Problem:** Test was failing with assertion error that set_ecdh_curve was
never called (expected 1 call, got 0).
**Root Cause:** Same caching issue as above. Additionally, the test needed to
use a real SSLContext instance instead of a MagicMock because _create_ssl_context
calls methods like set_ciphers() and minimum_version that require a real context.
**Fix:**
- Clear the SSL context cache at the start of the test
- Use a real SSLContext instance and patch set_ecdh_curve on it specifically
- Added explanatory comment about why a real context is needed
## Test Isolation Issues Fixed
### 3. test_ssl_security_level
**Problem:** Test was failing because it expected LiteLLMAiohttpTransport but
got httpx.AsyncHTTPTransport instead.
**Root Cause:** Test isolation issue. Other tests in the file (test_force_ipv4_transport,
test_aiohttp_disabled_transport) set litellm.disable_aiohttp_transport = True
but don't restore the original value. When this test runs after those tests,
aiohttp transport is disabled, causing it to use httpx transport instead.
**Fix:** Explicitly enable aiohttp transport at the start of the test and restore
the original value in a finally block, ensuring the test works regardless of
test execution order.
### 4. test_ssl_verification_with_aiohttp_transport
**Problem:** Same as above - expected LiteLLMAiohttpTransport but got
httpx.AsyncHTTPTransport.
**Root Cause:** Same test isolation issue - aiohttp transport disabled by
previous tests.
**Fix:** Same approach - explicitly enable aiohttp transport and restore
original value in finally block.
## Why These Fixes Work
1. **Cache clearing:** By clearing _ssl_context_cache before each test, we
ensure that get_ssl_configuration() creates a fresh SSL context, allowing
mocks to be properly called and verified.
2. **Test isolation:** By saving and restoring the disable_aiohttp_transport
setting, tests are independent of each other and work correctly regardless
of execution order.
These are minimal, targeted fixes that address the root causes without
modifying production code or affecting other functionality.
* Cache realtime websocket request body
Move the realtime request payload builder out of the websocket handler and wrap it with an LRU cache so repeated connections reuse the same bytes object. This keeps the JSON formatting cost down while bounding memory usage.
* Optimize realtime websocket caching
Refactored /v1/realtime to use cached helpers for both the JSON body and query params, introduced a reusable request-scope template, and optimized header handling to avoid redundant work.
* Refine realtime websocket header handling
* Reuse websocket scope headers in auth
* Refactor realtime request body helper
Move the realtime request body formatter into proxy common utils so it can be reused across modules. Reuse it in the websocket auth flow to share LRU caching and avoid ad hoc byte builders.
* fix: revert to old pattern
The old pattern was necessary, we can just return the optimized function instead.
* Reuse SSL context for realtime
Create a shared SSLContext for OpenAI realtime websocket dials and pass it into websockets.connect so we stop re-reading verify paths on every session.
* feat: reuse shared TLS context for realtime websockets
- add `SHARED_REALTIME_SSL_CONTEXT` helper so all realtime websocket clients share the same TLS settings
- wire the shared context into OpenAI, Azure, custom HTTPX handlers, and realtime health checks
- update realtime tests to assert that the expected SSL context is passed to `websockets.connect`
This keeps TLS configuration consistent and avoids recreating SSL contexts per connection.
* Reuse HTTP SSL context for realtime
Remove the standalone realtime SSL helper, expose a shared context directly from the HTTP handler, and point all realtime websocket clients and tests to it. Add the websocket header comparison tool.
* Lazy-load shared realtime SSL context
Fix circular imports introduced by eagerly instantiating the shared TLS context. Make the HTTP handler lazily create the context and have realtime clients/tests fetch it on demand, keeping configuration consistent without breaking startup.
* add: unit test for realtime LRU caches
* fix: merge conflict with imports
* Add openai metadata filed in the request
* Add docs related to openai metadata
* Add utils
* test_completion_openai_metadata[True]
* Added support for though signature for gemini 3 in responses api (#16872)
* Added support for though signature for gemini 3
* Update docs with all supported endpoints and cost tracking
* Added config based routing support for batches and files
* fix lint errors
* Litellm anthropic image url support (#16868)
* Add image as url support to anthropic
* fix mypy errors
* fix tests
* Fix: Populate spend_logs_metadata in batch and files endpoints (#16921)
* Add spend-logs-metadata to the metadata
* Add tests for spend logs metadata in batches
* use better names
* Remove support for penalty param for gemini 3 (#16907)
* Remove support for penalty param
* remove halucinated model names
* fix mypy/test errors
* fix tests
* fix too many lines error
* fix too many lines error
* Add config for cicd test case
* Fix final tests
* fix batch tests
* fix batch tests
* attempt to implement the passthrough feature
* Formatting and small change
* Fix formatting
* Format test file
---------
Co-authored-by: Xiaohan Fu <xiaohan@grayswan.ai>
* fix(router): use cacheable prefix for prompt caching cache keys
Fix issue where requests with same cacheable prefix but different user
messages were routing to different deployments, preventing cached token
reuse. The cache key now correctly includes only the cacheable prefix
(up to and including the last cache_control block) instead of the
entire messages array.
## New Functions
### extract_cacheable_prefix()
Static method that extracts the cacheable prefix from messages for
prompt caching. The cacheable prefix is defined as everything UP TO
AND INCLUDING the LAST content block (across all messages) that has
cache_control with type "ephemeral". This includes ALL blocks
before the last cacheable block (even if they don't have cache_control
themselves).
- Finds the last content block with cache_control across all messages
- Returns all messages and content blocks up to and including that
last cacheable block
- Excludes everything after the last cacheable block (including user
messages that come after)
- Returns empty list if no cacheable blocks are found
## Changed Functions
### get_prompt_caching_cache_key()
Modified to use the cacheable prefix instead of the full messages array
when generating cache keys. This ensures that requests with the same
cacheable prefix but different user messages generate the same cache
key, enabling proper routing to the same deployment.
- Now calls extract_cacheable_prefix() to get only cacheable content
- Returns None if no cacheable prefix is found (can't generate key)
- Cache key is now based on cacheable prefix only, not full messages
### async_get_model_id()
Completely refactored to use the cacheable prefix directly instead of
the previous workaround that checked progressively shorter message
slices. The previous implementation was inefficient and unreliable.
- Removed progressive message slicing logic (messages[:-1], messages[:-2], etc.)
- Now uses single direct cache lookup with cacheable prefix-based key
- More efficient (1 lookup instead of up to 4)
- More reliable (uses correct cache key based on cacheable prefix)
- Returns None if no cacheable prefix found
### add_model_id()
Added None check for cache_key to prevent caching when no cacheable
prefix is found. This ensures we don't attempt to cache when there's
no meaningful cache key to use.
- Added guard: returns early if cache_key is None
- Prevents attempting to cache when no cacheable prefix exists
### async_add_model_id()
Added None check for cache_key to prevent caching when no cacheable
prefix is found. Matches the behavior of add_model_id() for consistency.
- Added guard: returns early if cache_key is None
- Prevents attempting to cache when no cacheable prefix exists
### get_model_id()
Added None check for cache_key to handle cases where no cacheable
prefix is found. Ensures consistent behavior across all cache methods.
- Added guard: returns None if cache_key is None
- Prevents calling get_cache() with None key
## Test
### test_router_prompt_caching_same_cacheable_prefix_routes_to_same_deployment()
New end-to-end test that validates the fix. Tests that requests with
the same cacheable prefix (system blocks with cache_control) but
different user messages:
1. Generate the same cache key
2. Successfully perform cache lookup
3. Route to the same deployment
This test reproduces the exact scenario from the user's bug report
where three requests with different user messages should route to the
same deployment but were previously routing to different ones.
Fixes issue where cached tokens couldn't be reused because requests
were routed to different providers due to different cache keys.
* fix(router): use cast() for proper type handling in extract_cacheable_prefix
Replace type annotation with type: ignore comment with proper cast()
from typing module, matching the pattern used throughout the
codebase for creating modified AllMessageValues dictionaries.
Previously, get_ssl_configuration() created a new SSL context on every
call, even when the configuration was identical. This caused continuous
memory allocation from ssl.create_default_context(), especially during:
- Proxy server startup
- Background health checks
- HTTP client creation
Solution:
- Added _ssl_context_cache to cache SSL contexts by configuration
parameters (cafile, ssl_security_level, ssl_ecdh_curve)
- Refactored SSL context creation into _create_ssl_context() helper
- Modified get_ssl_configuration() to reuse cached contexts when
configuration matches
This significantly reduces memory allocation while maintaining backward
compatibility. SSL contexts are now reused instead of being recreated
repeatedly, eliminating the memory leak observed in memray profiling.
Fixes memory allocation issue where create_default_context was allocating
6.282MB+ continuously even without any requests.
