- Fix bug where model names without slash (e.g., 'gpt-5') couldn't
match providers in polling_via_cache list
- Look up model in llm_router.model_name_to_deployment_indices
- Check ALL deployments for matching provider (supports load balancing)
- Check custom_llm_provider first, then extract from model string
- Add comprehensive tests for provider resolution logic
Committed-By-Agent: cursor
- Add support for all ResponsesAPIResponse fields in update_state
- Extract model, instructions, temperature, top_p, max_output_tokens,
previous_response_id, text, truncation, parallel_tool_calls, user,
store, and incomplete_details from response.completed event
- Pass all fields to final update_state call
Committed-By-Agent: cursor
- Create new background_streaming.py in response_polling/
- Update endpoints.py to import from new location
- Update __init__.py to export background_streaming_task
- Add tests for module imports and structure
Committed-By-Agent: cursor
* fix: lazy load utils.py imports
Lazy-load most functions and response types from utils.py to avoid loading
tiktoken and other heavy dependencies at import time. This significantly
reduces memory usage when importing completion from litellm.
* fix: prevent memory leak in aiohttp connection pooling
Add connection limits to aiohttp TCPConnector to prevent unbounded
connection growth that causes memory leaks. Without these limits,
aiohttp's _wrap_create_connection can accumulate connections
indefinitely in long-running processes.
Changes:
- Set default limit of 300 total connections and 50 per host
- Apply limits to shared proxy session initialization
- Apply limits to HTTP handler transport creation
- Configurable via AIOHTTP_CONNECTOR_LIMIT and
AIOHTTP_CONNECTOR_LIMIT_PER_HOST environment variables
- Set to 0 for unlimited (not recommended for production)
This fix covers:
- All standard LLM provider API calls (OpenAI, Anthropic, etc.)
- Proxy server shared session
- Most guardrail HTTP calls
Impact: Prevents memory exhaustion in high-traffic deployments and
long-running proxy servers that make thousands of API calls.
Testing: Verified connection limits are applied correctly and
existing functionality remains unchanged.
Add Agent Lightning, Microsoft's open-source framework for training
AI agents with RL, APO, and SFT. Uses LiteLLM Proxy for LLM routing
and trace collection.
* fix(initial-commit): adding a way to get the right response type based on the api route
* feat(unified_guardrail.py): support streaming guardrails
* test: update tests
* fix: fix linting errors
* test: update tests
