6486db3646
* fix(middleware): replace BaseHTTPMiddleware with pure ASGI middleware BaseHTTPMiddleware wraps streaming responses with receive_or_disconnect per chunk, blocking the event loop and causing severe throughput degradation under concurrent streaming load (53% of CPU in profiling). Converts PrometheusAuthMiddleware to a pure ASGI middleware using the __call__(scope, receive, send) protocol. * fix(streaming): remove expensive debug logging and optimize usage stripping - Remove print_verbose calls that format chunk/response Pydantic objects, triggering millions of __repr__ calls (8% of CPU in profiling) - Guard remaining verbose_logger.debug with isEnabledFor(DEBUG) and use lazy %s formatting instead of f-strings - Replace usage stripping round-trip (model_dump + delete + reconstruct) with a _usage_stripped flag, deferring exclusion to serialization time * fix(proxy): remove per-chunk debug log and use _usage_stripped flag - Remove verbose_proxy_logger.debug that formatted every streaming chunk - Honor _usage_stripped flag from streaming handler to exclude usage during model_dump_json serialization instead of reconstructing objects * fix(proxy): remove per-chunk debug log in async_data_generator Remove verbose_proxy_logger.debug that formatted every streaming chunk, which triggered expensive Pydantic serialization on the hot path. * fix indentation and add clarifying comment for usage stripping * fix: guard calculate_total_usage against None usage in chunks * fix: store chunk copy to preserve usage for calculate_total_usage |
||
|---|---|---|
| .. | ||
| _experimental/mcp_server | ||
| agent_endpoints | ||
| anthropic_endpoints | ||
| auth | ||
| client | ||
| common_utils | ||
| db | ||
| discovery_endpoints | ||
| experimental/mcp_server | ||
| google_endpoints | ||
| guardrails | ||
| health_endpoints | ||
| hooks | ||
| image_endpoints | ||
| management_endpoints | ||
| management_helpers | ||
| middleware | ||
| openai_files_endpoint | ||
| pass_through_endpoints | ||
| policy_engine | ||
| prompts | ||
| public_endpoints | ||
| response_api_endpoints | ||
| spend_tracking | ||
| test_configs | ||
| ui_crud_endpoints | ||
| vector_store_endpoints | ||
| __init__.py | ||
| conftest.py | ||
| test_api_key_masking_in_errors.py | ||
| test_batch_metadata_none_fix.py | ||
| test_caching_routes.py | ||
| test_chat_completion_metadata.py | ||
| test_common_request_processing.py | ||
| test_custom_proxy.py | ||
| test_empty_model_list.py | ||
| test_enforce_user_param.py | ||
| test_fallback_management_endpoints.py | ||
| test_fastapi_offline_routes.py | ||
| test_health_check_functions.py | ||
| test_litellm_pre_call_utils.py | ||
| test_model_id_header_propagation.py | ||
| test_proxy_cli.py | ||
| test_proxy_server.py | ||
| test_proxy_types.py | ||
| test_proxy_utils.py | ||
| test_pyroscope.py | ||
| test_response_model_sanitization.py | ||
| test_route_a2a_models.py | ||
| test_route_llm_request.py | ||
| test_shared_health_check.py | ||
| test_spend_log_cleanup.py | ||
| test_swagger_chat_completions.py | ||
| test_team_member_update.py | ||