litellm/litellm
Ishaan Jaff 6486db3646
fix: improve streaming proxy throughput by fixing middleware and logging bottlenecks (#21501)
* fix(middleware): replace BaseHTTPMiddleware with pure ASGI middleware

BaseHTTPMiddleware wraps streaming responses with receive_or_disconnect
per chunk, blocking the event loop and causing severe throughput
degradation under concurrent streaming load (53% of CPU in profiling).

Converts PrometheusAuthMiddleware to a pure ASGI middleware using the
__call__(scope, receive, send) protocol.

* fix(streaming): remove expensive debug logging and optimize usage stripping

- Remove print_verbose calls that format chunk/response Pydantic objects,
  triggering millions of __repr__ calls (8% of CPU in profiling)
- Guard remaining verbose_logger.debug with isEnabledFor(DEBUG) and use
  lazy %s formatting instead of f-strings
- Replace usage stripping round-trip (model_dump + delete + reconstruct)
  with a _usage_stripped flag, deferring exclusion to serialization time

* fix(proxy): remove per-chunk debug log and use _usage_stripped flag

- Remove verbose_proxy_logger.debug that formatted every streaming chunk
- Honor _usage_stripped flag from streaming handler to exclude usage
  during model_dump_json serialization instead of reconstructing objects

* fix(proxy): remove per-chunk debug log in async_data_generator

Remove verbose_proxy_logger.debug that formatted every streaming chunk,
which triggered expensive Pydantic serialization on the hot path.

* fix indentation and add clarifying comment for usage stripping

* fix: guard calculate_total_usage against None usage in chunks

* fix: store chunk copy to preserve usage for calculate_total_usage
2026-02-18 16:16:49 -08:00
..
a2a_protocol fix: prompt registry 2026-02-18 00:34:54 +05:30
anthropic_interface fix: prompt registry 2026-02-18 00:34:54 +05:30
assistants
batch_completion fix: prompt registry 2026-02-18 00:34:54 +05:30
batches fix: prompt registry 2026-02-18 00:34:54 +05:30
caching fix: prompt registry 2026-02-18 00:34:54 +05:30
completion_extras fix code quality tests and mypy 2026-02-18 17:36:36 +05:30
containers fix: prompt registry 2026-02-18 00:34:54 +05:30
endpoints/speech/speech_to_completion_bridge
evals fix: prompt registry 2026-02-18 00:34:54 +05:30
experimental_mcp_client fix: prompt registry 2026-02-18 00:34:54 +05:30
files fix: prompt registry 2026-02-18 00:34:54 +05:30
fine_tuning
google_genai fix: prompt registry 2026-02-18 00:34:54 +05:30
images fix: prompt registry 2026-02-18 00:34:54 +05:30
integrations fix: guard against None metadata in prometheus metrics (#21489) 2026-02-18 12:40:45 -08:00
interactions fix: prompt registry 2026-02-18 00:34:54 +05:30
litellm_core_utils fix: improve streaming proxy throughput by fixing middleware and logging bottlenecks (#21501) 2026-02-18 16:16:49 -08:00
llms Merge branch 'main' into litellm_sonnet_4_6_feat 2026-02-18 18:40:56 +05:30
ocr
passthrough fix: prompt registry 2026-02-18 00:34:54 +05:30
proxy fix: improve streaming proxy throughput by fixing middleware and logging bottlenecks (#21501) 2026-02-18 16:16:49 -08:00
proxy_auth fix: prompt registry 2026-02-18 00:34:54 +05:30
rag fix: prompt registry 2026-02-18 00:34:54 +05:30
realtime_api fix: prompt registry 2026-02-18 00:34:54 +05:30
rerank_api fix: prompt registry 2026-02-18 00:34:54 +05:30
responses fix: prompt registry 2026-02-18 00:34:54 +05:30
router_strategy fix: prompt registry 2026-02-18 00:34:54 +05:30
router_utils Merge main into affinity_callback and address deployment affinity review feedback 2026-02-18 10:01:11 -06:00
search
secret_managers fix: prompt registry 2026-02-18 00:34:54 +05:30
skills fix: prompt registry 2026-02-18 00:34:54 +05:30
types Merge main into affinity_callback and address deployment affinity review feedback 2026-02-18 10:01:11 -06:00
vector_store_files
vector_stores
videos
__init__.py Merge branch 'main' into litellm_oss_staging_02_17_2026 2026-02-18 17:26:33 +05:30
_lazy_imports_registry.py Merge branch 'main' into litellm_oss_staging_02_17_2026 2026-02-18 17:26:33 +05:30
_lazy_imports.py fix: prompt registry 2026-02-18 00:34:54 +05:30
_logging.py fix: prompt registry 2026-02-18 00:34:54 +05:30
_redis.py
_service_logger.py fix: prompt registry 2026-02-18 00:34:54 +05:30
_uuid.py
_version.py
anthropic_beta_headers_config.json fix: prompt registry 2026-02-18 00:34:54 +05:30
anthropic_beta_headers_manager.py fix: prompt registry 2026-02-18 00:34:54 +05:30
budget_manager.py
constants.py Merge branch 'main' into litellm_oss_staging_02_17_2026 2026-02-18 17:26:33 +05:30
cost_calculator.py fix: prompt registry 2026-02-18 00:34:54 +05:30
cost.json
exceptions.py fix: prompt registry 2026-02-18 00:34:54 +05:30
main.py fix: prompt registry 2026-02-18 00:34:54 +05:30
model_prices_and_context_window_backup.json Merge pull request #21448 from BerriAI/litellm_sonnet_4_6_feat 2026-02-18 18:44:36 +05:30
mypy.ini
policy_templates_backup.json feat: split EU AI Act Article 5 into 5 dedicated sub-guardrails per language (#21453) 2026-02-17 20:43:25 -08:00
py.typed
router.py Merge main into affinity_callback and address deployment affinity review feedback 2026-02-18 10:01:11 -06:00
scheduler.py fix: prompt registry 2026-02-18 00:34:54 +05:30
timeout.py
utils.py Merge pull request #21467 from BerriAI/litellm_add_duck_duck_go 2026-02-18 18:38:37 +05:30