diff --git a/docs/my-website/docs/providers/openai.md b/docs/my-website/docs/providers/openai.md index 1f4a1687e8..0c82f9be12 100644 --- a/docs/my-website/docs/providers/openai.md +++ b/docs/my-website/docs/providers/openai.md @@ -432,9 +432,59 @@ curl -X POST 'http://0.0.0.0:4000/chat/completions' \ | fine tuned `gpt-3.5-turbo-1106` | `response = completion(model="ft:gpt-3.5-turbo-1106", messages=messages)` | | fine tuned `gpt-3.5-turbo-0613` | `response = completion(model="ft:gpt-3.5-turbo-0613", messages=messages)` | -## Getting Reasoning Content in `/chat/completions` +## [BETA] Route all .completions requests to Responses API (better quality) + When enabled, LiteLLM sends OpenAI traffic from `litellm.completion()` and the proxy `/chat/completions` endpoint through the [Responses API](https://platform.openai.com/docs/api-reference/responses) instead of Chat Completions. That path generally matches OpenAI’s latest model behavior and quality (for example, reasoning output on GPT‑5 class models). -GPT-5 models return reasoning content when called via the Responses API. You can call these models via the `/chat/completions` endpoint by using the `openai/responses/` prefix. +You can opt in globally or per request: + +**Option A — per-request prefix:** Use the `openai/responses/` model prefix. + +**Option B — global flag (recommended):** Set `route_all_chat_openai_to_responses = True` to automatically route all OpenAI `/chat/completions` requests through the Responses API, no model prefix needed. + + + + +```python +import litellm + +litellm.route_all_chat_openai_to_responses = True + +response = litellm.completion( + model="gpt-5.4", + messages=[{"role": "user", "content": "What is the capital of France?"}], + reasoning_effort="low", +) +``` + + + + +Set in your proxy config: +```yaml +litellm_settings: + route_all_chat_openai_to_responses: true +``` + +Then call normally — no model prefix needed: +```bash +curl -X POST 'http://0.0.0.0:4000/chat/completions' \ +-H 'Content-Type: application/json' \ +-H 'Authorization: Bearer sk-1234' \ +-d '{ + "model": "gpt-5.4", + "messages": [{"role": "user", "content": "What is the capital of France?"}], + "reasoning_effort": "low" +}' +``` + + + + +:::note +`route_all_chat_openai_to_responses` only applies to the `openai` provider. Azure OpenAI is unaffected. You can also set it via env var: `LITELLM_ROUTE_ALL_CHAT_OPENAI_TO_RESPONSES=true`. +::: + +**Option A — per-request prefix:** You can also prefix individual model names with `openai/responses/` to route just that call through the Responses API. diff --git a/docs/my-website/docs/proxy/config_settings.md b/docs/my-website/docs/proxy/config_settings.md index a886a754f5..0bcca0bc3e 100644 --- a/docs/my-website/docs/proxy/config_settings.md +++ b/docs/my-website/docs/proxy/config_settings.md @@ -197,6 +197,7 @@ router_settings: | key_generation_settings | object | Restricts who can generate keys. [Further docs](./virtual_keys.md#restricting-key-generation) | | disable_add_transform_inline_image_block | boolean | For Fireworks AI models - if true, turns off the auto-add of `#transform=inline` to the url of the image_url, if the model is not a vision model. | | use_chat_completions_url_for_anthropic_messages | boolean | If true, routes OpenAI `/v1/messages` requests through chat/completions instead of the Responses API. Can also be set via env var `LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES=true`. | +| route_all_chat_openai_to_responses | boolean | If true, routes all OpenAI `/chat/completions` requests through the Responses API bridge. Recommended for OpenAI models. Can also be set via env var `LITELLM_ROUTE_ALL_CHAT_OPENAI_TO_RESPONSES=true`. | | skip_system_message_in_guardrail | boolean | If true, unified guardrails omit `role: system` from scanned input on **chat completions** and **Anthropic `/v1/messages`** only; the LLM still receives full messages. Per-guardrail override: `litellm_params.skip_system_message_in_guardrail` on each guardrail. [Guardrails quick start](./guardrails/quick_start#skip-system-messages-in-guardrail-evaluation) | | disable_hf_tokenizer_download | boolean | If true, it defaults to using the openai tokenizer for all models (including huggingface models). | | enable_json_schema_validation | boolean | If true, enables json schema validation for all requests. | @@ -868,6 +869,7 @@ router_settings: | LITELLM_SECRET_AWS_KMS_LITELLM_LICENSE | AWS KMS encrypted license for LiteLLM | LITELLM_TOKEN | Access token for LiteLLM integration | LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES | When set to "true", routes OpenAI /v1/messages requests through chat/completions instead of the Responses API for Anthropic models. Can also be set via `litellm_settings.use_chat_completions_url_for_anthropic_messages` +| LITELLM_ROUTE_ALL_CHAT_OPENAI_TO_RESPONSES | When set to "true", routes all OpenAI /chat/completions requests through the Responses API bridge. Recommended for OpenAI models. Can also be set via `litellm_settings.route_all_chat_openai_to_responses` | LITELLM_USER_AGENT | Custom user agent string for LiteLLM API requests. Used for partner telemetry attribution | LITELLM_WORKER_STARTUP_HOOKS | Comma-separated list of `module.path:function_name` callables to run in each worker process during startup. Runs early in the worker lifecycle (before config/DB loading). Useful for re-initializing per-process state like [gflags](https://github.com/google/python-gflags). See [Worker Startup Hooks](/proxy/worker_startup_hooks) for details | LITELLM_PRINT_STANDARD_LOGGING_PAYLOAD | If true, prints the standard logging payload to the console - useful for debugging diff --git a/litellm/__init__.py b/litellm/__init__.py index 89cef667c6..77fa48625d 100644 --- a/litellm/__init__.py +++ b/litellm/__init__.py @@ -221,6 +221,9 @@ modify_params = bool(os.getenv("LITELLM_MODIFY_PARAMS", False)) use_chat_completions_url_for_anthropic_messages: bool = bool( os.getenv("LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES", False) ) # When True, routes OpenAI /v1/messages requests to chat/completions instead of the Responses API +route_all_chat_openai_to_responses: bool = ( + os.getenv("LITELLM_ROUTE_ALL_CHAT_OPENAI_TO_RESPONSES", "false").lower() == "true" +) # When True, routes all OpenAI /chat/completions requests through the Responses API bridge retry = True ### AUTH ### api_key: Optional[str] = None diff --git a/litellm/main.py b/litellm/main.py index 73db4a11cb..a93aed51b2 100644 --- a/litellm/main.py +++ b/litellm/main.py @@ -941,6 +941,14 @@ def responses_api_bridge_check( reasoning_effort: Optional[Any] = None, ) -> Tuple[dict, str]: model_info: Dict[str, Any] = {} + + # Global flag: route ALL OpenAI chat completions through Responses API. + # Returns early with minimal model_info; callers only inspect the "mode" key. + if litellm.route_all_chat_openai_to_responses and custom_llm_provider == "openai": + model = model.replace("responses/", "") + model_info["mode"] = "responses" + return model_info, model + try: model_info = cast( dict, diff --git a/litellm/responses/litellm_completion_transformation/transformation.py b/litellm/responses/litellm_completion_transformation/transformation.py index 9740494700..48b12a5fba 100644 --- a/litellm/responses/litellm_completion_transformation/transformation.py +++ b/litellm/responses/litellm_completion_transformation/transformation.py @@ -32,6 +32,7 @@ from litellm.types.llms.openai import ( OpenAIWebSearchOptions, OpenAIWebSearchUserLocation, OutputTokensDetails, + Reasoning, ResponseAPIUsage, ResponsesAPIOptionalRequestParams, ResponsesAPIResponse, @@ -181,12 +182,19 @@ class LiteLLMCompletionResponsesConfig: ) # Extract reasoning_effort from reasoning parameter - reasoning_effort = None + reasoning_effort: Optional[Union[Reasoning, str]] = None reasoning_param = responses_api_request.get("reasoning") if reasoning_param: if isinstance(reasoning_param, dict): - # reasoning can be {"effort": "low|medium|high"} - reasoning_effort = reasoning_param.get("effort") + # reasoning can be {"effort": "low|medium|high", "summary": "detailed"} + # Keep the full dict when summary is set so the responses API bridge can + # forward it; otherwise use the effort string for chat completion (e.g. Gemini). + if "summary" in reasoning_param: + reasoning_effort = reasoning_param + elif "effort" in reasoning_param: + reasoning_effort = reasoning_param.get("effort") + else: + reasoning_effort = reasoning_param elif isinstance(reasoning_param, str): # reasoning could be a string directly reasoning_effort = reasoning_param diff --git a/tests/test_litellm/test_main.py b/tests/test_litellm/test_main.py index 1ed2382393..4358d0dc19 100644 --- a/tests/test_litellm/test_main.py +++ b/tests/test_litellm/test_main.py @@ -809,6 +809,50 @@ def test_responses_api_bridge_check_handles_exception(): assert model_info["mode"] == "responses" +def test_responses_api_bridge_check_global_flag_routes_openai(): + """When route_all_chat_openai_to_responses is True, any OpenAI model routes to responses.""" + from litellm.main import responses_api_bridge_check + + with patch.object(litellm, "route_all_chat_openai_to_responses", True): + model_info, model = responses_api_bridge_check( + model="gpt-4o", + custom_llm_provider="openai", + ) + + assert model == "gpt-4o" + assert model_info.get("mode") == "responses" + + +def test_responses_api_bridge_check_global_flag_does_not_affect_azure(): + """route_all_chat_openai_to_responses should not affect Azure models.""" + from litellm.main import responses_api_bridge_check + + with patch.object(litellm, "route_all_chat_openai_to_responses", True): + with patch("litellm.main._get_model_info_helper") as mock_get_model_info: + mock_get_model_info.return_value = {"max_tokens": 4096} + model_info, model = responses_api_bridge_check( + model="gpt-4o", + custom_llm_provider="azure", + ) + + assert model_info.get("mode") != "responses" + + +def test_responses_api_bridge_check_global_flag_default_false(): + """By default, route_all_chat_openai_to_responses is False and doesn't affect routing.""" + from litellm.main import responses_api_bridge_check + + with patch.object(litellm, "route_all_chat_openai_to_responses", False): + with patch("litellm.main._get_model_info_helper") as mock_get_model_info: + mock_get_model_info.return_value = {"max_tokens": 4096} + model_info, model = responses_api_bridge_check( + model="gpt-4o", + custom_llm_provider="openai", + ) + + assert model_info.get("mode") != "responses" + + @pytest.mark.asyncio async def test_async_mock_delay(): """Use asyncio await for mock delay on acompletion"""